supaspin/README.md

# supaspin

Version 1.3.0

Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
single-node [microk8s](https://microk8s.io) server, from a terminal menu.

Use it for testing and staging copies of an app. Each instance is numbered, has its own hostname,
its own database, capped disk and capped memory, and runs beside your other workloads without
endangering them. Nothing in it is specific to any one app.

## What each instance gets

| | |
|---|---|
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions), with a Let's Encrypt certificate |
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |

Studio and pg-meta (the admin UI and its schema API) are **removed from the public gateway**: `/`
and `/pg/` return 404 on the public hostname. Studio on port `300N` has **no login of its own**, so
keep that port off the internet (do not forward it on your router).

Instance *N* uses hostname slot *N* from your settings, so the number of slots is the maximum number
of instances at once. Numbers are reused: create picks the lowest free one.

## Requirements

- A Debian/Ubuntu server with **microk8s** (or any Kubernetes with `kubectl` and `helm` on the
  path), with an **ingress-nginx** ingress class and **cert-manager** installed.
- DNS `A` records for every hostname slot pointing at the server, and ports 80 and 443 forwarded to
  it (Let's Encrypt fetches a test file over port 80 to issue each certificate).
- Root access on that server. `python3` and `python3-yaml` (the installer adds the latter with `apt`).

## Install

```bash
curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
```

**If your Gitea requires sign-in even for public repositories** (the download then fails with
"Could not download"), either pass a read-only token (Gitea → Settings → Applications, scope
`repository: read`):

```bash
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
```

or clone the repository and install from the checkout:

```bash
git clone ssh://git@git.didgibot.com:2226/public/supaspin.git && sudo bash supaspin/install.sh
```

Run it again to upgrade. It installs `/usr/local/bin/supaspin`, its values template and the
uninstaller in `/usr/share/supaspin/`.

Then start the app; the first run asks for your settings and prepares the cluster:

```bash
sudo supaspin
```

| Setting | Meaning |
|---|---|
| Storage root | folder that holds every instance's images and `.env` (`<root>/<N>/…`) |
| Node name | Kubernetes node the volumes are pinned to (detected) |
| Server LAN IP | address the Postgres and Studio ports are bound to (detected) |
| Let's Encrypt e-mail | for certificate expiry notices |
| Ingress class | usually `public` on microk8s |
| Hostname slots | comma-separated; the first is instance 1, the second instance 2, … |
| Public SSH host / port / user | only used to print the tunnel command |
| Mail settings | default SMTP settings for the auth service (changeable later) |

Settings live in `/etc/supaspin/config.yaml` (readable by root only). The first run also creates,
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-letsencrypt` ClusterIssuer,
the `supaspin-low` PriorityClass and the `supaspin-certs` namespace, and downloads the pinned
Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/charts/`.

## Using it

`sudo supaspin` opens the menu (arrow keys, Enter, `q` to go back).

| Menu item | What it does |
|---|---|
| Create instance | picks the lowest free number, asks whether to use the saved mail settings or enter different ones, creates the disks, installs Supabase, waits until every pod is ready, writes `<root>/<N>/.env` |
| List instances | number, hostname, pods ready, memory in use, disk use per image, `.env` path, and for each instance the tunnel command with plain instructions (open it, keep it in the background, close it) |
| Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
| Delete instance | asks you to type the number; removes everything (see below) |
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
| Suspend instance | stops every pod of one instance and keeps everything else: its memory, processor time, pod slots and two server ports go back to the server, its disks and `.env` file stay |
| Resume instance | starts a suspended instance again, with the data it had |
| Mail settings | view and change the saved default mail settings |
| Settings | change any first-run setting |

The same actions are available without the menu:

```bash
sudo supaspin create [--ask-mail]     # --ask-mail: choose saved or different mail settings
sudo supaspin list
sudo supaspin show N                  # .env with secrets hidden, plus tunnel help
sudo supaspin env N                   # the full .env (contains secrets)
sudo supaspin tunnel N                # the tunnel commands, step by step
sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]   # see "Copying an instance's values"
sudo supaspin delete N [--yes]
sudo supaspin upgrade [N]             # apply this version's memory caps to running instances
sudo supaspin suspend N               # stop an instance, keep its data (see "Suspending an instance")
sudo supaspin resume N                # start it again
sudo supaspin mail                    # change the saved default mail settings
sudo supaspin setup                   # change settings
supaspin --version                    # prints "supaspin 1.3.0"
```

### Connecting your app

Each instance writes `<root>/<N>/.env` (mode 600, owned by root) with the usual Supabase variable
names: `SUPABASE_URL`, `PUBLIC_SUPABASE_URL`, `API_EXTERNAL_URL`, `ANON_KEY`,
`PUBLIC_SUPABASE_ANON_KEY`, `JWT_ANON_KEY`, `SERVICE_ROLE_KEY`, `JWT_SERVICE_KEY`, `JWT_SECRET`,
`JWT_EXPIRY`, `POSTGRES_*`, `DATABASE_URL`, `DASHBOARD_USERNAME`, `DASHBOARD_PASSWORD` and
`GOTRUE_SMTP_*`. Read it with `sudo supaspin env N` and copy what your app needs.

The web API is public, so the URL and keys work from anywhere. **Postgres and Studio are not
public.** To reach them from your own computer you open a *tunnel*: a private connection from your
computer to the server that makes the instance's database and Studio (its admin screen) appear on
your own computer, at `localhost:544N` (database) and `http://localhost:300N` (Studio), for as long
as the tunnel is open. `sudo supaspin list` prints the exact commands for every instance
(`sudo supaspin tunnel N` prints them with a few more hints). Run them in a terminal on **your**
computer, not on the server.

**Which computer you use matters.** The commands for keeping the tunnel in the background and for
closing it work on **Mac and Linux only**. They do **not** work on **Windows**, because the ssh
program that comes with Windows does not support the "control file" they need. On Windows use the
first command (keep the tunnel open in its own PowerShell window) and close it with Ctrl+C or by
closing the window.

- If you want to open the tunnel and keep it open in your terminal window (Mac, Linux, Windows),
  run the `ssh -N ...` command. The window looks stuck and prints nothing: that is normal, it means
  the tunnel is open. Press Ctrl+C to close it.
- If you want the tunnel to run in the background so your terminal stays free (Mac and Linux only),
  run the `ssh -f -N -M -S /tmp/supaspin-N.sock ...` command. It asks for your password if needed,
  then gives you your terminal back.
- If you no longer need the background tunnel and want to close it (Mac and Linux only), run
  `ssh -S /tmp/supaspin-N.sock -O exit you@your.server`. It answers "Exit request sent."
- If you want to check whether the background tunnel is running (Mac and Linux only), run
  `ssh -S /tmp/supaspin-N.sock -O check you@your.server`. It answers "Master running" if it is.
- If closing says "No such file or directory" but the tunnel still works, or starting a tunnel says
  "address already in use" and the check does not say "Master running", a leftover tunnel is in the
  way: run `pkill -f "[s]upaspin-N.sock"` (Mac and Linux only; the `[s]` is on purpose, it stops the command from
  stopping itself) and start the tunnel again.

While the tunnel is open, put `DATABASE_URL=postgresql://postgres:<password>@localhost:5441/postgres`
in your app's `.env` (the password is `POSTGRES_PASSWORD` in the instance's `.env`) and open Studio
at `http://localhost:3001`. On the server's own LAN you do not need a tunnel: use
`<LAN IP>:5441` and `http://<LAN IP>:3001`.

### Copying an instance's values into your app's .env

If you want your app's `.env` to point at an instance, run:

```bash
sudo supaspin push-env 1 igor@192.168.10.50:/Users/igor/app/.env      # a Mac or Linux computer
sudo supaspin push-env 1 "igor@192.168.10.50:C:/Users/igor/app/.env"  # a Windows computer
sudo supaspin push-env 1 /home/igor/app/.env                          # a file on this server
sudo supaspin push-env 1 --dry-run igor@192.168.10.50:/Users/igor/app/.env   # only show what would change
```

or choose "Copy instance values into another .env file" in the menu, which asks for everything
(instance, file, SSH user, port and key file). Options: `--port` (SSH port, default 22), `--key`
(key file), `--dry-run` (write nothing). If the SSH server needs a password, SSH asks for it in
your terminal, once.

What it does with the target file:

- Every `KEY=VALUE` line of the instance's `.env` is put in. If the key is already in the file
  (also as `export KEY=...`), its value is replaced. If it is not, the line is added at the end.
- Lines that start with `#` are ignored completely. `# ANON_KEY=old` is left as it is, and
  `ANON_KEY` counts as missing, so a new `ANON_KEY=...` line is added.
- Every other line (other keys, comments, blank lines, their order, line endings, the file's owner
  and permissions) stays exactly as it was. The file must already exist; it is never created.
- It never prints values, only key names: replaced, added, already correct, and how many other
  lines stayed untouched. It ends with SUCCESS or FAILED and the reason, and its exit status is
  0 or 1.

Safety net: before the file is changed, an untouched copy is kept in a private temporary folder
(in memory, in `/run`). After writing, the file is read back and compared with that copy: nothing
but the instance's keys may differ, and each of them must have the instance's value. If anything is
off, the copy is written back and checked. The copy is deleted at the end. Only if even the
write-back fails is it kept, and its path is printed so that you can restore the file by hand.

The other computer can be a Mac, a Linux computer or a Windows computer. It only needs an SSH server
with file copying (SFTP) switched on and this server must be able to reach it. Files are copied with
`scp`, so nothing has to be installed there. The SSH connection is made as the person who typed
`sudo`, so your own keys and known computers apply. If the connection fails, supaspin says why and
what to do: for example how to switch SSH on (Mac: System Settings > General > Sharing > Remote
Login; Linux: `sudo apt install openssh-server`; Windows: the optional feature "OpenSSH Server"),
how to install this server's key (`ssh-copy-id`), or how to trust a new computer.

### Deleting

Delete works on a half-created instance too and can be run again safely. It:

1. saves the instance's TLS certificate into namespace `supaspin-certs`,
2. uninstalls the Helm release, then deletes the namespace and the volumes' Kubernetes objects,
3. unmounts every image, removes its `/etc/fstab` line and deletes `<root>/<N>/`.

Creating that instance again restores the saved certificate, so repeated create/delete cycles do
not hit Let's Encrypt's limit of 5 certificates per hostname per week. If a create fails midway,
run `supaspin delete N` to clean up.

## Suspending an instance

An instance you are not using right now does not have to be deleted. `sudo supaspin suspend N`
stops all twelve of its pods and leaves everything else alone; `sudo supaspin resume N` starts
them again with the data they had.

```bash
sudo supaspin suspend 2     # about half a minute
sudo supaspin resume 2      # a minute or two: the database answers first, the rest wait for it
```

What the server gets back while an instance sleeps:

- the memory it had reserved (1.875 GiB) and everything it was actually using (1.5 to 2.5 GiB),
- its twelve pod slots, out of the 110 a node allows,
- its two ports on the server: Postgres on 544N and Studio on 300N are free for something else.

What is kept, untouched:

- the database and the uploaded files - the disk images stay mounted and keep their contents,
- the `.env` file with its passwords and keys, so your app needs no change when the instance is back,
- the HTTPS certificate, which goes on renewing itself while the instance sleeps,
- the hostname and every Kubernetes object. Until you resume it, the hostname answers "service
  unavailable" instead of not answering at all.

Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
because nothing is listening on its ports.

Suspending is written into the instance's stored settings, not done by hand to the running pods,
so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
stores the new memory limits and leaves it asleep; they take effect when you resume it.

## Memory per instance

| Service | Request | Limit | Highest use measured |
|---|---|---|---|
| db | 512 MiB | 2048 MiB | 273 MiB |
| analytics | 128 MiB | 1536 MiB | 694 MiB |
| vector | 128 MiB | 1024 MiB | 290 MiB |
| kong | 128 MiB | 768 MiB | 269 MiB |
| realtime | 128 MiB | 768 MiB | 300 MiB |
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
| **Total** | **1.875 GiB** | **9.5 GiB** | |

The request is what the scheduler sets aside for a service and can never be given to anything else.
The limit is the ceiling the kernel kills the container at. They do different jobs here.

The requests are deliberately small: the server is one machine shared with the production
workloads, and reserving what three instances really use would leave production nothing to start
in. The limits are roughly twice the highest use measured across three running instances on
2026-09-22 (right-hand column above).

The earlier limits sat right on top of normal use and the kernel kept killing the two logging
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
log files of every pod on the server and not only its own instance's, was killed 297 times on that
same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
its own.

Changing the numbers in this table only affects instances created afterwards. To put them on
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
the instance was installed, replaces the memory block in them and applies the chart again, so the
pods are replaced service by service and nothing else about the instance changes.

## Troubleshooting

- **Pods not ready.** `sudo supaspin` → Show instance → Pods and events, or
  `kubectl get pods -n supaspin-N` and `kubectl describe pod …`. `OOMKilled` means a pod hit its
  memory limit.
- **Certificate stays pending.** The hostname's DNS record must point at the server and port 80 must
  reach the ingress: `kubectl get certificate,order,challenge -n supaspin-N`.
- **"no free hostname slot".** Delete an instance or add a slot in Settings (and its DNS record).
- **Create failed.** `sudo supaspin delete N`, then create again.
- **Port already in use.** Postgres and Studio ports are fixed per number (`5440+N`, `3000+N`);
  another program is using it. Free the port or use another instance number.
- **After a reboot.** Images are mounted from `/etc/fstab` (`nofail`, after the storage root is
  available) and the pods restart on their own. Check the mounts with `findmnt | grep supabase`.
  Kubernetes can start a pod a moment before its image is mounted, so supaspin makes every empty
  mount-point folder read-only (`chattr +i`) until the image is mounted over it: an early pod fails
  and retries instead of setting up a fresh database on the bare folder.

## Uninstall

```bash
sudo bash /usr/share/supaspin/uninstall.sh
```

It asks before deleting the instances (their data is gone for good) and before removing the shared
cluster objects, then removes the app, its settings and its cache.

## Development

The tests use Python's built-in `unittest` and need no extra packages:

```bash
python3 -m unittest discover -s tests
```

### Versions

The version is written in three places that must always show the same number: the file `VERSION`, the
`VERSION` constant at the top of `supaspin`, and the line "Version x.y.z" at the top of this README. A test
fails if they differ. Every release is a git tag `vX.Y.Z` (for example `v1.1.0`) on the commit that has that
number in `VERSION`.

## License

MIT, see `LICENSE`.