supaspin/README.md

# supaspin

Version 1.6.4

Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
single-node [microk8s](https://microk8s.io) server, from a terminal menu.

Use it for testing and staging copies of an app. Each instance is numbered, has its own hostname,
its own database, capped disk and capped memory, and runs beside your other workloads without
endangering them. Nothing in it is specific to any one app.

## What each instance gets

| | |
|---|---|
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions); the certificate is from Let's Encrypt when that is enabled, otherwise self-signed |
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …); which addresses listen is set by **Port bind** |
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
| Disk | fixed-size ext4 image files, mounted at boot: **db 6 GB, storage 1 GB, every other volume 256 MB** |
| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |

Studio and pg-meta (the admin UI and its schema API) are **removed from the public gateway**: `/`
and `/pg/` return 404 on the public hostname. Studio on port `300N` has **no login of its own**, so
keep that port off the internet (do not forward it on your router).

Instance *N* uses hostname slot *N* from your settings, so the number of slots is the maximum number
of instances at once. Numbers are reused: create picks the lowest free one.

## Requirements

- A Debian/Ubuntu server with **microk8s** (or any Kubernetes with `kubectl` and `helm` on the
  path), with an ingress controller installed: **ingress-nginx** or **Traefik** (the microk8s
  `ingress` addon is either, depending on the version). With **Traefik**, its CRDs must be present
  on the cluster — they come with the Traefik installation itself; supaspin creates `Middleware`
  objects and does not install cluster-wide types.
- With **Let's Encrypt** enabled: **cert-manager** installed, DNS `A` records for every hostname
  slot pointing at the server, and ports 80 and 443 forwarded to it (Let's Encrypt fetches a test
  file over port 80 to issue each certificate). Without it: no cert-manager, no public DNS — each
  instance gets a self-signed certificate instead.
- Root access on that server. `python3` and `python3-yaml` (the installer adds the latter with `apt`).
  Self-signed certificates additionally need `openssl` on the server.

## Install

```bash
curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
```

**If your Gitea requires sign-in even for public repositories** (the download then fails with
"Could not download"), either pass a read-only token (Gitea → Settings → Applications, scope
`repository: read`):

```bash
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
```

or clone the repository and install from the checkout:

```bash
git clone ssh://git@git.didgibot.com:2226/public/supaspin.git && sudo bash supaspin/install.sh
```

Run it again to upgrade. It installs `/usr/local/bin/supaspin`, its values template and the
uninstaller in `/usr/share/supaspin/`.

Then start the app; the first run asks for your settings and prepares the cluster:

```bash
sudo supaspin
```

| Setting | Meaning |
|---|---|
| Storage root | folder that holds every instance's images and `.env` (`<root>/<N>/…`) |
| Node name | Kubernetes node the volumes are pinned to (detected) |
| Server LAN IP | address shown for the Postgres and Studio ports (detected) |
| Port bind | `lan` = LAN IP only (default; keeps Postgres and the login-less Studio off a public interface), `all` = every interface incl. localhost (for a box behind NAT running its own apps), `local` = localhost only |
| Ingress class | usually `public` on microk8s |
| Ingress controller | `traefik` or `nginx`; every instance's Ingress carries the nginx annotations (50 MiB upload cap, long timeouts) and, in Traefik mode, additionally points at a `Middleware` object with the same 50 MiB cap - so the cap is enforced no matter which of the two actually routes the hostname |
| Let's Encrypt | `y` fetches a real certificate per hostname (needs public DNS, ports 80/443, cert-manager); `n` gives every instance a self-signed certificate and creates no cert-manager objects |
| Let's Encrypt e-mail | for certificate expiry notices (asked only when Let's Encrypt is on) |
| Hostname slots | comma-separated; a name with a number in it keys that slot (`supa4.didgibot.com` = slot 4, so instance 4 gets it with ports 5444/3004), a plain name takes its position in the list |
| Public SSH host / port / user | only used to print the tunnel command |
| Mail settings | default SMTP settings for the auth service (changeable later) |

Settings live in `/etc/supaspin/config.yaml` (readable by root only). The first run also creates,
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-low` PriorityClass and the
`supaspin-certs` namespace; with Let's Encrypt on it also creates the `supaspin-letsencrypt`
ClusterIssuer. It then downloads the pinned Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once
into `/var/lib/supaspin/charts/`.

## Using it

`sudo supaspin` opens the menu (arrow keys, Enter, `q` to go back).

| Menu item | What it does |
|---|---|
| Create instance | picks the lowest free number, asks whether to use the saved mail settings or enter different ones, creates the disks, installs Supabase, waits until every pod is ready, writes `<root>/<N>/.env` |
| List instances | number, hostname, pods ready, memory in use, disk use per image, `.env` path, and for each instance the tunnel command with plain instructions (open it, keep it in the background, close it) |
| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
| Delete instance | asks you to type the number; removes everything (see below) |
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
| Mail settings | view and change the saved default mail settings |
| Settings | change any first-run setting |

The same actions are available without the menu:

```bash
sudo supaspin create [--ask-mail]     # --ask-mail: choose saved or different mail settings
sudo supaspin list
sudo supaspin show N                  # .env with secrets hidden, plus tunnel help
sudo supaspin env N                   # the full .env (contains secrets)
sudo supaspin tunnel N                # the tunnel commands, step by step
sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]   # see "Copying an instance's values"
sudo supaspin delete N [--yes]
sudo supaspin upgrade [N]             # apply this version's memory caps and log retention to running instances
sudo supaspin suspend N               # stop an instance, keep its data (see "Suspending an instance")
sudo supaspin resume N                # start it again
sudo supaspin mail                    # change the saved default mail settings
sudo supaspin setup                   # change settings
supaspin --version                    # prints "supaspin X.Y.Z"
```

### Connecting your app

Each instance writes `<root>/<N>/.env` (mode 600, owned by root) with the usual Supabase variable
names: `SUPABASE_URL`, `PUBLIC_SUPABASE_URL`, `API_EXTERNAL_URL`, `ANON_KEY`,
`PUBLIC_SUPABASE_ANON_KEY`, `JWT_ANON_KEY`, `SERVICE_ROLE_KEY`, `JWT_SERVICE_KEY`, `JWT_SECRET`,
`JWT_EXPIRY`, `POSTGRES_*`, `DATABASE_URL`, `DASHBOARD_USERNAME`, `DASHBOARD_PASSWORD` and
`GOTRUE_SMTP_*`. Read it with `sudo supaspin env N` and copy what your app needs.

The web API is public, so the URL and keys work from anywhere. **Postgres and Studio are not
public.** To reach them from your own computer you open a *tunnel*: a private connection from your
computer to the server that makes the instance's database and Studio (its admin screen) appear on
your own computer, at `localhost:544N` (database) and `http://localhost:300N` (Studio), for as long
as the tunnel is open. `sudo supaspin list` prints the exact commands for every instance
(`sudo supaspin tunnel N` prints them with a few more hints). Run them in a terminal on **your**
computer, not on the server.

**Which computer you use matters.** The commands for keeping the tunnel in the background and for
closing it work on **Mac and Linux only**. They do **not** work on **Windows**, because the ssh
program that comes with Windows does not support the "control file" they need. On Windows use the
first command (keep the tunnel open in its own PowerShell window) and close it with Ctrl+C or by
closing the window.

- If you want to open the tunnel and keep it open in your terminal window (Mac, Linux, Windows),
  run the `ssh -N ...` command. The window looks stuck and prints nothing: that is normal, it means
  the tunnel is open. Press Ctrl+C to close it.
- If you want the tunnel to run in the background so your terminal stays free (Mac and Linux only),
  run the `ssh -f -N -M -S /tmp/supaspin-N.sock ...` command. It asks for your password if needed,
  then gives you your terminal back.
- If you no longer need the background tunnel and want to close it (Mac and Linux only), run
  `ssh -S /tmp/supaspin-N.sock -O exit you@your.server`. It answers "Exit request sent."
- If you want to check whether the background tunnel is running (Mac and Linux only), run
  `ssh -S /tmp/supaspin-N.sock -O check you@your.server`. It answers "Master running" if it is.
- If closing says "No such file or directory" but the tunnel still works, or starting a tunnel says
  "address already in use" and the check does not say "Master running", a leftover tunnel is in the
  way: run `pkill -f "[s]upaspin-N.sock"` (Mac and Linux only; the `[s]` is on purpose, it stops the command from
  stopping itself) and start the tunnel again.

While the tunnel is open, put `DATABASE_URL=postgresql://postgres:<password>@localhost:5441/postgres`
in your app's `.env` (the password is `POSTGRES_PASSWORD` in the instance's `.env`) and open Studio
at `http://localhost:3001`. On the server's own LAN you do not need a tunnel: use
`<LAN IP>:5441` and `http://<LAN IP>:3001`.

### Copying an instance's values into your app's .env

If you want your app's `.env` to point at an instance, run:

```bash
sudo supaspin push-env 1 igor@192.168.10.50:/Users/igor/app/.env      # a Mac or Linux computer
sudo supaspin push-env 1 "igor@192.168.10.50:C:/Users/igor/app/.env"  # a Windows computer
sudo supaspin push-env 1 /home/igor/app/.env                          # a file on this server
sudo supaspin push-env 1 --dry-run igor@192.168.10.50:/Users/igor/app/.env   # only show what would change
```

or choose "Copy instance values into another .env file" in the menu, which asks for everything
(instance, file, SSH user, port and key file). Options: `--port` (SSH port, default 22), `--key`
(key file), `--dry-run` (write nothing). If the SSH server needs a password, SSH asks for it in
your terminal, once.

What it does with the target file:

- Every `KEY=VALUE` line of the instance's `.env` is put in. If the key is already in the file
  (also as `export KEY=...`), its value is replaced. If it is not, the line is added at the end.
- Lines that start with `#` are ignored completely. `# ANON_KEY=old` is left as it is, and
  `ANON_KEY` counts as missing, so a new `ANON_KEY=...` line is added.
- Every other line (other keys, comments, blank lines, their order, line endings, the file's owner
  and permissions) stays exactly as it was. The file must already exist; it is never created.
- It never prints values, only key names: replaced, added, already correct, and how many other
  lines stayed untouched. It ends with SUCCESS or FAILED and the reason, and its exit status is
  0 or 1.

Safety net: before the file is changed, an untouched copy is kept in a private temporary folder
(in memory, in `/run`). After writing, the file is read back and compared with that copy: nothing
but the instance's keys may differ, and each of them must have the instance's value. If anything is
off, the copy is written back and checked. The copy is deleted at the end. Only if even the
write-back fails is it kept, and its path is printed so that you can restore the file by hand.

The other computer can be a Mac, a Linux computer or a Windows computer. It only needs an SSH server
with file copying (SFTP) switched on and this server must be able to reach it. Files are copied with
`scp`, so nothing has to be installed there. The SSH connection is made as the person who typed
`sudo`, so your own keys and known computers apply. If the connection fails, supaspin says why and
what to do: for example how to switch SSH on (Mac: System Settings > General > Sharing > Remote
Login; Linux: `sudo apt install openssh-server`; Windows: the optional feature "OpenSSH Server"),
how to install this server's key (`ssh-copy-id`), or how to trust a new computer.

### Deleting

Delete works on a half-created instance too and can be run again safely. It:

1. saves the instance's TLS certificate into namespace `supaspin-certs`,
2. uninstalls the Helm release, then deletes the namespace and the volumes' Kubernetes objects,
3. unmounts every image, removes its `/etc/fstab` line and deletes `<root>/<N>/`.

Creating that instance again restores the saved certificate, so repeated create/delete cycles do
not hit Let's Encrypt's limit of 5 certificates per hostname per week. If a create fails midway,
run `supaspin delete N` to clean up.

## Suspending an instance

An instance you are not using right now does not have to be deleted. It can be stopped and
started again with its data, from the menu:

```
sudo supaspin  ->  Show instance (and suspend or resume it)  ->  pick the instance
                                                             ->  Suspend it (stop it, keep its data)
```

The picker names every instance with its state next to it, so you can see at a glance which ones
are awake. Inside an instance the same line reads **Resume it** when the instance is asleep, so
there is one line to look for either way and nothing to remember. Stopping takes about half a
minute; starting takes a minute or two, because the database answers first and the rest wait for it.

The same two actions without the menu, if you prefer typing:

```bash
sudo supaspin suspend 2
sudo supaspin resume 2
```

What the server gets back while an instance sleeps:

- the memory it had reserved (1.875 GiB) and everything it was actually using (1.5 to 2.5 GiB),
- its twelve pod slots, out of the 110 a node allows,
- its two ports on the server: Postgres on 544N and Studio on 300N are free for something else.

What is kept, untouched:

- the database and the uploaded files - the disk images stay mounted and keep their contents,
- the `.env` file with its passwords and keys, so your app needs no change when the instance is back,
- the HTTPS certificate, which goes on renewing itself while the instance sleeps,
- the hostname and every Kubernetes object. Until you resume it, the hostname answers "service
  unavailable" instead of not answering at all.

Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
because nothing is listening on its ports. Asking for the tunnel command of a sleeping instance
says so instead of handing you a command that cannot work.

Suspending is written into the instance's stored settings, not done by hand to the running pods,
so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
stores the new memory limits and leaves it asleep; they take effect when you resume it.

## Memory per instance

| Service | Request | Limit | Highest use measured |
|---|---|---|---|
| db | 512 MiB | 2048 MiB | 273 MiB |
| analytics | 128 MiB | 1536 MiB | 694 MiB |
| vector | 128 MiB | 512 MiB | 290 MiB |
| kong | 128 MiB | 768 MiB | 269 MiB |
| realtime | 128 MiB | 768 MiB | 300 MiB |
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
| **Total** | **1.875 GiB** | **9 GiB** | |

The request is what the scheduler sets aside for a service and can never be given to anything else.
The limit is the ceiling the kernel kills the container at. They do different jobs here.

The requests are deliberately small: the server is one machine shared with the production
workloads, and reserving what three instances really use would leave production nothing to start
in. The limits are roughly twice the highest use measured across three running instances on
2026-09-22 (right-hand column above).

The earlier limits sat right on top of normal use and the kernel kept killing the two logging
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
(imgproxy) reads the uploads from the storage volume, so it has no image of its own.

On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
night. pg_cron in this image only runs jobs from the `postgres` database, so the job lives there
and reaches the log tables in `_supabase` over a local dblink connection. The dblink extension
and the helper function live in the `extensions` schema, where the app's database owner has no
rights, so instance code can never call them. That bounds both the
db image and vector's backlog, so vector is back to the 512 MiB ceiling the production supabase
charts carry.

Changing the numbers in this table only affects instances created afterwards. To put them on
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
the instance was installed, replaces the memory block in them and applies the chart again, so the
pods are replaced service by service and nothing else about the instance changes.

## Troubleshooting

- **Pods not ready.** `sudo supaspin` → Show instance → Pods and events, or
  `kubectl get pods -n supaspin-N` and `kubectl describe pod …`. `OOMKilled` means a pod hit its
  memory limit.
- **Certificate stays pending.** The hostname's DNS record must point at the server and port 80 must
  reach the ingress: `kubectl get certificate,order,challenge -n supaspin-N`.
- **"no free hostname slot".** Delete an instance or add a slot in Settings (and its DNS record).
- **Create failed** with `no matches for kind "Middleware" in version "traefik.io/v1alpha1"`. The
  config says Traefik but the cluster has no Traefik CRDs: either install Traefik (its CRDs come
  with it) or set the ingress controller to `nginx` in Settings.
- **Create failed.** `sudo supaspin delete N`, then create again.
- **Port already in use.** Postgres and Studio ports are fixed per number (`5440+N`, `3000+N`);
  another program is using it. Free the port or use another instance number.
- **After a reboot.** Images are mounted from `/etc/fstab` (`nofail`, after the storage root is
  available) and the pods restart on their own. Check the mounts with `findmnt | grep supabase`.
  Kubernetes can start a pod a moment before its image is mounted, so supaspin makes every empty
  mount-point folder read-only (`chattr +i`) until the image is mounted over it: an early pod fails
  and retries instead of setting up a fresh database on the bare folder.

## Uninstall

```bash
sudo bash /usr/share/supaspin/uninstall.sh
```

It asks before deleting the instances (their data is gone for good) and before removing the shared
cluster objects, then removes the app, its settings and its cache.

## Development

The tests use Python's built-in `unittest` and need no extra packages:

```bash
python3 -m unittest discover -s tests
```

### Versions

The version is written in three places that must always show the same number: the file `VERSION`, the
`VERSION` constant at the top of `supaspin`, and the line "Version x.y.z" at the top of this README. A test
fails if they differ. Every release is a git tag `vX.Y.Z` (for example `v1.1.0`) on the commit that has that
number in `VERSION`.

## License

MIT, see `LICENSE`.