Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
777fc8aa9a | ||
|
|
4a267c5336 | ||
|
|
d4fedd718f | ||
|
|
2f47ddedf1 | ||
|
|
80b84afb1e | ||
|
|
4d0fa4fe69 | ||
|
|
2c6e383617 | ||
|
|
ea3b705008 | ||
|
|
a923839c47 | ||
|
|
4bcdf6907c | ||
|
|
2e4e46c878 | ||
|
|
99fdf7a0ab |
@@ -21,8 +21,14 @@ The number is `MAJOR.MINOR.PATCH` (semantic versioning: `1.1.1` is major 1, mino
|
||||
4. **Tag every release.** After the pull request is merged, create the git tag `vX.Y.Z` (for example `v1.1.1`) on
|
||||
the commit on `main` that contains that number in `VERSION`. The tag name must equal `VERSION` with a `v` in
|
||||
front. A version without a tag is not released. Never move or delete a tag that was already published; if a
|
||||
release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/igor/supaspin/tags` with
|
||||
release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/public/supaspin/tags` with
|
||||
`tag_name`, `target` (the commit) and `message`.
|
||||
|
||||
The owner in an API address must be `public`. The repository used to sit under the user `igor` and Gitea still
|
||||
forwards the old address for anything a browser or git does, but it answers an API call to
|
||||
`/api/v1/repos/igor/supaspin/...` with a redirect that curl does not follow for a POST: the command returns an
|
||||
empty body and no error, which looks exactly like a rejected token. Opening or merging a pull request needs the
|
||||
same `public` owner.
|
||||
5. **Check the result.** Reinstall from the merged `main` (`sudo bash install.sh`), then run `supaspin --version` and
|
||||
compare it with `VERSION` and the tag.
|
||||
6. If you find a version in `VERSION` that has no tag, or a tag that does not match `VERSION`, tell Igor before doing
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# supaspin
|
||||
|
||||
Version 1.2.0
|
||||
Version 1.6.2
|
||||
|
||||
Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
|
||||
single-node [microk8s](https://microk8s.io) server, from a terminal menu.
|
||||
@@ -13,11 +13,11 @@ endangering them. Nothing in it is specific to any one app.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions), with a Let's Encrypt certificate |
|
||||
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
|
||||
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions); the certificate is from Let's Encrypt when that is enabled, otherwise self-signed |
|
||||
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …); which addresses listen is set by **Port bind** |
|
||||
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
|
||||
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
|
||||
| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
|
||||
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
|
||||
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
|
||||
@@ -32,15 +32,19 @@ of instances at once. Numbers are reused: create picks the lowest free one.
|
||||
## Requirements
|
||||
|
||||
- A Debian/Ubuntu server with **microk8s** (or any Kubernetes with `kubectl` and `helm` on the
|
||||
path), with an **ingress-nginx** ingress class and **cert-manager** installed.
|
||||
- DNS `A` records for every hostname slot pointing at the server, and ports 80 and 443 forwarded to
|
||||
it (Let's Encrypt fetches a test file over port 80 to issue each certificate).
|
||||
path), with an ingress controller installed: **ingress-nginx** or **Traefik** (the microk8s
|
||||
`ingress` addon is either, depending on the version).
|
||||
- With **Let's Encrypt** enabled: **cert-manager** installed, DNS `A` records for every hostname
|
||||
slot pointing at the server, and ports 80 and 443 forwarded to it (Let's Encrypt fetches a test
|
||||
file over port 80 to issue each certificate). Without it: no cert-manager, no public DNS — each
|
||||
instance gets a self-signed certificate instead.
|
||||
- Root access on that server. `python3` and `python3-yaml` (the installer adds the latter with `apt`).
|
||||
Self-signed certificates additionally need `openssl` on the server.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
```
|
||||
|
||||
**If your Gitea requires sign-in even for public repositories** (the download then fails with
|
||||
@@ -48,13 +52,13 @@ curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | s
|
||||
`repository: read`):
|
||||
|
||||
```bash
|
||||
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
|
||||
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
|
||||
```
|
||||
|
||||
or clone the repository and install from the checkout:
|
||||
|
||||
```bash
|
||||
git clone ssh://git@git.didgibot.com:2226/igor/supaspin.git && sudo bash supaspin/install.sh
|
||||
git clone ssh://git@git.didgibot.com:2226/public/supaspin.git && sudo bash supaspin/install.sh
|
||||
```
|
||||
|
||||
Run it again to upgrade. It installs `/usr/local/bin/supaspin`, its values template and the
|
||||
@@ -70,17 +74,21 @@ sudo supaspin
|
||||
|---|---|
|
||||
| Storage root | folder that holds every instance's images and `.env` (`<root>/<N>/…`) |
|
||||
| Node name | Kubernetes node the volumes are pinned to (detected) |
|
||||
| Server LAN IP | address the Postgres and Studio ports are bound to (detected) |
|
||||
| Let's Encrypt e-mail | for certificate expiry notices |
|
||||
| Server LAN IP | address shown for the Postgres and Studio ports (detected) |
|
||||
| Port bind | `lan` = LAN IP only (default; keeps Postgres and the login-less Studio off a public interface), `all` = every interface incl. localhost (for a box behind NAT running its own apps), `local` = localhost only |
|
||||
| Ingress class | usually `public` on microk8s |
|
||||
| Hostname slots | comma-separated; the first is instance 1, the second instance 2, … |
|
||||
| Ingress controller | `traefik` or `nginx`; Traefik instances get a Middleware that caps uploads at 50 MiB, nginx gets the equivalent annotations |
|
||||
| Let's Encrypt | `y` fetches a real certificate per hostname (needs public DNS, ports 80/443, cert-manager); `n` gives every instance a self-signed certificate and creates no cert-manager objects |
|
||||
| Let's Encrypt e-mail | for certificate expiry notices (asked only when Let's Encrypt is on) |
|
||||
| Hostname slots | comma-separated; a name with a number in it keys that slot (`supa4.didgibot.com` = slot 4, so instance 4 gets it with ports 5444/3004), a plain name takes its position in the list |
|
||||
| Public SSH host / port / user | only used to print the tunnel command |
|
||||
| Mail settings | default SMTP settings for the auth service (changeable later) |
|
||||
|
||||
Settings live in `/etc/supaspin/config.yaml` (readable by root only). The first run also creates,
|
||||
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-letsencrypt` ClusterIssuer,
|
||||
the `supaspin-low` PriorityClass and the `supaspin-certs` namespace, and downloads the pinned
|
||||
Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/charts/`.
|
||||
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-low` PriorityClass and the
|
||||
`supaspin-certs` namespace; with Let's Encrypt on it also creates the `supaspin-letsencrypt`
|
||||
ClusterIssuer. It then downloads the pinned Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once
|
||||
into `/var/lib/supaspin/charts/`.
|
||||
|
||||
## Using it
|
||||
|
||||
@@ -90,10 +98,10 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
|
||||
|---|---|
|
||||
| Create instance | picks the lowest free number, asks whether to use the saved mail settings or enter different ones, creates the disks, installs Supabase, waits until every pod is ready, writes `<root>/<N>/.env` |
|
||||
| List instances | number, hostname, pods ready, memory in use, disk use per image, `.env` path, and for each instance the tunnel command with plain instructions (open it, keep it in the background, close it) |
|
||||
| Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
|
||||
| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
|
||||
| Delete instance | asks you to type the number; removes everything (see below) |
|
||||
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
|
||||
| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
|
||||
| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
|
||||
| Mail settings | view and change the saved default mail settings |
|
||||
| Settings | change any first-run setting |
|
||||
|
||||
@@ -108,9 +116,11 @@ sudo supaspin tunnel N # the tunnel commands, step by step
|
||||
sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run] # see "Copying an instance's values"
|
||||
sudo supaspin delete N [--yes]
|
||||
sudo supaspin upgrade [N] # apply this version's memory caps to running instances
|
||||
sudo supaspin suspend N # stop an instance, keep its data (see "Suspending an instance")
|
||||
sudo supaspin resume N # start it again
|
||||
sudo supaspin mail # change the saved default mail settings
|
||||
sudo supaspin setup # change settings
|
||||
supaspin --version # prints "supaspin 1.2.0"
|
||||
supaspin --version # prints "supaspin 1.3.1"
|
||||
```
|
||||
|
||||
### Connecting your app
|
||||
@@ -209,17 +219,62 @@ Creating that instance again restores the saved certificate, so repeated create/
|
||||
not hit Let's Encrypt's limit of 5 certificates per hostname per week. If a create fails midway,
|
||||
run `supaspin delete N` to clean up.
|
||||
|
||||
## Suspending an instance
|
||||
|
||||
An instance you are not using right now does not have to be deleted. It can be stopped and
|
||||
started again with its data, from the menu:
|
||||
|
||||
```
|
||||
sudo supaspin -> Show instance (and suspend or resume it) -> pick the instance
|
||||
-> Suspend it (stop it, keep its data)
|
||||
```
|
||||
|
||||
The picker names every instance with its state next to it, so you can see at a glance which ones
|
||||
are awake. Inside an instance the same line reads **Resume it** when the instance is asleep, so
|
||||
there is one line to look for either way and nothing to remember. Stopping takes about half a
|
||||
minute; starting takes a minute or two, because the database answers first and the rest wait for it.
|
||||
|
||||
The same two actions without the menu, if you prefer typing:
|
||||
|
||||
```bash
|
||||
sudo supaspin suspend 2
|
||||
sudo supaspin resume 2
|
||||
```
|
||||
|
||||
What the server gets back while an instance sleeps:
|
||||
|
||||
- the memory it had reserved (1.875 GiB) and everything it was actually using (1.5 to 2.5 GiB),
|
||||
- its twelve pod slots, out of the 110 a node allows,
|
||||
- its two ports on the server: Postgres on 544N and Studio on 300N are free for something else.
|
||||
|
||||
What is kept, untouched:
|
||||
|
||||
- the database and the uploaded files - the disk images stay mounted and keep their contents,
|
||||
- the `.env` file with its passwords and keys, so your app needs no change when the instance is back,
|
||||
- the HTTPS certificate, which goes on renewing itself while the instance sleeps,
|
||||
- the hostname and every Kubernetes object. Until you resume it, the hostname answers "service
|
||||
unavailable" instead of not answering at all.
|
||||
|
||||
Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
|
||||
state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
|
||||
because nothing is listening on its ports. Asking for the tunnel command of a sleeping instance
|
||||
says so instead of handing you a command that cannot work.
|
||||
|
||||
Suspending is written into the instance's stored settings, not done by hand to the running pods,
|
||||
so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
|
||||
stores the new memory limits and leaves it asleep; they take effect when you resume it.
|
||||
|
||||
## Memory per instance
|
||||
|
||||
| Service | Request | Limit | Highest use measured |
|
||||
|---|---|---|---|
|
||||
| db | 512 MiB | 2048 MiB | 273 MiB |
|
||||
| analytics | 128 MiB | 1536 MiB | 694 MiB |
|
||||
| vector | 128 MiB | 1024 MiB | 290 MiB |
|
||||
| vector | 128 MiB | 512 MiB | 290 MiB |
|
||||
| kong | 128 MiB | 768 MiB | 269 MiB |
|
||||
| realtime | 128 MiB | 768 MiB | 300 MiB |
|
||||
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
|
||||
| **Total** | **1.875 GiB** | **9.5 GiB** | |
|
||||
| **Total** | **1.875 GiB** | **9 GiB** | |
|
||||
|
||||
The request is what the scheduler sets aside for a service and can never be given to anything else.
|
||||
The limit is the ceiling the kernel kills the container at. They do different jobs here.
|
||||
@@ -231,13 +286,23 @@ in. The limits are roughly twice the highest use measured across three running i
|
||||
|
||||
The earlier limits sat right on top of normal use and the kernel kept killing the two logging
|
||||
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
|
||||
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
|
||||
log files of every pod on the server and not only its own instance's, was killed 297 times on that
|
||||
same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
|
||||
other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
|
||||
the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
|
||||
in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
|
||||
its own.
|
||||
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
|
||||
instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
|
||||
could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
|
||||
cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
|
||||
Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
|
||||
(imgproxy) reads the uploads from the storage volume, so it has no image of its own.
|
||||
|
||||
On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
|
||||
it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
|
||||
100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
|
||||
every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
|
||||
minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
|
||||
job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
|
||||
night. pg_cron in this image only runs jobs from the `postgres` database, so the job lives there
|
||||
and reaches the log tables in `_supabase` over a local dblink connection. That bounds both the
|
||||
db image and vector's backlog, so vector is back to the 512 MiB ceiling the production supabase
|
||||
charts carry.
|
||||
|
||||
Changing the numbers in this table only affects instances created afterwards. To put them on
|
||||
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# One-line install (run on the Kubernetes server):
|
||||
# curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
# curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
# Installs /usr/local/bin/supaspin and its Helm values template. Safe to run again to upgrade.
|
||||
#
|
||||
# If your Gitea requires sign-in even for public repositories, give a read-only token:
|
||||
@@ -12,7 +12,7 @@
|
||||
# SUPASPIN_TOKEN=<token> Gitea token (read access to the repository) for the download
|
||||
set -euo pipefail
|
||||
|
||||
REPO="https://git.didgibot.com/igor/supaspin"
|
||||
REPO="https://git.didgibot.com/public/supaspin"
|
||||
REF="${SUPASPIN_REF:-main}"
|
||||
|
||||
[ "$(id -u)" -eq 0 ] || { echo "Run as root: curl -fsSL $REPO/raw/branch/main/install.sh | sudo bash" >&2; exit 1; }
|
||||
|
||||
@@ -8,6 +8,10 @@ Run as root: supaspin (menu)
|
||||
supaspin upgrade [N]
|
||||
Puts this version's memory caps on instances that already run, one service
|
||||
at a time. Without N every instance. Data and .env files are not touched.
|
||||
supaspin suspend N | resume N
|
||||
Suspend stops every pod of one instance and gives back its memory, its pod
|
||||
slots and its two server ports. The disks, the passwords in .env, the
|
||||
certificate and the hostname stay. Resume starts it again with its data.
|
||||
supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]
|
||||
TARGET = a .env file on this server (/path/.env) or on another computer
|
||||
(user@host:/path/.env, Windows: user@host:C:/path/.env). Puts the instance's
|
||||
@@ -35,7 +39,7 @@ import tty
|
||||
|
||||
import yaml
|
||||
|
||||
VERSION = "1.2.0"
|
||||
VERSION = "1.6.2"
|
||||
CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
|
||||
STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
|
||||
CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
|
||||
@@ -68,14 +72,62 @@ DB_UID, DB_GID = 100, 101
|
||||
# kong 260 / 269 / 265 MiB against 512
|
||||
# studio 203, storage 187, rest 162 MiB against 256
|
||||
# db 163 / 175 / 273 MiB against 2048 - left alone, it has room
|
||||
# On 2026-10-06 instance 3 filled its 2 GB database image with logs: analytics (Logflare) never
|
||||
# deletes what it stored, the db died of it, and vector's undelivered backlog was killed against
|
||||
# the 1024 MiB ceiling every few minutes. The fix is setup_log_retention: a pg_cron job deletes
|
||||
# analytics logs older than LOG_RETENTION in every instance, which also bounds vector's backlog,
|
||||
# so vector is back to the 512 MiB ceiling the production supabase charts carry.
|
||||
# Raising a limit costs nothing until a container actually asks for the memory; raising a request
|
||||
# would take the memory away from everything else on the node straight away.
|
||||
SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
|
||||
"realtime", "rest", "storage", "studio", "vector"]
|
||||
MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
|
||||
"vector": (128, 1024)}
|
||||
"vector": (128, 512)}
|
||||
DEFAULT_MEMORY = (128, 512)
|
||||
|
||||
# Analytics logs: deleted from every instance's _supabase database when older than this. Logflare
|
||||
# stores logs in _analytics.log_events_* tables and never deletes them; without this job the
|
||||
# capped db image fills up and the database dies (instance 3, 2026-10-06).
|
||||
# pg_cron in the supabase image only runs jobs from the database named in cron.database_name,
|
||||
# which the image pins to `postgres`, while the logs live in `_supabase`. The job therefore runs
|
||||
# in `postgres` and reaches the logs over a local dblink connection, which needs no password: the
|
||||
# image trusts local socket connections for supabase_admin.
|
||||
# dblink and the helper function MUST live in the `extensions` schema, not `public`: the app's
|
||||
# migrations revoke API-role privileges on every routine in public, and `postgres` (the role
|
||||
# Prisma connects as) is no superuser here - revoking someone else's routines fails with
|
||||
# "permission denied for function dblink_connect_u" (42501, maralago 0_init, 2026-10-08).
|
||||
LOG_RETENTION = "2 days"
|
||||
LOG_RETENTION_JOB = "supaspin-log-retention"
|
||||
RETENTION_SQL = f"""
|
||||
create extension if not exists pg_cron;
|
||||
create extension if not exists dblink with schema extensions;
|
||||
alter extension dblink set schema extensions;
|
||||
drop function if exists public.supaspin_delete_old_logs();
|
||||
create or replace function extensions.supaspin_delete_old_logs() returns void
|
||||
language plpgsql as $fn$
|
||||
declare
|
||||
conn text := 'supaspin_lr_' || substr(md5(clock_timestamp()::text), 1, 8);
|
||||
t text;
|
||||
begin
|
||||
perform extensions.dblink_connect(conn, 'dbname=_supabase user=supabase_admin');
|
||||
for t in
|
||||
select tablename from extensions.dblink(conn,
|
||||
'select tablename from pg_tables where schemaname = ''_analytics'' and tablename like ''log_events\\_%''')
|
||||
as log_tables(tablename text)
|
||||
loop
|
||||
perform extensions.dblink_exec(conn,
|
||||
format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t));
|
||||
end loop;
|
||||
perform extensions.dblink_disconnect(conn);
|
||||
exception when others then
|
||||
begin perform extensions.dblink_disconnect(conn); exception when others then null; end;
|
||||
raise;
|
||||
end $fn$;
|
||||
select extensions.supaspin_delete_old_logs();
|
||||
delete from cron.job where jobname = '{LOG_RETENTION_JOB}';
|
||||
select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select extensions.supaspin_delete_old_logs()');
|
||||
""".strip()
|
||||
|
||||
DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
|
||||
"admin_email": "admin@example.com", "sender_name": "Supabase"}
|
||||
|
||||
@@ -113,6 +165,54 @@ def tls_secret(n):
|
||||
return f"supa{n}-tls"
|
||||
|
||||
|
||||
_SLOT_NUM = re.compile(r"(\d+)")
|
||||
|
||||
|
||||
def parse_slots(text):
|
||||
"""Slots {number: hostname} from the comma-separated setup answer. A hostname with a
|
||||
number in it keys the slot by that number - supa4.didgibot.com is slot 4 - so the
|
||||
instance number, its ports and the hostname digit stay in step; a name without a
|
||||
number takes its position in the list (the first name is slot 1)."""
|
||||
slots = {}
|
||||
for i, h in enumerate([p.strip() for p in text.split(",") if p.strip()], 1):
|
||||
m = _SLOT_NUM.search(h)
|
||||
n = int(m.group(1)) if m else i
|
||||
if n in slots:
|
||||
raise ValueError(f"'{h}' wants slot {n}, which '{slots[n]}' already takes.")
|
||||
slots[n] = h
|
||||
return slots
|
||||
|
||||
|
||||
def buffering_middleware(n):
|
||||
"""Name of the Traefik Middleware that caps the request body size for instance n."""
|
||||
return f"supa{n}-buffering"
|
||||
|
||||
|
||||
def ingress_annotations(cfg, n):
|
||||
"""Ingress annotations for instance n.
|
||||
Both controllers' annotations are always present: an annotation only takes effect
|
||||
when a controller of that family actually routes the ingress, so on a Traefik
|
||||
cluster the nginx keys sit inert (and vice versa) and nothing needs reconfiguring
|
||||
when the router changes. The upload cap therefore lives in two places that agree:
|
||||
proxy-body-size for nginx, and the Middleware object the traefik annotation points
|
||||
at. The chart's own rewrite-target default must be nullified or every API path
|
||||
breaks. The cert-manager annotation orders a certificate from the
|
||||
supaspin-letsencrypt issuer; without it the TLS secret keeps whatever supaspin put
|
||||
there (a saved or a self-signed certificate)."""
|
||||
anns = {}
|
||||
if cfg.get("lets_encrypt", True):
|
||||
anns["cert-manager.io/cluster-issuer"] = ISSUER
|
||||
# The chart default rewrites every path to "/", which breaks all APIs; null removes it.
|
||||
anns["nginx.ingress.kubernetes.io/rewrite-target"] = None
|
||||
anns["nginx.ingress.kubernetes.io/proxy-body-size"] = "50m"
|
||||
anns["nginx.ingress.kubernetes.io/proxy-read-timeout"] = "3600"
|
||||
anns["nginx.ingress.kubernetes.io/proxy-send-timeout"] = "3600"
|
||||
if cfg.get("ingress_controller", "nginx") == "traefik":
|
||||
anns["traefik.ingress.kubernetes.io/router.middlewares"] = \
|
||||
f"{namespace(n)}-{buffering_middleware(n)}@kubernetescrd"
|
||||
return anns
|
||||
|
||||
|
||||
def lowest_free_number(slots, taken):
|
||||
"""Lowest slot number that is not in `taken`; None if every slot is used."""
|
||||
for n in sorted(int(k) for k in slots):
|
||||
@@ -175,13 +275,15 @@ def instance_values(cfg, n, sec, mail, template_text):
|
||||
subs.update({
|
||||
"PUBLIC_URL": f"https://{host}", "HOST": host,
|
||||
"TLS_SECRET": tls_secret(n), "INGRESS_CLASS": cfg["ingress_class"],
|
||||
"CLUSTER_ISSUER": ISSUER, "PROJECT_NAME": f"supaspin-{n}",
|
||||
"PROJECT_NAME": f"supaspin-{n}",
|
||||
"INGRESS_ANNOTATIONS": ingress_annotations(cfg, n),
|
||||
"SMTP_HOST": mail["host"], "SMTP_PORT": mail["port"], "SMTP_USER": mail["user"],
|
||||
"SMTP_PASS": mail["password"], "SMTP_ADMIN_EMAIL": mail["admin_email"],
|
||||
"SMTP_SENDER_NAME": mail["sender_name"],
|
||||
})
|
||||
# json.dumps gives a quoted YAML-safe string; the port is quoted as well (env values are strings).
|
||||
quoted = {k: json.dumps(str(v)) for k, v in subs.items()}
|
||||
# A dict is dumped as JSON too: JSON flow syntax is valid YAML, so the annotation map drops in.
|
||||
quoted = {k: (json.dumps(v) if isinstance(v, dict) else json.dumps(str(v))) for k, v in subs.items()}
|
||||
text = re.sub(r"\$\{(\w+)\}", lambda m: quoted[m.group(1)], template_text)
|
||||
values = yaml.safe_load(text)
|
||||
apply_memory(values)
|
||||
@@ -208,6 +310,24 @@ def apply_memory(values):
|
||||
return values
|
||||
|
||||
|
||||
def set_replicas(values, count):
|
||||
"""How many copies of each service to run. Zero stops the instance: the pods go away and give
|
||||
back their memory, their processor time, their pod slots and the two ports they hold on the
|
||||
server, while the disks, the passwords, the certificate and the hostname stay exactly as they
|
||||
are. One starts it again."""
|
||||
dep = values.setdefault("deployment", {})
|
||||
for svc in SERVICES:
|
||||
dep.setdefault(svc, {})["replicaCount"] = count
|
||||
return values
|
||||
|
||||
|
||||
def is_suspended(values):
|
||||
"""True when the stored settings say to run no copy of any service."""
|
||||
dep = values.get("deployment") or {}
|
||||
counts = [(dep.get(svc) or {}).get("replicaCount", 1) for svc in SERVICES]
|
||||
return all(c == 0 for c in counts)
|
||||
|
||||
|
||||
def memory_limits(values):
|
||||
"""{service: limit} as the values dictionary has it, for telling before from after."""
|
||||
dep = values.get("deployment") or {}
|
||||
@@ -414,15 +534,16 @@ def strip_kong_services(text, names):
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def post_render(text, n, lan_ip):
|
||||
def post_render(text, n, cfg):
|
||||
"""Edit the rendered chart: fixed host ports on db and studio, no public Studio/pg-meta route."""
|
||||
bind_ip = host_bind_ip(cfg)
|
||||
docs = [d for d in yaml.safe_load_all(text) if d]
|
||||
for d in docs:
|
||||
kind, name = d.get("kind"), d.get("metadata", {}).get("name", "")
|
||||
if kind == "StatefulSet" and name == f"{fullname(n)}-db":
|
||||
_host_port(d, pg_port(n), lan_ip)
|
||||
_host_port(d, pg_port(n), bind_ip)
|
||||
elif kind == "Deployment" and name == f"{fullname(n)}-studio":
|
||||
_host_port(d, studio_port(n), lan_ip)
|
||||
_host_port(d, studio_port(n), bind_ip)
|
||||
elif kind == "ConfigMap" and "kong.yml" in (d.get("data") or {}):
|
||||
d["data"]["kong.yml"] = strip_kong_services(d["data"]["kong.yml"], ("meta", "dashboard"))
|
||||
for d in docs: # keep multi-line scripts readable in the release manifest
|
||||
@@ -432,10 +553,21 @@ def post_render(text, n, lan_ip):
|
||||
return yaml.safe_dump_all(docs, sort_keys=False, default_flow_style=False)
|
||||
|
||||
|
||||
def host_bind_ip(cfg):
|
||||
"""Address the Postgres/Studio host ports listen on. `lan` (the default) is the LAN IP
|
||||
only; on a machine with a public interface that keeps Postgres and the login-less Studio
|
||||
off the internet. `all` is every interface - LAN and localhost - for a box behind NAT
|
||||
that runs its own apps talking to localhost. `local` is localhost only."""
|
||||
return {"lan": cfg["lan_ip"], "local": "127.0.0.1", "all": None}[cfg.get("port_bind", "lan")]
|
||||
|
||||
|
||||
def _host_port(doc, port, ip):
|
||||
c = doc["spec"]["template"]["spec"]["containers"][0]
|
||||
p = c["ports"][0]
|
||||
p["hostPort"] = port
|
||||
if ip is not None:
|
||||
# None = bind every interface (the `all` mode); omitting hostIP makes the kubelet
|
||||
# listen on 0.0.0.0, which a hostIP cannot express.
|
||||
p["hostIP"] = ip
|
||||
if doc.get("kind") == "Deployment":
|
||||
# A port on the server itself can be held by one pod at a time. The default rollout starts
|
||||
@@ -638,18 +770,21 @@ def ensure_cluster_objects(cfg, log):
|
||||
{"apiVersion": "storage.k8s.io/v1", "kind": "StorageClass",
|
||||
"metadata": {"name": STORAGE_CLASS}, "provisioner": "kubernetes.io/no-provisioner",
|
||||
"reclaimPolicy": "Retain", "volumeBindingMode": "WaitForFirstConsumer"},
|
||||
{"apiVersion": "cert-manager.io/v1", "kind": "ClusterIssuer",
|
||||
"metadata": {"name": ISSUER},
|
||||
"spec": {"acme": {"server": "https://acme-v02.api.letsencrypt.org/directory",
|
||||
"email": cfg["acme_email"],
|
||||
"privateKeySecretRef": {"name": ISSUER + "-account-key"},
|
||||
"solvers": [{"http01": {"ingress": {"class": cfg["ingress_class"]}}}]}}},
|
||||
{"apiVersion": "scheduling.k8s.io/v1", "kind": "PriorityClass",
|
||||
"metadata": {"name": PRIORITY_CLASS}, "value": -100,
|
||||
"preemptionPolicy": "Never", "globalDefault": False,
|
||||
"description": "supaspin test instances: evicted first, never evict others"},
|
||||
{"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": CERTS_NS}},
|
||||
]
|
||||
if cfg.get("lets_encrypt", True):
|
||||
# cert-manager needs this object to know where to order certificates from; without
|
||||
# Let's Encrypt it is dead weight, and cert-manager may not even be installed.
|
||||
docs.insert(1, {"apiVersion": "cert-manager.io/v1", "kind": "ClusterIssuer",
|
||||
"metadata": {"name": ISSUER},
|
||||
"spec": {"acme": {"server": "https://acme-v02.api.letsencrypt.org/directory",
|
||||
"email": cfg["acme_email"],
|
||||
"privateKeySecretRef": {"name": ISSUER + "-account-key"},
|
||||
"solvers": [{"http01": {"ingress": {"class": cfg["ingress_class"]}}}]}}})
|
||||
for d in docs:
|
||||
# Create only what is missing; never overwrite something already there.
|
||||
r = kubectl("get", d["kind"].lower(), d["metadata"]["name"], check=False)
|
||||
@@ -746,6 +881,55 @@ def wait_ready(n, log, timeout=1200):
|
||||
raise RuntimeError("pods did not become ready in time")
|
||||
|
||||
|
||||
def setup_log_retention(cfg, n, log):
|
||||
"""Give the instance a pg_cron job that deletes analytics logs older than LOG_RETENTION every
|
||||
night. Logflare stores the logs it receives in the _supabase database and never deletes them;
|
||||
without this job the capped db image fills up and the database dies (instance 3 on
|
||||
2026-10-06). The job runs in the postgres database, the only one pg_cron accepts in this
|
||||
image, and deletes in _supabase over a local dblink connection. Runs the first cleanup right
|
||||
away, so an upgrade also frees a full disk. Runs after create, upgrade and resume;
|
||||
scheduling under a fixed job name makes it idempotent."""
|
||||
log(f" log retention: deleting analytics logs older than {LOG_RETENTION}, every night")
|
||||
kubectl("-n", namespace(n), "exec", f"{fullname(n)}-db-0", "-c", "supabase-db", "--",
|
||||
"psql", "-U", "supabase_admin", "-d", "postgres", "-v", "ON_ERROR_STOP=1",
|
||||
"-c", RETENTION_SQL)
|
||||
log(" log retention is set")
|
||||
|
||||
|
||||
def middleware_manifest(n):
|
||||
"""The Traefik request-size cap the ingress annotation of instance n points at. nginx gets
|
||||
the same cap from proxy-body-size; Traefik has no such annotation, the rule lives here."""
|
||||
return {"apiVersion": "traefik.io/v1alpha1", "kind": "Middleware",
|
||||
"metadata": {"name": buffering_middleware(n), "namespace": namespace(n)},
|
||||
"spec": {"buffering": {"maxRequestBodyBytes": 50 * 1024 * 1024}}} # 50 MiB
|
||||
|
||||
|
||||
def ensure_middleware(cfg, n, log):
|
||||
"""Create the Middleware for instance n on Traefik. On nginx the annotation is not
|
||||
emitted, and applying a traefik.io object would fail there - the CRD is not installed."""
|
||||
if cfg.get("ingress_controller", "nginx") != "traefik":
|
||||
return
|
||||
apply(middleware_manifest(n))
|
||||
log(f" created Middleware {buffering_middleware(n)} (50 MiB request cap)")
|
||||
|
||||
|
||||
def ensure_self_signed_cert(cfg, n, host, log):
|
||||
"""A self-signed certificate for the hostname, so https answers without Let's Encrypt.
|
||||
Skipped when a certificate is already in place (restored from supaspin-certs)."""
|
||||
r = kubectl("get", "secret", tls_secret(n), "-n", namespace(n), check=False)
|
||||
if r.returncode == 0:
|
||||
return
|
||||
with tempfile.TemporaryDirectory(prefix="supaspin-cert-") as d:
|
||||
key, crt = f"{d}/tls.key", f"{d}/tls.crt"
|
||||
run(["openssl", "req", "-x509", "-newkey", "rsa:2048", "-sha256", "-nodes",
|
||||
"-keyout", key, "-out", crt, "-days", "3650",
|
||||
"-subj", f"/CN={host}", "-addext", f"subjectAltName=DNS:{host}"])
|
||||
kubectl("create", "secret", "tls", tls_secret(n), "-n", namespace(n),
|
||||
"--cert", crt, "--key", key)
|
||||
log(f" self-signed certificate for {host} (valid 10 years);"
|
||||
" clients must trust it or ignore the warning")
|
||||
|
||||
|
||||
def create_instance(cfg, mail, log, number=None):
|
||||
taken = existing_numbers(cfg)
|
||||
n = number if number is not None else lowest_free_number(cfg["slots"], taken)
|
||||
@@ -763,10 +947,13 @@ def create_instance(cfg, mail, log, number=None):
|
||||
kubectl("create", "namespace", namespace(n))
|
||||
apply([pv_manifest(cfg, n, v) for v in VOLUMES])
|
||||
restore_cert(n, log)
|
||||
if not cfg.get("lets_encrypt", True):
|
||||
ensure_self_signed_cert(cfg, n, host, log)
|
||||
sec = generate_secrets()
|
||||
with open(find_template()) as f:
|
||||
values = instance_values(cfg, n, sec, mail, f.read())
|
||||
vfile = write_values_file(n, values)
|
||||
ensure_middleware(cfg, n, log)
|
||||
log(" installing chart")
|
||||
try:
|
||||
helm("install", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
|
||||
@@ -774,6 +961,7 @@ def create_instance(cfg, mail, log, number=None):
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
env_path = f"{inst_dir(cfg, n)}/.env"
|
||||
fd = os.open(env_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f:
|
||||
@@ -807,20 +995,81 @@ def post_render_args(n):
|
||||
"--post-renderer-args", "post-render", "--post-renderer-args", str(n)]
|
||||
|
||||
|
||||
def upgrade_instance(cfg, n, log):
|
||||
"""Bring a running instance up to the memory caps this version of supaspin carries, without
|
||||
deleting anything. The values Helm kept from the install are read back, the memory block in
|
||||
them is replaced, and the chart is applied again: the pods are replaced one service at a time
|
||||
and the database, the uploads and the .env file are untouched."""
|
||||
def stored_values(cfg, n):
|
||||
"""The settings Helm kept from the install. They carry every secret of the instance, which is
|
||||
why supaspin deletes its own copy of them as soon as Helm has read it."""
|
||||
ns, rel = namespace(n), release(n)
|
||||
host = cfg["slots"].get(n, "?")
|
||||
log(f"Instance {n} ({host})")
|
||||
r = helm("get", "values", rel, "-n", ns, "-o", "yaml", check=False)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(f"instance {n}: no Helm release {rel} in namespace {ns}\n{r.stderr.strip()}")
|
||||
values = yaml.safe_load(r.stdout) or {}
|
||||
if not values.get("deployment"):
|
||||
raise RuntimeError(f"instance {n}: the stored values of {rel} look empty; not touching it")
|
||||
raise RuntimeError(f"instance {n}: the stored settings of {rel} look empty; not touching it")
|
||||
return values
|
||||
|
||||
|
||||
def apply_values(cfg, n, values):
|
||||
"""Hand the settings to Helm, which works out what to change and changes only that."""
|
||||
vfile = write_values_file(n, values)
|
||||
try:
|
||||
helm("upgrade", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
|
||||
*post_render_args(n))
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
|
||||
|
||||
def wait_gone(n, log, timeout=300):
|
||||
"""Wait until the instance's own pods have stopped."""
|
||||
end = time.time() + timeout
|
||||
while time.time() < end:
|
||||
_, total, _ = pod_summary(n)
|
||||
log(f" pods still running: {total}", live=True)
|
||||
if total == 0:
|
||||
log("")
|
||||
return
|
||||
time.sleep(3)
|
||||
raise RuntimeError("the pods did not stop in time")
|
||||
|
||||
|
||||
def suspend_instance(cfg, n, log):
|
||||
"""Stop every pod of an instance and keep everything else. The database, the uploads, the
|
||||
passwords in .env, the certificate and the hostname are untouched; the instance answers
|
||||
nothing until it is resumed."""
|
||||
log(f"Suspending instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
if is_suspended(values):
|
||||
log(" already suspended")
|
||||
return False
|
||||
apply_values(cfg, n, set_replicas(values, 0))
|
||||
wait_gone(n, log)
|
||||
log(f" instance {n} is suspended; its disks and its .env file are untouched")
|
||||
log(" start it again in this instance's menu: \"Resume it\"")
|
||||
return True
|
||||
|
||||
|
||||
def resume_instance(cfg, n, log):
|
||||
"""Start a suspended instance again, with the data it had when it was suspended."""
|
||||
log(f"Resuming instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
if not is_suspended(values):
|
||||
log(" already running")
|
||||
return False
|
||||
apply_values(cfg, n, set_replicas(values, 1))
|
||||
log(" the database answers first; the services that need it retry until it does")
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
log(f" instance {n} is running again: https://{cfg['slots'].get(n, '?')}")
|
||||
return True
|
||||
|
||||
|
||||
def upgrade_instance(cfg, n, log):
|
||||
"""Bring a running instance up to the memory caps this version of supaspin carries, without
|
||||
deleting anything. The values Helm kept from the install are read back, the memory block in
|
||||
them is replaced, and the chart is applied again: the pods are replaced one service at a time
|
||||
and the database, the uploads and the .env file are untouched."""
|
||||
log(f"Instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
asleep = is_suspended(values)
|
||||
before = memory_limits(values)
|
||||
after = memory_limits(apply_memory(values))
|
||||
changed = [svc for svc in SERVICES if before[svc] != after[svc]]
|
||||
@@ -831,12 +1080,14 @@ def upgrade_instance(cfg, n, log):
|
||||
# differently (the Studio rollout, the host ports, the gateway routes are all decided
|
||||
# while the manifests are rendered). Helm changes nothing if nothing differs.
|
||||
log(" caps already current; re-applying the chart in case anything else differs")
|
||||
vfile = write_values_file(n, values)
|
||||
try:
|
||||
helm("upgrade", rel, chart_path(cfg), "-n", ns, "-f", vfile, *post_render_args(n))
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
apply_values(cfg, n, values)
|
||||
if asleep:
|
||||
# A suspended instance stays suspended: the new caps are stored and take effect the moment
|
||||
# it is resumed. Waiting for its pods would wait for ever - by design it has none.
|
||||
log(f" instance {n} is suspended; the new caps and log retention apply when you resume it")
|
||||
return bool(changed)
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
log(f" instance {n} is running with the current settings")
|
||||
return bool(changed)
|
||||
|
||||
@@ -923,12 +1174,29 @@ def disk_use(cfg, n):
|
||||
return ", ".join(parts)
|
||||
|
||||
|
||||
def instance_state(n):
|
||||
"""What the cluster is currently asked to run for this instance: "running", "suspended",
|
||||
"partly suspended" (some services stopped, which should not happen on its own) or "unknown"
|
||||
when the instance has no workloads at all."""
|
||||
want = []
|
||||
for kind in ("deployments", "statefulsets"):
|
||||
data = kjson("get", kind, "-n", namespace(n), "-l", f"app.kubernetes.io/instance={release(n)}")
|
||||
for item in (data or {}).get("items", []):
|
||||
want.append(item["spec"].get("replicas", 1))
|
||||
if not want:
|
||||
return "unknown"
|
||||
if all(w == 0 for w in want):
|
||||
return "suspended"
|
||||
return "partly suspended" if any(w == 0 for w in want) else "running"
|
||||
|
||||
|
||||
def list_rows(cfg):
|
||||
rows = []
|
||||
for n in sorted(existing_numbers(cfg)):
|
||||
ready, total, _ = pod_summary(n)
|
||||
env = f"{inst_dir(cfg, n)}/.env"
|
||||
rows.append({"n": n, "host": cfg["slots"].get(n, "?"), "pods": f"{ready}/{total}",
|
||||
"state": instance_state(n),
|
||||
"memory": memory_use(n), "disk": disk_use(cfg, n),
|
||||
"env": env if os.path.exists(env) else "(none)"})
|
||||
return rows
|
||||
@@ -1560,7 +1828,7 @@ def pick_instance(cfg, title):
|
||||
if not nums:
|
||||
print("No instances.")
|
||||
return None
|
||||
i = menu(title, [f"{n} {cfg['slots'].get(n, '?')}" for n in nums])
|
||||
i = menu(title, [f"{n} {cfg['slots'].get(n, '?')} ({instance_state(n)})" for n in nums])
|
||||
return None if i is None else nums[i]
|
||||
|
||||
|
||||
@@ -1602,11 +1870,35 @@ def setup_dialog(cfg, log):
|
||||
cfg["storage_root"] = ask("Storage root (images live here)", cfg.get("storage_root", "/home/igor/k8s-storage/supabase"))
|
||||
cfg["node_name"] = ask("Kubernetes node name", cfg.get("node_name", d["node_name"]))
|
||||
cfg["lan_ip"] = ask("Server LAN IP (Postgres/Studio ports bind here)", cfg.get("lan_ip", d["lan_ip"]))
|
||||
cfg["acme_email"] = ask("Let's Encrypt e-mail", cfg.get("acme_email", ""))
|
||||
bind_default = cfg.get("port_bind", "lan")
|
||||
bind = ask("Port bind: lan (LAN only), all (LAN + localhost), local (localhost only)",
|
||||
bind_default).lower()
|
||||
while bind not in ("lan", "all", "local"):
|
||||
print(f"{RED}Answer 'lan', 'all' or 'local'.{OFF}")
|
||||
bind = ask("Port bind: lan (LAN only), all (LAN + localhost), local (localhost only)",
|
||||
bind_default).lower()
|
||||
cfg["port_bind"] = bind
|
||||
cfg["ingress_class"] = ask("Ingress class", cfg.get("ingress_class", "public"))
|
||||
controller_default = cfg.get("ingress_controller", "nginx")
|
||||
controller = ask("Ingress controller, traefik or nginx", controller_default).lower()
|
||||
while controller not in ("traefik", "nginx"):
|
||||
print(f"{RED}Answer 'traefik' or 'nginx'.{OFF}")
|
||||
controller = ask("Ingress controller, traefik or nginx", controller_default).lower()
|
||||
cfg["ingress_controller"] = controller
|
||||
le_default = "y" if cfg.get("lets_encrypt", True) else "n"
|
||||
le = ask("Fetch certificates from Let's Encrypt? Needs public DNS and ports 80/443 (Y/n)", le_default)
|
||||
cfg["lets_encrypt"] = le.lower().startswith("y")
|
||||
if cfg["lets_encrypt"]:
|
||||
cfg["acme_email"] = ask("Let's Encrypt e-mail", cfg.get("acme_email", ""))
|
||||
cur = ",".join(cfg["slots"][k] for k in sorted(cfg.get("slots", {}))) if cfg.get("slots") else ""
|
||||
hosts = [h.strip() for h in ask("Hostname slots, comma separated (first = instance 1)", cur).split(",") if h.strip()]
|
||||
cfg["slots"] = {i + 1: h for i, h in enumerate(hosts)}
|
||||
while True:
|
||||
try:
|
||||
cfg["slots"] = parse_slots(ask(
|
||||
"Hostname slots, comma separated (a name with a number keys that slot: "
|
||||
"supa4 = slot 4; a plain name takes its position)", cur))
|
||||
break
|
||||
except ValueError as e:
|
||||
print(f"{RED}{e}{OFF}")
|
||||
cfg["ssh_host"] = ask("Public SSH host for tunnels", cfg.get("ssh_host", ""))
|
||||
cfg["ssh_port"] = int(ask("Public SSH port", cfg.get("ssh_port", 22)))
|
||||
cfg["ssh_user"] = ask("SSH user", cfg.get("ssh_user", d["ssh_user"]))
|
||||
@@ -1619,11 +1911,19 @@ def setup_dialog(cfg, log):
|
||||
return cfg
|
||||
|
||||
|
||||
def show_menu(cfg, n):
|
||||
def show_menu(cfg, n, log=None):
|
||||
"""Everything you can do to one instance, including stopping and starting it. The stop/start
|
||||
line is worded for the state the instance is in when the menu is drawn, and the state is read
|
||||
again every time you come back to it, so the menu never offers something that makes no sense."""
|
||||
log = log or log_line
|
||||
while True:
|
||||
i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')}",
|
||||
[".env", ".env (secrets shown)", "Tunnel command", "Pods and events", "Back"])
|
||||
if i is None or i == 4:
|
||||
state = instance_state(n)
|
||||
asleep = state == "suspended"
|
||||
switch = ("Resume it (start it again with its data)" if asleep
|
||||
else "Suspend it (stop it, keep its data)")
|
||||
items = [".env", ".env (secrets shown)", "Tunnel command", "Pods and events", switch, "Back"]
|
||||
i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')} - {state}", items)
|
||||
if i is None or i == 5:
|
||||
return
|
||||
print()
|
||||
try:
|
||||
@@ -1631,10 +1931,14 @@ def show_menu(cfg, n):
|
||||
text = read_env(cfg, n)
|
||||
print(f"{inst_dir(cfg, n)}/.env\n\n" + (text if i == 1 else mask_env(text)))
|
||||
elif i == 2:
|
||||
print(tunnel_help(cfg, n))
|
||||
else:
|
||||
print(tunnel_help(cfg, n) if not asleep else
|
||||
f"Instance {n} is suspended: nothing is listening on its ports.\n"
|
||||
"Resume it from this menu and the tunnel commands work again.")
|
||||
elif i == 3:
|
||||
print(kubectl("get", "pods", "-n", namespace(n), "-o", "wide").stdout)
|
||||
print(kubectl("get", "events", "-n", namespace(n), "--sort-by=.lastTimestamp").stdout[-3000:])
|
||||
else:
|
||||
(resume_instance if asleep else suspend_instance)(cfg, n, log)
|
||||
except Exception as e:
|
||||
print(f"{RED}{e}{OFF}")
|
||||
pause()
|
||||
@@ -1646,8 +1950,12 @@ def print_list(cfg):
|
||||
print("No instances.")
|
||||
return
|
||||
for r in rows:
|
||||
print(f"{BOLD}{r['n']}{OFF} {r['host']} pods {r['pods']} memory {r['memory']}")
|
||||
state = r.get("state", "running")
|
||||
print(f"{BOLD}{r['n']}{OFF} {r['host']} {state} pods {r['pods']} memory {r['memory']}")
|
||||
print(f" disk {r['disk']}\n env {r['env']}")
|
||||
if state == "suspended":
|
||||
print(f" start it again in this instance's menu: \"Resume it\"\n")
|
||||
continue # the tunnel is pointless while nothing listens on the ports
|
||||
for line in tunnel_guide(cfg, r["n"]):
|
||||
print(f" {line}" if line else "")
|
||||
print()
|
||||
@@ -1655,8 +1963,10 @@ def print_list(cfg):
|
||||
|
||||
def main_menu(cfg, log):
|
||||
while True:
|
||||
i = menu(f"supaspin {VERSION}", ["Create instance", "List instances", "Show instance",
|
||||
"Delete instance", "Copy instance values into another .env file",
|
||||
i = menu(f"supaspin {VERSION}",
|
||||
["Create instance", "List instances",
|
||||
"Show instance (and suspend or resume it)", "Delete instance",
|
||||
"Copy instance values into another .env file",
|
||||
"Apply current memory caps to running instances",
|
||||
"Mail settings", "Settings", "Quit"])
|
||||
if i is None or i == 8:
|
||||
@@ -1668,9 +1978,9 @@ def main_menu(cfg, log):
|
||||
elif i == 1:
|
||||
print_list(cfg)
|
||||
elif i == 2:
|
||||
n = pick_instance(cfg, "Show which instance?")
|
||||
n = pick_instance(cfg, "Which instance?")
|
||||
if n:
|
||||
show_menu(cfg, n)
|
||||
show_menu(cfg, n, log)
|
||||
continue
|
||||
elif i == 3:
|
||||
n = pick_instance(cfg, "Delete which instance?")
|
||||
@@ -1697,7 +2007,7 @@ def main_menu(cfg, log):
|
||||
def main(argv):
|
||||
if len(argv) >= 2 and argv[1] == "post-render": # called by Helm; no root check, no output but YAML
|
||||
cfg = load_config()
|
||||
sys.stdout.write(post_render(sys.stdin.read(), int(argv[2]), cfg["lan_ip"]))
|
||||
sys.stdout.write(post_render(sys.stdin.read(), int(argv[2]), cfg))
|
||||
return 0
|
||||
if len(argv) > 1 and argv[1] in ("help", "-h", "--help"):
|
||||
print(__doc__)
|
||||
@@ -1738,6 +2048,14 @@ def main(argv):
|
||||
delete_instance(cfg, n, log_line)
|
||||
elif cmd == "upgrade":
|
||||
upgrade_all(cfg, log_line, n)
|
||||
elif cmd in ("suspend", "resume"):
|
||||
if n is None:
|
||||
# No "all" here on purpose: stopping every instance is not something to type by
|
||||
# accident, and resuming them all at once would start 36 pods in one go.
|
||||
print(f"supaspin {cmd} N - which instance? ({cmd} takes one number)",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
(suspend_instance if cmd == "suspend" else resume_instance)(cfg, n, log_line)
|
||||
elif cmd == "push-env":
|
||||
return push_env_command(cfg, argv[2:])
|
||||
elif cmd == "mail":
|
||||
|
||||
@@ -71,13 +71,9 @@ environment:
|
||||
ingress:
|
||||
enabled: true
|
||||
className: ${INGRESS_CLASS}
|
||||
annotations:
|
||||
# The chart default rewrites every path to "/", which breaks all APIs; null removes it.
|
||||
nginx.ingress.kubernetes.io/rewrite-target: null
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: 50m
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
|
||||
cert-manager.io/cluster-issuer: ${CLUSTER_ISSUER}
|
||||
# The annotation map is built in code (ingress_annotations): nginx gets proxy-* and a
|
||||
# nulled rewrite-target, Traefik gets its Middleware reference, cert-manager its issuer.
|
||||
annotations: ${INGRESS_ANNOTATIONS}
|
||||
tls:
|
||||
- hosts: [${HOST}]
|
||||
secretName: ${TLS_SECRET}
|
||||
|
||||
+429
-8
@@ -51,6 +51,25 @@ class Numbering(unittest.TestCase):
|
||||
def test_no_free_slot(self):
|
||||
self.assertIsNone(sp.lowest_free_number(CFG["slots"], {1, 2, 3}))
|
||||
|
||||
def test_slots_are_keyed_by_the_hostname_number(self):
|
||||
# The digit in the name is the slot number, so instance N, its ports and the
|
||||
# hostname stay in step on every machine.
|
||||
self.assertEqual(sp.parse_slots("supa4.didgibot.com, supa5.didgibot.com"),
|
||||
{4: "supa4.didgibot.com", 5: "supa5.didgibot.com"})
|
||||
|
||||
def test_plain_names_take_their_position(self):
|
||||
self.assertEqual(sp.parse_slots("db.example.com, db2.example.com"),
|
||||
{1: "db.example.com", 2: "db2.example.com"})
|
||||
self.assertEqual(sp.parse_slots("supa4.didgibot.com, spare.example.com"),
|
||||
{4: "supa4.didgibot.com", 2: "spare.example.com"})
|
||||
|
||||
def test_duplicates_are_an_error(self):
|
||||
with self.assertRaises(ValueError):
|
||||
sp.parse_slots("supa4.didgibot.com, supa4.other.com")
|
||||
|
||||
def test_empty_answer_gives_no_slots(self):
|
||||
self.assertEqual(sp.parse_slots(""), {})
|
||||
|
||||
def test_ports_and_names(self):
|
||||
self.assertEqual((sp.pg_port(1), sp.pg_port(3)), (5441, 5443))
|
||||
self.assertEqual((sp.studio_port(1), sp.studio_port(3)), (3001, 3003))
|
||||
@@ -90,14 +109,17 @@ class Values(unittest.TestCase):
|
||||
|
||||
def test_memory_caps_add_up(self):
|
||||
req, lim = sp.total_memory_mib()
|
||||
# db 2048, analytics 1536, kong 768, realtime 768, vector 1024, the other seven 512 each
|
||||
self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 1024 + 7 * 512)
|
||||
self.assertEqual(lim / 1024, 9.5)
|
||||
# db 2048, analytics 1536, kong 768, realtime 768, vector 512, the other seven 512 each
|
||||
self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 512 + 7 * 512)
|
||||
self.assertEqual(lim / 1024, 9.0)
|
||||
self.assertEqual(req, 512 + 11 * 128) # 1.875 GiB
|
||||
# Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
|
||||
# highest use, or the kernel kills the container in a loop the way it did on instance 1.
|
||||
self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536) # used 694 MiB, died at 768
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 1024) # used 290 MiB, died at 512
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 512) # used 290 MiB; the nightly log
|
||||
# retention keeps its delivery backlog from growing past this the way it did on
|
||||
# 2026-10-06, when instance 3's full db image killed analytics and vector filled with
|
||||
# the undeliverable backlog every few minutes against 1024 MiB.
|
||||
self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768) # used 300 MiB against 384
|
||||
self.assertGreaterEqual(sp.MEMORY["kong"][1], 768) # used 269 MiB against 512
|
||||
self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512) # studio used 203 against 256
|
||||
@@ -128,6 +150,27 @@ class Values(unittest.TestCase):
|
||||
self.assertEqual(set(sp.VOLUMES), {"db", "storage", "functions", "snippets", "deno", "pgsodium"})
|
||||
self.assertTrue(all(p[v]["storageClassName"] == "supaspin-local-sc" for v in sp.VOLUMES))
|
||||
|
||||
def test_log_retention_sql(self):
|
||||
# Logflare stores logs in _analytics.log_events_* and never deletes them; the job deletes
|
||||
# rows older than LOG_RETENTION and is scheduled under a fixed name so re-runs overwrite.
|
||||
# pg_cron in the supabase image only runs jobs from the `postgres` database, so the job
|
||||
# reaches the log tables in `_supabase` over a local dblink connection (trusted there).
|
||||
# dblink and the helper function live in the `extensions` schema: in `public` they belong
|
||||
# to someone else's routines and the app's `REVOKE ALL ON ALL ROUTINES IN SCHEMA public`
|
||||
# fails as the non-superuser postgres role (42501, dblink_connect_u).
|
||||
self.assertIn("schemaname = ''_analytics''", sp.RETENTION_SQL)
|
||||
self.assertIn("''log_events\\_%''", sp.RETENTION_SQL)
|
||||
self.assertIn(f"interval ''{sp.LOG_RETENTION}''", sp.RETENTION_SQL)
|
||||
self.assertIn(f"'{sp.LOG_RETENTION_JOB}'", sp.RETENTION_SQL)
|
||||
self.assertIn("cron.schedule", sp.RETENTION_SQL)
|
||||
self.assertIn("extensions.dblink_connect", sp.RETENTION_SQL)
|
||||
self.assertIn("user=supabase_admin", sp.RETENTION_SQL)
|
||||
self.assertIn("create extension if not exists dblink with schema extensions", sp.RETENTION_SQL)
|
||||
self.assertIn("alter extension dblink set schema extensions", sp.RETENTION_SQL)
|
||||
self.assertIn("extensions.supaspin_delete_old_logs", sp.RETENTION_SQL)
|
||||
self.assertIn("drop function if exists public.supaspin_delete_old_logs", sp.RETENTION_SQL)
|
||||
self.assertNotIn("\ndblink_", sp.RETENTION_SQL) # no unqualified dblink call remains
|
||||
|
||||
def test_imgproxy_reads_the_storage_volume(self):
|
||||
img = self.v["persistence"]["imgproxy"]
|
||||
self.assertEqual(img["existingClaim"], "supa2-supabase-storage")
|
||||
@@ -155,6 +198,120 @@ class Values(unittest.TestCase):
|
||||
self.assertEqual(v["secret"]["smtp"]["password"], 'p"a$s: #x')
|
||||
|
||||
|
||||
class IngressShaping(unittest.TestCase):
|
||||
"""The ingress annotation map per controller and per Let's Encrypt setting."""
|
||||
def setUp(self):
|
||||
self.sec = sp.generate_secrets()
|
||||
|
||||
def ann(self, cfg=None, n=2):
|
||||
return sp.instance_values(dict(CFG, **(cfg or {})), n, self.sec, MAIL, template()) \
|
||||
["ingress"]["annotations"]
|
||||
|
||||
def test_default_is_nginx_with_lets_encrypt(self):
|
||||
# Configs written before these settings existed must render exactly as before.
|
||||
a = self.ann()
|
||||
self.assertEqual(a["cert-manager.io/cluster-issuer"], "supaspin-letsencrypt")
|
||||
self.assertIsNone(a["nginx.ingress.kubernetes.io/rewrite-target"])
|
||||
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-body-size"], "50m")
|
||||
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-read-timeout"], "3600")
|
||||
self.assertNotIn("traefik.ingress.kubernetes.io/router.middlewares", a)
|
||||
|
||||
def test_traefik_also_keeps_nginx_annotations(self):
|
||||
# The nginx keys must be present in traefik mode too: while nginx routes the
|
||||
# ingress they enforce the cap, and they are inert the day Traefik takes over.
|
||||
a = self.ann({"ingress_controller": "traefik"})
|
||||
self.assertEqual(a["traefik.ingress.kubernetes.io/router.middlewares"],
|
||||
"supaspin-2-supa2-buffering@kubernetescrd")
|
||||
self.assertIsNone(a["nginx.ingress.kubernetes.io/rewrite-target"])
|
||||
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-body-size"], "50m")
|
||||
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-read-timeout"], "3600")
|
||||
self.assertEqual(a["cert-manager.io/cluster-issuer"], "supaspin-letsencrypt")
|
||||
|
||||
def test_lets_encrypt_off_drops_cert_manager_annotation(self):
|
||||
self.assertNotIn("cert-manager.io/cluster-issuer", self.ann({"lets_encrypt": False}))
|
||||
self.assertNotIn("cert-manager.io/cluster-issuer",
|
||||
self.ann({"lets_encrypt": False, "ingress_controller": "traefik"}))
|
||||
|
||||
def test_middleware_manifest(self):
|
||||
m = sp.middleware_manifest(2)
|
||||
self.assertEqual(m["apiVersion"], "traefik.io/v1alpha1")
|
||||
self.assertEqual(m["kind"], "Middleware")
|
||||
self.assertEqual(m["metadata"], {"name": "supa2-buffering", "namespace": "supaspin-2"})
|
||||
self.assertEqual(m["spec"]["buffering"]["maxRequestBodyBytes"], 50 * 1024 * 1024)
|
||||
|
||||
|
||||
class ClusterObjects(unittest.TestCase):
|
||||
"""ensure_cluster_objects skips the ClusterIssuer when Let's Encrypt is off."""
|
||||
def created_docs(self, cfg):
|
||||
applied = []
|
||||
|
||||
class R:
|
||||
returncode = 1
|
||||
stdout = stderr = ""
|
||||
|
||||
def fake_kubectl(*args, **kw):
|
||||
if args and args[0] == "apply":
|
||||
applied.extend(yaml.safe_load_all(kw["input_"]))
|
||||
return R()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp, \
|
||||
mock.patch.object(sp, "STATE_DIR", tmp), \
|
||||
mock.patch.object(sp, "kubectl", side_effect=fake_kubectl), \
|
||||
mock.patch.object(sp, "helm"):
|
||||
sp.ensure_cluster_objects(cfg, lambda *a: None)
|
||||
return applied
|
||||
|
||||
def test_issuer_created_when_lets_encrypt_on(self):
|
||||
self.assertIn("ClusterIssuer", [d["kind"] for d in self.created_docs(CFG)])
|
||||
|
||||
def test_no_issuer_without_lets_encrypt(self):
|
||||
kinds = [d["kind"] for d in self.created_docs(dict(CFG, lets_encrypt=False))]
|
||||
self.assertNotIn("ClusterIssuer", kinds)
|
||||
self.assertIn("StorageClass", kinds)
|
||||
self.assertIn("PriorityClass", kinds)
|
||||
self.assertIn("Namespace", kinds)
|
||||
|
||||
|
||||
class SelfSignedCert(unittest.TestCase):
|
||||
def test_generated_when_missing(self):
|
||||
calls = []
|
||||
|
||||
class R:
|
||||
returncode = 1
|
||||
stdout = stderr = ""
|
||||
|
||||
def fake_kubectl(*args, **kw):
|
||||
calls.append(args)
|
||||
return R()
|
||||
|
||||
def fake_run(args, **kw):
|
||||
calls.append(("run", args))
|
||||
|
||||
class P:
|
||||
returncode = 0
|
||||
return P()
|
||||
|
||||
with mock.patch.object(sp, "kubectl", side_effect=fake_kubectl), \
|
||||
mock.patch.object(sp, "run", side_effect=fake_run):
|
||||
sp.ensure_self_signed_cert(CFG, 2, "supa2.example.com", lambda *a: None)
|
||||
openssl = next(c[1] for c in calls if c[0] == "run" and c[1][0] == "openssl")
|
||||
self.assertIn("subjectAltName=DNS:supa2.example.com", openssl)
|
||||
create = next(c for c in calls if c[:3] == ("create", "secret", "tls"))
|
||||
self.assertEqual(create[3], "supa2-tls")
|
||||
self.assertIn("supaspin-2", create)
|
||||
|
||||
def test_kept_when_present(self):
|
||||
class R:
|
||||
returncode = 0
|
||||
stdout = stderr = ""
|
||||
|
||||
with mock.patch.object(sp, "kubectl", return_value=R()) as k, \
|
||||
mock.patch.object(sp, "run") as r:
|
||||
sp.ensure_self_signed_cert(CFG, 2, "supa2.example.com", lambda *a: None)
|
||||
k.assert_called_once()
|
||||
r.assert_not_called()
|
||||
|
||||
|
||||
class EnvFile(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.sec = sp.generate_secrets()
|
||||
@@ -254,7 +411,7 @@ data:
|
||||
|
||||
class PostRender(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, "192.168.10.30")))
|
||||
self.docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, CFG)))
|
||||
|
||||
def find(self, kind, name):
|
||||
return next(d for d in self.docs if d["kind"] == kind and d["metadata"]["name"] == name)
|
||||
@@ -265,6 +422,23 @@ class PostRender(unittest.TestCase):
|
||||
st = self.find("Deployment", "supa1-supabase-studio")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertEqual((st["hostPort"], st["hostIP"]), (3001, "192.168.10.30"))
|
||||
|
||||
def test_all_binds_localhost_too(self):
|
||||
# `all` omits hostIP: the kubelet then listens on every interface, localhost included.
|
||||
docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, dict(CFG, port_bind="all"))))
|
||||
find = lambda k, name: next(d for d in docs if d["kind"] == k and d["metadata"]["name"] == name)
|
||||
db = find("StatefulSet", "supa1-supabase-db")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
st = find("Deployment", "supa1-supabase-studio")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertEqual(db["hostPort"], 5441)
|
||||
self.assertNotIn("hostIP", db)
|
||||
self.assertEqual(st["hostPort"], 3001)
|
||||
self.assertNotIn("hostIP", st)
|
||||
|
||||
def test_local_binds_localhost_only(self):
|
||||
docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, dict(CFG, port_bind="local"))))
|
||||
find = lambda k, name: next(d for d in docs if d["kind"] == k and d["metadata"]["name"] == name)
|
||||
db = find("StatefulSet", "supa1-supabase-db")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertEqual((db["hostPort"], db["hostIP"]), (5441, "127.0.0.1"))
|
||||
|
||||
def test_other_pods_untouched(self):
|
||||
rest = self.find("Deployment", "supa1-supabase-rest")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertNotIn("hostPort", rest)
|
||||
@@ -294,9 +468,9 @@ class PostRender(unittest.TestCase):
|
||||
self.assertEqual(cm["data"]["kong-entrypoint.sh"], "echo hi\necho there\n")
|
||||
|
||||
def test_instance_number_selects_ports(self):
|
||||
docs = list(yaml.safe_load_all(sp.post_render(RENDERED.replace("supa1", "supa3"), 3, "10.0.0.1")))
|
||||
docs = list(yaml.safe_load_all(sp.post_render(RENDERED.replace("supa1", "supa3"), 3, CFG)))
|
||||
db = next(d for d in docs if d["kind"] == "StatefulSet")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertEqual((db["hostPort"], db["hostIP"]), (5443, "10.0.0.1"))
|
||||
self.assertEqual((db["hostPort"], db["hostIP"]), (5443, "192.168.10.30"))
|
||||
|
||||
|
||||
class Manifests(unittest.TestCase):
|
||||
@@ -975,6 +1149,7 @@ class Upgrade(unittest.TestCase):
|
||||
|
||||
with mock.patch.object(sp, "helm", helm), \
|
||||
mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
|
||||
mock.patch.object(sp, "setup_log_retention", lambda cfg, n, log: None), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
@@ -1086,7 +1261,7 @@ class HostPortRollout(unittest.TestCase):
|
||||
"template": {"spec": {"containers": [
|
||||
{"name": "rest", "ports": [{"containerPort": 3000}]}]}}}},
|
||||
])
|
||||
out = list(yaml.safe_load_all(sp.post_render(text, n, CFG["lan_ip"])))
|
||||
out = list(yaml.safe_load_all(sp.post_render(text, n, CFG)))
|
||||
return {d["metadata"]["name"]: d for d in out}
|
||||
|
||||
def test_studio_stops_the_old_pod_before_starting_the_new_one(self):
|
||||
@@ -1105,3 +1280,249 @@ class HostPortRollout(unittest.TestCase):
|
||||
self.assertNotIn("strategy", d["spec"])
|
||||
self.assertEqual(d["spec"]["template"]["spec"]["containers"][0]["ports"][0]["hostPort"],
|
||||
sp.pg_port(1))
|
||||
|
||||
|
||||
class SuspendResume(unittest.TestCase):
|
||||
"""Suspending an instance stops its pods and keeps everything else: the disks, the passwords,
|
||||
the certificate and the hostname. Resuming starts it again with the same data."""
|
||||
|
||||
RUNNING = {
|
||||
"deployment": {
|
||||
"analytics": {"enabled": True,
|
||||
"resources": {"limits": {"memory": "1536Mi"},
|
||||
"requests": {"memory": "128Mi"}}},
|
||||
"db": {"resources": {"limits": {"memory": "2048Mi"},
|
||||
"requests": {"memory": "512Mi"}}},
|
||||
},
|
||||
"secret": {"jwt": {"secret": "keep-me"}, "db": {"password": "keep-me-too"}},
|
||||
"persistence": {"db": {"enabled": True, "size": "2048Mi"}},
|
||||
}
|
||||
|
||||
def setUp(self):
|
||||
self.calls = []
|
||||
self.written = {}
|
||||
self.log = []
|
||||
self.waited = []
|
||||
|
||||
@contextlib.contextmanager
|
||||
def fake_helm(self, stored):
|
||||
P = collections.namedtuple("P", "returncode stdout stderr")
|
||||
|
||||
def helm(*args, check=True):
|
||||
self.calls.append(list(args))
|
||||
if args[0] == "get":
|
||||
return P(0, yaml.safe_dump(stored), "")
|
||||
if args[0] == "upgrade":
|
||||
with open(args[args.index("-f") + 1]) as f:
|
||||
self.written = yaml.safe_load(f)
|
||||
return P(0, "", "")
|
||||
|
||||
with mock.patch.object(sp, "helm", helm), \
|
||||
mock.patch.object(sp, "wait_ready",
|
||||
lambda n, log, **kw: self.waited.append(("ready", n))), \
|
||||
mock.patch.object(sp, "wait_gone",
|
||||
lambda n, log, **kw: self.waited.append(("gone", n))), \
|
||||
mock.patch.object(sp, "setup_log_retention",
|
||||
lambda cfg, n, log: self.waited.append(("retention", n))), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
def suspended(self):
|
||||
return sp.set_replicas(yaml.safe_load(yaml.safe_dump(self.RUNNING)), 0)
|
||||
|
||||
# ---- the setting itself ----
|
||||
|
||||
def test_set_replicas_covers_every_service(self):
|
||||
v = sp.set_replicas({}, 0)
|
||||
self.assertEqual(sorted(v["deployment"]), sorted(sp.SERVICES))
|
||||
self.assertTrue(all(d["replicaCount"] == 0 for d in v["deployment"].values()))
|
||||
self.assertTrue(sp.is_suspended(v))
|
||||
|
||||
def test_a_service_missing_the_setting_counts_as_running(self):
|
||||
self.assertFalse(sp.is_suspended(self.RUNNING))
|
||||
self.assertFalse(sp.is_suspended({"deployment": {}}))
|
||||
|
||||
def test_one_service_left_running_is_not_suspended(self):
|
||||
v = sp.set_replicas({}, 0)
|
||||
v["deployment"]["kong"]["replicaCount"] = 1
|
||||
self.assertFalse(sp.is_suspended(v))
|
||||
|
||||
# ---- suspend ----
|
||||
|
||||
def test_suspend_stops_every_service_and_keeps_the_rest(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
self.assertTrue(sp.suspend_instance(CFG, 1, self.log.append))
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.written["deployment"][svc]["replicaCount"], 0)
|
||||
self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
|
||||
self.assertEqual(self.written["secret"]["db"]["password"], "keep-me-too")
|
||||
self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
|
||||
# the memory caps are kept as they were: they apply again on resume
|
||||
self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
|
||||
"1536Mi")
|
||||
self.assertEqual(self.waited, [("gone", 1)])
|
||||
|
||||
def test_suspend_waits_for_the_pods_to_be_gone_not_for_them_to_be_ready(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.suspend_instance(CFG, 2, self.log.append)
|
||||
self.assertEqual([w[0] for w in self.waited], ["gone"])
|
||||
|
||||
def test_suspending_a_suspended_instance_changes_nothing(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
self.assertFalse(sp.suspend_instance(CFG, 1, self.log.append))
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
self.assertEqual(self.waited, [])
|
||||
|
||||
def test_the_values_file_with_the_secrets_is_deleted_again(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.suspend_instance(CFG, 1, self.log.append)
|
||||
path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
|
||||
self.assertFalse(os.path.exists(path))
|
||||
|
||||
# ---- resume ----
|
||||
|
||||
def test_resume_starts_every_service_again(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
self.assertTrue(sp.resume_instance(CFG, 3, self.log.append))
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.written["deployment"][svc]["replicaCount"], 1)
|
||||
self.assertFalse(sp.is_suspended(self.written))
|
||||
self.assertEqual(self.waited, [("ready", 3), ("retention", 3)])
|
||||
|
||||
def test_resuming_a_running_instance_changes_nothing(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
self.assertFalse(sp.resume_instance(CFG, 1, self.log.append))
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
self.assertEqual(self.waited, [])
|
||||
|
||||
def test_a_suspended_instance_keeps_its_data_settings(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
sp.resume_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
|
||||
self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
|
||||
|
||||
# ---- upgrade must not trip over a suspended instance ----
|
||||
|
||||
def test_upgrade_leaves_a_suspended_instance_suspended_and_does_not_wait(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertTrue(sp.is_suspended(self.written)) # not woken up
|
||||
self.assertEqual(self.waited, []) # would never finish
|
||||
self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
|
||||
f"{sp.MEMORY['analytics'][1]}Mi") # new caps stored for the resume
|
||||
|
||||
def test_upgrade_still_waits_for_a_running_instance(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual(self.waited, [("ready", 1), ("retention", 1)])
|
||||
|
||||
|
||||
class InstanceState(unittest.TestCase):
|
||||
"""What `supaspin list` says about an instance, read from what the cluster is asked to run."""
|
||||
|
||||
def state(self, deployments, statefulsets):
|
||||
def kjson(*args):
|
||||
kind = args[1]
|
||||
counts = deployments if kind == "deployments" else statefulsets
|
||||
return {"items": [{"spec": {"replicas": c}} for c in counts]}
|
||||
|
||||
with mock.patch.object(sp, "kjson", kjson):
|
||||
return sp.instance_state(1)
|
||||
|
||||
def test_running(self):
|
||||
self.assertEqual(self.state([1] * 11, [1]), "running")
|
||||
|
||||
def test_suspended(self):
|
||||
self.assertEqual(self.state([0] * 11, [0]), "suspended")
|
||||
|
||||
def test_the_database_alone_still_running_is_reported_honestly(self):
|
||||
self.assertEqual(self.state([0] * 11, [1]), "partly suspended")
|
||||
|
||||
def test_nothing_there(self):
|
||||
self.assertEqual(self.state([], []), "unknown")
|
||||
|
||||
def test_the_list_shows_the_state_and_no_tunnel_for_a_sleeping_instance(self):
|
||||
rows = [{"n": 1, "host": "supa1.example.com", "pods": "0/0", "state": "suspended",
|
||||
"memory": "n/a", "disk": "db 1/2M", "env": "/data/supabase/1/.env"}]
|
||||
out = io.StringIO()
|
||||
with mock.patch.object(sp, "list_rows", lambda cfg: rows), \
|
||||
contextlib.redirect_stdout(out):
|
||||
sp.print_list(CFG)
|
||||
text = out.getvalue()
|
||||
self.assertIn("suspended", text)
|
||||
self.assertIn("Resume it", text) # the menu line, not a command to remember
|
||||
self.assertNotIn("ssh -N -L", text)
|
||||
|
||||
|
||||
class InstanceMenu(unittest.TestCase):
|
||||
"""Stopping and starting an instance lives inside the instance you are looking at: one line,
|
||||
worded for the state the instance is in, so there is no second command to remember."""
|
||||
|
||||
def run_menu(self, state, choose):
|
||||
"""Draw the menu for an instance in `state`, pick the item whose text `choose` matches
|
||||
(None: leave at once). Returns (the item texts, what was called)."""
|
||||
seen, called = [], []
|
||||
|
||||
def menu(title, items):
|
||||
seen.append((title, list(items)))
|
||||
if choose is None:
|
||||
return None
|
||||
for i, text in enumerate(items):
|
||||
if choose in text:
|
||||
return i
|
||||
raise AssertionError(f"no item matching {choose!r} in {items}")
|
||||
|
||||
calls = {"suspend": lambda cfg, n, log: called.append(("suspend", n)),
|
||||
"resume": lambda cfg, n, log: called.append(("resume", n))}
|
||||
out = io.StringIO()
|
||||
with mock.patch.object(sp, "menu", menu), \
|
||||
mock.patch.object(sp, "instance_state", lambda n: state), \
|
||||
mock.patch.object(sp, "pause", lambda: (_ for _ in ()).throw(StopIteration)), \
|
||||
mock.patch.object(sp, "suspend_instance", calls["suspend"]), \
|
||||
mock.patch.object(sp, "resume_instance", calls["resume"]), \
|
||||
mock.patch.object(sp, "read_env", lambda cfg, n: "A=b\n"), \
|
||||
contextlib.redirect_stdout(out):
|
||||
try:
|
||||
sp.show_menu(CFG, 2, lambda *a, **k: None)
|
||||
except StopIteration:
|
||||
pass # the menu would loop for ever; stop at the first "press Enter"
|
||||
return seen[0][1], called, out.getvalue(), seen[0][0]
|
||||
|
||||
def test_a_running_instance_offers_to_suspend_it(self):
|
||||
items, called, _, title = self.run_menu("running", None)
|
||||
self.assertTrue(any("Suspend it" in t for t in items), items)
|
||||
self.assertFalse(any("Resume" in t for t in items), items)
|
||||
self.assertIn("running", title)
|
||||
|
||||
def test_a_suspended_instance_offers_to_resume_it(self):
|
||||
items, called, _, title = self.run_menu("suspended", None)
|
||||
self.assertTrue(any("Resume it" in t for t in items), items)
|
||||
self.assertFalse(any("Suspend" in t for t in items), items)
|
||||
self.assertIn("suspended", title)
|
||||
|
||||
def test_choosing_it_suspends_this_instance(self):
|
||||
_, called, _, _ = self.run_menu("running", "Suspend it")
|
||||
self.assertEqual(called, [("suspend", 2)])
|
||||
|
||||
def test_choosing_it_resumes_this_instance(self):
|
||||
_, called, _, _ = self.run_menu("suspended", "Resume it")
|
||||
self.assertEqual(called, [("resume", 2)])
|
||||
|
||||
def test_the_tunnel_of_a_sleeping_instance_says_why_it_is_pointless(self):
|
||||
_, _, text, _ = self.run_menu("suspended", "Tunnel")
|
||||
self.assertIn("suspended", text)
|
||||
self.assertNotIn("ssh -N -L", text)
|
||||
|
||||
def test_the_picker_names_the_state_of_every_instance(self):
|
||||
seen = []
|
||||
|
||||
def menu(title, items):
|
||||
seen.append(list(items))
|
||||
return None
|
||||
|
||||
with mock.patch.object(sp, "menu", menu), \
|
||||
mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2}), \
|
||||
mock.patch.object(sp, "instance_state", lambda n: "suspended" if n == 2 else "running"):
|
||||
sp.pick_instance(CFG, "Which instance?")
|
||||
self.assertEqual(seen[0], ["1 supa1.example.com (running)",
|
||||
"2 supa2.example.com (suspended)"])
|
||||
|
||||
Reference in New Issue
Block a user