Author SHA1 Message Date
Igor K.andClaude Sonnet 5.5 b1ae4f1daa fix: make the database disk image 6 GB instead of 2 GB
- VOLUMES["db"] goes from 2048 to 6144 MiB, so every new instance gets a 6 GB database image.
  Instance 4 filled its 2 GB image with analytics logs on 2026-10-14: Postgres crashed and could
  not finish crash recovery without free space. Instance 2 had already been grown to 6 GB by hand.
- Measured on instance 2: about 1.2 GB of logs a day under browser-test traffic, kept for
  LOG_RETENTION (2 days), plus up to 1 GB of write-ahead log. That peaks near 4.5 GB.
- Only instances created after this change get the larger image; existing ones keep their size.
- README disk line, the two unit tests that assert the db volume size, and the version
  (1.6.3 -> 1.6.4 in VERSION, supaspin and README.md) are updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-14 08:46:50 -04:00
igor da8f02310a docs: bring the README in line with the last commits, release 1.6.3 (#14) 2026-10-09 17:46:54 +00:00
igor ccc49ca7d1 fix: keep the nginx ingress annotations when the ingress controller is traefik (#13) 2026-10-09 17:43:21 +00:00
igor 4a267c5336 Merge pull request 'feat: Traefik ingress, optional Lets Encrypt, configurable port bind, slot numbers from hostnames' (#12) from feat/traefik-optional-le-port-bind-slot-numbers into main 2026-10-09 14:32:43 +00:00
Igor K. d4fedd718f feat: Traefik support, optional Let's Encrypt, configurable port bind, hostname-numbered slots, dblink in extensions schema
- Traefik is a first-class ingress controller choice: per-instance Middleware
  caps uploads at 50 MiB, nginx keeps its proxy-* annotations; picked in setup
- Let's Encrypt is optional: off skips the ClusterIssuer and gives each instance
  a 10-year self-signed certificate instead of a Let's Encrypt one
- New port_bind setting: lan (default, LAN IP only), all (LAN + localhost),
  local (localhost only)
- Hostname slots are keyed by the digit in the name (supa4 -> slot 4), so the
  instance number, ports and hostname stay in step on every machine
- The log-retention job installs dblink and its helper into the extensions
  schema instead of public: in public they break the app's
  REVOKE ALL ON ALL ROUTINES IN SCHEMA public for the non-superuser postgres
  role (42501 permission denied for function dblink_connect_u)
2026-10-08 14:40:24 -04:00
igor 2f47ddedf1 Merge pull request 'fix: run the log-retention SQL in the postgres database' (#11) from fix/log-retention-run-in-postgres into main 2026-10-06 17:35:54 +00:00
Igor K. 80b84afb1e fix: run the log-retention SQL in the postgres database
1.3.3 put the dblink job into the SQL but still ran psql against
_supabase, where pg_cron refuses the extension; the psql session now runs
in postgres, the only database pg_cron accepts in this image
version 1.3.4
2026-10-06 20:35:43 +03:00
igor 4d0fa4fe69 Merge pull request 'fix: reach the log-retention job into _supabase over dblink' (#10) from fix/log-retention-pg-cron-dblink into main 2026-10-06 17:34:48 +00:00
Igor K. 2c6e383617 fix: reach the log-retention job into _supabase over dblink
the supabase image pins cron.database_name to postgres, so pg_cron refuses
jobs scheduled from _supabase, where the log tables live; 1.3.2's upgrade
failed on the first live run with 'can only create extension in database
postgres'. The job now runs in postgres and deletes the _analytics
log_events_* rows in _supabase over a local dblink connection, which needs
no password because the image trusts local connections for supabase_admin.
version 1.3.3
2026-10-06 20:34:39 +03:00
5 changed files with 386 additions and 60 deletions
+34 -17
View File
@@ -1,6 +1,6 @@
# supaspin
Version 1.3.2
Version 1.6.4
Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
single-node [microk8s](https://microk8s.io) server, from a terminal menu.
@@ -13,10 +13,10 @@ endangering them. Nothing in it is specific to any one app.
| | |
|---|---|
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions), with a Let's Encrypt certificate |
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
| Web API | `https://<slot hostname>` (auth, rest, graphql, realtime, storage, functions); the certificate is from Let's Encrypt when that is enabled, otherwise self-signed |
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …); which addresses listen is set by **Port bind** |
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
| Disk | fixed-size ext4 image files, mounted at boot: **db 6 GB, storage 1 GB, every other volume 256 MB** |
| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
@@ -32,10 +32,16 @@ of instances at once. Numbers are reused: create picks the lowest free one.
## Requirements
- A Debian/Ubuntu server with **microk8s** (or any Kubernetes with `kubectl` and `helm` on the
path), with an **ingress-nginx** ingress class and **cert-manager** installed.
- DNS `A` records for every hostname slot pointing at the server, and ports 80 and 443 forwarded to
it (Let's Encrypt fetches a test file over port 80 to issue each certificate).
path), with an ingress controller installed: **ingress-nginx** or **Traefik** (the microk8s
`ingress` addon is either, depending on the version). With **Traefik**, its CRDs must be present
on the cluster — they come with the Traefik installation itself; supaspin creates `Middleware`
objects and does not install cluster-wide types.
- With **Let's Encrypt** enabled: **cert-manager** installed, DNS `A` records for every hostname
slot pointing at the server, and ports 80 and 443 forwarded to it (Let's Encrypt fetches a test
file over port 80 to issue each certificate). Without it: no cert-manager, no public DNS — each
instance gets a self-signed certificate instead.
- Root access on that server. `python3` and `python3-yaml` (the installer adds the latter with `apt`).
Self-signed certificates additionally need `openssl` on the server.
## Install
@@ -70,17 +76,21 @@ sudo supaspin
|---|---|
| Storage root | folder that holds every instance's images and `.env` (`<root>/<N>/…`) |
| Node name | Kubernetes node the volumes are pinned to (detected) |
| Server LAN IP | address the Postgres and Studio ports are bound to (detected) |
| Let's Encrypt e-mail | for certificate expiry notices |
| Server LAN IP | address shown for the Postgres and Studio ports (detected) |
| Port bind | `lan` = LAN IP only (default; keeps Postgres and the login-less Studio off a public interface), `all` = every interface incl. localhost (for a box behind NAT running its own apps), `local` = localhost only |
| Ingress class | usually `public` on microk8s |
| Hostname slots | comma-separated; the first is instance 1, the second instance 2, … |
| Ingress controller | `traefik` or `nginx`; every instance's Ingress carries the nginx annotations (50 MiB upload cap, long timeouts) and, in Traefik mode, additionally points at a `Middleware` object with the same 50 MiB cap - so the cap is enforced no matter which of the two actually routes the hostname |
| Let's Encrypt | `y` fetches a real certificate per hostname (needs public DNS, ports 80/443, cert-manager); `n` gives every instance a self-signed certificate and creates no cert-manager objects |
| Let's Encrypt e-mail | for certificate expiry notices (asked only when Let's Encrypt is on) |
| Hostname slots | comma-separated; a name with a number in it keys that slot (`supa4.didgibot.com` = slot 4, so instance 4 gets it with ports 5444/3004), a plain name takes its position in the list |
| Public SSH host / port / user | only used to print the tunnel command |
| Mail settings | default SMTP settings for the auth service (changeable later) |
Settings live in `/etc/supaspin/config.yaml` (readable by root only). The first run also creates,
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-letsencrypt` ClusterIssuer,
the `supaspin-low` PriorityClass and the `supaspin-certs` namespace, and downloads the pinned
Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/charts/`.
only if missing: the `supaspin-local-sc` StorageClass, the `supaspin-low` PriorityClass and the
`supaspin-certs` namespace; with Let's Encrypt on it also creates the `supaspin-letsencrypt`
ClusterIssuer. It then downloads the pinned Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once
into `/var/lib/supaspin/charts/`.
## Using it
@@ -107,12 +117,12 @@ sudo supaspin env N # the full .env (contains secrets)
sudo supaspin tunnel N # the tunnel commands, step by step
sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run] # see "Copying an instance's values"
sudo supaspin delete N [--yes]
sudo supaspin upgrade [N] # apply this version's memory caps to running instances
sudo supaspin upgrade [N] # apply this version's memory caps and log retention to running instances
sudo supaspin suspend N # stop an instance, keep its data (see "Suspending an instance")
sudo supaspin resume N # start it again
sudo supaspin mail # change the saved default mail settings
sudo supaspin setup # change settings
supaspin --version # prints "supaspin 1.3.1"
supaspin --version # prints "supaspin X.Y.Z"
```
### Connecting your app
@@ -291,8 +301,12 @@ it receives in the `_supabase` database and never deletes them, so the 2 GB db i
every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
night. That bounds both the db image and vector's backlog, so vector is back to the 512 MiB
ceiling the production supabase charts carry.
night. pg_cron in this image only runs jobs from the `postgres` database, so the job lives there
and reaches the log tables in `_supabase` over a local dblink connection. The dblink extension
and the helper function live in the `extensions` schema, where the app's database owner has no
rights, so instance code can never call them. That bounds both the
db image and vector's backlog, so vector is back to the 512 MiB ceiling the production supabase
charts carry.
Changing the numbers in this table only affects instances created afterwards. To put them on
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
@@ -307,6 +321,9 @@ pods are replaced service by service and nothing else about the instance changes
- **Certificate stays pending.** The hostname's DNS record must point at the server and port 80 must
reach the ingress: `kubectl get certificate,order,challenge -n supaspin-N`.
- **"no free hostname slot".** Delete an instance or add a slot in Settings (and its DNS record).
- **Create failed** with `no matches for kind "Middleware" in version "traefik.io/v1alpha1"`. The
config says Traefik but the cluster has no Traefik CRDs: either install Traefik (its CRDs come
with it) or set the ingress controller to `nginx` in Settings.
- **Create failed.** `sudo supaspin delete N`, then create again.
- **Port already in use.** Postgres and Studio ports are fixed per number (`5440+N`, `3000+N`);
another program is using it. Free the port or use another instance number.
+1 -1
View File
@@ -1 +1 @@
1.3.2
1.6.4
+178 -27
View File
@@ -39,7 +39,7 @@ import tty
import yaml
VERSION = "1.3.2"
VERSION = "1.6.4"
CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
@@ -53,7 +53,10 @@ CERTS_NS = "supaspin-certs"
# Image cap per volume, in MiB. The chart's separate imgproxy volume is not used: imgproxy has to
# read the uploaded files, so it mounts the storage volume instead (see instance_values).
VOLUMES = {"db": 2048, "storage": 1024, "functions": 256,
# The db image is 6 GB because of the analytics logs: browser-test traffic wrote about 1.2 GB of
# logs a day on instance 2 (2026-10-14), the nightly cleanup keeps LOG_RETENTION of them, and the
# write-ahead log may take up to 1 GB on top. That peaks near 4.5 GB; 2 GB filled up on instance 4.
VOLUMES = {"db": 6144, "storage": 1024, "functions": 256,
"snippets": 256, "deno": 256, "pgsodium": 256}
# The user Postgres runs as inside supabase/postgres 17.6.1.136 (`id postgres` in the image).
# Production's older image used 2000; do not copy that number.
@@ -88,24 +91,44 @@ DEFAULT_MEMORY = (128, 512)
# Analytics logs: deleted from every instance's _supabase database when older than this. Logflare
# stores logs in _analytics.log_events_* tables and never deletes them; without this job the
# capped db image fills up and the database dies (instance 3, 2026-10-06).
# pg_cron in the supabase image only runs jobs from the database named in cron.database_name,
# which the image pins to `postgres`, while the logs live in `_supabase`. The job therefore runs
# in `postgres` and reaches the logs over a local dblink connection, which needs no password: the
# image trusts local socket connections for supabase_admin.
# dblink and the helper function MUST live in the `extensions` schema, not `public`: the app's
# migrations revoke API-role privileges on every routine in public, and `postgres` (the role
# Prisma connects as) is no superuser here - revoking someone else's routines fails with
# "permission denied for function dblink_connect_u" (42501, maralago 0_init, 2026-10-08).
LOG_RETENTION = "2 days"
LOG_RETENTION_JOB = "supaspin-log-retention"
RETENTION_SQL = f"""
create extension if not exists pg_cron;
create or replace function _analytics.delete_old_logs() returns void language plpgsql as $fn$
create extension if not exists dblink with schema extensions;
alter extension dblink set schema extensions;
drop function if exists public.supaspin_delete_old_logs();
create or replace function extensions.supaspin_delete_old_logs() returns void
language plpgsql as $fn$
declare
conn text := 'supaspin_lr_' || substr(md5(clock_timestamp()::text), 1, 8);
t text;
begin
perform extensions.dblink_connect(conn, 'dbname=_supabase user=supabase_admin');
for t in
select tablename from pg_tables
where schemaname = '_analytics' and tablename like 'log_events\\_%'
select tablename from extensions.dblink(conn,
'select tablename from pg_tables where schemaname = ''_analytics'' and tablename like ''log_events\\_%''')
as log_tables(tablename text)
loop
execute format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t);
perform extensions.dblink_exec(conn,
format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t));
end loop;
perform extensions.dblink_disconnect(conn);
exception when others then
begin perform extensions.dblink_disconnect(conn); exception when others then null; end;
raise;
end $fn$;
select _analytics.delete_old_logs();
select extensions.supaspin_delete_old_logs();
delete from cron.job where jobname = '{LOG_RETENTION_JOB}';
select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select _analytics.delete_old_logs();');
select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select extensions.supaspin_delete_old_logs()');
""".strip()
DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
@@ -145,6 +168,54 @@ def tls_secret(n):
return f"supa{n}-tls"
_SLOT_NUM = re.compile(r"(\d+)")
def parse_slots(text):
"""Slots {number: hostname} from the comma-separated setup answer. A hostname with a
number in it keys the slot by that number - supa4.didgibot.com is slot 4 - so the
instance number, its ports and the hostname digit stay in step; a name without a
number takes its position in the list (the first name is slot 1)."""
slots = {}
for i, h in enumerate([p.strip() for p in text.split(",") if p.strip()], 1):
m = _SLOT_NUM.search(h)
n = int(m.group(1)) if m else i
if n in slots:
raise ValueError(f"'{h}' wants slot {n}, which '{slots[n]}' already takes.")
slots[n] = h
return slots
def buffering_middleware(n):
"""Name of the Traefik Middleware that caps the request body size for instance n."""
return f"supa{n}-buffering"
def ingress_annotations(cfg, n):
"""Ingress annotations for instance n.
Both controllers' annotations are always present: an annotation only takes effect
when a controller of that family actually routes the ingress, so on a Traefik
cluster the nginx keys sit inert (and vice versa) and nothing needs reconfiguring
when the router changes. The upload cap therefore lives in two places that agree:
proxy-body-size for nginx, and the Middleware object the traefik annotation points
at. The chart's own rewrite-target default must be nullified or every API path
breaks. The cert-manager annotation orders a certificate from the
supaspin-letsencrypt issuer; without it the TLS secret keeps whatever supaspin put
there (a saved or a self-signed certificate)."""
anns = {}
if cfg.get("lets_encrypt", True):
anns["cert-manager.io/cluster-issuer"] = ISSUER
# The chart default rewrites every path to "/", which breaks all APIs; null removes it.
anns["nginx.ingress.kubernetes.io/rewrite-target"] = None
anns["nginx.ingress.kubernetes.io/proxy-body-size"] = "50m"
anns["nginx.ingress.kubernetes.io/proxy-read-timeout"] = "3600"
anns["nginx.ingress.kubernetes.io/proxy-send-timeout"] = "3600"
if cfg.get("ingress_controller", "nginx") == "traefik":
anns["traefik.ingress.kubernetes.io/router.middlewares"] = \
f"{namespace(n)}-{buffering_middleware(n)}@kubernetescrd"
return anns
def lowest_free_number(slots, taken):
"""Lowest slot number that is not in `taken`; None if every slot is used."""
for n in sorted(int(k) for k in slots):
@@ -207,13 +278,15 @@ def instance_values(cfg, n, sec, mail, template_text):
subs.update({
"PUBLIC_URL": f"https://{host}", "HOST": host,
"TLS_SECRET": tls_secret(n), "INGRESS_CLASS": cfg["ingress_class"],
"CLUSTER_ISSUER": ISSUER, "PROJECT_NAME": f"supaspin-{n}",
"PROJECT_NAME": f"supaspin-{n}",
"INGRESS_ANNOTATIONS": ingress_annotations(cfg, n),
"SMTP_HOST": mail["host"], "SMTP_PORT": mail["port"], "SMTP_USER": mail["user"],
"SMTP_PASS": mail["password"], "SMTP_ADMIN_EMAIL": mail["admin_email"],
"SMTP_SENDER_NAME": mail["sender_name"],
})
# json.dumps gives a quoted YAML-safe string; the port is quoted as well (env values are strings).
quoted = {k: json.dumps(str(v)) for k, v in subs.items()}
# A dict is dumped as JSON too: JSON flow syntax is valid YAML, so the annotation map drops in.
quoted = {k: (json.dumps(v) if isinstance(v, dict) else json.dumps(str(v))) for k, v in subs.items()}
text = re.sub(r"\$\{(\w+)\}", lambda m: quoted[m.group(1)], template_text)
values = yaml.safe_load(text)
apply_memory(values)
@@ -464,15 +537,16 @@ def strip_kong_services(text, names):
return "".join(out)
def post_render(text, n, lan_ip):
def post_render(text, n, cfg):
"""Edit the rendered chart: fixed host ports on db and studio, no public Studio/pg-meta route."""
bind_ip = host_bind_ip(cfg)
docs = [d for d in yaml.safe_load_all(text) if d]
for d in docs:
kind, name = d.get("kind"), d.get("metadata", {}).get("name", "")
if kind == "StatefulSet" and name == f"{fullname(n)}-db":
_host_port(d, pg_port(n), lan_ip)
_host_port(d, pg_port(n), bind_ip)
elif kind == "Deployment" and name == f"{fullname(n)}-studio":
_host_port(d, studio_port(n), lan_ip)
_host_port(d, studio_port(n), bind_ip)
elif kind == "ConfigMap" and "kong.yml" in (d.get("data") or {}):
d["data"]["kong.yml"] = strip_kong_services(d["data"]["kong.yml"], ("meta", "dashboard"))
for d in docs: # keep multi-line scripts readable in the release manifest
@@ -482,11 +556,22 @@ def post_render(text, n, lan_ip):
return yaml.safe_dump_all(docs, sort_keys=False, default_flow_style=False)
def host_bind_ip(cfg):
"""Address the Postgres/Studio host ports listen on. `lan` (the default) is the LAN IP
only; on a machine with a public interface that keeps Postgres and the login-less Studio
off the internet. `all` is every interface - LAN and localhost - for a box behind NAT
that runs its own apps talking to localhost. `local` is localhost only."""
return {"lan": cfg["lan_ip"], "local": "127.0.0.1", "all": None}[cfg.get("port_bind", "lan")]
def _host_port(doc, port, ip):
c = doc["spec"]["template"]["spec"]["containers"][0]
p = c["ports"][0]
p["hostPort"] = port
p["hostIP"] = ip
if ip is not None:
# None = bind every interface (the `all` mode); omitting hostIP makes the kubelet
# listen on 0.0.0.0, which a hostIP cannot express.
p["hostIP"] = ip
if doc.get("kind") == "Deployment":
# A port on the server itself can be held by one pod at a time. The default rollout starts
# the replacement pod before stopping the old one, so the replacement never gets the port
@@ -688,18 +773,21 @@ def ensure_cluster_objects(cfg, log):
{"apiVersion": "storage.k8s.io/v1", "kind": "StorageClass",
"metadata": {"name": STORAGE_CLASS}, "provisioner": "kubernetes.io/no-provisioner",
"reclaimPolicy": "Retain", "volumeBindingMode": "WaitForFirstConsumer"},
{"apiVersion": "cert-manager.io/v1", "kind": "ClusterIssuer",
"metadata": {"name": ISSUER},
"spec": {"acme": {"server": "https://acme-v02.api.letsencrypt.org/directory",
"email": cfg["acme_email"],
"privateKeySecretRef": {"name": ISSUER + "-account-key"},
"solvers": [{"http01": {"ingress": {"class": cfg["ingress_class"]}}}]}}},
{"apiVersion": "scheduling.k8s.io/v1", "kind": "PriorityClass",
"metadata": {"name": PRIORITY_CLASS}, "value": -100,
"preemptionPolicy": "Never", "globalDefault": False,
"description": "supaspin test instances: evicted first, never evict others"},
{"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": CERTS_NS}},
]
if cfg.get("lets_encrypt", True):
# cert-manager needs this object to know where to order certificates from; without
# Let's Encrypt it is dead weight, and cert-manager may not even be installed.
docs.insert(1, {"apiVersion": "cert-manager.io/v1", "kind": "ClusterIssuer",
"metadata": {"name": ISSUER},
"spec": {"acme": {"server": "https://acme-v02.api.letsencrypt.org/directory",
"email": cfg["acme_email"],
"privateKeySecretRef": {"name": ISSUER + "-account-key"},
"solvers": [{"http01": {"ingress": {"class": cfg["ingress_class"]}}}]}}})
for d in docs:
# Create only what is missing; never overwrite something already there.
r = kubectl("get", d["kind"].lower(), d["metadata"]["name"], check=False)
@@ -800,15 +888,51 @@ def setup_log_retention(cfg, n, log):
"""Give the instance a pg_cron job that deletes analytics logs older than LOG_RETENTION every
night. Logflare stores the logs it receives in the _supabase database and never deletes them;
without this job the capped db image fills up and the database dies (instance 3 on
2026-10-06). Runs the first cleanup right away, so an upgrade also frees a full disk. Runs
after create, upgrade and resume; scheduling under a fixed job name makes it idempotent."""
2026-10-06). The job runs in the postgres database, the only one pg_cron accepts in this
image, and deletes in _supabase over a local dblink connection. Runs the first cleanup right
away, so an upgrade also frees a full disk. Runs after create, upgrade and resume;
scheduling under a fixed job name makes it idempotent."""
log(f" log retention: deleting analytics logs older than {LOG_RETENTION}, every night")
kubectl("-n", namespace(n), "exec", f"{fullname(n)}-db-0", "-c", "supabase-db", "--",
"psql", "-U", "supabase_admin", "-d", "_supabase", "-v", "ON_ERROR_STOP=1",
"psql", "-U", "supabase_admin", "-d", "postgres", "-v", "ON_ERROR_STOP=1",
"-c", RETENTION_SQL)
log(" log retention is set")
def middleware_manifest(n):
"""The Traefik request-size cap the ingress annotation of instance n points at. nginx gets
the same cap from proxy-body-size; Traefik has no such annotation, the rule lives here."""
return {"apiVersion": "traefik.io/v1alpha1", "kind": "Middleware",
"metadata": {"name": buffering_middleware(n), "namespace": namespace(n)},
"spec": {"buffering": {"maxRequestBodyBytes": 50 * 1024 * 1024}}} # 50 MiB
def ensure_middleware(cfg, n, log):
"""Create the Middleware for instance n on Traefik. On nginx the annotation is not
emitted, and applying a traefik.io object would fail there - the CRD is not installed."""
if cfg.get("ingress_controller", "nginx") != "traefik":
return
apply(middleware_manifest(n))
log(f" created Middleware {buffering_middleware(n)} (50 MiB request cap)")
def ensure_self_signed_cert(cfg, n, host, log):
"""A self-signed certificate for the hostname, so https answers without Let's Encrypt.
Skipped when a certificate is already in place (restored from supaspin-certs)."""
r = kubectl("get", "secret", tls_secret(n), "-n", namespace(n), check=False)
if r.returncode == 0:
return
with tempfile.TemporaryDirectory(prefix="supaspin-cert-") as d:
key, crt = f"{d}/tls.key", f"{d}/tls.crt"
run(["openssl", "req", "-x509", "-newkey", "rsa:2048", "-sha256", "-nodes",
"-keyout", key, "-out", crt, "-days", "3650",
"-subj", f"/CN={host}", "-addext", f"subjectAltName=DNS:{host}"])
kubectl("create", "secret", "tls", tls_secret(n), "-n", namespace(n),
"--cert", crt, "--key", key)
log(f" self-signed certificate for {host} (valid 10 years);"
" clients must trust it or ignore the warning")
def create_instance(cfg, mail, log, number=None):
taken = existing_numbers(cfg)
n = number if number is not None else lowest_free_number(cfg["slots"], taken)
@@ -826,10 +950,13 @@ def create_instance(cfg, mail, log, number=None):
kubectl("create", "namespace", namespace(n))
apply([pv_manifest(cfg, n, v) for v in VOLUMES])
restore_cert(n, log)
if not cfg.get("lets_encrypt", True):
ensure_self_signed_cert(cfg, n, host, log)
sec = generate_secrets()
with open(find_template()) as f:
values = instance_values(cfg, n, sec, mail, f.read())
vfile = write_values_file(n, values)
ensure_middleware(cfg, n, log)
log(" installing chart")
try:
helm("install", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
@@ -1746,11 +1873,35 @@ def setup_dialog(cfg, log):
cfg["storage_root"] = ask("Storage root (images live here)", cfg.get("storage_root", "/home/igor/k8s-storage/supabase"))
cfg["node_name"] = ask("Kubernetes node name", cfg.get("node_name", d["node_name"]))
cfg["lan_ip"] = ask("Server LAN IP (Postgres/Studio ports bind here)", cfg.get("lan_ip", d["lan_ip"]))
cfg["acme_email"] = ask("Let's Encrypt e-mail", cfg.get("acme_email", ""))
bind_default = cfg.get("port_bind", "lan")
bind = ask("Port bind: lan (LAN only), all (LAN + localhost), local (localhost only)",
bind_default).lower()
while bind not in ("lan", "all", "local"):
print(f"{RED}Answer 'lan', 'all' or 'local'.{OFF}")
bind = ask("Port bind: lan (LAN only), all (LAN + localhost), local (localhost only)",
bind_default).lower()
cfg["port_bind"] = bind
cfg["ingress_class"] = ask("Ingress class", cfg.get("ingress_class", "public"))
controller_default = cfg.get("ingress_controller", "nginx")
controller = ask("Ingress controller, traefik or nginx", controller_default).lower()
while controller not in ("traefik", "nginx"):
print(f"{RED}Answer 'traefik' or 'nginx'.{OFF}")
controller = ask("Ingress controller, traefik or nginx", controller_default).lower()
cfg["ingress_controller"] = controller
le_default = "y" if cfg.get("lets_encrypt", True) else "n"
le = ask("Fetch certificates from Let's Encrypt? Needs public DNS and ports 80/443 (Y/n)", le_default)
cfg["lets_encrypt"] = le.lower().startswith("y")
if cfg["lets_encrypt"]:
cfg["acme_email"] = ask("Let's Encrypt e-mail", cfg.get("acme_email", ""))
cur = ",".join(cfg["slots"][k] for k in sorted(cfg.get("slots", {}))) if cfg.get("slots") else ""
hosts = [h.strip() for h in ask("Hostname slots, comma separated (first = instance 1)", cur).split(",") if h.strip()]
cfg["slots"] = {i + 1: h for i, h in enumerate(hosts)}
while True:
try:
cfg["slots"] = parse_slots(ask(
"Hostname slots, comma separated (a name with a number keys that slot: "
"supa4 = slot 4; a plain name takes its position)", cur))
break
except ValueError as e:
print(f"{RED}{e}{OFF}")
cfg["ssh_host"] = ask("Public SSH host for tunnels", cfg.get("ssh_host", ""))
cfg["ssh_port"] = int(ask("Public SSH port", cfg.get("ssh_port", 22)))
cfg["ssh_user"] = ask("SSH user", cfg.get("ssh_user", d["ssh_user"]))
@@ -1859,7 +2010,7 @@ def main_menu(cfg, log):
def main(argv):
if len(argv) >= 2 and argv[1] == "post-render": # called by Helm; no root check, no output but YAML
cfg = load_config()
sys.stdout.write(post_render(sys.stdin.read(), int(argv[2]), cfg["lan_ip"]))
sys.stdout.write(post_render(sys.stdin.read(), int(argv[2]), cfg))
return 0
if len(argv) > 1 and argv[1] in ("help", "-h", "--help"):
print(__doc__)
+3 -7
View File
@@ -71,13 +71,9 @@ environment:
ingress:
enabled: true
className: ${INGRESS_CLASS}
annotations:
# The chart default rewrites every path to "/", which breaks all APIs; null removes it.
nginx.ingress.kubernetes.io/rewrite-target: null
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
cert-manager.io/cluster-issuer: ${CLUSTER_ISSUER}
# The annotation map is built in code (ingress_annotations): nginx gets proxy-* and a
# nulled rewrite-target, Traefik gets its Middleware reference, cert-manager its issuer.
annotations: ${INGRESS_ANNOTATIONS}
tls:
- hosts: [${HOST}]
secretName: ${TLS_SECRET}
+170 -8
View File
@@ -51,6 +51,25 @@ class Numbering(unittest.TestCase):
def test_no_free_slot(self):
self.assertIsNone(sp.lowest_free_number(CFG["slots"], {1, 2, 3}))
def test_slots_are_keyed_by_the_hostname_number(self):
# The digit in the name is the slot number, so instance N, its ports and the
# hostname stay in step on every machine.
self.assertEqual(sp.parse_slots("supa4.didgibot.com, supa5.didgibot.com"),
{4: "supa4.didgibot.com", 5: "supa5.didgibot.com"})
def test_plain_names_take_their_position(self):
self.assertEqual(sp.parse_slots("db.example.com, db2.example.com"),
{1: "db.example.com", 2: "db2.example.com"})
self.assertEqual(sp.parse_slots("supa4.didgibot.com, spare.example.com"),
{4: "supa4.didgibot.com", 2: "spare.example.com"})
def test_duplicates_are_an_error(self):
with self.assertRaises(ValueError):
sp.parse_slots("supa4.didgibot.com, supa4.other.com")
def test_empty_answer_gives_no_slots(self):
self.assertEqual(sp.parse_slots(""), {})
def test_ports_and_names(self):
self.assertEqual((sp.pg_port(1), sp.pg_port(3)), (5441, 5443))
self.assertEqual((sp.studio_port(1), sp.studio_port(3)), (3001, 3003))
@@ -124,7 +143,7 @@ class Values(unittest.TestCase):
def test_volume_caps(self):
p = self.v["persistence"]
self.assertEqual(p["db"]["size"], "2048Mi")
self.assertEqual(p["db"]["size"], "6144Mi")
self.assertEqual(p["storage"]["size"], "1024Mi")
for vol in ("functions", "snippets", "deno", "pgsodium"):
self.assertEqual(p[vol]["size"], "256Mi")
@@ -134,11 +153,23 @@ class Values(unittest.TestCase):
def test_log_retention_sql(self):
# Logflare stores logs in _analytics.log_events_* and never deletes them; the job deletes
# rows older than LOG_RETENTION and is scheduled under a fixed name so re-runs overwrite.
self.assertIn("schemaname = '_analytics'", sp.RETENTION_SQL)
self.assertIn("like 'log_events\\_%'", sp.RETENTION_SQL)
# pg_cron in the supabase image only runs jobs from the `postgres` database, so the job
# reaches the log tables in `_supabase` over a local dblink connection (trusted there).
# dblink and the helper function live in the `extensions` schema: in `public` they belong
# to someone else's routines and the app's `REVOKE ALL ON ALL ROUTINES IN SCHEMA public`
# fails as the non-superuser postgres role (42501, dblink_connect_u).
self.assertIn("schemaname = ''_analytics''", sp.RETENTION_SQL)
self.assertIn("''log_events\\_%''", sp.RETENTION_SQL)
self.assertIn(f"interval ''{sp.LOG_RETENTION}''", sp.RETENTION_SQL)
self.assertIn(f"'{sp.LOG_RETENTION_JOB}'", sp.RETENTION_SQL)
self.assertIn("cron.schedule", sp.RETENTION_SQL)
self.assertIn("extensions.dblink_connect", sp.RETENTION_SQL)
self.assertIn("user=supabase_admin", sp.RETENTION_SQL)
self.assertIn("create extension if not exists dblink with schema extensions", sp.RETENTION_SQL)
self.assertIn("alter extension dblink set schema extensions", sp.RETENTION_SQL)
self.assertIn("extensions.supaspin_delete_old_logs", sp.RETENTION_SQL)
self.assertIn("drop function if exists public.supaspin_delete_old_logs", sp.RETENTION_SQL)
self.assertNotIn("\ndblink_", sp.RETENTION_SQL) # no unqualified dblink call remains
def test_imgproxy_reads_the_storage_volume(self):
img = self.v["persistence"]["imgproxy"]
@@ -167,6 +198,120 @@ class Values(unittest.TestCase):
self.assertEqual(v["secret"]["smtp"]["password"], 'p"a$s: #x')
class IngressShaping(unittest.TestCase):
"""The ingress annotation map per controller and per Let's Encrypt setting."""
def setUp(self):
self.sec = sp.generate_secrets()
def ann(self, cfg=None, n=2):
return sp.instance_values(dict(CFG, **(cfg or {})), n, self.sec, MAIL, template()) \
["ingress"]["annotations"]
def test_default_is_nginx_with_lets_encrypt(self):
# Configs written before these settings existed must render exactly as before.
a = self.ann()
self.assertEqual(a["cert-manager.io/cluster-issuer"], "supaspin-letsencrypt")
self.assertIsNone(a["nginx.ingress.kubernetes.io/rewrite-target"])
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-body-size"], "50m")
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-read-timeout"], "3600")
self.assertNotIn("traefik.ingress.kubernetes.io/router.middlewares", a)
def test_traefik_also_keeps_nginx_annotations(self):
# The nginx keys must be present in traefik mode too: while nginx routes the
# ingress they enforce the cap, and they are inert the day Traefik takes over.
a = self.ann({"ingress_controller": "traefik"})
self.assertEqual(a["traefik.ingress.kubernetes.io/router.middlewares"],
"supaspin-2-supa2-buffering@kubernetescrd")
self.assertIsNone(a["nginx.ingress.kubernetes.io/rewrite-target"])
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-body-size"], "50m")
self.assertEqual(a["nginx.ingress.kubernetes.io/proxy-read-timeout"], "3600")
self.assertEqual(a["cert-manager.io/cluster-issuer"], "supaspin-letsencrypt")
def test_lets_encrypt_off_drops_cert_manager_annotation(self):
self.assertNotIn("cert-manager.io/cluster-issuer", self.ann({"lets_encrypt": False}))
self.assertNotIn("cert-manager.io/cluster-issuer",
self.ann({"lets_encrypt": False, "ingress_controller": "traefik"}))
def test_middleware_manifest(self):
m = sp.middleware_manifest(2)
self.assertEqual(m["apiVersion"], "traefik.io/v1alpha1")
self.assertEqual(m["kind"], "Middleware")
self.assertEqual(m["metadata"], {"name": "supa2-buffering", "namespace": "supaspin-2"})
self.assertEqual(m["spec"]["buffering"]["maxRequestBodyBytes"], 50 * 1024 * 1024)
class ClusterObjects(unittest.TestCase):
"""ensure_cluster_objects skips the ClusterIssuer when Let's Encrypt is off."""
def created_docs(self, cfg):
applied = []
class R:
returncode = 1
stdout = stderr = ""
def fake_kubectl(*args, **kw):
if args and args[0] == "apply":
applied.extend(yaml.safe_load_all(kw["input_"]))
return R()
with tempfile.TemporaryDirectory() as tmp, \
mock.patch.object(sp, "STATE_DIR", tmp), \
mock.patch.object(sp, "kubectl", side_effect=fake_kubectl), \
mock.patch.object(sp, "helm"):
sp.ensure_cluster_objects(cfg, lambda *a: None)
return applied
def test_issuer_created_when_lets_encrypt_on(self):
self.assertIn("ClusterIssuer", [d["kind"] for d in self.created_docs(CFG)])
def test_no_issuer_without_lets_encrypt(self):
kinds = [d["kind"] for d in self.created_docs(dict(CFG, lets_encrypt=False))]
self.assertNotIn("ClusterIssuer", kinds)
self.assertIn("StorageClass", kinds)
self.assertIn("PriorityClass", kinds)
self.assertIn("Namespace", kinds)
class SelfSignedCert(unittest.TestCase):
def test_generated_when_missing(self):
calls = []
class R:
returncode = 1
stdout = stderr = ""
def fake_kubectl(*args, **kw):
calls.append(args)
return R()
def fake_run(args, **kw):
calls.append(("run", args))
class P:
returncode = 0
return P()
with mock.patch.object(sp, "kubectl", side_effect=fake_kubectl), \
mock.patch.object(sp, "run", side_effect=fake_run):
sp.ensure_self_signed_cert(CFG, 2, "supa2.example.com", lambda *a: None)
openssl = next(c[1] for c in calls if c[0] == "run" and c[1][0] == "openssl")
self.assertIn("subjectAltName=DNS:supa2.example.com", openssl)
create = next(c for c in calls if c[:3] == ("create", "secret", "tls"))
self.assertEqual(create[3], "supa2-tls")
self.assertIn("supaspin-2", create)
def test_kept_when_present(self):
class R:
returncode = 0
stdout = stderr = ""
with mock.patch.object(sp, "kubectl", return_value=R()) as k, \
mock.patch.object(sp, "run") as r:
sp.ensure_self_signed_cert(CFG, 2, "supa2.example.com", lambda *a: None)
k.assert_called_once()
r.assert_not_called()
class EnvFile(unittest.TestCase):
def setUp(self):
self.sec = sp.generate_secrets()
@@ -266,7 +411,7 @@ data:
class PostRender(unittest.TestCase):
def setUp(self):
self.docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, "192.168.10.30")))
self.docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, CFG)))
def find(self, kind, name):
return next(d for d in self.docs if d["kind"] == kind and d["metadata"]["name"] == name)
@@ -277,6 +422,23 @@ class PostRender(unittest.TestCase):
st = self.find("Deployment", "supa1-supabase-studio")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
self.assertEqual((st["hostPort"], st["hostIP"]), (3001, "192.168.10.30"))
def test_all_binds_localhost_too(self):
# `all` omits hostIP: the kubelet then listens on every interface, localhost included.
docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, dict(CFG, port_bind="all"))))
find = lambda k, name: next(d for d in docs if d["kind"] == k and d["metadata"]["name"] == name)
db = find("StatefulSet", "supa1-supabase-db")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
st = find("Deployment", "supa1-supabase-studio")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
self.assertEqual(db["hostPort"], 5441)
self.assertNotIn("hostIP", db)
self.assertEqual(st["hostPort"], 3001)
self.assertNotIn("hostIP", st)
def test_local_binds_localhost_only(self):
docs = list(yaml.safe_load_all(sp.post_render(RENDERED, 1, dict(CFG, port_bind="local"))))
find = lambda k, name: next(d for d in docs if d["kind"] == k and d["metadata"]["name"] == name)
db = find("StatefulSet", "supa1-supabase-db")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
self.assertEqual((db["hostPort"], db["hostIP"]), (5441, "127.0.0.1"))
def test_other_pods_untouched(self):
rest = self.find("Deployment", "supa1-supabase-rest")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
self.assertNotIn("hostPort", rest)
@@ -306,9 +468,9 @@ class PostRender(unittest.TestCase):
self.assertEqual(cm["data"]["kong-entrypoint.sh"], "echo hi\necho there\n")
def test_instance_number_selects_ports(self):
docs = list(yaml.safe_load_all(sp.post_render(RENDERED.replace("supa1", "supa3"), 3, "10.0.0.1")))
docs = list(yaml.safe_load_all(sp.post_render(RENDERED.replace("supa1", "supa3"), 3, CFG)))
db = next(d for d in docs if d["kind"] == "StatefulSet")["spec"]["template"]["spec"]["containers"][0]["ports"][0]
self.assertEqual((db["hostPort"], db["hostIP"]), (5443, "10.0.0.1"))
self.assertEqual((db["hostPort"], db["hostIP"]), (5443, "192.168.10.30"))
class Manifests(unittest.TestCase):
@@ -316,7 +478,7 @@ class Manifests(unittest.TestCase):
pv = sp.pv_manifest(CFG, 2, "db")
self.assertEqual(pv["spec"]["claimRef"], {"namespace": "supaspin-2", "name": "supa2-supabase-db"})
self.assertEqual(pv["spec"]["local"]["path"], "/data/supabase/2/db/data")
self.assertEqual(pv["spec"]["capacity"]["storage"], "2048Mi")
self.assertEqual(pv["spec"]["capacity"]["storage"], "6144Mi")
self.assertEqual(pv["spec"]["persistentVolumeReclaimPolicy"], "Retain")
self.assertEqual(pv["spec"]["nodeAffinity"]["required"]["nodeSelectorTerms"][0]
["matchExpressions"][0]["values"], ["node1"])
@@ -1099,7 +1261,7 @@ class HostPortRollout(unittest.TestCase):
"template": {"spec": {"containers": [
{"name": "rest", "ports": [{"containerPort": 3000}]}]}}}},
])
out = list(yaml.safe_load_all(sp.post_render(text, n, CFG["lan_ip"])))
out = list(yaml.safe_load_all(sp.post_render(text, n, CFG)))
return {d["metadata"]["name"]: d for d in out}
def test_studio_stops_the_old_pod_before_starting_the_new_one(self):