Merge pull request 'fix: delete analytics logs older than 2 days and give vector a 512 MiB cap' (#9) from fix/log-retention-vector-512 into main

- every instance gets a pg_cron job (set up on create, upgrade and resume)
  that deletes _analytics.log_events_* rows older than 2 days and runs the
  first cleanup right away; Logflare never deletes stored logs, and instance
  3's 2 GB db image died 100% full of them on 2026-10-06, after which
  analytics 500'd every delivery and vector was OOM-killed by its
  undeliverable backlog every few minutes
- vector's memory limit back to 512 MiB, the ceiling the production
  supabase charts carry; the retention job bounds its backlog, so 1024 MiB
  is not needed anymore
- version 1.3.2

CLAUDE.md

@@ -21,8 +21,14 @@ The number is `MAJOR.MINOR.PATCH` (semantic versioning: `1.1.1` is major 1, mino
 4. **Tag every release.** After the pull request is merged, create the git tag `vX.Y.Z` (for example `v1.1.1`) on
    the commit on `main` that contains that number in `VERSION`. The tag name must equal `VERSION` with a `v` in
    front. A version without a tag is not released. Never move or delete a tag that was already published; if a
-   release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/igor/supaspin/tags` with
+   release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/public/supaspin/tags` with
    `tag_name`, `target` (the commit) and `message`.
+
+   The owner in an API address must be `public`. The repository used to sit under the user `igor` and Gitea still
+   forwards the old address for anything a browser or git does, but it answers an API call to
+   `/api/v1/repos/igor/supaspin/...` with a redirect that curl does not follow for a POST: the command returns an
+   empty body and no error, which looks exactly like a rejected token. Opening or merging a pull request needs the
+   same `public` owner.
 5. **Check the result.** Reinstall from the merged `main` (`sudo bash install.sh`), then run `supaspin --version` and
    compare it with `VERSION` and the tag.
 6. If you find a version in `VERSION` that has no tag, or a tag that does not match `VERSION`, tell Igor before doing

README.md

@@ -1,6 +1,6 @@
 # supaspin
 
-Version 1.1.1
+Version 1.3.2
 
 Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
 single-node [microk8s](https://microk8s.io) server, from a terminal menu.
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
 | Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
 | Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
 | Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
-| Memory | limits add up to **5.625 GiB** per instance, requests to 1.875 GiB (table below) |
+| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
 | Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
 | Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
 | Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
@@ -40,7 +40,7 @@ of instances at once. Numbers are reused: create picks the lowest free one.
 ## Install
 
 ```bash
-curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
+curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
 ```
 
 **If your Gitea requires sign-in even for public repositories** (the download then fails with
@@ -48,13 +48,13 @@ curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | s
 `repository: read`):
 
 ```bash
-curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
+curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
 ```
 
 or clone the repository and install from the checkout:
 
 ```bash
-git clone ssh://git@git.didgibot.com:2226/igor/supaspin.git && sudo bash supaspin/install.sh
+git clone ssh://git@git.didgibot.com:2226/public/supaspin.git && sudo bash supaspin/install.sh
 ```
 
 Run it again to upgrade. It installs `/usr/local/bin/supaspin`, its values template and the
@@ -90,9 +90,10 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
 |---|---|
 | Create instance | picks the lowest free number, asks whether to use the saved mail settings or enter different ones, creates the disks, installs Supabase, waits until every pod is ready, writes `<root>/<N>/.env` |
 | List instances | number, hostname, pods ready, memory in use, disk use per image, `.env` path, and for each instance the tunnel command with plain instructions (open it, keep it in the background, close it) |
-| Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
+| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
 | Delete instance | asks you to type the number; removes everything (see below) |
 | Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
+| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
 | Mail settings | view and change the saved default mail settings |
 | Settings | change any first-run setting |
 
@@ -106,9 +107,12 @@ sudo supaspin env N                   # the full .env (contains secrets)
 sudo supaspin tunnel N                # the tunnel commands, step by step
 sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]   # see "Copying an instance's values"
 sudo supaspin delete N [--yes]
+sudo supaspin upgrade [N]             # apply this version's memory caps to running instances
+sudo supaspin suspend N               # stop an instance, keep its data (see "Suspending an instance")
+sudo supaspin resume N                # start it again
 sudo supaspin mail                    # change the saved default mail settings
 sudo supaspin setup                   # change settings
-supaspin --version                    # prints "supaspin 1.1.1"
+supaspin --version                    # prints "supaspin 1.3.1"
 ```
 
 ### Connecting your app
@@ -207,21 +211,93 @@ Creating that instance again restores the saved certificate, so repeated create/
 not hit Let's Encrypt's limit of 5 certificates per hostname per week. If a create fails midway,
 run `supaspin delete N` to clean up.
 
+## Suspending an instance
+
+An instance you are not using right now does not have to be deleted. It can be stopped and
+started again with its data, from the menu:
+
+```
+sudo supaspin  ->  Show instance (and suspend or resume it)  ->  pick the instance
+                                                             ->  Suspend it (stop it, keep its data)
+```
+
+The picker names every instance with its state next to it, so you can see at a glance which ones
+are awake. Inside an instance the same line reads **Resume it** when the instance is asleep, so
+there is one line to look for either way and nothing to remember. Stopping takes about half a
+minute; starting takes a minute or two, because the database answers first and the rest wait for it.
+
+The same two actions without the menu, if you prefer typing:
+
+```bash
+sudo supaspin suspend 2
+sudo supaspin resume 2
+```
+
+What the server gets back while an instance sleeps:
+
+- the memory it had reserved (1.875 GiB) and everything it was actually using (1.5 to 2.5 GiB),
+- its twelve pod slots, out of the 110 a node allows,
+- its two ports on the server: Postgres on 544N and Studio on 300N are free for something else.
+
+What is kept, untouched:
+
+- the database and the uploaded files - the disk images stay mounted and keep their contents,
+- the `.env` file with its passwords and keys, so your app needs no change when the instance is back,
+- the HTTPS certificate, which goes on renewing itself while the instance sleeps,
+- the hostname and every Kubernetes object. Until you resume it, the hostname answers "service
+  unavailable" instead of not answering at all.
+
+Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
+state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
+because nothing is listening on its ports. Asking for the tunnel command of a sleeping instance
+says so instead of handing you a command that cannot work.
+
+Suspending is written into the instance's stored settings, not done by hand to the running pods,
+so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
+stores the new memory limits and leaves it asleep; they take effect when you resume it.
+
 ## Memory per instance
 
-| Service | Request | Limit |
-|---|---|---|
-| db | 512 MiB | 2048 MiB |
-| analytics | 128 MiB | 768 MiB |
-| kong | 128 MiB | 512 MiB |
-| realtime | 128 MiB | 384 MiB |
-| auth, functions, imgproxy, meta, rest, storage, studio, vector | 128 MiB each | 256 MiB each |
-| **Total** | **1.875 GiB** | **5.625 GiB** |
+| Service | Request | Limit | Highest use measured |
+|---|---|---|---|
+| db | 512 MiB | 2048 MiB | 273 MiB |
+| analytics | 128 MiB | 1536 MiB | 694 MiB |
+| vector | 128 MiB | 512 MiB | 290 MiB |
+| kong | 128 MiB | 768 MiB | 269 MiB |
+| realtime | 128 MiB | 768 MiB | 300 MiB |
+| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
+| **Total** | **1.875 GiB** | **9 GiB** | |
 
-Most limits are half of what a production setup would use. Analytics and realtime are higher: at
-half of production they were killed for running out of memory in a loop (measured on a fresh
-instance: analytics peaks at 624 MiB, realtime at 280 MiB). Image resizing (imgproxy) reads the
-uploads from the storage volume, so it has no image of its own.
+The request is what the scheduler sets aside for a service and can never be given to anything else.
+The limit is the ceiling the kernel kills the container at. They do different jobs here.
+
+The requests are deliberately small: the server is one machine shared with the production
+workloads, and reserving what three instances really use would leave production nothing to start
+in. The limits are roughly twice the highest use measured across three running instances on
+2026-09-22 (right-hand column above).
+
+The earlier limits sat right on top of normal use and the kernel kept killing the two logging
+services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
+768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
+instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
+could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
+cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
+Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
+(imgproxy) reads the uploads from the storage volume, so it has no image of its own.
+
+On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
+it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
+100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
+every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
+minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
+job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
+night. That bounds both the db image and vector's backlog, so vector is back to the 512 MiB
+ceiling the production supabase charts carry.
+
+Changing the numbers in this table only affects instances created afterwards. To put them on
+instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
+the instance was installed, replaces the memory block in them and applies the chart again, so the
+pods are replaced service by service and nothing else about the instance changes.
 
 ## Troubleshooting
 

VERSION

@@ -1 +1 @@
-1.1.1
+1.3.2

install.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # One-line install (run on the Kubernetes server):
-#   curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
+#   curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
 # Installs /usr/local/bin/supaspin and its Helm values template. Safe to run again to upgrade.
 #
 # If your Gitea requires sign-in even for public repositories, give a read-only token:
@@ -12,7 +12,7 @@
 # SUPASPIN_TOKEN=<token>        Gitea token (read access to the repository) for the download
 set -euo pipefail
 
-REPO="https://git.didgibot.com/igor/supaspin"
+REPO="https://git.didgibot.com/public/supaspin"
 REF="${SUPASPIN_REF:-main}"
 
 [ "$(id -u)" -eq 0 ] || { echo "Run as root: curl -fsSL $REPO/raw/branch/main/install.sh | sudo bash" >&2; exit 1; }

supaspin

@@ -5,6 +5,13 @@ Each numbered instance gets its own namespace, capped disk images, capped memory
 an HTTPS hostname and fixed Postgres/Studio ports. Standard library plus python3-yaml.
 Run as root: supaspin            (menu)
              supaspin create|list|show N|env N|tunnel N|delete N|setup|mail
+             supaspin upgrade [N]
+               Puts this version's memory caps on instances that already run, one service
+               at a time. Without N every instance. Data and .env files are not touched.
+             supaspin suspend N | resume N
+               Suspend stops every pod of one instance and gives back its memory, its pod
+               slots and its two server ports. The disks, the passwords in .env, the
+               certificate and the hostname stay. Resume starts it again with its data.
              supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]
                TARGET = a .env file on this server (/path/.env) or on another computer
                (user@host:/path/.env, Windows: user@host:C:/path/.env). Puts the instance's
@@ -32,7 +39,7 @@ import tty
 
 import yaml
 
-VERSION = "1.1.1"
+VERSION = "1.3.2"
 CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
 STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
 CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
@@ -53,13 +60,53 @@ VOLUMES = {"db": 2048, "storage": 1024, "functions": 256,
 DB_OWNED = ("db", "pgsodium")
 DB_UID, DB_GID = 100, 101
 
-# Memory per service: (request MiB, limit MiB). Half of maralago.ru production, except analytics
-# and realtime: at half of production they were killed for running out of memory in a loop
-# (measured on a fresh instance: analytics peaks at 624 MiB, realtime at 280 MiB).
+# Memory per service: (request MiB, limit MiB). The limit is the ceiling the kernel kills the
+# container at; the request is what the scheduler sets aside for it. The requests stay small on
+# purpose: this is one machine shared with production, and reserving what an instance really uses
+# three times over would leave production nothing to start in. The limits are about twice the
+# highest use measured across three running instances on 2026-09-22, because the two logging
+# services grow with their backlog and the old ceilings sat right on top of normal use:
+#   analytics 612 / 694 / 662 MiB against a 768 MiB ceiling - killed 36 times on instance 1
+#   vector    287 / 290 MiB against 512 - killed 297 times on instance 1, where it had a backlog
+#   realtime  269 / 287 / 300 MiB against 384
+#   kong      260 / 269 / 265 MiB against 512
+#   studio 203, storage 187, rest 162 MiB against 256
+#   db        163 / 175 / 273 MiB against 2048 - left alone, it has room
+# On 2026-10-06 instance 3 filled its 2 GB database image with logs: analytics (Logflare) never
+# deletes what it stored, the db died of it, and vector's undelivered backlog was killed against
+# the 1024 MiB ceiling every few minutes. The fix is setup_log_retention: a pg_cron job deletes
+# analytics logs older than LOG_RETENTION in every instance, which also bounds vector's backlog,
+# so vector is back to the 512 MiB ceiling the production supabase charts carry.
+# Raising a limit costs nothing until a container actually asks for the memory; raising a request
+# would take the memory away from everything else on the node straight away.
 SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
             "realtime", "rest", "storage", "studio", "vector"]
-MEMORY = {"db": (512, 2048), "analytics": (128, 768), "kong": (128, 512), "realtime": (128, 384)}
-DEFAULT_MEMORY = (128, 256)
+MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
+          "vector": (128, 512)}
+DEFAULT_MEMORY = (128, 512)
+
+# Analytics logs: deleted from every instance's _supabase database when older than this. Logflare
+# stores logs in _analytics.log_events_* tables and never deletes them; without this job the
+# capped db image fills up and the database dies (instance 3, 2026-10-06).
+LOG_RETENTION = "2 days"
+LOG_RETENTION_JOB = "supaspin-log-retention"
+RETENTION_SQL = f"""
+create extension if not exists pg_cron;
+create or replace function _analytics.delete_old_logs() returns void language plpgsql as $fn$
+declare
+  t text;
+begin
+  for t in
+    select tablename from pg_tables
+    where schemaname = '_analytics' and tablename like 'log_events\\_%'
+  loop
+    execute format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t);
+  end loop;
+end $fn$;
+select _analytics.delete_old_logs();
+delete from cron.job where jobname = '{LOG_RETENTION_JOB}';
+select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select _analytics.delete_old_logs();');
+""".strip()
 
 DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
                 "admin_email": "admin@example.com", "sender_name": "Supabase"}
@@ -169,12 +216,7 @@ def instance_values(cfg, n, sec, mail, template_text):
     quoted = {k: json.dumps(str(v)) for k, v in subs.items()}
     text = re.sub(r"\$\{(\w+)\}", lambda m: quoted[m.group(1)], template_text)
     values = yaml.safe_load(text)
-    dep = values.setdefault("deployment", {})
-    for svc in SERVICES:
-        req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
-        d = dep.setdefault(svc, {})
-        d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
-        d["priorityClassName"] = PRIORITY_CLASS
+    apply_memory(values)
     persistence = values.setdefault("persistence", {})
     for vol, mib in VOLUMES.items():
         persistence[vol] = {"enabled": True, "storageClassName": STORAGE_CLASS,
@@ -185,6 +227,47 @@ def instance_values(cfg, n, sec, mail, template_text):
     return values
 
 
+def apply_memory(values):
+    """Write the memory caps and the priority class of every service into a values dictionary,
+    replacing whatever was there. Used both when an instance is created and when a running one is
+    brought up to the current caps."""
+    dep = values.setdefault("deployment", {})
+    for svc in SERVICES:
+        req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
+        d = dep.setdefault(svc, {})
+        d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
+        d["priorityClassName"] = PRIORITY_CLASS
+    return values
+
+
+def set_replicas(values, count):
+    """How many copies of each service to run. Zero stops the instance: the pods go away and give
+    back their memory, their processor time, their pod slots and the two ports they hold on the
+    server, while the disks, the passwords, the certificate and the hostname stay exactly as they
+    are. One starts it again."""
+    dep = values.setdefault("deployment", {})
+    for svc in SERVICES:
+        dep.setdefault(svc, {})["replicaCount"] = count
+    return values
+
+
+def is_suspended(values):
+    """True when the stored settings say to run no copy of any service."""
+    dep = values.get("deployment") or {}
+    counts = [(dep.get(svc) or {}).get("replicaCount", 1) for svc in SERVICES]
+    return all(c == 0 for c in counts)
+
+
+def memory_limits(values):
+    """{service: limit} as the values dictionary has it, for telling before from after."""
+    dep = values.get("deployment") or {}
+    out = {}
+    for svc in SERVICES:
+        lim = ((dep.get(svc) or {}).get("resources") or {}).get("limits") or {}
+        out[svc] = lim.get("memory")
+    return out
+
+
 def total_memory_mib():
     """(requests, limits) in MiB, summed over all services."""
     req = sum(MEMORY.get(s, DEFAULT_MEMORY)[0] for s in SERVICES)
@@ -404,6 +487,12 @@ def _host_port(doc, port, ip):
     p = c["ports"][0]
     p["hostPort"] = port
     p["hostIP"] = ip
+    if doc.get("kind") == "Deployment":
+        # A port on the server itself can be held by one pod at a time. The default rollout starts
+        # the replacement pod before stopping the old one, so the replacement never gets the port
+        # and waits for ever ("didn't have free ports for the requested pod ports"). Recreate stops
+        # the old pod first. A StatefulSet already replaces its pod that way.
+        doc["spec"]["strategy"] = {"type": "Recreate"}
 
 
 def pv_manifest(cfg, n, vol):
@@ -707,6 +796,19 @@ def wait_ready(n, log, timeout=1200):
     raise RuntimeError("pods did not become ready in time")
 
 
+def setup_log_retention(cfg, n, log):
+    """Give the instance a pg_cron job that deletes analytics logs older than LOG_RETENTION every
+    night. Logflare stores the logs it receives in the _supabase database and never deletes them;
+    without this job the capped db image fills up and the database dies (instance 3 on
+    2026-10-06). Runs the first cleanup right away, so an upgrade also frees a full disk. Runs
+    after create, upgrade and resume; scheduling under a fixed job name makes it idempotent."""
+    log(f"  log retention: deleting analytics logs older than {LOG_RETENTION}, every night")
+    kubectl("-n", namespace(n), "exec", f"{fullname(n)}-db-0", "-c", "supabase-db", "--",
+            "psql", "-U", "supabase_admin", "-d", "_supabase", "-v", "ON_ERROR_STOP=1",
+            "-c", RETENTION_SQL)
+    log("  log retention is set")
+
+
 def create_instance(cfg, mail, log, number=None):
     taken = existing_numbers(cfg)
     n = number if number is not None else lowest_free_number(cfg["slots"], taken)
@@ -727,19 +829,15 @@ def create_instance(cfg, mail, log, number=None):
         sec = generate_secrets()
         with open(find_template()) as f:
             values = instance_values(cfg, n, sec, mail, f.read())
-        vfile = f"{STATE_DIR}/values-{n}.yaml"
-        os.makedirs(STATE_DIR, exist_ok=True)
-        fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
-        with os.fdopen(fd, "w") as f:
-            yaml.safe_dump(values, f, sort_keys=False)
+        vfile = write_values_file(n, values)
         log("  installing chart")
         try:
             helm("install", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
-                 "--post-renderer", os.path.realpath(sys.argv[0]),
-                 "--post-renderer-args", "post-render", "--post-renderer-args", str(n))
+                 *post_render_args(n))
         finally:
             os.remove(vfile)  # holds every secret
         wait_ready(n, log)
+        setup_log_retention(cfg, n, log)
         env_path = f"{inst_dir(cfg, n)}/.env"
         fd = os.open(env_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
         with os.fdopen(fd, "w") as f:
@@ -755,6 +853,140 @@ def create_instance(cfg, mail, log, number=None):
         raise
 
 
+def write_values_file(n, values):
+    """The values Helm is given. It carries every secret of the instance, so it is written
+    readable by root only and the caller deletes it as soon as Helm has read it."""
+    vfile = f"{STATE_DIR}/values-{n}.yaml"
+    os.makedirs(STATE_DIR, exist_ok=True)
+    fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    with os.fdopen(fd, "w") as f:
+        yaml.safe_dump(values, f, sort_keys=False)
+    return vfile
+
+
+def post_render_args(n):
+    """Helm hands the finished manifests to supaspin itself, which fixes the parts the chart
+    cannot express (the Postgres and Studio host ports, the gateway routes that are not used)."""
+    return ["--post-renderer", os.path.realpath(sys.argv[0]),
+            "--post-renderer-args", "post-render", "--post-renderer-args", str(n)]
+
+
+def stored_values(cfg, n):
+    """The settings Helm kept from the install. They carry every secret of the instance, which is
+    why supaspin deletes its own copy of them as soon as Helm has read it."""
+    ns, rel = namespace(n), release(n)
+    r = helm("get", "values", rel, "-n", ns, "-o", "yaml", check=False)
+    if r.returncode != 0:
+        raise RuntimeError(f"instance {n}: no Helm release {rel} in namespace {ns}\n{r.stderr.strip()}")
+    values = yaml.safe_load(r.stdout) or {}
+    if not values.get("deployment"):
+        raise RuntimeError(f"instance {n}: the stored settings of {rel} look empty; not touching it")
+    return values
+
+
+def apply_values(cfg, n, values):
+    """Hand the settings to Helm, which works out what to change and changes only that."""
+    vfile = write_values_file(n, values)
+    try:
+        helm("upgrade", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
+             *post_render_args(n))
+    finally:
+        os.remove(vfile)  # holds every secret
+
+
+def wait_gone(n, log, timeout=300):
+    """Wait until the instance's own pods have stopped."""
+    end = time.time() + timeout
+    while time.time() < end:
+        _, total, _ = pod_summary(n)
+        log(f"  pods still running: {total}", live=True)
+        if total == 0:
+            log("")
+            return
+        time.sleep(3)
+    raise RuntimeError("the pods did not stop in time")
+
+
+def suspend_instance(cfg, n, log):
+    """Stop every pod of an instance and keep everything else. The database, the uploads, the
+    passwords in .env, the certificate and the hostname are untouched; the instance answers
+    nothing until it is resumed."""
+    log(f"Suspending instance {n} ({cfg['slots'].get(n, '?')})")
+    values = stored_values(cfg, n)
+    if is_suspended(values):
+        log("  already suspended")
+        return False
+    apply_values(cfg, n, set_replicas(values, 0))
+    wait_gone(n, log)
+    log(f"  instance {n} is suspended; its disks and its .env file are untouched")
+    log("  start it again in this instance's menu: \"Resume it\"")
+    return True
+
+
+def resume_instance(cfg, n, log):
+    """Start a suspended instance again, with the data it had when it was suspended."""
+    log(f"Resuming instance {n} ({cfg['slots'].get(n, '?')})")
+    values = stored_values(cfg, n)
+    if not is_suspended(values):
+        log("  already running")
+        return False
+    apply_values(cfg, n, set_replicas(values, 1))
+    log("  the database answers first; the services that need it retry until it does")
+    wait_ready(n, log)
+    setup_log_retention(cfg, n, log)
+    log(f"  instance {n} is running again: https://{cfg['slots'].get(n, '?')}")
+    return True
+
+
+def upgrade_instance(cfg, n, log):
+    """Bring a running instance up to the memory caps this version of supaspin carries, without
+    deleting anything. The values Helm kept from the install are read back, the memory block in
+    them is replaced, and the chart is applied again: the pods are replaced one service at a time
+    and the database, the uploads and the .env file are untouched."""
+    log(f"Instance {n} ({cfg['slots'].get(n, '?')})")
+    values = stored_values(cfg, n)
+    asleep = is_suspended(values)
+    before = memory_limits(values)
+    after = memory_limits(apply_memory(values))
+    changed = [svc for svc in SERVICES if before[svc] != after[svc]]
+    for svc in changed:
+        log(f"  {svc}: {before[svc] or 'unset'} -> {after[svc]}")
+    if not changed:
+        # The chart is applied even so: the caps are not the only thing this version may produce
+        # differently (the Studio rollout, the host ports, the gateway routes are all decided
+        # while the manifests are rendered). Helm changes nothing if nothing differs.
+        log("  caps already current; re-applying the chart in case anything else differs")
+    apply_values(cfg, n, values)
+    if asleep:
+        # A suspended instance stays suspended: the new caps are stored and take effect the moment
+        # it is resumed. Waiting for its pods would wait for ever - by design it has none.
+        log(f"  instance {n} is suspended; the new caps and log retention apply when you resume it")
+        return bool(changed)
+    wait_ready(n, log)
+    setup_log_retention(cfg, n, log)
+    log(f"  instance {n} is running with the current settings")
+    return bool(changed)
+
+
+def upgrade_all(cfg, log, numbers=None):
+    """Upgrade one instance, or every instance that exists."""
+    nums = [numbers] if isinstance(numbers, int) else sorted(numbers or existing_numbers(cfg))
+    if not nums:
+        log("No instances.")
+        return
+    req, lim = total_memory_mib()
+    log(f"Caps in this version: {req} MiB reserved and up to {lim} MiB per instance.")
+    failed = []
+    for n in nums:
+        try:
+            upgrade_instance(cfg, n, log)
+        except RuntimeError as e:
+            failed.append(n)
+            log(f"{RED}  instance {n} failed: {e}{OFF}")
+    if failed:
+        raise RuntimeError("these instances were not upgraded: " + ", ".join(str(n) for n in failed))
+
+
 def delete_instance(cfg, n, log):
     log(f"Deleting instance {n}")
     save_cert(n, log)
@@ -818,12 +1050,29 @@ def disk_use(cfg, n):
     return ", ".join(parts)
 
 
+def instance_state(n):
+    """What the cluster is currently asked to run for this instance: "running", "suspended",
+    "partly suspended" (some services stopped, which should not happen on its own) or "unknown"
+    when the instance has no workloads at all."""
+    want = []
+    for kind in ("deployments", "statefulsets"):
+        data = kjson("get", kind, "-n", namespace(n), "-l", f"app.kubernetes.io/instance={release(n)}")
+        for item in (data or {}).get("items", []):
+            want.append(item["spec"].get("replicas", 1))
+    if not want:
+        return "unknown"
+    if all(w == 0 for w in want):
+        return "suspended"
+    return "partly suspended" if any(w == 0 for w in want) else "running"
+
+
 def list_rows(cfg):
     rows = []
     for n in sorted(existing_numbers(cfg)):
         ready, total, _ = pod_summary(n)
         env = f"{inst_dir(cfg, n)}/.env"
         rows.append({"n": n, "host": cfg["slots"].get(n, "?"), "pods": f"{ready}/{total}",
+                     "state": instance_state(n),
                      "memory": memory_use(n), "disk": disk_use(cfg, n),
                      "env": env if os.path.exists(env) else "(none)"})
     return rows
@@ -1455,7 +1704,7 @@ def pick_instance(cfg, title):
     if not nums:
         print("No instances.")
         return None
-    i = menu(title, [f"{n}  {cfg['slots'].get(n, '?')}" for n in nums])
+    i = menu(title, [f"{n}  {cfg['slots'].get(n, '?')}  ({instance_state(n)})" for n in nums])
     return None if i is None else nums[i]
 
 
@@ -1514,11 +1763,19 @@ def setup_dialog(cfg, log):
     return cfg
 
 
-def show_menu(cfg, n):
+def show_menu(cfg, n, log=None):
+    """Everything you can do to one instance, including stopping and starting it. The stop/start
+    line is worded for the state the instance is in when the menu is drawn, and the state is read
+    again every time you come back to it, so the menu never offers something that makes no sense."""
+    log = log or log_line
     while True:
-        i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')}",
-                 [".env", ".env (secrets shown)", "Tunnel command", "Pods and events", "Back"])
-        if i is None or i == 4:
+        state = instance_state(n)
+        asleep = state == "suspended"
+        switch = ("Resume it (start it again with its data)" if asleep
+                  else "Suspend it (stop it, keep its data)")
+        items = [".env", ".env (secrets shown)", "Tunnel command", "Pods and events", switch, "Back"]
+        i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')} - {state}", items)
+        if i is None or i == 5:
             return
         print()
         try:
@@ -1526,10 +1783,14 @@ def show_menu(cfg, n):
                 text = read_env(cfg, n)
                 print(f"{inst_dir(cfg, n)}/.env\n\n" + (text if i == 1 else mask_env(text)))
             elif i == 2:
-                print(tunnel_help(cfg, n))
-            else:
+                print(tunnel_help(cfg, n) if not asleep else
+                      f"Instance {n} is suspended: nothing is listening on its ports.\n"
+                      "Resume it from this menu and the tunnel commands work again.")
+            elif i == 3:
                 print(kubectl("get", "pods", "-n", namespace(n), "-o", "wide").stdout)
                 print(kubectl("get", "events", "-n", namespace(n), "--sort-by=.lastTimestamp").stdout[-3000:])
+            else:
+                (resume_instance if asleep else suspend_instance)(cfg, n, log)
         except Exception as e:
             print(f"{RED}{e}{OFF}")
         pause()
@@ -1541,8 +1802,12 @@ def print_list(cfg):
         print("No instances.")
         return
     for r in rows:
-        print(f"{BOLD}{r['n']}{OFF}  {r['host']}  pods {r['pods']}  memory {r['memory']}")
+        state = r.get("state", "running")
+        print(f"{BOLD}{r['n']}{OFF}  {r['host']}  {state}  pods {r['pods']}  memory {r['memory']}")
         print(f"     disk {r['disk']}\n     env  {r['env']}")
+        if state == "suspended":
+            print(f"     start it again in this instance's menu: \"Resume it\"\n")
+            continue  # the tunnel is pointless while nothing listens on the ports
         for line in tunnel_guide(cfg, r["n"]):
             print(f"     {line}" if line else "")
         print()
@@ -1550,10 +1815,13 @@ def print_list(cfg):
 
 def main_menu(cfg, log):
     while True:
-        i = menu(f"supaspin {VERSION}", ["Create instance", "List instances", "Show instance",
-                                          "Delete instance", "Copy instance values into another .env file",
+        i = menu(f"supaspin {VERSION}",
+                 ["Create instance", "List instances",
+                  "Show instance (and suspend or resume it)", "Delete instance",
+                  "Copy instance values into another .env file",
+                  "Apply current memory caps to running instances",
                   "Mail settings", "Settings", "Quit"])
-        if i is None or i == 7:
+        if i is None or i == 8:
             return
         print()
         try:
@@ -1562,9 +1830,9 @@ def main_menu(cfg, log):
             elif i == 1:
                 print_list(cfg)
             elif i == 2:
-                n = pick_instance(cfg, "Show which instance?")
+                n = pick_instance(cfg, "Which instance?")
                 if n:
-                    show_menu(cfg, n)
+                    show_menu(cfg, n, log)
                     continue
             elif i == 3:
                 n = pick_instance(cfg, "Delete which instance?")
@@ -1577,9 +1845,11 @@ def main_menu(cfg, log):
                     print()
                     push_env(cfg, n, spec, log)
             elif i == 5:
+                upgrade_all(cfg, log)
+            elif i == 6:
                 cfg["mail"] = mail_dialog(cfg)
                 save_config(cfg)
-            elif i == 6:
+            elif i == 7:
                 cfg = setup_dialog(cfg, log)
         except Exception as e:
             print(f"{RED}{e}{OFF}")
@@ -1628,6 +1898,16 @@ def main(argv):
         elif cmd == "delete" and n:
             if "--yes" in argv or confirm_number(n, "deleting it"):
                 delete_instance(cfg, n, log_line)
+        elif cmd == "upgrade":
+            upgrade_all(cfg, log_line, n)
+        elif cmd in ("suspend", "resume"):
+            if n is None:
+                # No "all" here on purpose: stopping every instance is not something to type by
+                # accident, and resuming them all at once would start 36 pods in one go.
+                print(f"supaspin {cmd} N  - which instance? ({cmd} takes one number)",
+                      file=sys.stderr)
+                return 2
+            (suspend_instance if cmd == "suspend" else resume_instance)(cfg, n, log_line)
         elif cmd == "push-env":
             return push_env_command(cfg, argv[2:])
         elif cmd == "mail":

tests/test_supaspin.py

@@ -90,13 +90,23 @@ class Values(unittest.TestCase):
 
     def test_memory_caps_add_up(self):
         req, lim = sp.total_memory_mib()
-        # db 2048, analytics 768, kong 512, realtime 384, the other eight 256 each
-        self.assertEqual(lim, 2048 + 768 + 512 + 384 + 8 * 256)
-        self.assertEqual(lim / 1024, 5.625)
+        # db 2048, analytics 1536, kong 768, realtime 768, vector 512, the other seven 512 each
+        self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 512 + 7 * 512)
+        self.assertEqual(lim / 1024, 9.0)
         self.assertEqual(req, 512 + 11 * 128)  # 1.875 GiB
-        # analytics and realtime were killed for running out of memory at half of production
-        self.assertGreaterEqual(sp.MEMORY["analytics"][1], 768)
-        self.assertGreaterEqual(sp.MEMORY["realtime"][1], 384)
+        # Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
+        # highest use, or the kernel kills the container in a loop the way it did on instance 1.
+        self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536)  # used 694 MiB, died at 768
+        self.assertGreaterEqual(sp.MEMORY["vector"][1], 512)      # used 290 MiB; the nightly log
+        #  retention keeps its delivery backlog from growing past this the way it did on
+        #  2026-10-06, when instance 3's full db image killed analytics and vector filled with
+        #  the undeliverable backlog every few minutes against 1024 MiB.
+        self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768)    # used 300 MiB against 384
+        self.assertGreaterEqual(sp.MEMORY["kong"][1], 768)        # used 269 MiB against 512
+        self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512)        # studio used 203 against 256
+        # The requests stay where they were: the node is shared with production and reserving
+        # what the instances really use would leave production nothing to schedule into.
+        self.assertTrue(all(m[0] == 128 for s, m in sp.MEMORY.items() if s != "db"))
         self.assertEqual(req / 1024, 1.875)
         for svc in sp.SERVICES:
             self.assertEqual(self.v["deployment"][svc]["priorityClassName"], "supaspin-low")
@@ -121,6 +131,15 @@ class Values(unittest.TestCase):
         self.assertEqual(set(sp.VOLUMES), {"db", "storage", "functions", "snippets", "deno", "pgsodium"})
         self.assertTrue(all(p[v]["storageClassName"] == "supaspin-local-sc" for v in sp.VOLUMES))
 
+    def test_log_retention_sql(self):
+        # Logflare stores logs in _analytics.log_events_* and never deletes them; the job deletes
+        # rows older than LOG_RETENTION and is scheduled under a fixed name so re-runs overwrite.
+        self.assertIn("schemaname = '_analytics'", sp.RETENTION_SQL)
+        self.assertIn("like 'log_events\\_%'", sp.RETENTION_SQL)
+        self.assertIn(f"interval ''{sp.LOG_RETENTION}''", sp.RETENTION_SQL)
+        self.assertIn(f"'{sp.LOG_RETENTION_JOB}'", sp.RETENTION_SQL)
+        self.assertIn("cron.schedule", sp.RETENTION_SQL)
+
     def test_imgproxy_reads_the_storage_volume(self):
         img = self.v["persistence"]["imgproxy"]
         self.assertEqual(img["existingClaim"], "supa2-supabase-storage")
@@ -928,3 +947,420 @@ class PushEnvCommand(unittest.TestCase):
 
 if __name__ == "__main__":
     unittest.main()
+
+
+class Upgrade(unittest.TestCase):
+    """`supaspin upgrade` puts the current memory caps on an instance that is already running.
+    It must change the caps and nothing else: no volume is resized, no secret is regenerated."""
+
+    STORED = {
+        "deployment": {
+            "analytics": {"enabled": True, "priorityClassName": "supaspin-low",
+                          "resources": {"limits": {"memory": "768Mi"},
+                                        "requests": {"memory": "128Mi"}}},
+            "vector": {"enabled": True, "priorityClassName": "supaspin-low",
+                       "resources": {"limits": {"memory": "512Mi"},
+                                     "requests": {"memory": "128Mi"}}},
+        },
+        "secret": {"jwt": {"secret": "keep-me", "anonKey": "anon"}},
+        "persistence": {"db": {"enabled": True, "size": "2048Mi"}},
+    }
+
+    def setUp(self):
+        self.calls = []
+        self.written = {}
+        self.log = []
+
+    @contextlib.contextmanager
+    def fake_helm(self, stored=None, rc=0):
+        stored = self.STORED if stored is None else stored
+
+        def helm(*args, check=True):
+            self.calls.append(list(args))
+            if args[0] == "get":
+                out = yaml.safe_dump(stored) if stored is not None else ""
+                return collections.namedtuple("P", "returncode stdout stderr")(rc, out, "boom")
+            if args[0] == "upgrade":
+                with open(args[args.index("-f") + 1]) as f:
+                    self.written = yaml.safe_load(f)
+            return collections.namedtuple("P", "returncode stdout stderr")(0, "", "")
+
+        with mock.patch.object(sp, "helm", helm), \
+                mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
+                mock.patch.object(sp, "setup_log_retention", lambda cfg, n, log: None), \
+                mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
+            yield
+
+    def test_raises_the_caps_and_keeps_everything_else(self):
+        with self.fake_helm():
+            changed = sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertTrue(changed)
+        dep = self.written["deployment"]
+        self.assertEqual(dep["analytics"]["resources"]["limits"]["memory"],
+                         f"{sp.MEMORY['analytics'][1]}Mi")
+        self.assertEqual(dep["vector"]["resources"]["limits"]["memory"],
+                         f"{sp.MEMORY['vector'][1]}Mi")
+        self.assertTrue(dep["analytics"]["enabled"])              # untouched
+        self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
+        self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
+
+    def test_every_service_gets_a_cap_even_if_the_stored_values_had_none(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        for svc in sp.SERVICES:
+            req, lim = sp.MEMORY.get(svc, sp.DEFAULT_MEMORY)
+            got = self.written["deployment"][svc]["resources"]
+            self.assertEqual(got["limits"]["memory"], f"{lim}Mi")
+            self.assertEqual(got["requests"]["memory"], f"{req}Mi")
+
+    def test_the_values_file_with_the_secrets_is_deleted_again(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
+        self.assertFalse(os.path.exists(path))
+
+    def test_helm_gets_the_post_renderer(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 2, self.log.append)
+        up = [c for c in self.calls if c[0] == "upgrade"][0]
+        self.assertIn("--post-renderer", up)
+        self.assertEqual(up[up.index("--post-renderer-args") + 1], "post-render")
+        self.assertIn("2", up)
+
+    def test_an_instance_already_at_the_caps_is_still_re_applied(self):
+        # The caps are not the only thing a new version can change: the Studio rollout, the host
+        # ports and the gateway routes are decided while the manifests are rendered, and none of
+        # them shows up in the stored values.
+        stored = {"deployment": sp.apply_memory({})["deployment"]}
+        with self.fake_helm(stored):
+            changed = sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertFalse(changed)                                   # no cap moved
+        self.assertEqual([c[0] for c in self.calls], ["get", "upgrade"])  # applied anyway
+
+    def test_a_missing_release_is_refused_before_anything_is_written(self):
+        with self.fake_helm(rc=1):
+            with self.assertRaises(RuntimeError):
+                sp.upgrade_instance(CFG, 3, self.log.append)
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+
+    def test_empty_stored_values_are_refused(self):
+        with self.fake_helm({}):
+            with self.assertRaises(RuntimeError):
+                sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+
+    def test_upgrade_all_does_every_instance_and_reports_the_failures(self):
+        done = []
+
+        def one(cfg, n, log):
+            if n == 2:
+                raise RuntimeError("no release")
+            done.append(n)
+            return True
+
+        with mock.patch.object(sp, "upgrade_instance", one), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
+            with self.assertRaises(RuntimeError) as e:
+                sp.upgrade_all(CFG, self.log.append)
+        self.assertEqual(done, [1, 3])
+        self.assertIn("2", str(e.exception))
+
+    def test_upgrade_all_with_a_number_does_only_that_instance(self):
+        done = []
+        with mock.patch.object(sp, "upgrade_instance", lambda cfg, n, log: done.append(n)), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
+            sp.upgrade_all(CFG, self.log.append, 2)
+        self.assertEqual(done, [2])
+
+
+class HostPortRollout(unittest.TestCase):
+    """Studio and Postgres bind a port on the server itself, and only one pod can hold a port.
+    The Studio deployment must therefore stop its old pod before starting the new one, or an
+    upgrade leaves the replacement pod Pending for ever."""
+
+    def rendered(self, n=1):
+        sec = sp.generate_secrets()
+        values = sp.instance_values(CFG, n, sec, MAIL, template())
+        with mock.patch.object(sp, "helm"), mock.patch.object(sp, "kubectl"):
+            text = yaml.safe_dump_all([
+                {"apiVersion": "apps/v1", "kind": "Deployment",
+                 "metadata": {"name": f"{sp.fullname(n)}-studio"},
+                 "spec": {"strategy": {"type": "RollingUpdate",
+                                       "rollingUpdate": {"maxSurge": "25%"}},
+                          "template": {"spec": {"containers": [
+                              {"name": "studio", "ports": [{"containerPort": 3000}]}]}}}},
+                {"apiVersion": "apps/v1", "kind": "StatefulSet",
+                 "metadata": {"name": f"{sp.fullname(n)}-db"},
+                 "spec": {"template": {"spec": {"containers": [
+                     {"name": "db", "ports": [{"containerPort": 5432}]}]}}}},
+                {"apiVersion": "apps/v1", "kind": "Deployment",
+                 "metadata": {"name": f"{sp.fullname(n)}-rest"},
+                 "spec": {"strategy": {"type": "RollingUpdate"},
+                          "template": {"spec": {"containers": [
+                              {"name": "rest", "ports": [{"containerPort": 3000}]}]}}}},
+            ])
+        out = list(yaml.safe_load_all(sp.post_render(text, n, CFG["lan_ip"])))
+        return {d["metadata"]["name"]: d for d in out}
+
+    def test_studio_stops_the_old_pod_before_starting_the_new_one(self):
+        d = self.rendered()[f"{sp.fullname(1)}-studio"]
+        self.assertEqual(d["spec"]["strategy"], {"type": "Recreate"})
+        port = d["spec"]["template"]["spec"]["containers"][0]["ports"][0]
+        self.assertEqual(port["hostPort"], sp.studio_port(1))
+        self.assertEqual(port["hostIP"], CFG["lan_ip"])
+
+    def test_a_deployment_without_a_host_port_keeps_the_normal_rollout(self):
+        d = self.rendered()[f"{sp.fullname(1)}-rest"]
+        self.assertEqual(d["spec"]["strategy"]["type"], "RollingUpdate")
+
+    def test_the_database_keeps_its_host_port_and_gains_no_strategy(self):
+        d = self.rendered()[f"{sp.fullname(1)}-db"]
+        self.assertNotIn("strategy", d["spec"])
+        self.assertEqual(d["spec"]["template"]["spec"]["containers"][0]["ports"][0]["hostPort"],
+                         sp.pg_port(1))
+
+
+class SuspendResume(unittest.TestCase):
+    """Suspending an instance stops its pods and keeps everything else: the disks, the passwords,
+    the certificate and the hostname. Resuming starts it again with the same data."""
+
+    RUNNING = {
+        "deployment": {
+            "analytics": {"enabled": True,
+                          "resources": {"limits": {"memory": "1536Mi"},
+                                        "requests": {"memory": "128Mi"}}},
+            "db": {"resources": {"limits": {"memory": "2048Mi"},
+                                 "requests": {"memory": "512Mi"}}},
+        },
+        "secret": {"jwt": {"secret": "keep-me"}, "db": {"password": "keep-me-too"}},
+        "persistence": {"db": {"enabled": True, "size": "2048Mi"}},
+    }
+
+    def setUp(self):
+        self.calls = []
+        self.written = {}
+        self.log = []
+        self.waited = []
+
+    @contextlib.contextmanager
+    def fake_helm(self, stored):
+        P = collections.namedtuple("P", "returncode stdout stderr")
+
+        def helm(*args, check=True):
+            self.calls.append(list(args))
+            if args[0] == "get":
+                return P(0, yaml.safe_dump(stored), "")
+            if args[0] == "upgrade":
+                with open(args[args.index("-f") + 1]) as f:
+                    self.written = yaml.safe_load(f)
+            return P(0, "", "")
+
+        with mock.patch.object(sp, "helm", helm), \
+                mock.patch.object(sp, "wait_ready",
+                                  lambda n, log, **kw: self.waited.append(("ready", n))), \
+                mock.patch.object(sp, "wait_gone",
+                                  lambda n, log, **kw: self.waited.append(("gone", n))), \
+                mock.patch.object(sp, "setup_log_retention",
+                                  lambda cfg, n, log: self.waited.append(("retention", n))), \
+                mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
+            yield
+
+    def suspended(self):
+        return sp.set_replicas(yaml.safe_load(yaml.safe_dump(self.RUNNING)), 0)
+
+    # ---- the setting itself ----
+
+    def test_set_replicas_covers_every_service(self):
+        v = sp.set_replicas({}, 0)
+        self.assertEqual(sorted(v["deployment"]), sorted(sp.SERVICES))
+        self.assertTrue(all(d["replicaCount"] == 0 for d in v["deployment"].values()))
+        self.assertTrue(sp.is_suspended(v))
+
+    def test_a_service_missing_the_setting_counts_as_running(self):
+        self.assertFalse(sp.is_suspended(self.RUNNING))
+        self.assertFalse(sp.is_suspended({"deployment": {}}))
+
+    def test_one_service_left_running_is_not_suspended(self):
+        v = sp.set_replicas({}, 0)
+        v["deployment"]["kong"]["replicaCount"] = 1
+        self.assertFalse(sp.is_suspended(v))
+
+    # ---- suspend ----
+
+    def test_suspend_stops_every_service_and_keeps_the_rest(self):
+        with self.fake_helm(self.RUNNING):
+            self.assertTrue(sp.suspend_instance(CFG, 1, self.log.append))
+        for svc in sp.SERVICES:
+            self.assertEqual(self.written["deployment"][svc]["replicaCount"], 0)
+        self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
+        self.assertEqual(self.written["secret"]["db"]["password"], "keep-me-too")
+        self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
+        # the memory caps are kept as they were: they apply again on resume
+        self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
+                         "1536Mi")
+        self.assertEqual(self.waited, [("gone", 1)])
+
+    def test_suspend_waits_for_the_pods_to_be_gone_not_for_them_to_be_ready(self):
+        with self.fake_helm(self.RUNNING):
+            sp.suspend_instance(CFG, 2, self.log.append)
+        self.assertEqual([w[0] for w in self.waited], ["gone"])
+
+    def test_suspending_a_suspended_instance_changes_nothing(self):
+        with self.fake_helm(self.suspended()):
+            self.assertFalse(sp.suspend_instance(CFG, 1, self.log.append))
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+        self.assertEqual(self.waited, [])
+
+    def test_the_values_file_with_the_secrets_is_deleted_again(self):
+        with self.fake_helm(self.RUNNING):
+            sp.suspend_instance(CFG, 1, self.log.append)
+        path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
+        self.assertFalse(os.path.exists(path))
+
+    # ---- resume ----
+
+    def test_resume_starts_every_service_again(self):
+        with self.fake_helm(self.suspended()):
+            self.assertTrue(sp.resume_instance(CFG, 3, self.log.append))
+        for svc in sp.SERVICES:
+            self.assertEqual(self.written["deployment"][svc]["replicaCount"], 1)
+        self.assertFalse(sp.is_suspended(self.written))
+        self.assertEqual(self.waited, [("ready", 3), ("retention", 3)])
+
+    def test_resuming_a_running_instance_changes_nothing(self):
+        with self.fake_helm(self.RUNNING):
+            self.assertFalse(sp.resume_instance(CFG, 1, self.log.append))
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+        self.assertEqual(self.waited, [])
+
+    def test_a_suspended_instance_keeps_its_data_settings(self):
+        with self.fake_helm(self.suspended()):
+            sp.resume_instance(CFG, 1, self.log.append)
+        self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
+        self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
+
+    # ---- upgrade must not trip over a suspended instance ----
+
+    def test_upgrade_leaves_a_suspended_instance_suspended_and_does_not_wait(self):
+        with self.fake_helm(self.suspended()):
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertTrue(sp.is_suspended(self.written))        # not woken up
+        self.assertEqual(self.waited, [])                     # would never finish
+        self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
+                         f"{sp.MEMORY['analytics'][1]}Mi")    # new caps stored for the resume
+
+    def test_upgrade_still_waits_for_a_running_instance(self):
+        with self.fake_helm(self.RUNNING):
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertEqual(self.waited, [("ready", 1), ("retention", 1)])
+
+
+class InstanceState(unittest.TestCase):
+    """What `supaspin list` says about an instance, read from what the cluster is asked to run."""
+
+    def state(self, deployments, statefulsets):
+        def kjson(*args):
+            kind = args[1]
+            counts = deployments if kind == "deployments" else statefulsets
+            return {"items": [{"spec": {"replicas": c}} for c in counts]}
+
+        with mock.patch.object(sp, "kjson", kjson):
+            return sp.instance_state(1)
+
+    def test_running(self):
+        self.assertEqual(self.state([1] * 11, [1]), "running")
+
+    def test_suspended(self):
+        self.assertEqual(self.state([0] * 11, [0]), "suspended")
+
+    def test_the_database_alone_still_running_is_reported_honestly(self):
+        self.assertEqual(self.state([0] * 11, [1]), "partly suspended")
+
+    def test_nothing_there(self):
+        self.assertEqual(self.state([], []), "unknown")
+
+    def test_the_list_shows_the_state_and_no_tunnel_for_a_sleeping_instance(self):
+        rows = [{"n": 1, "host": "supa1.example.com", "pods": "0/0", "state": "suspended",
+                 "memory": "n/a", "disk": "db 1/2M", "env": "/data/supabase/1/.env"}]
+        out = io.StringIO()
+        with mock.patch.object(sp, "list_rows", lambda cfg: rows), \
+                contextlib.redirect_stdout(out):
+            sp.print_list(CFG)
+        text = out.getvalue()
+        self.assertIn("suspended", text)
+        self.assertIn("Resume it", text)          # the menu line, not a command to remember
+        self.assertNotIn("ssh -N -L", text)
+
+
+class InstanceMenu(unittest.TestCase):
+    """Stopping and starting an instance lives inside the instance you are looking at: one line,
+    worded for the state the instance is in, so there is no second command to remember."""
+
+    def run_menu(self, state, choose):
+        """Draw the menu for an instance in `state`, pick the item whose text `choose` matches
+        (None: leave at once). Returns (the item texts, what was called)."""
+        seen, called = [], []
+
+        def menu(title, items):
+            seen.append((title, list(items)))
+            if choose is None:
+                return None
+            for i, text in enumerate(items):
+                if choose in text:
+                    return i
+            raise AssertionError(f"no item matching {choose!r} in {items}")
+
+        calls = {"suspend": lambda cfg, n, log: called.append(("suspend", n)),
+                 "resume": lambda cfg, n, log: called.append(("resume", n))}
+        out = io.StringIO()
+        with mock.patch.object(sp, "menu", menu), \
+                mock.patch.object(sp, "instance_state", lambda n: state), \
+                mock.patch.object(sp, "pause", lambda: (_ for _ in ()).throw(StopIteration)), \
+                mock.patch.object(sp, "suspend_instance", calls["suspend"]), \
+                mock.patch.object(sp, "resume_instance", calls["resume"]), \
+                mock.patch.object(sp, "read_env", lambda cfg, n: "A=b\n"), \
+                contextlib.redirect_stdout(out):
+            try:
+                sp.show_menu(CFG, 2, lambda *a, **k: None)
+            except StopIteration:
+                pass  # the menu would loop for ever; stop at the first "press Enter"
+        return seen[0][1], called, out.getvalue(), seen[0][0]
+
+    def test_a_running_instance_offers_to_suspend_it(self):
+        items, called, _, title = self.run_menu("running", None)
+        self.assertTrue(any("Suspend it" in t for t in items), items)
+        self.assertFalse(any("Resume" in t for t in items), items)
+        self.assertIn("running", title)
+
+    def test_a_suspended_instance_offers_to_resume_it(self):
+        items, called, _, title = self.run_menu("suspended", None)
+        self.assertTrue(any("Resume it" in t for t in items), items)
+        self.assertFalse(any("Suspend" in t for t in items), items)
+        self.assertIn("suspended", title)
+
+    def test_choosing_it_suspends_this_instance(self):
+        _, called, _, _ = self.run_menu("running", "Suspend it")
+        self.assertEqual(called, [("suspend", 2)])
+
+    def test_choosing_it_resumes_this_instance(self):
+        _, called, _, _ = self.run_menu("suspended", "Resume it")
+        self.assertEqual(called, [("resume", 2)])
+
+    def test_the_tunnel_of_a_sleeping_instance_says_why_it_is_pointless(self):
+        _, _, text, _ = self.run_menu("suspended", "Tunnel")
+        self.assertIn("suspended", text)
+        self.assertNotIn("ssh -N -L", text)
+
+    def test_the_picker_names_the_state_of_every_instance(self):
+        seen = []
+
+        def menu(title, items):
+            seen.append(list(items))
+            return None
+
+        with mock.patch.object(sp, "menu", menu), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2}), \
+                mock.patch.object(sp, "instance_state", lambda n: "suspended" if n == 2 else "running"):
+            sp.pick_instance(CFG, "Which instance?")
+        self.assertEqual(seen[0], ["1  supa1.example.com  (running)",
+                                   "2  supa2.example.com  (suspended)"])