Merge pull request 'fix: run the log-retention SQL in the postgres database' (#11) from fix/log-retention-run-in-postgres into main

1.3.3 put the dblink job into the SQL but still ran psql against
_supabase, where pg_cron refuses the extension; the psql session now runs
in postgres, the only database pg_cron accepts in this image
version 1.3.4

README.md

@@ -1,6 +1,6 @@
 # supaspin
 
-Version 1.3.0
+Version 1.3.4
 
 Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
 single-node [microk8s](https://microk8s.io) server, from a terminal menu.
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
 | Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
 | Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
 | Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
-| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
+| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
 | Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
 | Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
 | Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
@@ -90,12 +90,10 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
 |---|---|
 | Create instance | picks the lowest free number, asks whether to use the saved mail settings or enter different ones, creates the disks, installs Supabase, waits until every pod is ready, writes `<root>/<N>/.env` |
 | List instances | number, hostname, pods ready, memory in use, disk use per image, `.env` path, and for each instance the tunnel command with plain instructions (open it, keep it in the background, close it) |
-| Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
+| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
 | Delete instance | asks you to type the number; removes everything (see below) |
 | Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
-| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
+| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
-| Suspend instance | stops every pod of one instance and keeps everything else: its memory, processor time, pod slots and two server ports go back to the server, its disks and `.env` file stay |
-| Resume instance | starts a suspended instance again, with the data it had |
 | Mail settings | view and change the saved default mail settings |
 | Settings | change any first-run setting |
 
@@ -114,7 +112,7 @@ sudo supaspin suspend N               # stop an instance, keep its data (see "Su
 sudo supaspin resume N                # start it again
 sudo supaspin mail                    # change the saved default mail settings
 sudo supaspin setup                   # change settings
-supaspin --version                    # prints "supaspin 1.3.0"
+supaspin --version                    # prints "supaspin 1.3.1"
 ```
 
 ### Connecting your app
@@ -215,13 +213,24 @@ run `supaspin delete N` to clean up.
 
 ## Suspending an instance
 
-An instance you are not using right now does not have to be deleted. `sudo supaspin suspend N`
+An instance you are not using right now does not have to be deleted. It can be stopped and
-stops all twelve of its pods and leaves everything else alone; `sudo supaspin resume N` starts
+started again with its data, from the menu:
-them again with the data they had.
+
+```
+sudo supaspin  ->  Show instance (and suspend or resume it)  ->  pick the instance
+                                                             ->  Suspend it (stop it, keep its data)
+```
+
+The picker names every instance with its state next to it, so you can see at a glance which ones
+are awake. Inside an instance the same line reads **Resume it** when the instance is asleep, so
+there is one line to look for either way and nothing to remember. Stopping takes about half a
+minute; starting takes a minute or two, because the database answers first and the rest wait for it.
+
+The same two actions without the menu, if you prefer typing:
 
 ```bash
-sudo supaspin suspend 2     # about half a minute
+sudo supaspin suspend 2
-sudo supaspin resume 2      # a minute or two: the database answers first, the rest wait for it
+sudo supaspin resume 2
 ```
 
 What the server gets back while an instance sleeps:
@@ -240,7 +249,8 @@ What is kept, untouched:
 
 Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
 state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
-because nothing is listening on its ports.
+because nothing is listening on its ports. Asking for the tunnel command of a sleeping instance
+says so instead of handing you a command that cannot work.
 
 Suspending is written into the instance's stored settings, not done by hand to the running pods,
 so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
@@ -252,11 +262,11 @@ stores the new memory limits and leaves it asleep; they take effect when you res
 |---|---|---|---|
 | db | 512 MiB | 2048 MiB | 273 MiB |
 | analytics | 128 MiB | 1536 MiB | 694 MiB |
-| vector | 128 MiB | 1024 MiB | 290 MiB |
+| vector | 128 MiB | 512 MiB | 290 MiB |
 | kong | 128 MiB | 768 MiB | 269 MiB |
 | realtime | 128 MiB | 768 MiB | 300 MiB |
 | auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
-| **Total** | **1.875 GiB** | **9.5 GiB** | |
+| **Total** | **1.875 GiB** | **9 GiB** | |
 
 The request is what the scheduler sets aside for a service and can never be given to anything else.
 The limit is the ceiling the kernel kills the container at. They do different jobs here.
@@ -268,13 +278,23 @@ in. The limits are roughly twice the highest use measured across three running i
 
 The earlier limits sat right on top of normal use and the kernel kept killing the two logging
 services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
-768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
+768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
-log files of every pod on the server and not only its own instance's, was killed 297 times on that
+instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
-same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
+could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
-other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
+cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
-the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
+Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
-in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
+(imgproxy) reads the uploads from the storage volume, so it has no image of its own.
-its own.
+
+On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
+it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
+100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
+every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
+minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
+job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
+night. pg_cron in this image only runs jobs from the `postgres` database, so the job lives there
+and reaches the log tables in `_supabase` over a local dblink connection. That bounds both the
+db image and vector's backlog, so vector is back to the 512 MiB ceiling the production supabase
+charts carry.
 
 Changing the numbers in this table only affects instances created afterwards. To put them on
 instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when

VERSION

@@ -1 +1 @@
-1.3.0
+1.3.4

supaspin

@@ -39,7 +39,7 @@ import tty
 
 import yaml
 
-VERSION = "1.3.0"
+VERSION = "1.3.4"
 CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
 STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
 CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
@@ -72,14 +72,56 @@ DB_UID, DB_GID = 100, 101
 #   kong      260 / 269 / 265 MiB against 512
 #   studio 203, storage 187, rest 162 MiB against 256
 #   db        163 / 175 / 273 MiB against 2048 - left alone, it has room
+# On 2026-10-06 instance 3 filled its 2 GB database image with logs: analytics (Logflare) never
+# deletes what it stored, the db died of it, and vector's undelivered backlog was killed against
+# the 1024 MiB ceiling every few minutes. The fix is setup_log_retention: a pg_cron job deletes
+# analytics logs older than LOG_RETENTION in every instance, which also bounds vector's backlog,
+# so vector is back to the 512 MiB ceiling the production supabase charts carry.
 # Raising a limit costs nothing until a container actually asks for the memory; raising a request
 # would take the memory away from everything else on the node straight away.
 SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
             "realtime", "rest", "storage", "studio", "vector"]
 MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
-          "vector": (128, 1024)}
+          "vector": (128, 512)}
 DEFAULT_MEMORY = (128, 512)
 
+# Analytics logs: deleted from every instance's _supabase database when older than this. Logflare
+# stores logs in _analytics.log_events_* tables and never deletes them; without this job the
+# capped db image fills up and the database dies (instance 3, 2026-10-06).
+# pg_cron in the supabase image only runs jobs from the database named in cron.database_name,
+# which the image pins to `postgres`, while the logs live in `_supabase`. The job therefore runs
+# in `postgres` and reaches the logs over a local dblink connection, which needs no password: the
+# image trusts local socket connections for supabase_admin.
+LOG_RETENTION = "2 days"
+LOG_RETENTION_JOB = "supaspin-log-retention"
+RETENTION_SQL = f"""
+create extension if not exists pg_cron;
+create extension if not exists dblink;
+create or replace function supaspin_delete_old_logs() returns void
+language plpgsql as $fn$
+declare
+  conn text := 'supaspin_lr_' || substr(md5(clock_timestamp()::text), 1, 8);
+  t text;
+begin
+  perform dblink_connect(conn, 'dbname=_supabase user=supabase_admin');
+  for t in
+    select tablename from dblink(conn,
+      'select tablename from pg_tables where schemaname = ''_analytics'' and tablename like ''log_events\\_%''')
+      as log_tables(tablename text)
+  loop
+    perform dblink_exec(conn,
+      format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t));
+  end loop;
+  perform dblink_disconnect(conn);
+exception when others then
+  begin perform dblink_disconnect(conn); exception when others then null; end;
+  raise;
+end $fn$;
+select supaspin_delete_old_logs();
+delete from cron.job where jobname = '{LOG_RETENTION_JOB}';
+select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select supaspin_delete_old_logs()');
+""".strip()
+
 DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
                 "admin_email": "admin@example.com", "sender_name": "Supabase"}
 
@@ -768,6 +810,21 @@ def wait_ready(n, log, timeout=1200):
     raise RuntimeError("pods did not become ready in time")
 
 
+def setup_log_retention(cfg, n, log):
+    """Give the instance a pg_cron job that deletes analytics logs older than LOG_RETENTION every
+    night. Logflare stores the logs it receives in the _supabase database and never deletes them;
+    without this job the capped db image fills up and the database dies (instance 3 on
+    2026-10-06). The job runs in the postgres database, the only one pg_cron accepts in this
+    image, and deletes in _supabase over a local dblink connection. Runs the first cleanup right
+    away, so an upgrade also frees a full disk. Runs after create, upgrade and resume;
+    scheduling under a fixed job name makes it idempotent."""
+    log(f"  log retention: deleting analytics logs older than {LOG_RETENTION}, every night")
+    kubectl("-n", namespace(n), "exec", f"{fullname(n)}-db-0", "-c", "supabase-db", "--",
+            "psql", "-U", "supabase_admin", "-d", "postgres", "-v", "ON_ERROR_STOP=1",
+            "-c", RETENTION_SQL)
+    log("  log retention is set")
+
+
 def create_instance(cfg, mail, log, number=None):
     taken = existing_numbers(cfg)
     n = number if number is not None else lowest_free_number(cfg["slots"], taken)
@@ -796,6 +853,7 @@ def create_instance(cfg, mail, log, number=None):
         finally:
             os.remove(vfile)  # holds every secret
         wait_ready(n, log)
+        setup_log_retention(cfg, n, log)
         env_path = f"{inst_dir(cfg, n)}/.env"
         fd = os.open(env_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
         with os.fdopen(fd, "w") as f:
@@ -877,7 +935,7 @@ def suspend_instance(cfg, n, log):
     apply_values(cfg, n, set_replicas(values, 0))
     wait_gone(n, log)
     log(f"  instance {n} is suspended; its disks and its .env file are untouched")
-    log(f"  start it again with: supaspin resume {n}")
+    log("  start it again in this instance's menu: \"Resume it\"")
     return True
 
 
@@ -891,6 +949,7 @@ def resume_instance(cfg, n, log):
     apply_values(cfg, n, set_replicas(values, 1))
     log("  the database answers first; the services that need it retry until it does")
     wait_ready(n, log)
+    setup_log_retention(cfg, n, log)
     log(f"  instance {n} is running again: https://{cfg['slots'].get(n, '?')}")
     return True
 
@@ -917,9 +976,10 @@ def upgrade_instance(cfg, n, log):
     if asleep:
         # A suspended instance stays suspended: the new caps are stored and take effect the moment
         # it is resumed. Waiting for its pods would wait for ever - by design it has none.
-        log(f"  instance {n} is suspended; the new caps apply when you resume it")
+        log(f"  instance {n} is suspended; the new caps and log retention apply when you resume it")
         return bool(changed)
     wait_ready(n, log)
+    setup_log_retention(cfg, n, log)
     log(f"  instance {n} is running with the current settings")
     return bool(changed)
 
@@ -1660,7 +1720,7 @@ def pick_instance(cfg, title):
     if not nums:
         print("No instances.")
         return None
-    i = menu(title, [f"{n}  {cfg['slots'].get(n, '?')}" for n in nums])
+    i = menu(title, [f"{n}  {cfg['slots'].get(n, '?')}  ({instance_state(n)})" for n in nums])
     return None if i is None else nums[i]
 
 
@@ -1719,11 +1779,19 @@ def setup_dialog(cfg, log):
     return cfg
 
 
-def show_menu(cfg, n):
+def show_menu(cfg, n, log=None):
+    """Everything you can do to one instance, including stopping and starting it. The stop/start
+    line is worded for the state the instance is in when the menu is drawn, and the state is read
+    again every time you come back to it, so the menu never offers something that makes no sense."""
+    log = log or log_line
     while True:
-        i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')}",
+        state = instance_state(n)
-                 [".env", ".env (secrets shown)", "Tunnel command", "Pods and events", "Back"])
+        asleep = state == "suspended"
-        if i is None or i == 4:
+        switch = ("Resume it (start it again with its data)" if asleep
+                  else "Suspend it (stop it, keep its data)")
+        items = [".env", ".env (secrets shown)", "Tunnel command", "Pods and events", switch, "Back"]
+        i = menu(f"Instance {n} - {cfg['slots'].get(n, '?')} - {state}", items)
+        if i is None or i == 5:
             return
         print()
         try:
@@ -1731,10 +1799,14 @@ def show_menu(cfg, n):
                 text = read_env(cfg, n)
                 print(f"{inst_dir(cfg, n)}/.env\n\n" + (text if i == 1 else mask_env(text)))
             elif i == 2:
-                print(tunnel_help(cfg, n))
+                print(tunnel_help(cfg, n) if not asleep else
-            else:
+                      f"Instance {n} is suspended: nothing is listening on its ports.\n"
+                      "Resume it from this menu and the tunnel commands work again.")
+            elif i == 3:
                 print(kubectl("get", "pods", "-n", namespace(n), "-o", "wide").stdout)
                 print(kubectl("get", "events", "-n", namespace(n), "--sort-by=.lastTimestamp").stdout[-3000:])
+            else:
+                (resume_instance if asleep else suspend_instance)(cfg, n, log)
         except Exception as e:
             print(f"{RED}{e}{OFF}")
         pause()
@@ -1750,7 +1822,7 @@ def print_list(cfg):
         print(f"{BOLD}{r['n']}{OFF}  {r['host']}  {state}  pods {r['pods']}  memory {r['memory']}")
         print(f"     disk {r['disk']}\n     env  {r['env']}")
         if state == "suspended":
-            print(f"     start it again with: supaspin resume {r['n']}\n")
+            print(f"     start it again in this instance's menu: \"Resume it\"\n")
             continue  # the tunnel is pointless while nothing listens on the ports
         for line in tunnel_guide(cfg, r["n"]):
             print(f"     {line}" if line else "")
@@ -1759,12 +1831,13 @@ def print_list(cfg):
 
 def main_menu(cfg, log):
     while True:
-        i = menu(f"supaspin {VERSION}", ["Create instance", "List instances", "Show instance",
+        i = menu(f"supaspin {VERSION}",
-                                          "Delete instance", "Copy instance values into another .env file",
+                 ["Create instance", "List instances",
-                                          "Apply current memory caps to running instances",
+                  "Show instance (and suspend or resume it)", "Delete instance",
-                                          "Suspend instance (stop it, keep its data)",
+                  "Copy instance values into another .env file",
-                                          "Resume instance", "Mail settings", "Settings", "Quit"])
+                  "Apply current memory caps to running instances",
-        if i is None or i == 10:
+                  "Mail settings", "Settings", "Quit"])
+        if i is None or i == 8:
             return
         print()
         try:
@@ -1773,9 +1846,9 @@ def main_menu(cfg, log):
             elif i == 1:
                 print_list(cfg)
             elif i == 2:
-                n = pick_instance(cfg, "Show which instance?")
+                n = pick_instance(cfg, "Which instance?")
                 if n:
-                    show_menu(cfg, n)
+                    show_menu(cfg, n, log)
                     continue
             elif i == 3:
                 n = pick_instance(cfg, "Delete which instance?")
@@ -1790,17 +1863,9 @@ def main_menu(cfg, log):
             elif i == 5:
                 upgrade_all(cfg, log)
             elif i == 6:
-                n = pick_instance(cfg, "Suspend which instance?")
-                if n:
-                    suspend_instance(cfg, n, log)
-            elif i == 7:
-                n = pick_instance(cfg, "Resume which instance?")
-                if n:
-                    resume_instance(cfg, n, log)
-            elif i == 8:
                 cfg["mail"] = mail_dialog(cfg)
                 save_config(cfg)
-            elif i == 9:
+            elif i == 7:
                 cfg = setup_dialog(cfg, log)
         except Exception as e:
             print(f"{RED}{e}{OFF}")

tests/test_supaspin.py

@@ -90,14 +90,17 @@ class Values(unittest.TestCase):
 
     def test_memory_caps_add_up(self):
         req, lim = sp.total_memory_mib()
-        # db 2048, analytics 1536, kong 768, realtime 768, vector 1024, the other seven 512 each
+        # db 2048, analytics 1536, kong 768, realtime 768, vector 512, the other seven 512 each
-        self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 1024 + 7 * 512)
+        self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 512 + 7 * 512)
-        self.assertEqual(lim / 1024, 9.5)
+        self.assertEqual(lim / 1024, 9.0)
         self.assertEqual(req, 512 + 11 * 128)  # 1.875 GiB
         # Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
         # highest use, or the kernel kills the container in a loop the way it did on instance 1.
         self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536)  # used 694 MiB, died at 768
-        self.assertGreaterEqual(sp.MEMORY["vector"][1], 1024)     # used 290 MiB, died at 512
+        self.assertGreaterEqual(sp.MEMORY["vector"][1], 512)      # used 290 MiB; the nightly log
+        #  retention keeps its delivery backlog from growing past this the way it did on
+        #  2026-10-06, when instance 3's full db image killed analytics and vector filled with
+        #  the undeliverable backlog every few minutes against 1024 MiB.
         self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768)    # used 300 MiB against 384
         self.assertGreaterEqual(sp.MEMORY["kong"][1], 768)        # used 269 MiB against 512
         self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512)        # studio used 203 against 256
@@ -128,6 +131,19 @@ class Values(unittest.TestCase):
         self.assertEqual(set(sp.VOLUMES), {"db", "storage", "functions", "snippets", "deno", "pgsodium"})
         self.assertTrue(all(p[v]["storageClassName"] == "supaspin-local-sc" for v in sp.VOLUMES))
 
+    def test_log_retention_sql(self):
+        # Logflare stores logs in _analytics.log_events_* and never deletes them; the job deletes
+        # rows older than LOG_RETENTION and is scheduled under a fixed name so re-runs overwrite.
+        # pg_cron in the supabase image only runs jobs from the `postgres` database, so the job
+        # reaches the log tables in `_supabase` over a local dblink connection (trusted there).
+        self.assertIn("schemaname = ''_analytics''", sp.RETENTION_SQL)
+        self.assertIn("''log_events\\_%''", sp.RETENTION_SQL)
+        self.assertIn(f"interval ''{sp.LOG_RETENTION}''", sp.RETENTION_SQL)
+        self.assertIn(f"'{sp.LOG_RETENTION_JOB}'", sp.RETENTION_SQL)
+        self.assertIn("cron.schedule", sp.RETENTION_SQL)
+        self.assertIn("dblink_connect", sp.RETENTION_SQL)
+        self.assertIn("user=supabase_admin", sp.RETENTION_SQL)
+
     def test_imgproxy_reads_the_storage_volume(self):
         img = self.v["persistence"]["imgproxy"]
         self.assertEqual(img["existingClaim"], "supa2-supabase-storage")
@@ -975,6 +991,7 @@ class Upgrade(unittest.TestCase):
 
         with mock.patch.object(sp, "helm", helm), \
                 mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
+                mock.patch.object(sp, "setup_log_retention", lambda cfg, n, log: None), \
                 mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
             yield
 
@@ -1147,6 +1164,8 @@ class SuspendResume(unittest.TestCase):
                                   lambda n, log, **kw: self.waited.append(("ready", n))), \
                 mock.patch.object(sp, "wait_gone",
                                   lambda n, log, **kw: self.waited.append(("gone", n))), \
+                mock.patch.object(sp, "setup_log_retention",
+                                  lambda cfg, n, log: self.waited.append(("retention", n))), \
                 mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
             yield
 
@@ -1210,7 +1229,7 @@ class SuspendResume(unittest.TestCase):
         for svc in sp.SERVICES:
             self.assertEqual(self.written["deployment"][svc]["replicaCount"], 1)
         self.assertFalse(sp.is_suspended(self.written))
-        self.assertEqual(self.waited, [("ready", 3)])
+        self.assertEqual(self.waited, [("ready", 3), ("retention", 3)])
 
     def test_resuming_a_running_instance_changes_nothing(self):
         with self.fake_helm(self.RUNNING):
@@ -1237,7 +1256,7 @@ class SuspendResume(unittest.TestCase):
     def test_upgrade_still_waits_for_a_running_instance(self):
         with self.fake_helm(self.RUNNING):
             sp.upgrade_instance(CFG, 1, self.log.append)
-        self.assertEqual(self.waited, [("ready", 1)])
+        self.assertEqual(self.waited, [("ready", 1), ("retention", 1)])
 
 
 class InstanceState(unittest.TestCase):
@@ -1273,5 +1292,79 @@ class InstanceState(unittest.TestCase):
             sp.print_list(CFG)
         text = out.getvalue()
         self.assertIn("suspended", text)
-        self.assertIn("supaspin resume 1", text)
+        self.assertIn("Resume it", text)          # the menu line, not a command to remember
         self.assertNotIn("ssh -N -L", text)
+
+
+class InstanceMenu(unittest.TestCase):
+    """Stopping and starting an instance lives inside the instance you are looking at: one line,
+    worded for the state the instance is in, so there is no second command to remember."""
+
+    def run_menu(self, state, choose):
+        """Draw the menu for an instance in `state`, pick the item whose text `choose` matches
+        (None: leave at once). Returns (the item texts, what was called)."""
+        seen, called = [], []
+
+        def menu(title, items):
+            seen.append((title, list(items)))
+            if choose is None:
+                return None
+            for i, text in enumerate(items):
+                if choose in text:
+                    return i
+            raise AssertionError(f"no item matching {choose!r} in {items}")
+
+        calls = {"suspend": lambda cfg, n, log: called.append(("suspend", n)),
+                 "resume": lambda cfg, n, log: called.append(("resume", n))}
+        out = io.StringIO()
+        with mock.patch.object(sp, "menu", menu), \
+                mock.patch.object(sp, "instance_state", lambda n: state), \
+                mock.patch.object(sp, "pause", lambda: (_ for _ in ()).throw(StopIteration)), \
+                mock.patch.object(sp, "suspend_instance", calls["suspend"]), \
+                mock.patch.object(sp, "resume_instance", calls["resume"]), \
+                mock.patch.object(sp, "read_env", lambda cfg, n: "A=b\n"), \
+                contextlib.redirect_stdout(out):
+            try:
+                sp.show_menu(CFG, 2, lambda *a, **k: None)
+            except StopIteration:
+                pass  # the menu would loop for ever; stop at the first "press Enter"
+        return seen[0][1], called, out.getvalue(), seen[0][0]
+
+    def test_a_running_instance_offers_to_suspend_it(self):
+        items, called, _, title = self.run_menu("running", None)
+        self.assertTrue(any("Suspend it" in t for t in items), items)
+        self.assertFalse(any("Resume" in t for t in items), items)
+        self.assertIn("running", title)
+
+    def test_a_suspended_instance_offers_to_resume_it(self):
+        items, called, _, title = self.run_menu("suspended", None)
+        self.assertTrue(any("Resume it" in t for t in items), items)
+        self.assertFalse(any("Suspend" in t for t in items), items)
+        self.assertIn("suspended", title)
+
+    def test_choosing_it_suspends_this_instance(self):
+        _, called, _, _ = self.run_menu("running", "Suspend it")
+        self.assertEqual(called, [("suspend", 2)])
+
+    def test_choosing_it_resumes_this_instance(self):
+        _, called, _, _ = self.run_menu("suspended", "Resume it")
+        self.assertEqual(called, [("resume", 2)])
+
+    def test_the_tunnel_of_a_sleeping_instance_says_why_it_is_pointless(self):
+        _, _, text, _ = self.run_menu("suspended", "Tunnel")
+        self.assertIn("suspended", text)
+        self.assertNotIn("ssh -N -L", text)
+
+    def test_the_picker_names_the_state_of_every_instance(self):
+        seen = []
+
+        def menu(title, items):
+            seen.append(list(items))
+            return None
+
+        with mock.patch.object(sp, "menu", menu), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2}), \
+                mock.patch.object(sp, "instance_state", lambda n: "suspended" if n == 2 else "running"):
+            sp.pick_instance(CFG, "Which instance?")
+        self.assertEqual(seen[0], ["1  supa1.example.com  (running)",
+                                   "2  supa2.example.com  (suspended)"])