feat: raise the memory ceilings and add supaspin upgrade

The two logging services were being killed by the kernel in a loop on instance 1 (supa1.didgibot.com): vector 297 times, analytics 36 times, both exit code 137 (out of memory). The old ceilings sat right on top of what the services normally use, so any backlog crossed them. Measured across the three running instances on 2026-09-22: - analytics 612 / 694 / 662 MiB against a 768 MiB ceiling -> 1536 MiB - vector 287 / 290 MiB against 512 MiB -> 1024 MiB - realtime 269 / 287 / 300 MiB against 384 MiB -> 768 MiB - kong 260 / 269 / 265 MiB against 512 MiB -> 768 MiB - studio 203, storage 187, rest 162 MiB against 256 MiB -> 512 MiB - db 163 / 175 / 273 MiB against 2048 MiB -> unchanged Limits only. The requests stay at 128 MiB (512 for db): the server is one machine shared with the production workloads and already has 80% of its memory reserved, so raising the requests would leave production nothing to schedule into. Raising a limit reserves nothing - it only moves the point at which the kernel kills. Per-instance totals go from 5.875 GiB to 9.5 GiB of limit; reserved memory is unchanged at 1.875 GiB. New numbers only reach instances created afterwards, so this also adds `sudo supaspin upgrade [N]`: it reads back the values Helm stored at install time, replaces the memory block in them and applies the chart again. Pods are replaced service by service; databases, uploads, certificates and .env files are untouched. Without a number it does every instance and reports which ones failed. Also available from the menu. Fixes a rollout that could never finish, found by the first upgrade: Studio binds port 3001 (3002, 3003) on the server itself so it can be reached over an SSH tunnel, and one port can be held by one pod at a time. The default rollout starts the replacement pod before stopping the old one, so the replacement never got the port and stayed Pending for ever. The Studio deployment now stops the old pod first. Postgres was never affected: a StatefulSet already replaces its pod that way. The values file that carries the instance's secrets and the post-renderer arguments are now built by shared helpers used by both create and upgrade. Tests: 113 pass (12 new). Version 1.1.2 -> 1.2.0 in VERSION, the VERSION constant and README. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 21:50:14 +03:00
4 changed files with 351 additions and 49 deletions
@@ -1,6 +1,6 @@
 # supaspin

-Version 1.1.2
+Version 1.2.0

 Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
 single-node [microk8s](https://microk8s.io) server, from a terminal menu.
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
 | Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
 | Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
 | Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
-| Memory | limits add up to **5.875 GiB** per instance, requests to 1.875 GiB (table below) |
+| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
 | Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
 | Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
 | Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
@@ -93,6 +93,7 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
 | Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
 | Delete instance | asks you to type the number; removes everything (see below) |
 | Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
+| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
 | Mail settings | view and change the saved default mail settings |
 | Settings | change any first-run setting |

@@ -106,9 +107,10 @@ sudo supaspin env N                   # the full .env (contains secrets)
 sudo supaspin tunnel N                # the tunnel commands, step by step
 sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]   # see "Copying an instance's values"
 sudo supaspin delete N [--yes]
+sudo supaspin upgrade [N]             # apply this version's memory caps to running instances
 sudo supaspin mail                    # change the saved default mail settings
 sudo supaspin setup                   # change settings
-supaspin --version                    # prints "supaspin 1.1.2"
+supaspin --version                    # prints "supaspin 1.2.0"
 ```

 ### Connecting your app
@@ -209,22 +211,38 @@ run `supaspin delete N` to clean up.

 ## Memory per instance

-| Service | Request | Limit |
-|---|---|---|
-| db | 512 MiB | 2048 MiB |
-| analytics | 128 MiB | 768 MiB |
-| kong | 128 MiB | 512 MiB |
-| realtime | 128 MiB | 384 MiB |
-| vector | 128 MiB | 512 MiB |
-| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 256 MiB each |
-| **Total** | **1.875 GiB** | **5.875 GiB** |
+| Service | Request | Limit | Highest use measured |
+|---|---|---|---|
+| db | 512 MiB | 2048 MiB | 273 MiB |
+| analytics | 128 MiB | 1536 MiB | 694 MiB |
+| vector | 128 MiB | 1024 MiB | 290 MiB |
+| kong | 128 MiB | 768 MiB | 269 MiB |
+| realtime | 128 MiB | 768 MiB | 300 MiB |
+| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
+| **Total** | **1.875 GiB** | **9.5 GiB** | |

-Most limits are half of what a production setup would use. Analytics, realtime and vector are higher:
-at half of production they were killed for running out of memory in a loop (measured on a fresh
-instance: analytics peaks at 624 MiB, realtime at 280 MiB). Vector is the log collector: it reads the
-log files of every pod on the server, not only its own instance's, so its memory need follows the
-server's total log volume; at 256 MiB it was killed in a loop on one instance and used 89% on another. Image resizing (imgproxy) reads the
-uploads from the storage volume, so it has no image of its own.
+The request is what the scheduler sets aside for a service and can never be given to anything else.
+The limit is the ceiling the kernel kills the container at. They do different jobs here.
+
+The requests are deliberately small: the server is one machine shared with the production
+workloads, and reserving what three instances really use would leave production nothing to start
+in. The limits are roughly twice the highest use measured across three running instances on
+2026-09-22 (right-hand column above).
+
+The earlier limits sat right on top of normal use and the kernel kept killing the two logging
+services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
+768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
+log files of every pod on the server and not only its own instance's, was killed 297 times on that
+same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
+other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
+the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
+in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
+its own.
+
+Changing the numbers in this table only affects instances created afterwards. To put them on
+instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
+the instance was installed, replaces the memory block in them and applies the chart again, so the
+pods are replaced service by service and nothing else about the instance changes.

 ## Troubleshooting

@@ -1 +1 @@
-1.1.2
+1.2.0
@@ -5,6 +5,9 @@ Each numbered instance gets its own namespace, capped disk images, capped memory
 an HTTPS hostname and fixed Postgres/Studio ports. Standard library plus python3-yaml.
 Run as root: supaspin            (menu)
             supaspin create|list|show N|env N|tunnel N|delete N|setup|mail
+             supaspin upgrade [N]
+               Puts this version's memory caps on instances that already run, one service
+               at a time. Without N every instance. Data and .env files are not touched.
             supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]
               TARGET = a .env file on this server (/path/.env) or on another computer
               (user@host:/path/.env, Windows: user@host:C:/path/.env). Puts the instance's
@@ -32,7 +35,7 @@ import tty

 import yaml

-VERSION = "1.1.2"
+VERSION = "1.2.0"
 CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
 STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
 CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
@@ -53,15 +56,25 @@ VOLUMES = {"db": 2048, "storage": 1024, "functions": 256,
 DB_OWNED = ("db", "pgsodium")
 DB_UID, DB_GID = 100, 101

-# Memory per service: (request MiB, limit MiB). Half of maralago.ru production, except analytics,
-# realtime and vector: at half of production they were killed for running out of memory in a loop
-# (measured on a fresh instance: analytics peaks at 624 MiB, realtime at 280 MiB; vector reads the log
-# files of every pod on the node, not only its own instance's, and was killed at 256 MiB on 2026-09-19).
+# Memory per service: (request MiB, limit MiB). The limit is the ceiling the kernel kills the
+# container at; the request is what the scheduler sets aside for it. The requests stay small on
+# purpose: this is one machine shared with production, and reserving what an instance really uses
+# three times over would leave production nothing to start in. The limits are about twice the
+# highest use measured across three running instances on 2026-09-22, because the two logging
+# services grow with their backlog and the old ceilings sat right on top of normal use:
+#   analytics 612 / 694 / 662 MiB against a 768 MiB ceiling - killed 36 times on instance 1
+#   vector    287 / 290 MiB against 512 - killed 297 times on instance 1, where it had a backlog
+#   realtime  269 / 287 / 300 MiB against 384
+#   kong      260 / 269 / 265 MiB against 512
+#   studio 203, storage 187, rest 162 MiB against 256
+#   db        163 / 175 / 273 MiB against 2048 - left alone, it has room
+# Raising a limit costs nothing until a container actually asks for the memory; raising a request
+# would take the memory away from everything else on the node straight away.
 SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
            "realtime", "rest", "storage", "studio", "vector"]
-MEMORY = {"db": (512, 2048), "analytics": (128, 768), "kong": (128, 512), "realtime": (128, 384),
-          "vector": (128, 512)}
-DEFAULT_MEMORY = (128, 256)
+MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
+          "vector": (128, 1024)}
+DEFAULT_MEMORY = (128, 512)

 DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
                "admin_email": "admin@example.com", "sender_name": "Supabase"}
@@ -171,12 +184,7 @@ def instance_values(cfg, n, sec, mail, template_text):
    quoted = {k: json.dumps(str(v)) for k, v in subs.items()}
    text = re.sub(r"\$\{(\w+)\}", lambda m: quoted[m.group(1)], template_text)
    values = yaml.safe_load(text)
-    dep = values.setdefault("deployment", {})
-    for svc in SERVICES:
-        req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
-        d = dep.setdefault(svc, {})
-        d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
-        d["priorityClassName"] = PRIORITY_CLASS
+    apply_memory(values)
    persistence = values.setdefault("persistence", {})
    for vol, mib in VOLUMES.items():
        persistence[vol] = {"enabled": True, "storageClassName": STORAGE_CLASS,
@@ -187,6 +195,29 @@ def instance_values(cfg, n, sec, mail, template_text):
    return values


+def apply_memory(values):
+    """Write the memory caps and the priority class of every service into a values dictionary,
+    replacing whatever was there. Used both when an instance is created and when a running one is
+    brought up to the current caps."""
+    dep = values.setdefault("deployment", {})
+    for svc in SERVICES:
+        req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
+        d = dep.setdefault(svc, {})
+        d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
+        d["priorityClassName"] = PRIORITY_CLASS
+    return values
+
+
+def memory_limits(values):
+    """{service: limit} as the values dictionary has it, for telling before from after."""
+    dep = values.get("deployment") or {}
+    out = {}
+    for svc in SERVICES:
+        lim = ((dep.get(svc) or {}).get("resources") or {}).get("limits") or {}
+        out[svc] = lim.get("memory")
+    return out
+
+
 def total_memory_mib():
    """(requests, limits) in MiB, summed over all services."""
    req = sum(MEMORY.get(s, DEFAULT_MEMORY)[0] for s in SERVICES)
@@ -406,6 +437,12 @@ def _host_port(doc, port, ip):
    p = c["ports"][0]
    p["hostPort"] = port
    p["hostIP"] = ip
+    if doc.get("kind") == "Deployment":
+        # A port on the server itself can be held by one pod at a time. The default rollout starts
+        # the replacement pod before stopping the old one, so the replacement never gets the port
+        # and waits for ever ("didn't have free ports for the requested pod ports"). Recreate stops
+        # the old pod first. A StatefulSet already replaces its pod that way.
+        doc["spec"]["strategy"] = {"type": "Recreate"}


 def pv_manifest(cfg, n, vol):
@@ -729,16 +766,11 @@ def create_instance(cfg, mail, log, number=None):
        sec = generate_secrets()
        with open(find_template()) as f:
            values = instance_values(cfg, n, sec, mail, f.read())
-        vfile = f"{STATE_DIR}/values-{n}.yaml"
-        os.makedirs(STATE_DIR, exist_ok=True)
-        fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
-        with os.fdopen(fd, "w") as f:
-            yaml.safe_dump(values, f, sort_keys=False)
+        vfile = write_values_file(n, values)
        log("  installing chart")
        try:
            helm("install", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
-                 "--post-renderer", os.path.realpath(sys.argv[0]),
-                 "--post-renderer-args", "post-render", "--post-renderer-args", str(n))
+                 *post_render_args(n))
        finally:
            os.remove(vfile)  # holds every secret
        wait_ready(n, log)
@@ -757,6 +789,77 @@ def create_instance(cfg, mail, log, number=None):
        raise


+def write_values_file(n, values):
+    """The values Helm is given. It carries every secret of the instance, so it is written
+    readable by root only and the caller deletes it as soon as Helm has read it."""
+    vfile = f"{STATE_DIR}/values-{n}.yaml"
+    os.makedirs(STATE_DIR, exist_ok=True)
+    fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    with os.fdopen(fd, "w") as f:
+        yaml.safe_dump(values, f, sort_keys=False)
+    return vfile
+
+
+def post_render_args(n):
+    """Helm hands the finished manifests to supaspin itself, which fixes the parts the chart
+    cannot express (the Postgres and Studio host ports, the gateway routes that are not used)."""
+    return ["--post-renderer", os.path.realpath(sys.argv[0]),
+            "--post-renderer-args", "post-render", "--post-renderer-args", str(n)]
+
+
+def upgrade_instance(cfg, n, log):
+    """Bring a running instance up to the memory caps this version of supaspin carries, without
+    deleting anything. The values Helm kept from the install are read back, the memory block in
+    them is replaced, and the chart is applied again: the pods are replaced one service at a time
+    and the database, the uploads and the .env file are untouched."""
+    ns, rel = namespace(n), release(n)
+    host = cfg["slots"].get(n, "?")
+    log(f"Instance {n} ({host})")
+    r = helm("get", "values", rel, "-n", ns, "-o", "yaml", check=False)
+    if r.returncode != 0:
+        raise RuntimeError(f"instance {n}: no Helm release {rel} in namespace {ns}\n{r.stderr.strip()}")
+    values = yaml.safe_load(r.stdout) or {}
+    if not values.get("deployment"):
+        raise RuntimeError(f"instance {n}: the stored values of {rel} look empty; not touching it")
+    before = memory_limits(values)
+    after = memory_limits(apply_memory(values))
+    changed = [svc for svc in SERVICES if before[svc] != after[svc]]
+    for svc in changed:
+        log(f"  {svc}: {before[svc] or 'unset'} -> {after[svc]}")
+    if not changed:
+        # The chart is applied even so: the caps are not the only thing this version may produce
+        # differently (the Studio rollout, the host ports, the gateway routes are all decided
+        # while the manifests are rendered). Helm changes nothing if nothing differs.
+        log("  caps already current; re-applying the chart in case anything else differs")
+    vfile = write_values_file(n, values)
+    try:
+        helm("upgrade", rel, chart_path(cfg), "-n", ns, "-f", vfile, *post_render_args(n))
+    finally:
+        os.remove(vfile)  # holds every secret
+    wait_ready(n, log)
+    log(f"  instance {n} is running with the current settings")
+    return bool(changed)
+
+
+def upgrade_all(cfg, log, numbers=None):
+    """Upgrade one instance, or every instance that exists."""
+    nums = [numbers] if isinstance(numbers, int) else sorted(numbers or existing_numbers(cfg))
+    if not nums:
+        log("No instances.")
+        return
+    req, lim = total_memory_mib()
+    log(f"Caps in this version: {req} MiB reserved and up to {lim} MiB per instance.")
+    failed = []
+    for n in nums:
+        try:
+            upgrade_instance(cfg, n, log)
+        except RuntimeError as e:
+            failed.append(n)
+            log(f"{RED}  instance {n} failed: {e}{OFF}")
+    if failed:
+        raise RuntimeError("these instances were not upgraded: " + ", ".join(str(n) for n in failed))
+
+
 def delete_instance(cfg, n, log):
    log(f"Deleting instance {n}")
    save_cert(n, log)
@@ -1554,8 +1657,9 @@ def main_menu(cfg, log):
    while True:
        i = menu(f"supaspin {VERSION}", ["Create instance", "List instances", "Show instance",
                                          "Delete instance", "Copy instance values into another .env file",
+                                          "Apply current memory caps to running instances",
                                          "Mail settings", "Settings", "Quit"])
-        if i is None or i == 7:
+        if i is None or i == 8:
            return
        print()
        try:
@@ -1579,9 +1683,11 @@ def main_menu(cfg, log):
                    print()
                    push_env(cfg, n, spec, log)
            elif i == 5:
+                upgrade_all(cfg, log)
+            elif i == 6:
                cfg["mail"] = mail_dialog(cfg)
                save_config(cfg)
-            elif i == 6:
+            elif i == 7:
                cfg = setup_dialog(cfg, log)
        except Exception as e:
            print(f"{RED}{e}{OFF}")
@@ -1630,6 +1736,8 @@ def main(argv):
        elif cmd == "delete" and n:
            if "--yes" in argv or confirm_number(n, "deleting it"):
                delete_instance(cfg, n, log_line)
+        elif cmd == "upgrade":
+            upgrade_all(cfg, log_line, n)
        elif cmd == "push-env":
            return push_env_command(cfg, argv[2:])
        elif cmd == "mail":
@@ -90,14 +90,20 @@ class Values(unittest.TestCase):

    def test_memory_caps_add_up(self):
        req, lim = sp.total_memory_mib()
-        # db 2048, analytics 768, kong 512, realtime 384, vector 512, the other seven 256 each
-        self.assertEqual(lim, 2048 + 768 + 512 + 384 + 512 + 7 * 256)
-        self.assertEqual(lim / 1024, 5.875)
+        # db 2048, analytics 1536, kong 768, realtime 768, vector 1024, the other seven 512 each
+        self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 1024 + 7 * 512)
+        self.assertEqual(lim / 1024, 9.5)
        self.assertEqual(req, 512 + 11 * 128)  # 1.875 GiB
-        # analytics, realtime and vector were killed for running out of memory at half of production
-        self.assertGreaterEqual(sp.MEMORY["analytics"][1], 768)
-        self.assertGreaterEqual(sp.MEMORY["realtime"][1], 384)
-        self.assertGreaterEqual(sp.MEMORY["vector"][1], 512)
+        # Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
+        # highest use, or the kernel kills the container in a loop the way it did on instance 1.
+        self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536)  # used 694 MiB, died at 768
+        self.assertGreaterEqual(sp.MEMORY["vector"][1], 1024)     # used 290 MiB, died at 512
+        self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768)    # used 300 MiB against 384
+        self.assertGreaterEqual(sp.MEMORY["kong"][1], 768)        # used 269 MiB against 512
+        self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512)        # studio used 203 against 256
+        # The requests stay where they were: the node is shared with production and reserving
+        # what the instances really use would leave production nothing to schedule into.
+        self.assertTrue(all(m[0] == 128 for s, m in sp.MEMORY.items() if s != "db"))
        self.assertEqual(req / 1024, 1.875)
        for svc in sp.SERVICES:
            self.assertEqual(self.v["deployment"][svc]["priorityClassName"], "supaspin-low")
@@ -929,3 +935,173 @@ class PushEnvCommand(unittest.TestCase):

 if __name__ == "__main__":
    unittest.main()
+
+
+class Upgrade(unittest.TestCase):
+    """`supaspin upgrade` puts the current memory caps on an instance that is already running.
+    It must change the caps and nothing else: no volume is resized, no secret is regenerated."""
+
+    STORED = {
+        "deployment": {
+            "analytics": {"enabled": True, "priorityClassName": "supaspin-low",
+                          "resources": {"limits": {"memory": "768Mi"},
+                                        "requests": {"memory": "128Mi"}}},
+            "vector": {"enabled": True, "priorityClassName": "supaspin-low",
+                       "resources": {"limits": {"memory": "512Mi"},
+                                     "requests": {"memory": "128Mi"}}},
+        },
+        "secret": {"jwt": {"secret": "keep-me", "anonKey": "anon"}},
+        "persistence": {"db": {"enabled": True, "size": "2048Mi"}},
+    }
+
+    def setUp(self):
+        self.calls = []
+        self.written = {}
+        self.log = []
+
+    @contextlib.contextmanager
+    def fake_helm(self, stored=None, rc=0):
+        stored = self.STORED if stored is None else stored
+
+        def helm(*args, check=True):
+            self.calls.append(list(args))
+            if args[0] == "get":
+                out = yaml.safe_dump(stored) if stored is not None else ""
+                return collections.namedtuple("P", "returncode stdout stderr")(rc, out, "boom")
+            if args[0] == "upgrade":
+                with open(args[args.index("-f") + 1]) as f:
+                    self.written = yaml.safe_load(f)
+            return collections.namedtuple("P", "returncode stdout stderr")(0, "", "")
+
+        with mock.patch.object(sp, "helm", helm), \
+                mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
+                mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
+            yield
+
+    def test_raises_the_caps_and_keeps_everything_else(self):
+        with self.fake_helm():
+            changed = sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertTrue(changed)
+        dep = self.written["deployment"]
+        self.assertEqual(dep["analytics"]["resources"]["limits"]["memory"],
+                         f"{sp.MEMORY['analytics'][1]}Mi")
+        self.assertEqual(dep["vector"]["resources"]["limits"]["memory"],
+                         f"{sp.MEMORY['vector'][1]}Mi")
+        self.assertTrue(dep["analytics"]["enabled"])              # untouched
+        self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
+        self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
+
+    def test_every_service_gets_a_cap_even_if_the_stored_values_had_none(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        for svc in sp.SERVICES:
+            req, lim = sp.MEMORY.get(svc, sp.DEFAULT_MEMORY)
+            got = self.written["deployment"][svc]["resources"]
+            self.assertEqual(got["limits"]["memory"], f"{lim}Mi")
+            self.assertEqual(got["requests"]["memory"], f"{req}Mi")
+
+    def test_the_values_file_with_the_secrets_is_deleted_again(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 1, self.log.append)
+        path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
+        self.assertFalse(os.path.exists(path))
+
+    def test_helm_gets_the_post_renderer(self):
+        with self.fake_helm():
+            sp.upgrade_instance(CFG, 2, self.log.append)
+        up = [c for c in self.calls if c[0] == "upgrade"][0]
+        self.assertIn("--post-renderer", up)
+        self.assertEqual(up[up.index("--post-renderer-args") + 1], "post-render")
+        self.assertIn("2", up)
+
+    def test_an_instance_already_at_the_caps_is_still_re_applied(self):
+        # The caps are not the only thing a new version can change: the Studio rollout, the host
+        # ports and the gateway routes are decided while the manifests are rendered, and none of
+        # them shows up in the stored values.
+        stored = {"deployment": sp.apply_memory({})["deployment"]}
+        with self.fake_helm(stored):
+            changed = sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertFalse(changed)                                   # no cap moved
+        self.assertEqual([c[0] for c in self.calls], ["get", "upgrade"])  # applied anyway
+
+    def test_a_missing_release_is_refused_before_anything_is_written(self):
+        with self.fake_helm(rc=1):
+            with self.assertRaises(RuntimeError):
+                sp.upgrade_instance(CFG, 3, self.log.append)
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+
+    def test_empty_stored_values_are_refused(self):
+        with self.fake_helm({}):
+            with self.assertRaises(RuntimeError):
+                sp.upgrade_instance(CFG, 1, self.log.append)
+        self.assertEqual([c[0] for c in self.calls], ["get"])
+
+    def test_upgrade_all_does_every_instance_and_reports_the_failures(self):
+        done = []
+
+        def one(cfg, n, log):
+            if n == 2:
+                raise RuntimeError("no release")
+            done.append(n)
+            return True
+
+        with mock.patch.object(sp, "upgrade_instance", one), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
+            with self.assertRaises(RuntimeError) as e:
+                sp.upgrade_all(CFG, self.log.append)
+        self.assertEqual(done, [1, 3])
+        self.assertIn("2", str(e.exception))
+
+    def test_upgrade_all_with_a_number_does_only_that_instance(self):
+        done = []
+        with mock.patch.object(sp, "upgrade_instance", lambda cfg, n, log: done.append(n)), \
+                mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
+            sp.upgrade_all(CFG, self.log.append, 2)
+        self.assertEqual(done, [2])
+
+
+class HostPortRollout(unittest.TestCase):
+    """Studio and Postgres bind a port on the server itself, and only one pod can hold a port.
+    The Studio deployment must therefore stop its old pod before starting the new one, or an
+    upgrade leaves the replacement pod Pending for ever."""
+
+    def rendered(self, n=1):
+        sec = sp.generate_secrets()
+        values = sp.instance_values(CFG, n, sec, MAIL, template())
+        with mock.patch.object(sp, "helm"), mock.patch.object(sp, "kubectl"):
+            text = yaml.safe_dump_all([
+                {"apiVersion": "apps/v1", "kind": "Deployment",
+                 "metadata": {"name": f"{sp.fullname(n)}-studio"},
+                 "spec": {"strategy": {"type": "RollingUpdate",
+                                       "rollingUpdate": {"maxSurge": "25%"}},
+                          "template": {"spec": {"containers": [
+                              {"name": "studio", "ports": [{"containerPort": 3000}]}]}}}},
+                {"apiVersion": "apps/v1", "kind": "StatefulSet",
+                 "metadata": {"name": f"{sp.fullname(n)}-db"},
+                 "spec": {"template": {"spec": {"containers": [
+                     {"name": "db", "ports": [{"containerPort": 5432}]}]}}}},
+                {"apiVersion": "apps/v1", "kind": "Deployment",
+                 "metadata": {"name": f"{sp.fullname(n)}-rest"},
+                 "spec": {"strategy": {"type": "RollingUpdate"},
+                          "template": {"spec": {"containers": [
+                              {"name": "rest", "ports": [{"containerPort": 3000}]}]}}}},
+            ])
+        out = list(yaml.safe_load_all(sp.post_render(text, n, CFG["lan_ip"])))
+        return {d["metadata"]["name"]: d for d in out}
+
+    def test_studio_stops_the_old_pod_before_starting_the_new_one(self):
+        d = self.rendered()[f"{sp.fullname(1)}-studio"]
+        self.assertEqual(d["spec"]["strategy"], {"type": "Recreate"})
+        port = d["spec"]["template"]["spec"]["containers"][0]["ports"][0]
+        self.assertEqual(port["hostPort"], sp.studio_port(1))
+        self.assertEqual(port["hostIP"], CFG["lan_ip"])
+
+    def test_a_deployment_without_a_host_port_keeps_the_normal_rollout(self):
+        d = self.rendered()[f"{sp.fullname(1)}-rest"]
+        self.assertEqual(d["spec"]["strategy"]["type"], "RollingUpdate")
+
+    def test_the_database_keeps_its_host_port_and_gains_no_strategy(self):
+        d = self.rendered()[f"{sp.fullname(1)}-db"]
+        self.assertNotIn("strategy", d["spec"])
+        self.assertEqual(d["spec"]["template"]["spec"]["containers"][0]["ports"][0]["hostPort"],
+                         sp.pg_port(1))
@@ -1 +1 @@
 .1.2
 .2.0