Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 IgorandClaude Opus 5
|
2e4e46c878 | ||
|
IgorandClaude Opus 5
|
99fdf7a0ab | ||
|
IgorandClaude Opus 5
|
1d49234400 |
@@ -21,8 +21,14 @@ The number is `MAJOR.MINOR.PATCH` (semantic versioning: `1.1.1` is major 1, mino
|
||||
4. **Tag every release.** After the pull request is merged, create the git tag `vX.Y.Z` (for example `v1.1.1`) on
|
||||
the commit on `main` that contains that number in `VERSION`. The tag name must equal `VERSION` with a `v` in
|
||||
front. A version without a tag is not released. Never move or delete a tag that was already published; if a
|
||||
release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/igor/supaspin/tags` with
|
||||
release is wrong, ship a new PATCH instead. With the Gitea API: `POST /api/v1/repos/public/supaspin/tags` with
|
||||
`tag_name`, `target` (the commit) and `message`.
|
||||
|
||||
The owner in an API address must be `public`. The repository used to sit under the user `igor` and Gitea still
|
||||
forwards the old address for anything a browser or git does, but it answers an API call to
|
||||
`/api/v1/repos/igor/supaspin/...` with a redirect that curl does not follow for a POST: the command returns an
|
||||
empty body and no error, which looks exactly like a rejected token. Opening or merging a pull request needs the
|
||||
same `public` owner.
|
||||
5. **Check the result.** Reinstall from the merged `main` (`sudo bash install.sh`), then run `supaspin --version` and
|
||||
compare it with `VERSION` and the tag.
|
||||
6. If you find a version in `VERSION` that has no tag, or a tag that does not match `VERSION`, tell Igor before doing
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# supaspin
|
||||
|
||||
Version 1.1.2
|
||||
Version 1.3.0
|
||||
|
||||
Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
|
||||
single-node [microk8s](https://microk8s.io) server, from a terminal menu.
|
||||
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
|
||||
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
|
||||
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
|
||||
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
|
||||
| Memory | limits add up to **5.875 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
|
||||
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
|
||||
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
|
||||
@@ -40,7 +40,7 @@ of instances at once. Numbers are reused: create picks the lowest free one.
|
||||
## Install
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
```
|
||||
|
||||
**If your Gitea requires sign-in even for public repositories** (the download then fails with
|
||||
@@ -48,13 +48,13 @@ curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | s
|
||||
`repository: read`):
|
||||
|
||||
```bash
|
||||
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
|
||||
curl -fsSL -H "Authorization: token $TOKEN" https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo SUPASPIN_TOKEN=$TOKEN bash
|
||||
```
|
||||
|
||||
or clone the repository and install from the checkout:
|
||||
|
||||
```bash
|
||||
git clone ssh://git@git.didgibot.com:2226/igor/supaspin.git && sudo bash supaspin/install.sh
|
||||
git clone ssh://git@git.didgibot.com:2226/public/supaspin.git && sudo bash supaspin/install.sh
|
||||
```
|
||||
|
||||
Run it again to upgrade. It installs `/usr/local/bin/supaspin`, its values template and the
|
||||
@@ -93,6 +93,9 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
|
||||
| Show instance | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events |
|
||||
| Delete instance | asks you to type the number; removes everything (see below) |
|
||||
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
|
||||
| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
|
||||
| Suspend instance | stops every pod of one instance and keeps everything else: its memory, processor time, pod slots and two server ports go back to the server, its disks and `.env` file stay |
|
||||
| Resume instance | starts a suspended instance again, with the data it had |
|
||||
| Mail settings | view and change the saved default mail settings |
|
||||
| Settings | change any first-run setting |
|
||||
|
||||
@@ -106,9 +109,12 @@ sudo supaspin env N # the full .env (contains secrets)
|
||||
sudo supaspin tunnel N # the tunnel commands, step by step
|
||||
sudo supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run] # see "Copying an instance's values"
|
||||
sudo supaspin delete N [--yes]
|
||||
sudo supaspin upgrade [N] # apply this version's memory caps to running instances
|
||||
sudo supaspin suspend N # stop an instance, keep its data (see "Suspending an instance")
|
||||
sudo supaspin resume N # start it again
|
||||
sudo supaspin mail # change the saved default mail settings
|
||||
sudo supaspin setup # change settings
|
||||
supaspin --version # prints "supaspin 1.1.2"
|
||||
supaspin --version # prints "supaspin 1.3.0"
|
||||
```
|
||||
|
||||
### Connecting your app
|
||||
@@ -207,24 +213,73 @@ Creating that instance again restores the saved certificate, so repeated create/
|
||||
not hit Let's Encrypt's limit of 5 certificates per hostname per week. If a create fails midway,
|
||||
run `supaspin delete N` to clean up.
|
||||
|
||||
## Suspending an instance
|
||||
|
||||
An instance you are not using right now does not have to be deleted. `sudo supaspin suspend N`
|
||||
stops all twelve of its pods and leaves everything else alone; `sudo supaspin resume N` starts
|
||||
them again with the data they had.
|
||||
|
||||
```bash
|
||||
sudo supaspin suspend 2 # about half a minute
|
||||
sudo supaspin resume 2 # a minute or two: the database answers first, the rest wait for it
|
||||
```
|
||||
|
||||
What the server gets back while an instance sleeps:
|
||||
|
||||
- the memory it had reserved (1.875 GiB) and everything it was actually using (1.5 to 2.5 GiB),
|
||||
- its twelve pod slots, out of the 110 a node allows,
|
||||
- its two ports on the server: Postgres on 544N and Studio on 300N are free for something else.
|
||||
|
||||
What is kept, untouched:
|
||||
|
||||
- the database and the uploaded files - the disk images stay mounted and keep their contents,
|
||||
- the `.env` file with its passwords and keys, so your app needs no change when the instance is back,
|
||||
- the HTTPS certificate, which goes on renewing itself while the instance sleeps,
|
||||
- the hostname and every Kubernetes object. Until you resume it, the hostname answers "service
|
||||
unavailable" instead of not answering at all.
|
||||
|
||||
Disk space is **not** given back: that is the data you are keeping. `sudo supaspin list` shows the
|
||||
state of every instance, and a suspended one is listed as `suspended` with no tunnel commands,
|
||||
because nothing is listening on its ports.
|
||||
|
||||
Suspending is written into the instance's stored settings, not done by hand to the running pods,
|
||||
so it survives anything that re-applies the chart. `sudo supaspin upgrade` on a suspended instance
|
||||
stores the new memory limits and leaves it asleep; they take effect when you resume it.
|
||||
|
||||
## Memory per instance
|
||||
|
||||
| Service | Request | Limit |
|
||||
|---|---|---|
|
||||
| db | 512 MiB | 2048 MiB |
|
||||
| analytics | 128 MiB | 768 MiB |
|
||||
| kong | 128 MiB | 512 MiB |
|
||||
| realtime | 128 MiB | 384 MiB |
|
||||
| vector | 128 MiB | 512 MiB |
|
||||
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 256 MiB each |
|
||||
| **Total** | **1.875 GiB** | **5.875 GiB** |
|
||||
| Service | Request | Limit | Highest use measured |
|
||||
|---|---|---|---|
|
||||
| db | 512 MiB | 2048 MiB | 273 MiB |
|
||||
| analytics | 128 MiB | 1536 MiB | 694 MiB |
|
||||
| vector | 128 MiB | 1024 MiB | 290 MiB |
|
||||
| kong | 128 MiB | 768 MiB | 269 MiB |
|
||||
| realtime | 128 MiB | 768 MiB | 300 MiB |
|
||||
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
|
||||
| **Total** | **1.875 GiB** | **9.5 GiB** | |
|
||||
|
||||
Most limits are half of what a production setup would use. Analytics, realtime and vector are higher:
|
||||
at half of production they were killed for running out of memory in a loop (measured on a fresh
|
||||
instance: analytics peaks at 624 MiB, realtime at 280 MiB). Vector is the log collector: it reads the
|
||||
log files of every pod on the server, not only its own instance's, so its memory need follows the
|
||||
server's total log volume; at 256 MiB it was killed in a loop on one instance and used 89% on another. Image resizing (imgproxy) reads the
|
||||
uploads from the storage volume, so it has no image of its own.
|
||||
The request is what the scheduler sets aside for a service and can never be given to anything else.
|
||||
The limit is the ceiling the kernel kills the container at. They do different jobs here.
|
||||
|
||||
The requests are deliberately small: the server is one machine shared with the production
|
||||
workloads, and reserving what three instances really use would leave production nothing to start
|
||||
in. The limits are roughly twice the highest use measured across three running instances on
|
||||
2026-09-22 (right-hand column above).
|
||||
|
||||
The earlier limits sat right on top of normal use and the kernel kept killing the two logging
|
||||
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
|
||||
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
|
||||
log files of every pod on the server and not only its own instance's, was killed 297 times on that
|
||||
same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
|
||||
other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
|
||||
the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
|
||||
in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
|
||||
its own.
|
||||
|
||||
Changing the numbers in this table only affects instances created afterwards. To put them on
|
||||
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
|
||||
the instance was installed, replaces the memory block in them and applies the chart again, so the
|
||||
pods are replaced service by service and nothing else about the instance changes.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# One-line install (run on the Kubernetes server):
|
||||
# curl -fsSL https://git.didgibot.com/igor/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
# curl -fsSL https://git.didgibot.com/public/supaspin/raw/branch/main/install.sh | sudo bash
|
||||
# Installs /usr/local/bin/supaspin and its Helm values template. Safe to run again to upgrade.
|
||||
#
|
||||
# If your Gitea requires sign-in even for public repositories, give a read-only token:
|
||||
@@ -12,7 +12,7 @@
|
||||
# SUPASPIN_TOKEN=<token> Gitea token (read access to the repository) for the download
|
||||
set -euo pipefail
|
||||
|
||||
REPO="https://git.didgibot.com/igor/supaspin"
|
||||
REPO="https://git.didgibot.com/public/supaspin"
|
||||
REF="${SUPASPIN_REF:-main}"
|
||||
|
||||
[ "$(id -u)" -eq 0 ] || { echo "Run as root: curl -fsSL $REPO/raw/branch/main/install.sh | sudo bash" >&2; exit 1; }
|
||||
|
||||
@@ -5,6 +5,13 @@ Each numbered instance gets its own namespace, capped disk images, capped memory
|
||||
an HTTPS hostname and fixed Postgres/Studio ports. Standard library plus python3-yaml.
|
||||
Run as root: supaspin (menu)
|
||||
supaspin create|list|show N|env N|tunnel N|delete N|setup|mail
|
||||
supaspin upgrade [N]
|
||||
Puts this version's memory caps on instances that already run, one service
|
||||
at a time. Without N every instance. Data and .env files are not touched.
|
||||
supaspin suspend N | resume N
|
||||
Suspend stops every pod of one instance and gives back its memory, its pod
|
||||
slots and its two server ports. The disks, the passwords in .env, the
|
||||
certificate and the hostname stay. Resume starts it again with its data.
|
||||
supaspin push-env N TARGET [--port P] [--key FILE] [--dry-run]
|
||||
TARGET = a .env file on this server (/path/.env) or on another computer
|
||||
(user@host:/path/.env, Windows: user@host:C:/path/.env). Puts the instance's
|
||||
@@ -32,7 +39,7 @@ import tty
|
||||
|
||||
import yaml
|
||||
|
||||
VERSION = "1.1.2"
|
||||
VERSION = "1.3.0"
|
||||
CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
|
||||
STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
|
||||
CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
|
||||
@@ -53,15 +60,25 @@ VOLUMES = {"db": 2048, "storage": 1024, "functions": 256,
|
||||
DB_OWNED = ("db", "pgsodium")
|
||||
DB_UID, DB_GID = 100, 101
|
||||
|
||||
# Memory per service: (request MiB, limit MiB). Half of maralago.ru production, except analytics,
|
||||
# realtime and vector: at half of production they were killed for running out of memory in a loop
|
||||
# (measured on a fresh instance: analytics peaks at 624 MiB, realtime at 280 MiB; vector reads the log
|
||||
# files of every pod on the node, not only its own instance's, and was killed at 256 MiB on 2026-09-19).
|
||||
# Memory per service: (request MiB, limit MiB). The limit is the ceiling the kernel kills the
|
||||
# container at; the request is what the scheduler sets aside for it. The requests stay small on
|
||||
# purpose: this is one machine shared with production, and reserving what an instance really uses
|
||||
# three times over would leave production nothing to start in. The limits are about twice the
|
||||
# highest use measured across three running instances on 2026-09-22, because the two logging
|
||||
# services grow with their backlog and the old ceilings sat right on top of normal use:
|
||||
# analytics 612 / 694 / 662 MiB against a 768 MiB ceiling - killed 36 times on instance 1
|
||||
# vector 287 / 290 MiB against 512 - killed 297 times on instance 1, where it had a backlog
|
||||
# realtime 269 / 287 / 300 MiB against 384
|
||||
# kong 260 / 269 / 265 MiB against 512
|
||||
# studio 203, storage 187, rest 162 MiB against 256
|
||||
# db 163 / 175 / 273 MiB against 2048 - left alone, it has room
|
||||
# Raising a limit costs nothing until a container actually asks for the memory; raising a request
|
||||
# would take the memory away from everything else on the node straight away.
|
||||
SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
|
||||
"realtime", "rest", "storage", "studio", "vector"]
|
||||
MEMORY = {"db": (512, 2048), "analytics": (128, 768), "kong": (128, 512), "realtime": (128, 384),
|
||||
"vector": (128, 512)}
|
||||
DEFAULT_MEMORY = (128, 256)
|
||||
MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
|
||||
"vector": (128, 1024)}
|
||||
DEFAULT_MEMORY = (128, 512)
|
||||
|
||||
DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
|
||||
"admin_email": "admin@example.com", "sender_name": "Supabase"}
|
||||
@@ -171,12 +188,7 @@ def instance_values(cfg, n, sec, mail, template_text):
|
||||
quoted = {k: json.dumps(str(v)) for k, v in subs.items()}
|
||||
text = re.sub(r"\$\{(\w+)\}", lambda m: quoted[m.group(1)], template_text)
|
||||
values = yaml.safe_load(text)
|
||||
dep = values.setdefault("deployment", {})
|
||||
for svc in SERVICES:
|
||||
req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
|
||||
d = dep.setdefault(svc, {})
|
||||
d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
|
||||
d["priorityClassName"] = PRIORITY_CLASS
|
||||
apply_memory(values)
|
||||
persistence = values.setdefault("persistence", {})
|
||||
for vol, mib in VOLUMES.items():
|
||||
persistence[vol] = {"enabled": True, "storageClassName": STORAGE_CLASS,
|
||||
@@ -187,6 +199,47 @@ def instance_values(cfg, n, sec, mail, template_text):
|
||||
return values
|
||||
|
||||
|
||||
def apply_memory(values):
|
||||
"""Write the memory caps and the priority class of every service into a values dictionary,
|
||||
replacing whatever was there. Used both when an instance is created and when a running one is
|
||||
brought up to the current caps."""
|
||||
dep = values.setdefault("deployment", {})
|
||||
for svc in SERVICES:
|
||||
req, lim = MEMORY.get(svc, DEFAULT_MEMORY)
|
||||
d = dep.setdefault(svc, {})
|
||||
d["resources"] = {"requests": {"memory": f"{req}Mi"}, "limits": {"memory": f"{lim}Mi"}}
|
||||
d["priorityClassName"] = PRIORITY_CLASS
|
||||
return values
|
||||
|
||||
|
||||
def set_replicas(values, count):
|
||||
"""How many copies of each service to run. Zero stops the instance: the pods go away and give
|
||||
back their memory, their processor time, their pod slots and the two ports they hold on the
|
||||
server, while the disks, the passwords, the certificate and the hostname stay exactly as they
|
||||
are. One starts it again."""
|
||||
dep = values.setdefault("deployment", {})
|
||||
for svc in SERVICES:
|
||||
dep.setdefault(svc, {})["replicaCount"] = count
|
||||
return values
|
||||
|
||||
|
||||
def is_suspended(values):
|
||||
"""True when the stored settings say to run no copy of any service."""
|
||||
dep = values.get("deployment") or {}
|
||||
counts = [(dep.get(svc) or {}).get("replicaCount", 1) for svc in SERVICES]
|
||||
return all(c == 0 for c in counts)
|
||||
|
||||
|
||||
def memory_limits(values):
|
||||
"""{service: limit} as the values dictionary has it, for telling before from after."""
|
||||
dep = values.get("deployment") or {}
|
||||
out = {}
|
||||
for svc in SERVICES:
|
||||
lim = ((dep.get(svc) or {}).get("resources") or {}).get("limits") or {}
|
||||
out[svc] = lim.get("memory")
|
||||
return out
|
||||
|
||||
|
||||
def total_memory_mib():
|
||||
"""(requests, limits) in MiB, summed over all services."""
|
||||
req = sum(MEMORY.get(s, DEFAULT_MEMORY)[0] for s in SERVICES)
|
||||
@@ -406,6 +459,12 @@ def _host_port(doc, port, ip):
|
||||
p = c["ports"][0]
|
||||
p["hostPort"] = port
|
||||
p["hostIP"] = ip
|
||||
if doc.get("kind") == "Deployment":
|
||||
# A port on the server itself can be held by one pod at a time. The default rollout starts
|
||||
# the replacement pod before stopping the old one, so the replacement never gets the port
|
||||
# and waits for ever ("didn't have free ports for the requested pod ports"). Recreate stops
|
||||
# the old pod first. A StatefulSet already replaces its pod that way.
|
||||
doc["spec"]["strategy"] = {"type": "Recreate"}
|
||||
|
||||
|
||||
def pv_manifest(cfg, n, vol):
|
||||
@@ -729,16 +788,11 @@ def create_instance(cfg, mail, log, number=None):
|
||||
sec = generate_secrets()
|
||||
with open(find_template()) as f:
|
||||
values = instance_values(cfg, n, sec, mail, f.read())
|
||||
vfile = f"{STATE_DIR}/values-{n}.yaml"
|
||||
os.makedirs(STATE_DIR, exist_ok=True)
|
||||
fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f:
|
||||
yaml.safe_dump(values, f, sort_keys=False)
|
||||
vfile = write_values_file(n, values)
|
||||
log(" installing chart")
|
||||
try:
|
||||
helm("install", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
|
||||
"--post-renderer", os.path.realpath(sys.argv[0]),
|
||||
"--post-renderer-args", "post-render", "--post-renderer-args", str(n))
|
||||
*post_render_args(n))
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
wait_ready(n, log)
|
||||
@@ -757,6 +811,138 @@ def create_instance(cfg, mail, log, number=None):
|
||||
raise
|
||||
|
||||
|
||||
def write_values_file(n, values):
|
||||
"""The values Helm is given. It carries every secret of the instance, so it is written
|
||||
readable by root only and the caller deletes it as soon as Helm has read it."""
|
||||
vfile = f"{STATE_DIR}/values-{n}.yaml"
|
||||
os.makedirs(STATE_DIR, exist_ok=True)
|
||||
fd = os.open(vfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f:
|
||||
yaml.safe_dump(values, f, sort_keys=False)
|
||||
return vfile
|
||||
|
||||
|
||||
def post_render_args(n):
|
||||
"""Helm hands the finished manifests to supaspin itself, which fixes the parts the chart
|
||||
cannot express (the Postgres and Studio host ports, the gateway routes that are not used)."""
|
||||
return ["--post-renderer", os.path.realpath(sys.argv[0]),
|
||||
"--post-renderer-args", "post-render", "--post-renderer-args", str(n)]
|
||||
|
||||
|
||||
def stored_values(cfg, n):
|
||||
"""The settings Helm kept from the install. They carry every secret of the instance, which is
|
||||
why supaspin deletes its own copy of them as soon as Helm has read it."""
|
||||
ns, rel = namespace(n), release(n)
|
||||
r = helm("get", "values", rel, "-n", ns, "-o", "yaml", check=False)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(f"instance {n}: no Helm release {rel} in namespace {ns}\n{r.stderr.strip()}")
|
||||
values = yaml.safe_load(r.stdout) or {}
|
||||
if not values.get("deployment"):
|
||||
raise RuntimeError(f"instance {n}: the stored settings of {rel} look empty; not touching it")
|
||||
return values
|
||||
|
||||
|
||||
def apply_values(cfg, n, values):
|
||||
"""Hand the settings to Helm, which works out what to change and changes only that."""
|
||||
vfile = write_values_file(n, values)
|
||||
try:
|
||||
helm("upgrade", release(n), chart_path(cfg), "-n", namespace(n), "-f", vfile,
|
||||
*post_render_args(n))
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
|
||||
|
||||
def wait_gone(n, log, timeout=300):
|
||||
"""Wait until the instance's own pods have stopped."""
|
||||
end = time.time() + timeout
|
||||
while time.time() < end:
|
||||
_, total, _ = pod_summary(n)
|
||||
log(f" pods still running: {total}", live=True)
|
||||
if total == 0:
|
||||
log("")
|
||||
return
|
||||
time.sleep(3)
|
||||
raise RuntimeError("the pods did not stop in time")
|
||||
|
||||
|
||||
def suspend_instance(cfg, n, log):
|
||||
"""Stop every pod of an instance and keep everything else. The database, the uploads, the
|
||||
passwords in .env, the certificate and the hostname are untouched; the instance answers
|
||||
nothing until it is resumed."""
|
||||
log(f"Suspending instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
if is_suspended(values):
|
||||
log(" already suspended")
|
||||
return False
|
||||
apply_values(cfg, n, set_replicas(values, 0))
|
||||
wait_gone(n, log)
|
||||
log(f" instance {n} is suspended; its disks and its .env file are untouched")
|
||||
log(f" start it again with: supaspin resume {n}")
|
||||
return True
|
||||
|
||||
|
||||
def resume_instance(cfg, n, log):
|
||||
"""Start a suspended instance again, with the data it had when it was suspended."""
|
||||
log(f"Resuming instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
if not is_suspended(values):
|
||||
log(" already running")
|
||||
return False
|
||||
apply_values(cfg, n, set_replicas(values, 1))
|
||||
log(" the database answers first; the services that need it retry until it does")
|
||||
wait_ready(n, log)
|
||||
log(f" instance {n} is running again: https://{cfg['slots'].get(n, '?')}")
|
||||
return True
|
||||
|
||||
|
||||
def upgrade_instance(cfg, n, log):
|
||||
"""Bring a running instance up to the memory caps this version of supaspin carries, without
|
||||
deleting anything. The values Helm kept from the install are read back, the memory block in
|
||||
them is replaced, and the chart is applied again: the pods are replaced one service at a time
|
||||
and the database, the uploads and the .env file are untouched."""
|
||||
log(f"Instance {n} ({cfg['slots'].get(n, '?')})")
|
||||
values = stored_values(cfg, n)
|
||||
asleep = is_suspended(values)
|
||||
before = memory_limits(values)
|
||||
after = memory_limits(apply_memory(values))
|
||||
changed = [svc for svc in SERVICES if before[svc] != after[svc]]
|
||||
for svc in changed:
|
||||
log(f" {svc}: {before[svc] or 'unset'} -> {after[svc]}")
|
||||
if not changed:
|
||||
# The chart is applied even so: the caps are not the only thing this version may produce
|
||||
# differently (the Studio rollout, the host ports, the gateway routes are all decided
|
||||
# while the manifests are rendered). Helm changes nothing if nothing differs.
|
||||
log(" caps already current; re-applying the chart in case anything else differs")
|
||||
apply_values(cfg, n, values)
|
||||
if asleep:
|
||||
# A suspended instance stays suspended: the new caps are stored and take effect the moment
|
||||
# it is resumed. Waiting for its pods would wait for ever - by design it has none.
|
||||
log(f" instance {n} is suspended; the new caps apply when you resume it")
|
||||
return bool(changed)
|
||||
wait_ready(n, log)
|
||||
log(f" instance {n} is running with the current settings")
|
||||
return bool(changed)
|
||||
|
||||
|
||||
def upgrade_all(cfg, log, numbers=None):
|
||||
"""Upgrade one instance, or every instance that exists."""
|
||||
nums = [numbers] if isinstance(numbers, int) else sorted(numbers or existing_numbers(cfg))
|
||||
if not nums:
|
||||
log("No instances.")
|
||||
return
|
||||
req, lim = total_memory_mib()
|
||||
log(f"Caps in this version: {req} MiB reserved and up to {lim} MiB per instance.")
|
||||
failed = []
|
||||
for n in nums:
|
||||
try:
|
||||
upgrade_instance(cfg, n, log)
|
||||
except RuntimeError as e:
|
||||
failed.append(n)
|
||||
log(f"{RED} instance {n} failed: {e}{OFF}")
|
||||
if failed:
|
||||
raise RuntimeError("these instances were not upgraded: " + ", ".join(str(n) for n in failed))
|
||||
|
||||
|
||||
def delete_instance(cfg, n, log):
|
||||
log(f"Deleting instance {n}")
|
||||
save_cert(n, log)
|
||||
@@ -820,12 +1006,29 @@ def disk_use(cfg, n):
|
||||
return ", ".join(parts)
|
||||
|
||||
|
||||
def instance_state(n):
|
||||
"""What the cluster is currently asked to run for this instance: "running", "suspended",
|
||||
"partly suspended" (some services stopped, which should not happen on its own) or "unknown"
|
||||
when the instance has no workloads at all."""
|
||||
want = []
|
||||
for kind in ("deployments", "statefulsets"):
|
||||
data = kjson("get", kind, "-n", namespace(n), "-l", f"app.kubernetes.io/instance={release(n)}")
|
||||
for item in (data or {}).get("items", []):
|
||||
want.append(item["spec"].get("replicas", 1))
|
||||
if not want:
|
||||
return "unknown"
|
||||
if all(w == 0 for w in want):
|
||||
return "suspended"
|
||||
return "partly suspended" if any(w == 0 for w in want) else "running"
|
||||
|
||||
|
||||
def list_rows(cfg):
|
||||
rows = []
|
||||
for n in sorted(existing_numbers(cfg)):
|
||||
ready, total, _ = pod_summary(n)
|
||||
env = f"{inst_dir(cfg, n)}/.env"
|
||||
rows.append({"n": n, "host": cfg["slots"].get(n, "?"), "pods": f"{ready}/{total}",
|
||||
"state": instance_state(n),
|
||||
"memory": memory_use(n), "disk": disk_use(cfg, n),
|
||||
"env": env if os.path.exists(env) else "(none)"})
|
||||
return rows
|
||||
@@ -1543,8 +1746,12 @@ def print_list(cfg):
|
||||
print("No instances.")
|
||||
return
|
||||
for r in rows:
|
||||
print(f"{BOLD}{r['n']}{OFF} {r['host']} pods {r['pods']} memory {r['memory']}")
|
||||
state = r.get("state", "running")
|
||||
print(f"{BOLD}{r['n']}{OFF} {r['host']} {state} pods {r['pods']} memory {r['memory']}")
|
||||
print(f" disk {r['disk']}\n env {r['env']}")
|
||||
if state == "suspended":
|
||||
print(f" start it again with: supaspin resume {r['n']}\n")
|
||||
continue # the tunnel is pointless while nothing listens on the ports
|
||||
for line in tunnel_guide(cfg, r["n"]):
|
||||
print(f" {line}" if line else "")
|
||||
print()
|
||||
@@ -1554,8 +1761,10 @@ def main_menu(cfg, log):
|
||||
while True:
|
||||
i = menu(f"supaspin {VERSION}", ["Create instance", "List instances", "Show instance",
|
||||
"Delete instance", "Copy instance values into another .env file",
|
||||
"Mail settings", "Settings", "Quit"])
|
||||
if i is None or i == 7:
|
||||
"Apply current memory caps to running instances",
|
||||
"Suspend instance (stop it, keep its data)",
|
||||
"Resume instance", "Mail settings", "Settings", "Quit"])
|
||||
if i is None or i == 10:
|
||||
return
|
||||
print()
|
||||
try:
|
||||
@@ -1579,9 +1788,19 @@ def main_menu(cfg, log):
|
||||
print()
|
||||
push_env(cfg, n, spec, log)
|
||||
elif i == 5:
|
||||
upgrade_all(cfg, log)
|
||||
elif i == 6:
|
||||
n = pick_instance(cfg, "Suspend which instance?")
|
||||
if n:
|
||||
suspend_instance(cfg, n, log)
|
||||
elif i == 7:
|
||||
n = pick_instance(cfg, "Resume which instance?")
|
||||
if n:
|
||||
resume_instance(cfg, n, log)
|
||||
elif i == 8:
|
||||
cfg["mail"] = mail_dialog(cfg)
|
||||
save_config(cfg)
|
||||
elif i == 6:
|
||||
elif i == 9:
|
||||
cfg = setup_dialog(cfg, log)
|
||||
except Exception as e:
|
||||
print(f"{RED}{e}{OFF}")
|
||||
@@ -1630,6 +1849,16 @@ def main(argv):
|
||||
elif cmd == "delete" and n:
|
||||
if "--yes" in argv or confirm_number(n, "deleting it"):
|
||||
delete_instance(cfg, n, log_line)
|
||||
elif cmd == "upgrade":
|
||||
upgrade_all(cfg, log_line, n)
|
||||
elif cmd in ("suspend", "resume"):
|
||||
if n is None:
|
||||
# No "all" here on purpose: stopping every instance is not something to type by
|
||||
# accident, and resuming them all at once would start 36 pods in one go.
|
||||
print(f"supaspin {cmd} N - which instance? ({cmd} takes one number)",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
(suspend_instance if cmd == "suspend" else resume_instance)(cfg, n, log_line)
|
||||
elif cmd == "push-env":
|
||||
return push_env_command(cfg, argv[2:])
|
||||
elif cmd == "mail":
|
||||
|
||||
+353
-7
@@ -90,14 +90,20 @@ class Values(unittest.TestCase):
|
||||
|
||||
def test_memory_caps_add_up(self):
|
||||
req, lim = sp.total_memory_mib()
|
||||
# db 2048, analytics 768, kong 512, realtime 384, vector 512, the other seven 256 each
|
||||
self.assertEqual(lim, 2048 + 768 + 512 + 384 + 512 + 7 * 256)
|
||||
self.assertEqual(lim / 1024, 5.875)
|
||||
# db 2048, analytics 1536, kong 768, realtime 768, vector 1024, the other seven 512 each
|
||||
self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 1024 + 7 * 512)
|
||||
self.assertEqual(lim / 1024, 9.5)
|
||||
self.assertEqual(req, 512 + 11 * 128) # 1.875 GiB
|
||||
# analytics, realtime and vector were killed for running out of memory at half of production
|
||||
self.assertGreaterEqual(sp.MEMORY["analytics"][1], 768)
|
||||
self.assertGreaterEqual(sp.MEMORY["realtime"][1], 384)
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 512)
|
||||
# Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
|
||||
# highest use, or the kernel kills the container in a loop the way it did on instance 1.
|
||||
self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536) # used 694 MiB, died at 768
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 1024) # used 290 MiB, died at 512
|
||||
self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768) # used 300 MiB against 384
|
||||
self.assertGreaterEqual(sp.MEMORY["kong"][1], 768) # used 269 MiB against 512
|
||||
self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512) # studio used 203 against 256
|
||||
# The requests stay where they were: the node is shared with production and reserving
|
||||
# what the instances really use would leave production nothing to schedule into.
|
||||
self.assertTrue(all(m[0] == 128 for s, m in sp.MEMORY.items() if s != "db"))
|
||||
self.assertEqual(req / 1024, 1.875)
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.v["deployment"][svc]["priorityClassName"], "supaspin-low")
|
||||
@@ -929,3 +935,343 @@ class PushEnvCommand(unittest.TestCase):
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class Upgrade(unittest.TestCase):
|
||||
"""`supaspin upgrade` puts the current memory caps on an instance that is already running.
|
||||
It must change the caps and nothing else: no volume is resized, no secret is regenerated."""
|
||||
|
||||
STORED = {
|
||||
"deployment": {
|
||||
"analytics": {"enabled": True, "priorityClassName": "supaspin-low",
|
||||
"resources": {"limits": {"memory": "768Mi"},
|
||||
"requests": {"memory": "128Mi"}}},
|
||||
"vector": {"enabled": True, "priorityClassName": "supaspin-low",
|
||||
"resources": {"limits": {"memory": "512Mi"},
|
||||
"requests": {"memory": "128Mi"}}},
|
||||
},
|
||||
"secret": {"jwt": {"secret": "keep-me", "anonKey": "anon"}},
|
||||
"persistence": {"db": {"enabled": True, "size": "2048Mi"}},
|
||||
}
|
||||
|
||||
def setUp(self):
|
||||
self.calls = []
|
||||
self.written = {}
|
||||
self.log = []
|
||||
|
||||
@contextlib.contextmanager
|
||||
def fake_helm(self, stored=None, rc=0):
|
||||
stored = self.STORED if stored is None else stored
|
||||
|
||||
def helm(*args, check=True):
|
||||
self.calls.append(list(args))
|
||||
if args[0] == "get":
|
||||
out = yaml.safe_dump(stored) if stored is not None else ""
|
||||
return collections.namedtuple("P", "returncode stdout stderr")(rc, out, "boom")
|
||||
if args[0] == "upgrade":
|
||||
with open(args[args.index("-f") + 1]) as f:
|
||||
self.written = yaml.safe_load(f)
|
||||
return collections.namedtuple("P", "returncode stdout stderr")(0, "", "")
|
||||
|
||||
with mock.patch.object(sp, "helm", helm), \
|
||||
mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
def test_raises_the_caps_and_keeps_everything_else(self):
|
||||
with self.fake_helm():
|
||||
changed = sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertTrue(changed)
|
||||
dep = self.written["deployment"]
|
||||
self.assertEqual(dep["analytics"]["resources"]["limits"]["memory"],
|
||||
f"{sp.MEMORY['analytics'][1]}Mi")
|
||||
self.assertEqual(dep["vector"]["resources"]["limits"]["memory"],
|
||||
f"{sp.MEMORY['vector'][1]}Mi")
|
||||
self.assertTrue(dep["analytics"]["enabled"]) # untouched
|
||||
self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
|
||||
self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
|
||||
|
||||
def test_every_service_gets_a_cap_even_if_the_stored_values_had_none(self):
|
||||
with self.fake_helm():
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
for svc in sp.SERVICES:
|
||||
req, lim = sp.MEMORY.get(svc, sp.DEFAULT_MEMORY)
|
||||
got = self.written["deployment"][svc]["resources"]
|
||||
self.assertEqual(got["limits"]["memory"], f"{lim}Mi")
|
||||
self.assertEqual(got["requests"]["memory"], f"{req}Mi")
|
||||
|
||||
def test_the_values_file_with_the_secrets_is_deleted_again(self):
|
||||
with self.fake_helm():
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
|
||||
self.assertFalse(os.path.exists(path))
|
||||
|
||||
def test_helm_gets_the_post_renderer(self):
|
||||
with self.fake_helm():
|
||||
sp.upgrade_instance(CFG, 2, self.log.append)
|
||||
up = [c for c in self.calls if c[0] == "upgrade"][0]
|
||||
self.assertIn("--post-renderer", up)
|
||||
self.assertEqual(up[up.index("--post-renderer-args") + 1], "post-render")
|
||||
self.assertIn("2", up)
|
||||
|
||||
def test_an_instance_already_at_the_caps_is_still_re_applied(self):
|
||||
# The caps are not the only thing a new version can change: the Studio rollout, the host
|
||||
# ports and the gateway routes are decided while the manifests are rendered, and none of
|
||||
# them shows up in the stored values.
|
||||
stored = {"deployment": sp.apply_memory({})["deployment"]}
|
||||
with self.fake_helm(stored):
|
||||
changed = sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertFalse(changed) # no cap moved
|
||||
self.assertEqual([c[0] for c in self.calls], ["get", "upgrade"]) # applied anyway
|
||||
|
||||
def test_a_missing_release_is_refused_before_anything_is_written(self):
|
||||
with self.fake_helm(rc=1):
|
||||
with self.assertRaises(RuntimeError):
|
||||
sp.upgrade_instance(CFG, 3, self.log.append)
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
|
||||
def test_empty_stored_values_are_refused(self):
|
||||
with self.fake_helm({}):
|
||||
with self.assertRaises(RuntimeError):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
|
||||
def test_upgrade_all_does_every_instance_and_reports_the_failures(self):
|
||||
done = []
|
||||
|
||||
def one(cfg, n, log):
|
||||
if n == 2:
|
||||
raise RuntimeError("no release")
|
||||
done.append(n)
|
||||
return True
|
||||
|
||||
with mock.patch.object(sp, "upgrade_instance", one), \
|
||||
mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
|
||||
with self.assertRaises(RuntimeError) as e:
|
||||
sp.upgrade_all(CFG, self.log.append)
|
||||
self.assertEqual(done, [1, 3])
|
||||
self.assertIn("2", str(e.exception))
|
||||
|
||||
def test_upgrade_all_with_a_number_does_only_that_instance(self):
|
||||
done = []
|
||||
with mock.patch.object(sp, "upgrade_instance", lambda cfg, n, log: done.append(n)), \
|
||||
mock.patch.object(sp, "existing_numbers", lambda cfg: {1, 2, 3}):
|
||||
sp.upgrade_all(CFG, self.log.append, 2)
|
||||
self.assertEqual(done, [2])
|
||||
|
||||
|
||||
class HostPortRollout(unittest.TestCase):
|
||||
"""Studio and Postgres bind a port on the server itself, and only one pod can hold a port.
|
||||
The Studio deployment must therefore stop its old pod before starting the new one, or an
|
||||
upgrade leaves the replacement pod Pending for ever."""
|
||||
|
||||
def rendered(self, n=1):
|
||||
sec = sp.generate_secrets()
|
||||
values = sp.instance_values(CFG, n, sec, MAIL, template())
|
||||
with mock.patch.object(sp, "helm"), mock.patch.object(sp, "kubectl"):
|
||||
text = yaml.safe_dump_all([
|
||||
{"apiVersion": "apps/v1", "kind": "Deployment",
|
||||
"metadata": {"name": f"{sp.fullname(n)}-studio"},
|
||||
"spec": {"strategy": {"type": "RollingUpdate",
|
||||
"rollingUpdate": {"maxSurge": "25%"}},
|
||||
"template": {"spec": {"containers": [
|
||||
{"name": "studio", "ports": [{"containerPort": 3000}]}]}}}},
|
||||
{"apiVersion": "apps/v1", "kind": "StatefulSet",
|
||||
"metadata": {"name": f"{sp.fullname(n)}-db"},
|
||||
"spec": {"template": {"spec": {"containers": [
|
||||
{"name": "db", "ports": [{"containerPort": 5432}]}]}}}},
|
||||
{"apiVersion": "apps/v1", "kind": "Deployment",
|
||||
"metadata": {"name": f"{sp.fullname(n)}-rest"},
|
||||
"spec": {"strategy": {"type": "RollingUpdate"},
|
||||
"template": {"spec": {"containers": [
|
||||
{"name": "rest", "ports": [{"containerPort": 3000}]}]}}}},
|
||||
])
|
||||
out = list(yaml.safe_load_all(sp.post_render(text, n, CFG["lan_ip"])))
|
||||
return {d["metadata"]["name"]: d for d in out}
|
||||
|
||||
def test_studio_stops_the_old_pod_before_starting_the_new_one(self):
|
||||
d = self.rendered()[f"{sp.fullname(1)}-studio"]
|
||||
self.assertEqual(d["spec"]["strategy"], {"type": "Recreate"})
|
||||
port = d["spec"]["template"]["spec"]["containers"][0]["ports"][0]
|
||||
self.assertEqual(port["hostPort"], sp.studio_port(1))
|
||||
self.assertEqual(port["hostIP"], CFG["lan_ip"])
|
||||
|
||||
def test_a_deployment_without_a_host_port_keeps_the_normal_rollout(self):
|
||||
d = self.rendered()[f"{sp.fullname(1)}-rest"]
|
||||
self.assertEqual(d["spec"]["strategy"]["type"], "RollingUpdate")
|
||||
|
||||
def test_the_database_keeps_its_host_port_and_gains_no_strategy(self):
|
||||
d = self.rendered()[f"{sp.fullname(1)}-db"]
|
||||
self.assertNotIn("strategy", d["spec"])
|
||||
self.assertEqual(d["spec"]["template"]["spec"]["containers"][0]["ports"][0]["hostPort"],
|
||||
sp.pg_port(1))
|
||||
|
||||
|
||||
class SuspendResume(unittest.TestCase):
|
||||
"""Suspending an instance stops its pods and keeps everything else: the disks, the passwords,
|
||||
the certificate and the hostname. Resuming starts it again with the same data."""
|
||||
|
||||
RUNNING = {
|
||||
"deployment": {
|
||||
"analytics": {"enabled": True,
|
||||
"resources": {"limits": {"memory": "1536Mi"},
|
||||
"requests": {"memory": "128Mi"}}},
|
||||
"db": {"resources": {"limits": {"memory": "2048Mi"},
|
||||
"requests": {"memory": "512Mi"}}},
|
||||
},
|
||||
"secret": {"jwt": {"secret": "keep-me"}, "db": {"password": "keep-me-too"}},
|
||||
"persistence": {"db": {"enabled": True, "size": "2048Mi"}},
|
||||
}
|
||||
|
||||
def setUp(self):
|
||||
self.calls = []
|
||||
self.written = {}
|
||||
self.log = []
|
||||
self.waited = []
|
||||
|
||||
@contextlib.contextmanager
|
||||
def fake_helm(self, stored):
|
||||
P = collections.namedtuple("P", "returncode stdout stderr")
|
||||
|
||||
def helm(*args, check=True):
|
||||
self.calls.append(list(args))
|
||||
if args[0] == "get":
|
||||
return P(0, yaml.safe_dump(stored), "")
|
||||
if args[0] == "upgrade":
|
||||
with open(args[args.index("-f") + 1]) as f:
|
||||
self.written = yaml.safe_load(f)
|
||||
return P(0, "", "")
|
||||
|
||||
with mock.patch.object(sp, "helm", helm), \
|
||||
mock.patch.object(sp, "wait_ready",
|
||||
lambda n, log, **kw: self.waited.append(("ready", n))), \
|
||||
mock.patch.object(sp, "wait_gone",
|
||||
lambda n, log, **kw: self.waited.append(("gone", n))), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
def suspended(self):
|
||||
return sp.set_replicas(yaml.safe_load(yaml.safe_dump(self.RUNNING)), 0)
|
||||
|
||||
# ---- the setting itself ----
|
||||
|
||||
def test_set_replicas_covers_every_service(self):
|
||||
v = sp.set_replicas({}, 0)
|
||||
self.assertEqual(sorted(v["deployment"]), sorted(sp.SERVICES))
|
||||
self.assertTrue(all(d["replicaCount"] == 0 for d in v["deployment"].values()))
|
||||
self.assertTrue(sp.is_suspended(v))
|
||||
|
||||
def test_a_service_missing_the_setting_counts_as_running(self):
|
||||
self.assertFalse(sp.is_suspended(self.RUNNING))
|
||||
self.assertFalse(sp.is_suspended({"deployment": {}}))
|
||||
|
||||
def test_one_service_left_running_is_not_suspended(self):
|
||||
v = sp.set_replicas({}, 0)
|
||||
v["deployment"]["kong"]["replicaCount"] = 1
|
||||
self.assertFalse(sp.is_suspended(v))
|
||||
|
||||
# ---- suspend ----
|
||||
|
||||
def test_suspend_stops_every_service_and_keeps_the_rest(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
self.assertTrue(sp.suspend_instance(CFG, 1, self.log.append))
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.written["deployment"][svc]["replicaCount"], 0)
|
||||
self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
|
||||
self.assertEqual(self.written["secret"]["db"]["password"], "keep-me-too")
|
||||
self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
|
||||
# the memory caps are kept as they were: they apply again on resume
|
||||
self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
|
||||
"1536Mi")
|
||||
self.assertEqual(self.waited, [("gone", 1)])
|
||||
|
||||
def test_suspend_waits_for_the_pods_to_be_gone_not_for_them_to_be_ready(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.suspend_instance(CFG, 2, self.log.append)
|
||||
self.assertEqual([w[0] for w in self.waited], ["gone"])
|
||||
|
||||
def test_suspending_a_suspended_instance_changes_nothing(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
self.assertFalse(sp.suspend_instance(CFG, 1, self.log.append))
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
self.assertEqual(self.waited, [])
|
||||
|
||||
def test_the_values_file_with_the_secrets_is_deleted_again(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.suspend_instance(CFG, 1, self.log.append)
|
||||
path = [c[c.index("-f") + 1] for c in self.calls if c[0] == "upgrade"][0]
|
||||
self.assertFalse(os.path.exists(path))
|
||||
|
||||
# ---- resume ----
|
||||
|
||||
def test_resume_starts_every_service_again(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
self.assertTrue(sp.resume_instance(CFG, 3, self.log.append))
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.written["deployment"][svc]["replicaCount"], 1)
|
||||
self.assertFalse(sp.is_suspended(self.written))
|
||||
self.assertEqual(self.waited, [("ready", 3)])
|
||||
|
||||
def test_resuming_a_running_instance_changes_nothing(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
self.assertFalse(sp.resume_instance(CFG, 1, self.log.append))
|
||||
self.assertEqual([c[0] for c in self.calls], ["get"])
|
||||
self.assertEqual(self.waited, [])
|
||||
|
||||
def test_a_suspended_instance_keeps_its_data_settings(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
sp.resume_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual(self.written["persistence"]["db"]["size"], "2048Mi")
|
||||
self.assertEqual(self.written["secret"]["jwt"]["secret"], "keep-me")
|
||||
|
||||
# ---- upgrade must not trip over a suspended instance ----
|
||||
|
||||
def test_upgrade_leaves_a_suspended_instance_suspended_and_does_not_wait(self):
|
||||
with self.fake_helm(self.suspended()):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertTrue(sp.is_suspended(self.written)) # not woken up
|
||||
self.assertEqual(self.waited, []) # would never finish
|
||||
self.assertEqual(self.written["deployment"]["analytics"]["resources"]["limits"]["memory"],
|
||||
f"{sp.MEMORY['analytics'][1]}Mi") # new caps stored for the resume
|
||||
|
||||
def test_upgrade_still_waits_for_a_running_instance(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual(self.waited, [("ready", 1)])
|
||||
|
||||
|
||||
class InstanceState(unittest.TestCase):
|
||||
"""What `supaspin list` says about an instance, read from what the cluster is asked to run."""
|
||||
|
||||
def state(self, deployments, statefulsets):
|
||||
def kjson(*args):
|
||||
kind = args[1]
|
||||
counts = deployments if kind == "deployments" else statefulsets
|
||||
return {"items": [{"spec": {"replicas": c}} for c in counts]}
|
||||
|
||||
with mock.patch.object(sp, "kjson", kjson):
|
||||
return sp.instance_state(1)
|
||||
|
||||
def test_running(self):
|
||||
self.assertEqual(self.state([1] * 11, [1]), "running")
|
||||
|
||||
def test_suspended(self):
|
||||
self.assertEqual(self.state([0] * 11, [0]), "suspended")
|
||||
|
||||
def test_the_database_alone_still_running_is_reported_honestly(self):
|
||||
self.assertEqual(self.state([0] * 11, [1]), "partly suspended")
|
||||
|
||||
def test_nothing_there(self):
|
||||
self.assertEqual(self.state([], []), "unknown")
|
||||
|
||||
def test_the_list_shows_the_state_and_no_tunnel_for_a_sleeping_instance(self):
|
||||
rows = [{"n": 1, "host": "supa1.example.com", "pods": "0/0", "state": "suspended",
|
||||
"memory": "n/a", "disk": "db 1/2M", "env": "/data/supabase/1/.env"}]
|
||||
out = io.StringIO()
|
||||
with mock.patch.object(sp, "list_rows", lambda cfg: rows), \
|
||||
contextlib.redirect_stdout(out):
|
||||
sp.print_list(CFG)
|
||||
text = out.getvalue()
|
||||
self.assertIn("suspended", text)
|
||||
self.assertIn("supaspin resume 1", text)
|
||||
self.assertNotIn("ssh -N -L", text)
|
||||
|
||||
Reference in New Issue
Block a user