Merge pull request 'fix: delete analytics logs older than 2 days and give vector a 512 MiB cap' (#9) from fix/log-retention-vector-512 into main
This commit was merged in pull request #9.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# supaspin
|
||||
|
||||
Version 1.3.1
|
||||
Version 1.3.2
|
||||
|
||||
Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
|
||||
single-node [microk8s](https://microk8s.io) server, from a terminal menu.
|
||||
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
|
||||
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
|
||||
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
|
||||
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
|
||||
| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
|
||||
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
|
||||
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
|
||||
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
|
||||
@@ -93,7 +93,7 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
|
||||
| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
|
||||
| Delete instance | asks you to type the number; removes everything (see below) |
|
||||
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
|
||||
| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
|
||||
| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
|
||||
| Mail settings | view and change the saved default mail settings |
|
||||
| Settings | change any first-run setting |
|
||||
|
||||
@@ -262,11 +262,11 @@ stores the new memory limits and leaves it asleep; they take effect when you res
|
||||
|---|---|---|---|
|
||||
| db | 512 MiB | 2048 MiB | 273 MiB |
|
||||
| analytics | 128 MiB | 1536 MiB | 694 MiB |
|
||||
| vector | 128 MiB | 1024 MiB | 290 MiB |
|
||||
| vector | 128 MiB | 512 MiB | 290 MiB |
|
||||
| kong | 128 MiB | 768 MiB | 269 MiB |
|
||||
| realtime | 128 MiB | 768 MiB | 300 MiB |
|
||||
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
|
||||
| **Total** | **1.875 GiB** | **9.5 GiB** | |
|
||||
| **Total** | **1.875 GiB** | **9 GiB** | |
|
||||
|
||||
The request is what the scheduler sets aside for a service and can never be given to anything else.
|
||||
The limit is the ceiling the kernel kills the container at. They do different jobs here.
|
||||
@@ -278,13 +278,21 @@ in. The limits are roughly twice the highest use measured across three running i
|
||||
|
||||
The earlier limits sat right on top of normal use and the kernel kept killing the two logging
|
||||
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
|
||||
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
|
||||
log files of every pod on the server and not only its own instance's, was killed 297 times on that
|
||||
same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
|
||||
other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
|
||||
the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
|
||||
in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
|
||||
its own.
|
||||
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
|
||||
instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
|
||||
could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
|
||||
cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
|
||||
Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
|
||||
(imgproxy) reads the uploads from the storage volume, so it has no image of its own.
|
||||
|
||||
On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
|
||||
it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
|
||||
100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
|
||||
every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
|
||||
minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
|
||||
job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
|
||||
night. That bounds both the db image and vector's backlog, so vector is back to the 512 MiB
|
||||
ceiling the production supabase charts carry.
|
||||
|
||||
Changing the numbers in this table only affects instances created afterwards. To put them on
|
||||
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when
|
||||
|
||||
@@ -39,7 +39,7 @@ import tty
|
||||
|
||||
import yaml
|
||||
|
||||
VERSION = "1.3.1"
|
||||
VERSION = "1.3.2"
|
||||
CONFIG_PATH = os.environ.get("SUPASPIN_CONFIG", "/etc/supaspin/config.yaml")
|
||||
STATE_DIR = os.environ.get("SUPASPIN_STATE", "/var/lib/supaspin")
|
||||
CHART_REPO = "https://supabase-community.github.io/supabase-kubernetes"
|
||||
@@ -72,14 +72,42 @@ DB_UID, DB_GID = 100, 101
|
||||
# kong 260 / 269 / 265 MiB against 512
|
||||
# studio 203, storage 187, rest 162 MiB against 256
|
||||
# db 163 / 175 / 273 MiB against 2048 - left alone, it has room
|
||||
# On 2026-10-06 instance 3 filled its 2 GB database image with logs: analytics (Logflare) never
|
||||
# deletes what it stored, the db died of it, and vector's undelivered backlog was killed against
|
||||
# the 1024 MiB ceiling every few minutes. The fix is setup_log_retention: a pg_cron job deletes
|
||||
# analytics logs older than LOG_RETENTION in every instance, which also bounds vector's backlog,
|
||||
# so vector is back to the 512 MiB ceiling the production supabase charts carry.
|
||||
# Raising a limit costs nothing until a container actually asks for the memory; raising a request
|
||||
# would take the memory away from everything else on the node straight away.
|
||||
SERVICES = ["db", "analytics", "kong", "auth", "functions", "imgproxy", "meta",
|
||||
"realtime", "rest", "storage", "studio", "vector"]
|
||||
MEMORY = {"db": (512, 2048), "analytics": (128, 1536), "kong": (128, 768), "realtime": (128, 768),
|
||||
"vector": (128, 1024)}
|
||||
"vector": (128, 512)}
|
||||
DEFAULT_MEMORY = (128, 512)
|
||||
|
||||
# Analytics logs: deleted from every instance's _supabase database when older than this. Logflare
|
||||
# stores logs in _analytics.log_events_* tables and never deletes them; without this job the
|
||||
# capped db image fills up and the database dies (instance 3, 2026-10-06).
|
||||
LOG_RETENTION = "2 days"
|
||||
LOG_RETENTION_JOB = "supaspin-log-retention"
|
||||
RETENTION_SQL = f"""
|
||||
create extension if not exists pg_cron;
|
||||
create or replace function _analytics.delete_old_logs() returns void language plpgsql as $fn$
|
||||
declare
|
||||
t text;
|
||||
begin
|
||||
for t in
|
||||
select tablename from pg_tables
|
||||
where schemaname = '_analytics' and tablename like 'log_events\\_%'
|
||||
loop
|
||||
execute format('delete from _analytics.%I where timestamp < now() - interval ''{LOG_RETENTION}''', t);
|
||||
end loop;
|
||||
end $fn$;
|
||||
select _analytics.delete_old_logs();
|
||||
delete from cron.job where jobname = '{LOG_RETENTION_JOB}';
|
||||
select cron.schedule('{LOG_RETENTION_JOB}', '17 3 * * *', 'select _analytics.delete_old_logs();');
|
||||
""".strip()
|
||||
|
||||
DEFAULT_MAIL = {"host": "smtp.example.com", "port": 587, "user": "", "password": "",
|
||||
"admin_email": "admin@example.com", "sender_name": "Supabase"}
|
||||
|
||||
@@ -768,6 +796,19 @@ def wait_ready(n, log, timeout=1200):
|
||||
raise RuntimeError("pods did not become ready in time")
|
||||
|
||||
|
||||
def setup_log_retention(cfg, n, log):
|
||||
"""Give the instance a pg_cron job that deletes analytics logs older than LOG_RETENTION every
|
||||
night. Logflare stores the logs it receives in the _supabase database and never deletes them;
|
||||
without this job the capped db image fills up and the database dies (instance 3 on
|
||||
2026-10-06). Runs the first cleanup right away, so an upgrade also frees a full disk. Runs
|
||||
after create, upgrade and resume; scheduling under a fixed job name makes it idempotent."""
|
||||
log(f" log retention: deleting analytics logs older than {LOG_RETENTION}, every night")
|
||||
kubectl("-n", namespace(n), "exec", f"{fullname(n)}-db-0", "-c", "supabase-db", "--",
|
||||
"psql", "-U", "supabase_admin", "-d", "_supabase", "-v", "ON_ERROR_STOP=1",
|
||||
"-c", RETENTION_SQL)
|
||||
log(" log retention is set")
|
||||
|
||||
|
||||
def create_instance(cfg, mail, log, number=None):
|
||||
taken = existing_numbers(cfg)
|
||||
n = number if number is not None else lowest_free_number(cfg["slots"], taken)
|
||||
@@ -796,6 +837,7 @@ def create_instance(cfg, mail, log, number=None):
|
||||
finally:
|
||||
os.remove(vfile) # holds every secret
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
env_path = f"{inst_dir(cfg, n)}/.env"
|
||||
fd = os.open(env_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f:
|
||||
@@ -891,6 +933,7 @@ def resume_instance(cfg, n, log):
|
||||
apply_values(cfg, n, set_replicas(values, 1))
|
||||
log(" the database answers first; the services that need it retry until it does")
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
log(f" instance {n} is running again: https://{cfg['slots'].get(n, '?')}")
|
||||
return True
|
||||
|
||||
@@ -917,9 +960,10 @@ def upgrade_instance(cfg, n, log):
|
||||
if asleep:
|
||||
# A suspended instance stays suspended: the new caps are stored and take effect the moment
|
||||
# it is resumed. Waiting for its pods would wait for ever - by design it has none.
|
||||
log(f" instance {n} is suspended; the new caps apply when you resume it")
|
||||
log(f" instance {n} is suspended; the new caps and log retention apply when you resume it")
|
||||
return bool(changed)
|
||||
wait_ready(n, log)
|
||||
setup_log_retention(cfg, n, log)
|
||||
log(f" instance {n} is running with the current settings")
|
||||
return bool(changed)
|
||||
|
||||
|
||||
+21
-6
@@ -90,14 +90,17 @@ class Values(unittest.TestCase):
|
||||
|
||||
def test_memory_caps_add_up(self):
|
||||
req, lim = sp.total_memory_mib()
|
||||
# db 2048, analytics 1536, kong 768, realtime 768, vector 1024, the other seven 512 each
|
||||
self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 1024 + 7 * 512)
|
||||
self.assertEqual(lim / 1024, 9.5)
|
||||
# db 2048, analytics 1536, kong 768, realtime 768, vector 512, the other seven 512 each
|
||||
self.assertEqual(lim, 2048 + 1536 + 768 + 768 + 512 + 7 * 512)
|
||||
self.assertEqual(lim / 1024, 9.0)
|
||||
self.assertEqual(req, 512 + 11 * 128) # 1.875 GiB
|
||||
# Measured on three running instances on 2026-09-22; the ceiling has to stay clear of the
|
||||
# highest use, or the kernel kills the container in a loop the way it did on instance 1.
|
||||
self.assertGreaterEqual(sp.MEMORY["analytics"][1], 1536) # used 694 MiB, died at 768
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 1024) # used 290 MiB, died at 512
|
||||
self.assertGreaterEqual(sp.MEMORY["vector"][1], 512) # used 290 MiB; the nightly log
|
||||
# retention keeps its delivery backlog from growing past this the way it did on
|
||||
# 2026-10-06, when instance 3's full db image killed analytics and vector filled with
|
||||
# the undeliverable backlog every few minutes against 1024 MiB.
|
||||
self.assertGreaterEqual(sp.MEMORY["realtime"][1], 768) # used 300 MiB against 384
|
||||
self.assertGreaterEqual(sp.MEMORY["kong"][1], 768) # used 269 MiB against 512
|
||||
self.assertGreaterEqual(sp.DEFAULT_MEMORY[1], 512) # studio used 203 against 256
|
||||
@@ -128,6 +131,15 @@ class Values(unittest.TestCase):
|
||||
self.assertEqual(set(sp.VOLUMES), {"db", "storage", "functions", "snippets", "deno", "pgsodium"})
|
||||
self.assertTrue(all(p[v]["storageClassName"] == "supaspin-local-sc" for v in sp.VOLUMES))
|
||||
|
||||
def test_log_retention_sql(self):
|
||||
# Logflare stores logs in _analytics.log_events_* and never deletes them; the job deletes
|
||||
# rows older than LOG_RETENTION and is scheduled under a fixed name so re-runs overwrite.
|
||||
self.assertIn("schemaname = '_analytics'", sp.RETENTION_SQL)
|
||||
self.assertIn("like 'log_events\\_%'", sp.RETENTION_SQL)
|
||||
self.assertIn(f"interval ''{sp.LOG_RETENTION}''", sp.RETENTION_SQL)
|
||||
self.assertIn(f"'{sp.LOG_RETENTION_JOB}'", sp.RETENTION_SQL)
|
||||
self.assertIn("cron.schedule", sp.RETENTION_SQL)
|
||||
|
||||
def test_imgproxy_reads_the_storage_volume(self):
|
||||
img = self.v["persistence"]["imgproxy"]
|
||||
self.assertEqual(img["existingClaim"], "supa2-supabase-storage")
|
||||
@@ -975,6 +987,7 @@ class Upgrade(unittest.TestCase):
|
||||
|
||||
with mock.patch.object(sp, "helm", helm), \
|
||||
mock.patch.object(sp, "wait_ready", lambda n, log, **kw: None), \
|
||||
mock.patch.object(sp, "setup_log_retention", lambda cfg, n, log: None), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
@@ -1147,6 +1160,8 @@ class SuspendResume(unittest.TestCase):
|
||||
lambda n, log, **kw: self.waited.append(("ready", n))), \
|
||||
mock.patch.object(sp, "wait_gone",
|
||||
lambda n, log, **kw: self.waited.append(("gone", n))), \
|
||||
mock.patch.object(sp, "setup_log_retention",
|
||||
lambda cfg, n, log: self.waited.append(("retention", n))), \
|
||||
mock.patch.object(sp, "STATE_DIR", tempfile.mkdtemp()):
|
||||
yield
|
||||
|
||||
@@ -1210,7 +1225,7 @@ class SuspendResume(unittest.TestCase):
|
||||
for svc in sp.SERVICES:
|
||||
self.assertEqual(self.written["deployment"][svc]["replicaCount"], 1)
|
||||
self.assertFalse(sp.is_suspended(self.written))
|
||||
self.assertEqual(self.waited, [("ready", 3)])
|
||||
self.assertEqual(self.waited, [("ready", 3), ("retention", 3)])
|
||||
|
||||
def test_resuming_a_running_instance_changes_nothing(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
@@ -1237,7 +1252,7 @@ class SuspendResume(unittest.TestCase):
|
||||
def test_upgrade_still_waits_for_a_running_instance(self):
|
||||
with self.fake_helm(self.RUNNING):
|
||||
sp.upgrade_instance(CFG, 1, self.log.append)
|
||||
self.assertEqual(self.waited, [("ready", 1)])
|
||||
self.assertEqual(self.waited, [("ready", 1), ("retention", 1)])
|
||||
|
||||
|
||||
class InstanceState(unittest.TestCase):
|
||||
|
||||
Reference in New Issue
Block a user