public/supaspin
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

fix: delete analytics logs older than 2 days and give vector a 512 MiB cap

Browse Source 
- every instance gets a pg_cron job (set up on create, upgrade and resume)
  that deletes _analytics.log_events_* rows older than 2 days and runs the
  first cleanup right away; Logflare never deletes stored logs, and instance
  3's 2 GB db image died 100% full of them on 2026-10-06, after which
  analytics 500'd every delivery and vector was OOM-killed by its
  undeliverable backlog every few minutes
- vector's memory limit back to 512 MiB, the ceiling the production
  supabase charts carry; the retention job bounds its backlog, so 1024 MiB
  is not needed anymore
- version 1.3.2
This commit is contained in:
Igor K.
2026-10-06 20:28:29 +03:00
parent 4bcdf6907c
commit a923839c47
4 changed files with 89 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+20 -12
View File
@@ -1,6 +1,6 @@
# supaspin
Version 1.3.1
Version 1.3.2
Create, list, show and delete throwaway [Supabase](https://supabase.com) instances on a
single-node [microk8s](https://microk8s.io) server, from a terminal menu.
@@ -17,7 +17,7 @@ endangering them. Nothing in it is specific to any one app.
| Postgres | `<LAN IP>:544N` (instance 1 → 5441, 2 → 5442, …), bound on the LAN address only |
| Studio (admin UI) | `http://<LAN IP>:300N` (3001, 3002, …), not on the public hostname |
| Disk | fixed-size ext4 image files, mounted at boot: **db 2 GB, storage 1 GB, every other volume 256 MB** |
| Memory | limits add up to **9.5 GiB** per instance, requests to 1.875 GiB (table below) |
| Memory | limits add up to **9 GiB** per instance, requests to 1.875 GiB (table below) |
| Priority | pods run at priority -100 (`supaspin-low`); when the server runs short of memory they are evicted first and never evict anyone else |
| Secrets | generated fresh per instance: JWT secret, anon and service keys, Postgres password, Studio password, and the internal keys |
| Uploads | stored in the instance's own storage image (no MinIO); deleted with the instance |
@@ -93,7 +93,7 @@ Supabase Helm chart (0.8.0, Postgres 17.6.1.136) once into `/var/lib/supaspin/ch
| Show instance (and suspend or resume it) | the `.env` (secrets hidden unless you ask), the tunnel command, pod status and recent events - and the one line that stops this instance or starts it again, worded for whichever state it is in |
| Delete instance | asks you to type the number; removes everything (see below) |
| Copy instance values into another .env file | puts an instance's `.env` values into your app's `.env`, on this server or on another computer (see below) |
| Apply current memory caps to running instances | puts this version's memory limits on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
| Apply current memory caps and log retention to running instances | puts this version's memory limits and the log-retention job on instances that already exist, replacing their pods one service at a time; databases, uploads and `.env` files are not touched |
| Mail settings | view and change the saved default mail settings |
| Settings | change any first-run setting |
@@ -262,11 +262,11 @@ stores the new memory limits and leaves it asleep; they take effect when you res
|---|---|---|---|
| db | 512 MiB | 2048 MiB | 273 MiB |
| analytics | 128 MiB | 1536 MiB | 694 MiB |
| vector | 128 MiB | 1024 MiB | 290 MiB |
| vector | 128 MiB | 512 MiB | 290 MiB |
| kong | 128 MiB | 768 MiB | 269 MiB |
| realtime | 128 MiB | 768 MiB | 300 MiB |
| auth, functions, imgproxy, meta, rest, storage, studio | 128 MiB each | 512 MiB each | 203 MiB (studio) |
| **Total** | **1.875 GiB** | **9.5 GiB** | |
| **Total** | **1.875 GiB** | **9 GiB** | |
The request is what the scheduler sets aside for a service and can never be given to anything else.
The limit is the ceiling the kernel kills the container at. They do different jobs here.
@@ -278,13 +278,21 @@ in. The limits are roughly twice the highest use measured across three running i
The earlier limits sat right on top of normal use and the kernel kept killing the two logging
services. Analytics (Logflare, which stores the logs Studio shows) used 612, 694 and 662 MiB of a
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that reads the
log files of every pod on the server and not only its own instance's, was killed 297 times on that
same instance, where a backlog it could not deliver kept growing in its memory. The two feed each
other: analytics dies, vector cannot deliver, vector fills up and dies, and on restart it replays
the backlog into analytics. Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next
in line. Image resizing (imgproxy) reads the uploads from the storage volume, so it has no image of
its own.
768 MiB ceiling and was killed 36 times on one instance. Vector, the log collector that ships the
instance's pod logs to analytics, was killed 297 times on that same instance, where a backlog it
could not deliver kept growing in its memory. The two feed each other: analytics dies, vector
cannot deliver, vector fills up and dies, and on restart it replays the backlog into analytics.
Realtime at 300 MiB of 384 and studio at 203 MiB of 256 were the next in line. Image resizing
(imgproxy) reads the uploads from the storage volume, so it has no image of its own.
On 2026-10-06 instance 3 died of the same pipeline in a different place: Logflare stores the logs
it receives in the `_supabase` database and never deletes them, so the 2 GB db image filled to
100% over eight days, Postgres crashed and could not even start its recovery, analytics answered
every delivery with an error, and vector was killed against its 1024 MiB ceiling every few
minutes with the undeliverable backlog. The fix is log retention: every instance gets a pg_cron
job (set up on create, upgrade and resume) that deletes analytics logs older than **2 days** every
night. That bounds both the db image and vector's backlog, so vector is back to the 512 MiB
ceiling the production supabase charts carry.
Changing the numbers in this table only affects instances created afterwards. To put them on
instances that already run, use `sudo supaspin upgrade`: it reads back the values Helm stored when