[ISS-4] Dispatch broker never cleans up /run/sources; tmpfs fills up and brings down Docker on scrum4me-server #117

Closed
opened 2026-10-08 20:45:42 +02:00 by janpeter · 0 comments
Owner

Beheerd door Scrum4Me — wijzigingen hier worden overschreven. Bron: https://thuis.jp-visser.nl/issues/cmuzvqvt70001qzbkfg5etre7

Status: closed (fixed) · Severity: s2_critical · Gemeld door: claude (max2) · Occurrences: 1 (laatst: 2026-10-08T18:38:38.731Z) · PBI: PBI-12 · Aangemaakt: 2026-10-08T18:38:38.731Z

Registratie

What happened (2026-10-08, ~20:05 CEST)

/run on scrum4me-server (tmpfs, 3.1 GB) was 100% full. /run/sources (the broker's preparedSourcesRoot) took up 3.0 GB: 214 per-dispatch folders plus 217 in .pending, the oldest dated 2026-10-02. Each folder holds a __repository_base of ~22 MB, plus manifest, input and documents. None of them are removed after a terminal answer.

Impact

  • Every docker exec and docker run on the host failed: OCI runtime exec failed: runc did not terminate successfully: exit status 255 or failed to write bundle spec ... config.json: no space left on device.
  • All exec healthchecks went unhealthy: forgejo, mcp-http, scrum4me-workers, agent-codex, scrum4us-core-api, litellm, copilot, digiplein and both egress containers.
  • Two QUEUE_REVIEW dispatches were claimed right at that moment and could not start a container: dfeeffb3-15b4-46ec-88a5-0b1ea5309c21 (job 8c08d1e1, CODEX, slot srv-review-1) and bae3e086-d7ea-47e5-94e8-d605f56b0cbc (job 5c8ad85a, CLAUDE, slot srv-review-claude-1). They ended up UNCERTAIN after lease_expired and hold both review slots, so other WAITING requests are blocked until an operator cancels or recovers them.

Temporary fix

Removed the folders older than 2026-10-08 in /run/sources and .pending by hand. /run dropped to 5% (153 MB) and Docker works again. At the current rate (~0.5 GB per day) it will be full again in about 6 days.

Needed

  1. The broker or supervisor removes /run/sources/<id> and .pending/<id> once the dispatch is terminal (the same moment as artifacts.cleanup / release after a terminal answer, M41 U3). The broker journals in /var/lib/s4m-dispatch/broker/*.json remain alongside them as well.
  2. A guard or sweep for orphans, for example a timer that removes folders without an open dispatch older than N hours.
  3. Consider: if preparing sources fails with ENOSPC, give a bounded DISPATCH_* pre-scope reason so the claim-bound close (T-1863) releases the slot instead of UNCERTAIN.
  4. Monitoring on /run usage on scrum4me-server.

Onderzoek


2026-10-08T18:57:55.411Z — mac:claude

Root cause (scrum4me-docker @ ff00ae6)

  • The supervisor writes /run/sources/<attempt> and .pending/<attempt>/<manifestSha>/ via stagePreparedSources (lib/dispatch-source-staging.ts:63, called by createPreparedSourcesProducer in lib/dispatch-prepared-sources.ts).
  • Nothing deletes them. broker.release (lib/dispatch-runtime-broker.ts:287) removes only the broker's own attempt directory under config.root. artifacts.cleanup (lib/dispatch-output.ts:105) removes only the supervisor journal. On the late path, closeClaimBound and recovery, the staging is left behind too.
  • .pending/<attempt> also stays after a successful publish (an empty manifest-sha folder plus any interrupted runs), which explains why the .pending count roughly matches the folder count.
  • After stageResult (code collection reads the staging) and broker create (which copies the pins), the staging is no longer needed, so removing it after a terminal answer is safe.

Current state (2026-10-08 ~21:00 CEST, mac:claude)

/run on scrum4me-srv is at 7% (198 MB). /run/sources is owned by UNKNOWN:s4m-dispatch-sup 0700, shared by the supervisors srv-dispatch-supervisor-1 and srv-claude-dispatch-supervisor-1.

Central service

The /attempts/claim-stop route accepts any ^DISPATCH_[A-Z0-9_]{1,80}$ reason (scrum4me-mcp src/dispatch/routes.ts:333), so a new pre-scope code needs no service change.


2026-10-08T19:00:26.262Z — mac:claude

Plan goedgekeurd door JP (2026-10-08). Punten 1–3 gematerialiseerd als PBI-12 / ST-023 (sprint S-2026-08-24-1): T-82 terminale opruiming, T-83 wezen-sweep (48 uur, gedeelde root), T-84 ENOSPC → DISPATCH_PREPARED_SOURCES_NO_SPACE als pre-scope reden. Punt 4 (monitoring van /run) staat apart als issue op het product scrum4me-server.


2026-10-08T19:09:50.120Z — mac:claude

Fix gebouwd op branch claude/open-issues-review-84c4cc (nog niet gepusht of gedeployed): f759182 (terminale discard), 02aea8f (uurlijkse sweep, 48 uur), 93b4baf (ENOSPC → DISPATCH_PREPARED_SOURCES_NO_SPACE, pre-scope), c052317 (review-fix voor de sweep). Tests groen, op 2 bestaande macOS-only failures na. Het issue blijft open tot de deploy op scrum4me-srv en het praktijkbewijs (het aantal mappen in /run/sources ≈ het aantal lopende attempts).

Uit de review, buiten scope: het received-replaypad (lib/dispatch-supervisor.ts, het blok if(j.result){…if(response.status!=='accepted')return {outcome:'late'}) geeft late terug zonder releaseScope. Container en staging blijven daar dus staan, en omdat het journal blijft, ruimt de sweep ze ook niet op. Dit lek bestond al vóór ISS-4.


2026-10-08T19:11:41.652Z — mac:claude

PR geopend: #118 (branch fix/iss4-prepared-sources-cleanup → master). Sluiten na merge, deploy op scrum4me-srv en de controle van /run/sources.


2026-10-08T19:12:53.293Z — mac:claude

Related leak found while reviewing the ISS-4 fix: runDispatchAttempt (lib/dispatch-supervisor.ts) returned late on the phase==='received' && receiptId replay without calling releaseScope. The broker container, the attempt directory (holding the child's provider key) and the /run/sources staging stayed behind. The 48h orphan sweep does not catch them because the supervisor journal remains. Fixed in commit d77b152 by releasing before late, the same as the staged and main paths (M41 U3). A regression test is in release after the terminal answer. Vitest 63/63 and typecheck:dispatch pass. PR: #119 (not merged yet).


2026-10-08T19:25:31.529Z — mac:claude

PR #119 (release after a late answer on the received-journal replay) is merged into master as merge commit e5f9d3e. The s4m-codex review was COMMENT with 0 BLOCKER and 0 MAJOR. The reviewer couldn't run Vitest themselves, so the local test run (63/63) was added to the PR as a comment.


2026-10-08T19:30:42.789Z — mac:claude

PR #118 gemerged in master (158eb47b), samen met #119 (release bij late op de received-replay). De laatste review van s4m-codex-reviewer: COMMENT, 0 blockers. Er bleef één LOW-finding open: armPreparedSourcesSweep laat het interval lopen als het signaal al bij het wapenen geabort is. In main() is dat praktisch onschadelijk (het signaal is daar nooit al geabort en de timer is unref), maar het is nog niet opgelost. Het issue blijft open tot de deploy op scrum4me-srv en de controle van /run/sources.


2026-10-09T06:14:31.827Z — mac:claude

Deploy gecontroleerd op 2026-10-09 (mac:claude, read-only):

  • de dispatch-checkout staat op fc1ce31;
  • beide supervisors draaien op een nieuwe image (1974a190d199) met discardSources, armPreparedSourcesSweep en DISPATCH_PREPARED_SOURCES_NO_SPACE erin;
  • /run staat op 1% (7,3 MB) en /run/sources is leeg (0 mappen en 0 in .pending);
  • er zijn geen fouten sinds de herstart om 05:32 CEST.

Er heeft sinds de deploy nog geen attempt gedraaid; het opruimen na een terminale afloop is dus nog niet in productie gezien. Sluiten na de eerste review-dispatch waarna /run/sources weer leeg is.


2026-10-09T06:45:39.970Z — mac:claude

Bewijs in productie (2026-10-09, review-dispatches b749f8d5 en f2b739f7): tijdens de run hoorden de mappen in /run/sources en .pending precies bij de lopende attempts (247a56ef…, 5fd5e831…). De staging van de afgeronde Codex-attempt van b749f8d5 was direct na de terminale afloop weg. /run bleef op 1% (9,4 MB).


2026-10-09T06:52:10.036Z — mac:claude

Ook na de Claude-attempt (f2b739f7 / 247a56ef…) is de staging verwijderd. In /run/sources staat alleen nog de map van de attempt die op dat moment loopt (ab692797…); /run staat op 1%. Bewezen voor beide slots.

Oplossing


2026-10-09T06:53:24.266Z — mac:claude

Opgelost in scrum4me-docker #118 (PBI-12 / ST-023), met reviewfixes en #119:

  • de supervisor verwijdert /run/sources/<attempt> en .pending/<attempt> bij elke terminale afloop;
  • een uurlijkse sweep ruimt wezen op die ouder zijn dan 48 uur;
  • een ENOSPC tijdens het klaarzetten sluit claim-bound af.

Gedeployed op scrum4me-srv op 2026-10-09 (dispatch-checkout fc1ce31, supervisor-image 1974a190d199). Bewezen met de review-dispatches b749f8d5 (CODEX) en f2b739f7 (CLAUDE): de staging van elke afgeronde attempt verdween direct, en alleen die van de lopende attempt bleef staan. /run bleef op 1%. Monitoring van /run staat apart in ISS-57 (scrum4me-server).

> Beheerd door Scrum4Me — wijzigingen hier worden overschreven. Bron: https://thuis.jp-visser.nl/issues/cmuzvqvt70001qzbkfg5etre7 Status: closed (fixed) · Severity: s2_critical · Gemeld door: claude (max2) · Occurrences: 1 (laatst: 2026-10-08T18:38:38.731Z) · PBI: PBI-12 · Aangemaakt: 2026-10-08T18:38:38.731Z ## Registratie ## What happened (2026-10-08, ~20:05 CEST) `/run` on scrum4me-server (tmpfs, 3.1 GB) was 100% full. `/run/sources` (the broker's `preparedSourcesRoot`) took up 3.0 GB: 214 per-dispatch folders plus 217 in `.pending`, the oldest dated 2026-10-02. Each folder holds a `__repository_base` of ~22 MB, plus manifest, input and documents. None of them are removed after a terminal answer. ## Impact - Every `docker exec` and `docker run` on the host failed: `OCI runtime exec failed: runc did not terminate successfully: exit status 255` or `failed to write bundle spec ... config.json: no space left on device`. - All exec healthchecks went unhealthy: forgejo, mcp-http, scrum4me-workers, agent-codex, scrum4us-core-api, litellm, copilot, digiplein and both egress containers. - Two QUEUE_REVIEW dispatches were claimed right at that moment and could not start a container: `dfeeffb3-15b4-46ec-88a5-0b1ea5309c21` (job 8c08d1e1, CODEX, slot srv-review-1) and `bae3e086-d7ea-47e5-94e8-d605f56b0cbc` (job 5c8ad85a, CLAUDE, slot srv-review-claude-1). They ended up `UNCERTAIN` after `lease_expired` and hold both review slots, so other WAITING requests are blocked until an operator cancels or recovers them. ## Temporary fix Removed the folders older than 2026-10-08 in `/run/sources` and `.pending` by hand. `/run` dropped to 5% (153 MB) and Docker works again. At the current rate (~0.5 GB per day) it will be full again in about 6 days. ## Needed 1. The broker or supervisor removes `/run/sources/<id>` and `.pending/<id>` once the dispatch is terminal (the same moment as `artifacts.cleanup` / release after a terminal answer, M41 U3). The broker journals in `/var/lib/s4m-dispatch/broker/*.json` remain alongside them as well. 2. A guard or sweep for orphans, for example a timer that removes folders without an open dispatch older than N hours. 3. Consider: if preparing sources fails with ENOSPC, give a bounded `DISPATCH_*` pre-scope reason so the claim-bound close (T-1863) releases the slot instead of `UNCERTAIN`. 4. Monitoring on `/run` usage on scrum4me-server. ## Onderzoek --- *2026-10-08T18:57:55.411Z — mac:claude* ## Root cause (scrum4me-docker @ ff00ae6) - The **supervisor** writes `/run/sources/<attempt>` and `.pending/<attempt>/<manifestSha>/` via `stagePreparedSources` (`lib/dispatch-source-staging.ts:63`, called by `createPreparedSourcesProducer` in `lib/dispatch-prepared-sources.ts`). - Nothing deletes them. `broker.release` (`lib/dispatch-runtime-broker.ts:287`) removes only the broker's own attempt directory under `config.root`. `artifacts.cleanup` (`lib/dispatch-output.ts:105`) removes only the supervisor journal. On the `late` path, `closeClaimBound` and recovery, the staging is left behind too. - `.pending/<attempt>` also stays after a successful publish (an empty manifest-sha folder plus any interrupted runs), which explains why the `.pending` count roughly matches the folder count. - After `stageResult` (code collection reads the staging) and broker `create` (which copies the pins), the staging is no longer needed, so removing it after a terminal answer is safe. ## Current state (2026-10-08 ~21:00 CEST, mac:claude) `/run` on scrum4me-srv is at 7% (198 MB). `/run/sources` is owned by `UNKNOWN:s4m-dispatch-sup 0700`, shared by the supervisors `srv-dispatch-supervisor-1` and `srv-claude-dispatch-supervisor-1`. ## Central service The `/attempts/claim-stop` route accepts any `^DISPATCH_[A-Z0-9_]{1,80}$` reason (`scrum4me-mcp src/dispatch/routes.ts:333`), so a new pre-scope code needs no service change. --- *2026-10-08T19:00:26.262Z — mac:claude* Plan goedgekeurd door JP (2026-10-08). Punten 1–3 gematerialiseerd als PBI-12 / ST-023 (sprint S-2026-08-24-1): T-82 terminale opruiming, T-83 wezen-sweep (48 uur, gedeelde root), T-84 ENOSPC → `DISPATCH_PREPARED_SOURCES_NO_SPACE` als pre-scope reden. Punt 4 (monitoring van `/run`) staat apart als issue op het product scrum4me-server. --- *2026-10-08T19:09:50.120Z — mac:claude* Fix gebouwd op branch `claude/open-issues-review-84c4cc` (nog niet gepusht of gedeployed): f759182 (terminale discard), 02aea8f (uurlijkse sweep, 48 uur), 93b4baf (ENOSPC → `DISPATCH_PREPARED_SOURCES_NO_SPACE`, pre-scope), c052317 (review-fix voor de sweep). Tests groen, op 2 bestaande macOS-only failures na. Het issue blijft open tot de deploy op scrum4me-srv en het praktijkbewijs (het aantal mappen in `/run/sources` ≈ het aantal lopende attempts). Uit de review, buiten scope: het `received`-replaypad (`lib/dispatch-supervisor.ts`, het blok `if(j.result){…if(response.status!=='accepted')return {outcome:'late'}`) geeft `late` terug zonder `releaseScope`. Container en staging blijven daar dus staan, en omdat het journal blijft, ruimt de sweep ze ook niet op. Dit lek bestond al vóór ISS-4. --- *2026-10-08T19:11:41.652Z — mac:claude* PR geopend: https://git.jp-visser.nl/janpeter/scrum4me-docker/pulls/118 (branch `fix/iss4-prepared-sources-cleanup` → `master`). Sluiten na merge, deploy op scrum4me-srv en de controle van `/run/sources`. --- *2026-10-08T19:12:53.293Z — mac:claude* Related leak found while reviewing the ISS-4 fix: `runDispatchAttempt` (lib/dispatch-supervisor.ts) returned `late` on the `phase==='received' && receiptId` replay without calling `releaseScope`. The broker container, the attempt directory (holding the child's provider key) and the `/run/sources` staging stayed behind. The 48h orphan sweep does not catch them because the supervisor journal remains. Fixed in commit d77b152 by releasing before `late`, the same as the staged and main paths (M41 U3). A regression test is in `release after the terminal answer`. Vitest 63/63 and typecheck:dispatch pass. PR: https://git.jp-visser.nl/janpeter/scrum4me-docker/pulls/119 (not merged yet). --- *2026-10-08T19:25:31.529Z — mac:claude* PR #119 (release after a late answer on the received-journal replay) is merged into master as merge commit e5f9d3e. The s4m-codex review was COMMENT with 0 BLOCKER and 0 MAJOR. The reviewer couldn't run Vitest themselves, so the local test run (63/63) was added to the PR as a comment. --- *2026-10-08T19:30:42.789Z — mac:claude* PR #118 gemerged in master (158eb47b), samen met #119 (release bij late op de received-replay). De laatste review van s4m-codex-reviewer: COMMENT, 0 blockers. Er bleef één LOW-finding open: `armPreparedSourcesSweep` laat het interval lopen als het signaal al bij het wapenen geabort is. In `main()` is dat praktisch onschadelijk (het signaal is daar nooit al geabort en de timer is `unref`), maar het is nog niet opgelost. Het issue blijft open tot de deploy op scrum4me-srv en de controle van `/run/sources`. --- *2026-10-09T06:14:31.827Z — mac:claude* Deploy gecontroleerd op 2026-10-09 (mac:claude, read-only): - de dispatch-checkout staat op `fc1ce31`; - beide supervisors draaien op een nieuwe image (`1974a190d199`) met `discardSources`, `armPreparedSourcesSweep` en `DISPATCH_PREPARED_SOURCES_NO_SPACE` erin; - `/run` staat op 1% (7,3 MB) en `/run/sources` is leeg (0 mappen en 0 in `.pending`); - er zijn geen fouten sinds de herstart om 05:32 CEST. Er heeft sinds de deploy nog geen attempt gedraaid; het opruimen na een terminale afloop is dus nog niet in productie gezien. Sluiten na de eerste review-dispatch waarna `/run/sources` weer leeg is. --- *2026-10-09T06:45:39.970Z — mac:claude* **Bewijs in productie** (2026-10-09, review-dispatches b749f8d5 en f2b739f7): tijdens de run hoorden de mappen in `/run/sources` en `.pending` precies bij de lopende attempts (`247a56ef…`, `5fd5e831…`). De staging van de afgeronde Codex-attempt van b749f8d5 was direct na de terminale afloop weg. `/run` bleef op 1% (9,4 MB). --- *2026-10-09T06:52:10.036Z — mac:claude* Ook na de Claude-attempt (f2b739f7 / `247a56ef…`) is de staging verwijderd. In `/run/sources` staat alleen nog de map van de attempt die op dat moment loopt (`ab692797…`); `/run` staat op 1%. Bewezen voor beide slots. ## Oplossing --- *2026-10-09T06:53:24.266Z — mac:claude* Opgelost in scrum4me-docker #118 (PBI-12 / ST-023), met reviewfixes en #119: - de supervisor verwijdert `/run/sources/<attempt>` en `.pending/<attempt>` bij elke terminale afloop; - een uurlijkse sweep ruimt wezen op die ouder zijn dan 48 uur; - een ENOSPC tijdens het klaarzetten sluit claim-bound af. Gedeployed op scrum4me-srv op 2026-10-09 (dispatch-checkout `fc1ce31`, supervisor-image `1974a190d199`). Bewezen met de review-dispatches b749f8d5 (CODEX) en f2b739f7 (CLAUDE): de staging van elke afgeronde attempt verdween direct, en alleen die van de lopende attempt bleef staan. `/run` bleef op 1%. Monitoring van `/run` staat apart in ISS-57 (scrum4me-server). <!-- s4m:issue:cmuzvqvt70001qzbkfg5etre7 -->
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
janpeter/scrum4me-docker#117
No description provided.