[ISS-4] Dispatch broker never cleans up /run/sources; tmpfs fills up and brings down Docker on scrum4me-server #117
Labels
No labels
severity/s2
severity/s3
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
janpeter/scrum4me-docker#117
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Status: closed (fixed) · Severity: s2_critical · Gemeld door: claude (max2) · Occurrences: 1 (laatst: 2026-10-08T18:38:38.731Z) · PBI: PBI-12 · Aangemaakt: 2026-10-08T18:38:38.731Z
Registratie
What happened (2026-10-08, ~20:05 CEST)
/runon scrum4me-server (tmpfs, 3.1 GB) was 100% full./run/sources(the broker'spreparedSourcesRoot) took up 3.0 GB: 214 per-dispatch folders plus 217 in.pending, the oldest dated 2026-10-02. Each folder holds a__repository_baseof ~22 MB, plus manifest, input and documents. None of them are removed after a terminal answer.Impact
docker execanddocker runon the host failed:OCI runtime exec failed: runc did not terminate successfully: exit status 255orfailed to write bundle spec ... config.json: no space left on device.dfeeffb3-15b4-46ec-88a5-0b1ea5309c21(job 8c08d1e1, CODEX, slot srv-review-1) andbae3e086-d7ea-47e5-94e8-d605f56b0cbc(job 5c8ad85a, CLAUDE, slot srv-review-claude-1). They ended upUNCERTAINafterlease_expiredand hold both review slots, so other WAITING requests are blocked until an operator cancels or recovers them.Temporary fix
Removed the folders older than 2026-10-08 in
/run/sourcesand.pendingby hand./rundropped to 5% (153 MB) and Docker works again. At the current rate (~0.5 GB per day) it will be full again in about 6 days.Needed
/run/sources/<id>and.pending/<id>once the dispatch is terminal (the same moment asartifacts.cleanup/ release after a terminal answer, M41 U3). The broker journals in/var/lib/s4m-dispatch/broker/*.jsonremain alongside them as well.DISPATCH_*pre-scope reason so the claim-bound close (T-1863) releases the slot instead ofUNCERTAIN./runusage on scrum4me-server.Onderzoek
2026-10-08T18:57:55.411Z — mac:claude
Root cause (scrum4me-docker @
ff00ae6)/run/sources/<attempt>and.pending/<attempt>/<manifestSha>/viastagePreparedSources(lib/dispatch-source-staging.ts:63, called bycreatePreparedSourcesProducerinlib/dispatch-prepared-sources.ts).broker.release(lib/dispatch-runtime-broker.ts:287) removes only the broker's own attempt directory underconfig.root.artifacts.cleanup(lib/dispatch-output.ts:105) removes only the supervisor journal. On thelatepath,closeClaimBoundand recovery, the staging is left behind too..pending/<attempt>also stays after a successful publish (an empty manifest-sha folder plus any interrupted runs), which explains why the.pendingcount roughly matches the folder count.stageResult(code collection reads the staging) and brokercreate(which copies the pins), the staging is no longer needed, so removing it after a terminal answer is safe.Current state (2026-10-08 ~21:00 CEST, mac:claude)
/runon scrum4me-srv is at 7% (198 MB)./run/sourcesis owned byUNKNOWN:s4m-dispatch-sup 0700, shared by the supervisorssrv-dispatch-supervisor-1andsrv-claude-dispatch-supervisor-1.Central service
The
/attempts/claim-stoproute accepts any^DISPATCH_[A-Z0-9_]{1,80}$reason (scrum4me-mcp src/dispatch/routes.ts:333), so a new pre-scope code needs no service change.2026-10-08T19:00:26.262Z — mac:claude
Plan goedgekeurd door JP (2026-10-08). Punten 1–3 gematerialiseerd als PBI-12 / ST-023 (sprint S-2026-08-24-1): T-82 terminale opruiming, T-83 wezen-sweep (48 uur, gedeelde root), T-84 ENOSPC →
DISPATCH_PREPARED_SOURCES_NO_SPACEals pre-scope reden. Punt 4 (monitoring van/run) staat apart als issue op het product scrum4me-server.2026-10-08T19:09:50.120Z — mac:claude
Fix gebouwd op branch
claude/open-issues-review-84c4cc(nog niet gepusht of gedeployed):f759182(terminale discard),02aea8f(uurlijkse sweep, 48 uur),93b4baf(ENOSPC →DISPATCH_PREPARED_SOURCES_NO_SPACE, pre-scope),c052317(review-fix voor de sweep). Tests groen, op 2 bestaande macOS-only failures na. Het issue blijft open tot de deploy op scrum4me-srv en het praktijkbewijs (het aantal mappen in/run/sources≈ het aantal lopende attempts).Uit de review, buiten scope: het
received-replaypad (lib/dispatch-supervisor.ts, het blokif(j.result){…if(response.status!=='accepted')return {outcome:'late'}) geeftlateterug zonderreleaseScope. Container en staging blijven daar dus staan, en omdat het journal blijft, ruimt de sweep ze ook niet op. Dit lek bestond al vóór ISS-4.2026-10-08T19:11:41.652Z — mac:claude
PR geopend: #118 (branch
fix/iss4-prepared-sources-cleanup→master). Sluiten na merge, deploy op scrum4me-srv en de controle van/run/sources.2026-10-08T19:12:53.293Z — mac:claude
Related leak found while reviewing the ISS-4 fix:
runDispatchAttempt(lib/dispatch-supervisor.ts) returnedlateon thephase==='received' && receiptIdreplay without callingreleaseScope. The broker container, the attempt directory (holding the child's provider key) and the/run/sourcesstaging stayed behind. The 48h orphan sweep does not catch them because the supervisor journal remains. Fixed in commitd77b152by releasing beforelate, the same as the staged and main paths (M41 U3). A regression test is inrelease after the terminal answer. Vitest 63/63 and typecheck:dispatch pass. PR: #119 (not merged yet).2026-10-08T19:25:31.529Z — mac:claude
PR #119 (release after a late answer on the received-journal replay) is merged into master as merge commit
e5f9d3e. The s4m-codex review was COMMENT with 0 BLOCKER and 0 MAJOR. The reviewer couldn't run Vitest themselves, so the local test run (63/63) was added to the PR as a comment.2026-10-08T19:30:42.789Z — mac:claude
PR #118 gemerged in master (
158eb47b), samen met #119 (release bij late op de received-replay). De laatste review van s4m-codex-reviewer: COMMENT, 0 blockers. Er bleef één LOW-finding open:armPreparedSourcesSweeplaat het interval lopen als het signaal al bij het wapenen geabort is. Inmain()is dat praktisch onschadelijk (het signaal is daar nooit al geabort en de timer isunref), maar het is nog niet opgelost. Het issue blijft open tot de deploy op scrum4me-srv en de controle van/run/sources.2026-10-09T06:14:31.827Z — mac:claude
Deploy gecontroleerd op 2026-10-09 (mac:claude, read-only):
fc1ce31;1974a190d199) metdiscardSources,armPreparedSourcesSweepenDISPATCH_PREPARED_SOURCES_NO_SPACEerin;/runstaat op 1% (7,3 MB) en/run/sourcesis leeg (0 mappen en 0 in.pending);Er heeft sinds de deploy nog geen attempt gedraaid; het opruimen na een terminale afloop is dus nog niet in productie gezien. Sluiten na de eerste review-dispatch waarna
/run/sourcesweer leeg is.2026-10-09T06:45:39.970Z — mac:claude
Bewijs in productie (2026-10-09, review-dispatches b749f8d5 en f2b739f7): tijdens de run hoorden de mappen in
/run/sourcesen.pendingprecies bij de lopende attempts (247a56ef…,5fd5e831…). De staging van de afgeronde Codex-attempt van b749f8d5 was direct na de terminale afloop weg./runbleef op 1% (9,4 MB).2026-10-09T06:52:10.036Z — mac:claude
Ook na de Claude-attempt (f2b739f7 /
247a56ef…) is de staging verwijderd. In/run/sourcesstaat alleen nog de map van de attempt die op dat moment loopt (ab692797…);/runstaat op 1%. Bewezen voor beide slots.Oplossing
2026-10-09T06:53:24.266Z — mac:claude
Opgelost in scrum4me-docker #118 (PBI-12 / ST-023), met reviewfixes en #119:
/run/sources/<attempt>en.pending/<attempt>bij elke terminale afloop;Gedeployed op scrum4me-srv op 2026-10-09 (dispatch-checkout
fc1ce31, supervisor-image1974a190d199). Bewezen met de review-dispatches b749f8d5 (CODEX) en f2b739f7 (CLAUDE): de staging van elke afgeronde attempt verdween direct, en alleen die van de lopende attempt bleef staan./runbleef op 1%. Monitoring van/runstaat apart in ISS-57 (scrum4me-server).