Scrum4Me/docs/runbooks/review-plan-job.md

Review-Plan Job Orchestration

Implementation guide for the IDEA_REVIEW_PLAN job kind and multi-model iterative plan review.

---

Overview

The review-plan job is an autonomous agent that performs iterative multi-model review of implementation plans (YAML frontmatter + markdown documents). It coordinates three review stages (structure, logic/patterns, risk assessment), detects convergence, and either approves the plan or returns it for manual refinement.

Job Kind: IDEA_REVIEW_PLAN Triggerable From: PLAN_READY, PLAN_REVIEWED (re-review) Transitions To: PLAN_REVIEWED (approved) or PLAN_REVIEW_FAILED (rejected/abandoned)

---

System Design

Data Flow

User clicks "Review Plan" on PLAN_READY idea
  ↓
startReviewPlanJobAction() queues IDEA_REVIEW_PLAN job
  ↓
Worker claims job via wait_for_job (MCP)
  ↓
Review-plan prompt orchestrates:
  - Ronde 1: Structure check (YAML parsing, format correctness)
  - Ronde 2: Logic & patterns (dependencies, architecture fit)
  - Ronde 3: Risk assessment (edge cases, refactoring, type-safety)
  ↓
Convergence detection: if stable, ask approval
  ↓
On approval: update_idea_plan_reviewed(approval_status='approved')
  → Idea transitions to PLAN_REVIEWED
  → IdeaLog entry created with PLAN_REVIEW_RESULT
  ↓
On rejection: return for manual edit (status → PLAN_REVIEW_FAILED)

Review-Log JSON Schema

The orchestrator produces a detailed JSON log stored in idea.plan_review_log:

interface ReviewLog {
  plan_file: string;           // Idea code (e.g., "I-042")
  created_at: ISO8601;         // Review start timestamp
  
  rounds: Array<{
    round: number;             // 0, 1, 2 (structure, logic, risk)
    model: string;             // claude-3-5-haiku | claude-3-5-sonnet | claude-opus-4-7
    role: string;              // "Structure Review" | "Logic & Patterns" | "Risk Assessment"
    focus: string;             // Review focus summary
    plan_before: string;       // Original plan_md at round start
    plan_after: string;        // Revised plan after feedback
    issues: Array<{
      category: 'structure' | 'logic' | 'risk' | 'pattern';
      severity: 'error' | 'warning' | 'info';
      suggestion: string;      // Concrete fix recommendation
    }>;
    score: number;             // 0-100 review score
    plan_diff_lines: number;   // Changed lines in this round
    converged: boolean;        // Did this round trigger convergence?
    timestamp: ISO8601;        // Round completion time
  }>;
  
  convergence?: {
    stable_at_round: number;   // Round where convergence was detected
    final_diff_pct: number;    // Percentage of changed lines at convergence
    convergence_metric: string; // "plan_stability" (constant for now)
  };
  
  approval: {
    status: 'pending' | 'approved' | 'rejected';
    timestamp?: ISO8601;       // When user made decision
  };
  
  summary: string;            // 1–2 sentence summary for IdeaLog
}

---

Assumptions & Constraints

Prompt Assumptions

1. Plan Format: Idea's plan_md field contains YAML frontmatter (parsed at PLAN_READY) + markdown body.

   • Frontmatter keys: pbi, stories, tasks, priority, verify_required.
   • If parse fails, orchestrator transitions idea to PLAN_REVIEW_FAILED.

2. Context Availability: The job payload includes:

   • idea.plan_md: The plan to review (required)
   • idea.grill_md: Context from grill phase (optional but recommended)
   • product.definition_of_done: Product-level acceptance criteria
   • repo_url: Local repository for pattern inspection

3. User Availability: At least one worker is active (server-side check via countActiveWorkers).

4. No External APIs: Orchestrator performs reviews entirely with information from job context. No external codex or multi-model APIs are called directly.

   • Future improvement: Codex-injection from docs/patterns/**/*.md and docs/architecture/**/*.md.

Convergence Detection Assumptions

1. Stability Metric: Two consecutive rounds with < 5% line changes = convergence.

   • Threshold is hardcoded; future: make configurable per product.
   • Diff percentage = (changed_lines / total_lines) * 100.

2. Max Iterations: 3 initial rounds + 2 optional extra rounds (total max 5) before forced approval.

3. No Infinite Loops: If max iterations reached, approval gate enforces a decision.

Validation Assumptions

1. Plan is Mutable: Orchestrator can revise plan_md between rounds without breaking downstream parsing.

   • If YAML structure is corrupted, parsePlanMd (server-side) will fail on approval.
   • Orchestrator should never corrupt YAML syntax.

2. IdeaLog Persistence: MCP tool update_idea_plan_reviewed atomically saves:

   • idea.plan_review_log (full JSON)
   • idea.reviewed_at (timestamp)
   • idea.status (transition)
   • IdeaLog entry (audit)

3. User Decisions are Final: Once approved, plan-review log is immutable (until next re-review).

---

Implementation Details

Prompt Location

• Main Repo: lib/idea-prompts/review-plan-job.md
• MCP Server: scrum4me-mcp/src/prompts/idea/review-plan.md
• Synchronization: Manual (for now); future: sync-schema.sh-like mechanism.

Job Config Snapshot

Job created with config from lib/job-config.ts:

IDEA_REVIEW_PLAN: {
  model: 'claude-opus-4-7',       // Opus for final orchestration
  thinking_budget: 6000,          // Extended for multi-round analysis
  permission_mode: 'acceptEdits',
  max_turns: 1,
  allowed_tools: [
    'Read', 'Write', 'Grep', 'Glob',
    'mcp__scrum4me__update_idea_plan_reviewed',
    'mcp__scrum4me__log_idea_decision',
    'mcp__scrum4me__update_job_status',
    'mcp__scrum4me__ask_user_question',
  ],
}

Note: Model is fixed to Opus for orchestration. Individual review rounds are simulated (not actual model switching) within Opus's analysis. Future: Direct multi-model support via Claude API.

MCP Tool: update_idea_plan_reviewed

Location: scrum4me-mcp/src/tools/update-idea-plan-reviewed.ts

Input:

{
  idea_id: string;
  review_log: object;  // Full ReviewLog JSON
  approval_status?: 'pending' | 'approved' | 'rejected';
}

Behavior:

1. Validates user owns idea.
2. Transitions idea status:
   • approval_status='approved' → PLAN_REVIEWED
   • approval_status='rejected' → PLAN_REVIEW_FAILED
   • Default → PLAN_REVIEWED
3. Saves plan_review_log and reviewed_at atomically.
4. Creates IdeaLog entry with type PLAN_REVIEW_RESULT.

---

Dependencies

Database

• Idea Model: Must have fields plan_review_log (Json), reviewed_at (DateTime).
• IdeaStatus Enum: Must include REVIEWING_PLAN, PLAN_REVIEW_FAILED, PLAN_REVIEWED.
• IdeaLogType Enum: Must include PLAN_REVIEW_RESULT.

Server Actions

• startReviewPlanJobAction() — Queues job, enforces status transitions.
• cancelIdeaJobAction() — Allows user to cancel mid-review (reverts to PLAN_READY).

MCP Tools

• update_idea_plan_reviewed() — Saves review-log and transitions status.
• log_idea_decision() — Logs convergence/approval decisions.
• update_job_status() — Marks job as done/failed.
• ask_user_question() — Approval gate interaction.

Files

• lib/idea-prompts/review-plan-job.md — Orchestrator prompt.
• scrum4me-mcp/src/prompts/idea/review-plan.md — MCP server copy.
• scrum4me-mcp/src/lib/kind-prompts.ts — Prompt loader.
• scrum4me-mcp/src/tools/wait-for-job.ts — Job context builder.

---

Error Handling

Parse Failures

If plan_md cannot be parsed as valid YAML frontmatter:

1. Orchestrator logs error in review_log.
2. Calls update_job_status('failed', error: 'plan_parse_failed').
3. Idea remains in REVIEWING_PLAN (no transition).
4. User can manually edit plan_md and retry.

User Cancellation

If user cancels job via UI:

1. Server sets job status → CANCELLED.
2. Worker receives no further answer from ask_user_question.
3. Orchestrator gracefully saves partial review_log.
4. Calls update_job_status('skipped', ...).
5. Idea reverts to PLAN_READY.

Question Timeout

If approval question expires (24h):

1. Orchestrator logs timeout in review_log.
2. Calls update_job_status('failed', error: 'approval_timeout').
3. Idea reverts to PLAN_READY.

---

Testing Strategy

Unit Tests

• Mock ReviewLog Generation: Verify review-log JSON structure matches schema.
• Convergence Calculation: Diff percentage computation, stability threshold.
• Status Transitions: Valid state machine paths (PLAN_READY → REVIEWING_PLAN → PLAN_REVIEWED).

Integration Tests

• End-to-End: Draft idea → Grill → Plan → Review → PLAN_REVIEWED.
• Re-Review: PLAN_REVIEWED → REVIEWING_PLAN → PLAN_REVIEWED (no data loss).
• Cancellation: Mid-review cancellation → revert to PLAN_READY.
• Parse Errors: Malformed plan_md → PLAN_REVIEW_FAILED.

Manual Testing

1. Create test idea with PLAN_READY status.
2. Click "Review Plan".
3. Monitor job in Jobs dashboard.
4. Verify review-log in idea detail page.
5. Accept/reject approval.
6. Confirm status transition and IdeaLog entry.

---

Future Enhancements

1. Direct Multi-Model Calls: Use Claude API to invoke Haiku, Sonnet, Opus separately with model switching.
2. Codex Injection: Auto-load and inject docs/patterns/**/*.md and docs/architecture/**/*.md as context.
3. Configurable Thresholds: Allow product-level convergence percentage and max-rounds settings.
4. Review History: Preserve all review-logs for audit trail and re-review diffs.
5. Feedback Loop: Log user edits between review rounds and suggest re-run based on delta.
6. Scheduled Re-Review: Auto-trigger review after N days (staleness check).

---

References

• docs/architecture/jobs.md — Job system architecture.
• docs/patterns/server-action.md — Server action pattern (startReviewPlanJobAction).
• docs/api/rest-contract.md — API surface for plan-review.
• lib/idea-status.ts — Status transition graph and state machine.
• lib/idea-plan-parser.ts — Plan YAML parsing (validator for approved plans).