Engineering a Multi-Agent System for Production PR Reviews

Managing a high volume of pull requests often leads to "reviewer fatigue," where the quality of the 20th review rarely matches the energy of the first. Traditional AI solutions that simply dump a code diff into a single LLM prompt often result in hallucinations and "noise," failing to provide the selectivity senior engineers actually need. By building a multi-agent system grounded in a robust memory layer, teams can automate mechanical review tasks and reclaim senior attention for high-value architectural judgments.

The Context: Why Simple Prompts Fail

Traditional automation, such as linters or single-prompt LLM reviewers, lacks grounding—the deep context of the entire repository and past team decisions. A single prompt attempting to judge security, quality, and testing simultaneously often becomes overwhelmed, leading to maximal output rather than high-value findings. To solve this, we must transition from "automated reviews" to an agentic system that mirrors human reasoning.

The Meat: Architecture & Implementation

The system follows a fan-out/fan-in design pattern, using specialized agents and a unified data spine.

1. Asynchronous Ingress Service

GitHub webhooks require a fast acknowledgement (typically under 10 seconds), while LLM processing can take over a minute.

• Action: Build a stateless ingress service using FastAPI to validate the X-Hub-Signature-256 and queue the job into Reddis (ARQ).

• Invariant: Every payload must be verified against a shared secret before processing to prevent forged requests.

2. Multi-Agent Orchestration

Using LangGraph, define a directed graph where an Orchestrator fans out the PR to four specialist agents in parallel:

• Security Agent: Scans for vulnerabilities like SQL injection.

• Quality Agent: Checks logic, design patterns, and standards.

• Testing Agent: Identifies missing edge cases and coverage gaps.

• Docs Agent: Ensures readability and documentation alignment.

3. The Unified Memory Spine

Grounding is achieved by providing agents with three types of memory stored in Tiger Cloud (a managed PostgreSQL-compatible store with PG Vector and TimeScaleDB extensions):

• Semantic Memory: The codebase itself, stored as vector embeddings for similarity searches.

• Episodic Memory: Past reviews and disputed findings to learn from history.

• Procedural Memory: Team-specific rules and architectural conventions.

4. Aggregation & The HITL Gate

The Aggregator fanned-in findings, de-duplicates them, and computes a confidence score.

• Confidence > 0.6: Post the structured review directly to GitHub.

• Confidence < 0.6 or Critical Finding: Route to a Human-in-the-Loop (HITL) approval queue.

# Abstract interface for the Workflow Engine
class WorkflowEngine(ABC):
    @abstractmethod
    async def run_workflow(self, pr_id: str):
        pass

    @abstractmethod
    async def get_state(self, pr_id: str):
        pass

The Gotchas: Edge Cases & Trade-offs

• The "Almost Right" Problem: Agents may be 90% correct but subtly wrong. Mitigation requires rationales for every finding so they are auditable and disputable.

• Vector Dimension Mismatch: A common error occurs if the embedding dimension in your .env (e.g., 256) does not match your PG Vector schema (e.g., 1536), causing insertion failures.

• Orchestration Deadlock: If one specialist agent hangs, the aggregator may wait forever. Use timeouts on every node in your graph to ensure the system degrades gracefully.

• Idempotency: GitHub may retry webhook deliveries. Use an unaltered key (delivery ID) to de-duplicate jobs and avoid posting the same review twice.