Gate AI agent regressions on every pull request.
Drop agentdiff-check into your CI pipeline. Diff candidate execution graphs against golden baselines, post rich diagnostic blame trees to PR comments, and lock the merge button if thresholds fail.
Drop into any repo in 30 seconds.
Generate your gate in one command with agentdiff init --with-approve. If divergence, cost, or retry loops breach tolerance, the action fails with exit code 1 and posts a human-first PR verdict.
name: AgentDiff Gate
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Record Candidate Trace
run: |
pip install agent-trajectory-diff
agentdiff record my_agent:run \
--input '{"scenario": "customer_support"}' \
--output traces/pr_candidate.json
- name: Gate Against Statistical Baseline Envelope
uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
with:
baseline: baselines/customer_support.envelope.json
candidate: traces/pr_candidate.json
pr: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}What happens on every pull request.
Three deterministic safeguards run on every push, giving your team actionable feedback before bad code hits main.
Rich PR Diagnostic Commentary
Posts an interactive markdown summary directly to the PR with exact divergence ratios, step-by-step diff trees, token spikes, and isolated culprit root-cause lines.
Exit Code 1 Merge Blocker
Emits strict non-zero exit codes upon regression, automatically failing the GitHub check suite and preventing engineers from merging broken agent behaviors.
Interactive /agentdiff approve Bot
Reviewers bless intentional improvements directly by commenting /agentdiff approve. Candidate runs join the baseline envelope without local checkouts or manual JSON edits.
Bless improvements without leaving GitHub
A team member with write access comments /agentdiff approve on a PR flagged for path drift.
Path drift and cost deltas are blessable; infinite loops are never blessable. The bot refuses to bless cyclical bugs.
The bot updates the committed baseline envelope, pushes the commit, and flips the GitHub Check result to green.
Did the PR pass — or did the gate move?
When developers weaken test thresholds (e.g. raising max_divergence from 0.25 to 0.75 to force a broken agent to pass), AgentDiff compares the PR configuration against the main branch baseline config.
Threshold alterations are flagged as high-priority security warnings in the PR review timeline, preventing silent regression masking.
- name: Fetch Baseline Policy from Main
run: git show origin/main:agentdiff.toml > /tmp/baseline-agentdiff.toml
- name: Gate Against Baseline with Governance Guard
uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
with:
baseline: baselines/golden_run.json
candidate: traces/pr_candidate.json
baseline-config: /tmp/baseline-agentdiff.toml # Flags loosened thresholds
pr: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}Complete Action inputs specification.
Everything configurable in agentdiff-check:
| Parameter | Requirement | Description |
|---|---|---|
| baseline | Required | Path to the committed golden baseline trace JSON in your repository. |
| candidate | Required | Path to the candidate trace JSON generated by the agent during the current PR workflow run. |
| pr | Optional | Pull request number (e.g. ${{ github.event.number }}) to post the formatted divergence report and culprit blame tree to. |
| baseline-config | Optional | Path to the main branch agentdiff.toml. Enables the Goodhart guard to detect if PR authors silently weakened tolerance thresholds. |
| adapter | Optional | Telemetry parser: auto (default), langgraph, crewai, openai_agents, openinference, langfuse, langsmith, generic. |
| update-baseline | Optional | When true, automatically advances the stored golden baseline trace if the PR trajectory verdict passes all quality gates. |
| github-token | Optional | GitHub secret token (e.g. ${{ secrets.GITHUB_TOKEN }}) with pull-requests: write permission for automated commentary. |
Record. Diff. Gate. Ship with confidence.
Add one GitHub Action step to automate trajectory regression testing across your engineering team in under 5 minutes.