According to Linear, teams using AI agents now ship 5x as many pull requests as two years ago, while Cursor’s data shows developers produce 2.5x as much code as 18 months ago.
More code and faster shipping make documentation drift from the product. Documentation describes one behavior, while code produces another. This hinders adoption, breaks onboarding workflows, and can drive prospective users to competitors or frustrate existing ones.
Tools such as Mintlify offer features to catch documentation drift and automatically update your docs. However, Mintlify’s implementation may not fit your existing workflow, and it locks you into its platform. This guide shows you how to build the same capability without locking into any specific product.
This guide walks through the GitHub Actions workflow we use to catch documentation drift on pull requests with Pi, an open source agent harness. You can replicate the workflow on your own repository.
We’re already running this setup on VectorLint, an open source content review harness we maintain at TinyRocket.
Let’s get started.
What You’ll Build
By the end of this guide, you’ll be able to detect documentation drift in CI. You’ll have three pieces wired together:
- A GitHub App installed on the repository that gives the workflow permission to post comments
- An agent skill that tells the agent what counts as documentation drift
- A GitHub Actions workflow that orchestrates the check on demand, builds the agent’s routing message inline, and posts the result as a PR comment
Note: The workflow flags documentation drift; it doesn’t create a PR or issue. That’s intentional. There could be false positives where the agent flags drift that isn’t user-facing. Automatically creating a pull request for each false positive only adds review work. You can build a downstream workflow that takes this workflow’s output and creates an issue or PR with an agent.
Before You Start
You need a few things in place:
- A GitHub repository following a Docs as Code approach (documentation alongside the code, for example,
/docsor a README markdown file) - Owner or member permission on the repository
- A GitHub account with permission to create GitHub Apps (or an organization where you can create Apps)
- An API key from an AI provider that Pi supports (Anthropic, OpenAI, Google, and others). This guide uses Amazon Bedrock. If you use a different provider, the only change is the authentication parameters.
Pi is an open source coding agent that you run from the command line. It’s lightweight, provider-agnostic, and highly customizable, which is why we chose it for this guide. If you want to test the skill before committing it to CI, install Pi locally.
Note: You can replace Pi with any other agent and get similar results. You’ll need to adapt the setup to how your agent works.
Step 1: Create the Doc-Drift Skill
Copy the doc-drift skill from the VectorLint repository into your project at .agents/skills/doc-drift/. The directory contains:
SKILL.md: the main instruction the agent followsreferences/user-facing-criteria.md: defines what counts as a user-facing changereferences/comment.md: defines the output format for drift reports
The skill tells the agent to extract user-facing behavioral changes from a PR diff, search the documentation for claims those changes invalidate, and write a report for each finding.
A user-facing change could be a renamed command-line flag, a new environment variable, a changed config key, or a rewritten error message. The skill ignores internal refactors, test changes, and anything with no observable effect on the user.
Note: We built this skill for VectorLint, so it has hardcoded scope paths and repository references. To adapt it, change the scope paths and repository name in
SKILL.mdand update the criteria inreferences/user-facing-criteria.mdto match your product. You can also hand the skill to an agent and ask it to adapt the references to your codebase.
Step 2: Create and Install a GitHub App
The workflow needs to post comments on pull requests. You can authenticate with either a personal access token or a GitHub App. A GitHub App is safer because you scope its permissions to exactly what the workflow needs. A personal access token tends to carry more permissions than necessary, which increases the blast radius if it’s ever exposed.
Use GitHub’s guide on creating a GitHub App and grant it these permissions:
- Issues: Read & write to post PR comments
- Pull requests: Read & write for the eyes reaction and
gh pr diff - Contents: Read for checkout
After creating the App, grab the App ID and download your private key (a .pem file). Then install the GitHub App on the repository where the workflow runs. You’ll add both values as repository secrets in Step 4.
Step 3: Build the GitHub Actions Workflow
Create a file at .github/workflows/doc-drift.yml. Each section below covers one block of this file. The full workflow appears at the end for copy-pasting.
The Trigger and the Authorization Guard
To guard against cost and security overhead, a documentation check should only run when someone triggers it on a pull request, not on every push.
name: Doc Drift Check
on:
issue_comment:
types: [created]
jobs:
authorize:
name: Check authorization
runs-on: ubuntu-latest
# Only run on PR comments that start with /check-docs (skip bots)
if: |
github.event.issue.pull_request != null &&
startsWith(github.event.comment.body, '/check-docs') &&
!contains(github.event.comment.user.login, '[bot]')
permissions:
issues: write
pull-requests: read
contents: read
outputs:
authorized: ${{ steps.auth.outputs.authorized }}
The on: issue_comment trigger fires whenever a comment is created. The if: condition narrows triggering to comments on pull requests that start with /check-docs and skips any commenter whose login contains [bot], so bot accounts cannot trigger the workflow. The permissions block grants the minimum access the job needs; write access for comments comes from a GitHub App token (covered below), so GITHUB_TOKEN only needs read access to pull requests.
To guard against unauthorized reviews, only repository owners and members should be able to trigger the drift check. This limits the cost of running AI models and reduces the surface area for prompt injection.
steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- name: Check authorization
id: auth
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const association = context.payload.comment.author_association;
if (!['OWNER', 'MEMBER'].includes(association)) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: `@${context.payload.comment.user.login} Only repo owners or members can trigger doc drift checks.`
});
core.setOutput('authorized', 'false');
} else {
core.setOutput('authorized', 'true');
}
The step reads author_association from the comment payload and allows OWNER and MEMBER. If the commenter is neither, it posts a rejection comment and sets authorized to false. Otherwise, it sets the output to true, allowing downstream jobs to proceed.
Why a GitHub App token, not GITHUB_TOKEN: As covered in Step 2, issue_comment triggers on public repositories downgrade GITHUB_TOKEN to read-only. The App token you created in Step 2 bypasses this restriction, so every write step in the workflow uses it instead.
This mitigates the risk of prompt injection since pull request titles, bodies, and comments are user-controlled strings.
Note: Running an agent in a workflow carries some security risk. A malicious actor could create a pull request containing instructions to get your agent to expose secrets. To avoid that, only grant the minimum permissions your agents need to run. For more on this, read GitHub’s guide on mitigating cloud agent risk.
Check Out the PR and Fetch the Diff
The agent needs the diff to know what changed in the pull request and to check for behavioral changes.
check-docs:
name: Check documentation drift
runs-on: ubuntu-latest
needs: authorize
if: needs.authorize.outputs.authorized == 'true'
permissions:
issues: write
pull-requests: read
contents: read
steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- name: React to comment
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
content: 'eyes'
});
- name: Get PR head SHA
id: pr
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number
});
core.setOutput('head_sha', pr.data.head.sha);
- name: Checkout main branch
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ github.event.repository.default_branch }}
path: main
fetch-depth: 0
- name: Checkout PR branch
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ steps.pr.outputs.head_sha }}
path: pr
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: '22'
- name: Install Pi
run: npm install -g @earendil-works/pi-coding-agent@0.79.6
- name: Show Pi version
run: pi --version
- name: Fetch PR diff
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
DIFF_PATH="$GITHUB_WORKSPACE/pr/.doc-drift-input.diff"
gh pr diff --repo ${{ github.repository }} ${{ github.event.issue.number }} > "$DIFF_PATH"
echo "Diff size: $(wc -l < "$DIFF_PATH") lines"
The eyes reaction gives the reviewer immediate feedback that the command landed, before the run finishes. The Get PR head SHA step resolves the pull request’s current head commit, so the checkout targets the right ref.
The workflow checks out two copies of the repository: main (the default branch, into path: main) holds the doc-drift skill and the current documentation the agent checks against; pr (the pull request’s head commit, into path: pr) holds the changed code. fetch-depth: 0 gives the agent the full repository history, which it uses to reason about how code and documentation relate. setup-node installs Node version 22, which Pi requires (@earendil-works/pi-coding-agent@0.79.6 needs Node ≥22.19.0). Pi is installed at a pinned version (@0.79.6) for reproducibility. The diff is written to pr/.doc-drift-input.diff so the agent can read it from the PR checkout.
All actions are pinned to full commit SHAs for supply chain safety.
Build the Agent Message
You need to tell Pi where to find the diff, which skill to invoke, and where to write its reports. Rather than maintaining a separate script, the workflow builds this routing message with an inline bash heredoc.
- name: Build agent message
run: |
WORKSPACE="${{ github.workspace }}/pr"
DIFF_PATH="$WORKSPACE/.doc-drift-input.diff"
cat > /tmp/pi-message.txt <<EOF
You are running a doc drift check on a pull request in the VectorLint repository.
The pull request checkout to inspect is located at:
${WORKSPACE}
Read the PR diff from this file:
${DIFF_PATH}
Use the doc-drift skill. When you have finished, write one report file
per behavioral change you identified, named sequentially:
${WORKSPACE}/.doc-drift-1.md
${WORKSPACE}/.doc-drift-2.md
... and so on.
If there are no issues to report, write a single file ${WORKSPACE}/.doc-drift-1.md
containing the no-issues-found report.
Do not post anything to GitHub directly. The workflow will handle posting.
EOF
The heredoc produces a routing message that tells Pi where the PR checkout lives, where to read the diff, and where to write its reports. The variables expand at runtime against github.workspace.
The doc-drift skill it references lives at .agents/skills/doc-drift/, the same skill you created in Step 1.
Run the Drift Check
With the diff on disk, the routing message ready, and Node.js available, the workflow can run Pi and hand it the routing message. This is where the agent reasons about whether documentation needs updating.
A CI runner has no interactive terminal, so Pi runs in a headless, non-interactive mode.
- name: Run doc drift check
working-directory: pr
env:
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.PI_BEDROCK_API_KEY }}
AWS_REGION: ${{ secrets.PI_BEDROCK_REGION }}
PI_MODEL: ${{ secrets.PI_MODEL }}
run: |
pi --no-session -p \
--provider amazon-bedrock \
--model "$PI_MODEL" \
--skill ../main/.agents/skills/doc-drift \
"/skill:doc-drift $(cat /tmp/pi-message.txt)"
The step runs with working-directory: pr so Pi operates in the PR checkout. --skill ../main/.agents/skills/doc-drift points at the doc-drift skill in the main checkout (one directory up from pr). --no-session makes sure the session isn’t saved, and -p prints the response and exits. Both flags give you non-interactive execution.
When the agent finishes, it writes one report file per behavioral change to numbered paths like .doc-drift-1.md, which the next step picks up. AWS_BEARER_TOKEN_BEDROCK, AWS_REGION, and PI_MODEL come from repository secrets you configure in Step 4.
Post the Findings as PR Comments
The agent is a language model, inherently nondeterministic. Letting it post directly to GitHub would make posting unpredictable. It could emit partial output, retry mid-run, or fail halfway through a comment. A file-based handoff sidesteps that.
- name: Post report comments
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
REPORT_FILES=$(ls "$GITHUB_WORKSPACE"/pr/.doc-drift-*.md 2>/dev/null | sort -V)
if [ -z "$REPORT_FILES" ]; then
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} \
--body "Doc drift check completed but produced no output. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
else
for file in $REPORT_FILES; do
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} --body-file "$file"
done
fi
- name: Post failure comment
if: failure()
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} \
--body "Doc drift check failed. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
The step lists all .doc-drift-*.md files in the pr checkout and loops through them, posting each one as a PR comment with gh pr comment --body-file. The agent never posts to GitHub itself. It only writes files, and the shell loop posts the comments to the PR, producing the same behavior on every run. The workflow passes the App token to gh pr comment for the same reason as every other write: GITHUB_TOKEN would 403 on issue_comment triggers for public repositories.
The workflow owning all GitHub writes keeps the agent’s permissions minimal and makes every comment traceable.
Two fallback paths handle edge cases. If the run produces no report files, a comment tells the reviewer and links to the Actions log. If the run fails outright, the failure() step catches the failure and posts a failure notice with the same log link.
The Complete Workflow
Here’s the full workflow for copy-pasting.
name: Doc Drift Check
on:
issue_comment:
types: [created]
jobs:
authorize:
name: Check authorization
runs-on: ubuntu-latest
# Only run on PR comments that start with /check-docs (skip bots)
if: |
github.event.issue.pull_request != null &&
startsWith(github.event.comment.body, '/check-docs') &&
!contains(github.event.comment.user.login, '[bot]')
permissions:
issues: write
pull-requests: read
contents: read
outputs:
authorized: ${{ steps.auth.outputs.authorized }}
steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- name: Check authorization
id: auth
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const association = context.payload.comment.author_association;
if (!['OWNER', 'MEMBER'].includes(association)) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: `@${context.payload.comment.user.login} Only repo owners or members can trigger doc drift checks.`
});
core.setOutput('authorized', 'false');
} else {
core.setOutput('authorized', 'true');
}
check-docs:
name: Check documentation drift
runs-on: ubuntu-latest
needs: authorize
if: needs.authorize.outputs.authorized == 'true'
permissions:
issues: write
pull-requests: read
contents: read
steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- name: React to comment
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
content: 'eyes'
});
- name: Get PR head SHA
id: pr
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number
});
core.setOutput('head_sha', pr.data.head.sha);
- name: Checkout main branch
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ github.event.repository.default_branch }}
path: main
fetch-depth: 0
- name: Checkout PR branch
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ steps.pr.outputs.head_sha }}
path: pr
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: '22'
- name: Install Pi
run: npm install -g @earendil-works/pi-coding-agent@0.79.6
- name: Show Pi version
run: pi --version
- name: Fetch PR diff
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
DIFF_PATH="$GITHUB_WORKSPACE/pr/.doc-drift-input.diff"
gh pr diff --repo ${{ github.repository }} ${{ github.event.issue.number }} > "$DIFF_PATH"
echo "Diff size: $(wc -l < "$DIFF_PATH") lines"
- name: Build agent message
run: |
WORKSPACE="${{ github.workspace }}/pr"
DIFF_PATH="$WORKSPACE/.doc-drift-input.diff"
cat > /tmp/pi-message.txt <<EOF
You are running a doc drift check on a pull request in the VectorLint repository.
The pull request checkout to inspect is located at:
${WORKSPACE}
Read the PR diff from this file:
${DIFF_PATH}
Use the doc-drift skill. When you have finished, write one report file
per behavioral change you identified, named sequentially:
${WORKSPACE}/.doc-drift-1.md
${WORKSPACE}/.doc-drift-2.md
... and so on.
If there are no issues to report, write a single file ${WORKSPACE}/.doc-drift-1.md
containing the no-issues-found report.
Do not post anything to GitHub directly. The workflow will handle posting.
EOF
- name: Run doc drift check
working-directory: pr
env:
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.PI_BEDROCK_API_KEY }}
AWS_REGION: ${{ secrets.PI_BEDROCK_REGION }}
PI_MODEL: ${{ secrets.PI_MODEL }}
run: |
pi --no-session -p \
--provider amazon-bedrock \
--model "$PI_MODEL" \
--skill ../main/.agents/skills/doc-drift \
"/skill:doc-drift $(cat /tmp/pi-message.txt)"
- name: Post report comments
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
REPORT_FILES=$(ls "$GITHUB_WORKSPACE"/pr/.doc-drift-*.md 2>/dev/null | sort -V)
if [ -z "$REPORT_FILES" ]; then
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} \
--body "Doc drift check completed but produced no output. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
else
for file in $REPORT_FILES; do
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} --body-file "$file"
done
fi
- name: Post failure comment
if: failure()
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
gh pr comment --repo ${{ github.repository }} ${{ github.event.issue.number }} \
--body "Doc drift check failed. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
Step 4: Configure the Secrets
Add five repository secrets under Settings → Secrets and variables → Actions in your GitHub account.
GitHub App secrets:
– APP_ID: the numeric App ID from the App’s General settings page
– APP_PRIVATE_KEY: the contents of the .pem private key file
Provider secrets:
– PI_BEDROCK_API_KEY
– PI_BEDROCK_REGION
– PI_MODEL
For other providers, change the --provider flag in the workflow and the environment variable names to match. For example, Anthropic needs --provider anthropic and only two secrets ANTHROPIC_API_KEY and PI_MODEL. Check the Pi provider documentation for the environment variable your provider expects.
Step 5: Test the Workflow
On any pull request, leave a comment starting with /check-docs as a repo owner or member. The workflow confirms you’re authorized, runs the agent against the PR diff, and posts one comment per behavioral change the agent found. Here’s a drift finding:
## ⚠️ Doc drift - renamed the `--output` flag to `--format`
### `docs/guides/cli-reference.md` - Output formats
**What the doc claims:** "Run `vectorlint doc.md --output line` for terminal output."
**What's now true:** The flag is `--format`, not `--output`.
Fix prompt:
~~~
`docs/guides/cli-reference.md`, Output formats: "--output" is no longer accurate - the flag is now `--format`. Update it. Keep all existing structure, tone, and style.
~~~
Every surfaced drift finding has a fix prompt. The check tells you what broke, and the prompt gives you a starting point to fix it.
Not every PR drifts documentation. When the agent finds a new user-facing change that isn’t documented yet, the comment names the change and its location. When nothing needs attention, you get a confirmation comment with the files and changes it checked.
Handling False Positives and Negatives
AI models are probabilistic, so there’s bound to be some false positives and negatives.
The agent might flag product changes as documentation drift when they aren’t, or miss drifts when a PR genuinely invalidates documentation.
When you notice either problem, update the instruction in the docs-drift skill. Add a concrete instruction that names the pattern, states what the agent should do differently, and includes an example if the rule isn’t obvious. Then test it locally on the same code change to verify it covers the edge case.
These misses are less frequent with more capable models. If you observe consistent false positives or negatives, consider upgrading to a more capable model.
Resolving the Doc Drift
At this point, your workflow catches documentation drift at the PR level. When a PR changes documented behavior, the agent flags it in the PR itself, no platform required.
The simplest approach to resolving the drift is to update the documentation in the same PR. You can hand the fix message to an agent to update the relevant docs, or build an agent workflow to create a PR based on detected drift.
You’ll need guardrails, constraints, and verification signals to keep the agent’s output accurate, on-style, and useful. These would let you quickly confirm the agent’s work is correct and provide feedback so the agent can self-correct.
Building those guardrails is a project in itself and requires iteration. You need style-aware instructions, verification checks, and feedback loops that let you quickly assess the agent’s output and improve performance over time.
For teams without a dedicated docs engineer, hiring an external documentation engineering service is often more cost-effective and efficient.
At TinyRocket, we build automated workflows, agent skills, verification loops, and tools that help teams keep documentation in sync with their product. Our agentic workflows let your team focus on building while reducing documentation work to a matter of quick judgment. Book a call today to get a free documentation workflow audit.
Leave a Reply