Agent Pentest Methodology and Reporting
Yesterday you red teamed your own agent and produced a rough assessment. That was the exercise. Today we turn it into a procedure you can run on any agent, for any client, in any week, and get comparable results every time.
This matters because agent testing has a bad habit: it drifts into improvisation. Someone pastes a few jailbreak strings, the agent refuses, and the report says “no issues found.” Or the agent leaks something, the tester screenshots it, and nobody can say whether it was a fluke. A methodology fixes both problems. It tells you what to test, how to measure it, and how to write it down so the result survives contact with a skeptical engineering team.
What is different about testing an agent
A web app pentest has a stable target. Same request, same response. An agent pentest does not, and the methodology has to account for four differences.
The target is probabilistic. The same payload can land on attempt three and fail on attempt four. A single run proves almost nothing. Every result needs a rate.
The attack surface is data, not just endpoints. Anything the agent reads is an input. Tickets, files, web pages, tool descriptions, memory, retrieved documents. Your scope has to name these, not only URLs.
Impact depends on the tools, not the model. A prompt injection against an agent with no tools is a curiosity. The same injection against an agent with a database tool and an outbound channel is a breach. Severity comes from what the chain reaches.
Authorization is part of the test. The agent acts with real credentials. You need explicit permission for what those credentials can touch, and rules about what you will and will not do with them.
The six phases
Phase 0: Scope and rules of engagement
Before any testing, write down and get signed off:
The agent, its version, and the environments in scope (staging, production, or a clone)
Which tools and connected systems may be exercised, and which are off limits
Whether testing against production data is allowed, and what test accounts exist
What counts as proof of impact. For agents, agree on safe canaries: a harmless marker file, a test record, a callback to a server you control
Rate limits, cost limits, and a stop condition. Agents can loop, and loops cost money
A contact for when something unexpected happens
Never skip the canary agreement. It is what lets you demonstrate data reach without ever touching real customer data.
Phase 1: Map the target
Build the attacker’s view, the same one from Day 6 and Day 28.
Tools. Every tool the agent can call, including defaults nobody remembers enabling.
Identities. For each tool, which credential it uses and what that credential can reach.
Inputs. Every source of content that reaches the context window, and who can influence it.
Exits. Every path by which bytes can leave, including rendered markdown images, link previews, tool arguments, and logs.
Persistence. Memory stores, vector databases, and anything that survives between sessions.
Trust boundaries. Where one user’s data, tenant, or privilege level should not cross into another.
The output is a one page diagram and a trifecta checklist. If all three conditions hold (private data, untrusted content, outbound channel), you already know where to start.
Phase 2: Enumerate the chains
A chain is entry, then influence, then target, then exit. Write every plausible one down as a test case before you attack anything. Each test case gets:
An ID
The entry point (which input an outsider controls)
The intended target (which tool or data you are trying to reach)
The exit (how a result would leave)
The expected control that should stop it
This list is your test plan, and later it becomes your coverage statement. “We tested 41 chains, 9 reached their target” is a sentence a client can act on. “We tried a bunch of prompts” is not.
Phase 3: Test, as rates
For each chain, run the attack from the relevant earlier days: tool poisoning, rug pulls, shadowing, confused deputy, indirect injection through each delivery vector, memory and retrieval poisoning, exfiltration channels.
Record every attempt. For each chain, track:
Number of attempts (aim for at least 10 per payload family, more for borderline cases)
Outcome per attempt: did not land, landed and reached nothing, landed and reached the target, landed and exited
Any variation in the agent’s behavior, since the same chain can fail in different ways
Then report the rate. A chain that completes 3 of 20 attempts is a real finding. An attacker needs it to work once, and they can retry for free.
Vary the conditions too. Test with different models if the agent supports routing, different context lengths, and with the payload at different positions in the content. Long contexts and buried instructions behave differently.
Phase 4: Validate defenses
For every control the system claims to have, attack it directly.
Input filtering. Try encoded, translated, and split payloads. Note the bypass rate.
Allowlists. Test whether an allowed action can be redirected through its arguments or environment.
Human approval. Count how many prompts a normal task generates. If it is high, you have an approval fatigue finding.
Scoping and token binding. Confirm that an injection in one user’s session cannot reach another user’s data.
Egress controls. Test every exit from your map, including the quiet ones.
Logging. After each successful chain, check whether the logs let you reconstruct what happened. A chain you cannot reconstruct is a detection finding.
A control that exists but does not fire is a failed control. Mark it that way.
Phase 5: Validate findings and clean up
Before writing, reproduce each finding fresh. Confirm it is not an artifact of a stale session or leftover memory. Then clean up: remove planted memory entries, poisoned documents, test tickets, and any config the agent wrote. Document what you removed. An agent pentest that leaves persistent injected content behind is a hazard for the next user.
Severity for agent findings
Standard CVSS fits agents awkwardly, because likelihood is probabilistic and impact depends on connected tools. A practical approach is to score on four factors and state them explicitly in the report:
Reach. What did the chain touch? Nothing, own-session data, another user’s data, credentials, or write access to production systems.
Exit. Did data actually leave the boundary, or only reach the agent?
Reliability. The measured success rate, not a guess.
Attacker requirement. What does the attacker need to place the payload? A public form is lower effort than insider access.
A chain with cross-tenant reach, a working exit, a 30 percent success rate, and a public entry point is critical regardless of how the model “feels” about it. A chain that lands 1 in 50 times but only reaches the user’s own data is low. State the factors, then the rating, so the client can disagree with a specific number instead of the whole report.
Writing the report
A good agent pentest report has seven parts.
Executive summary. One page. What was tested, the headline result, the three most important fixes. Written for the person who decides budget.
Scope and method. The Phase 0 and Phase 1 material, condensed. Include the architecture diagram and the chain count.
Findings. Each one in a fixed template:
Title: plain language, names the chain (“Public support ticket can exfiltrate private order data via markdown image”)
Severity with the four factors
The chain: entry, influence, target, exit
Reproduction steps with the exact payload, the tool calls observed, and the canary used
Measured reliability: “Landed in 7 of 20 attempts”
Evidence: logs, tool call traces, screenshots, callback records
Why existing controls failed
Remediation: the specific control, not “improve prompt injection defenses”
What held. The defenses that worked, with evidence. This tells the client what not to break while fixing the rest.
Coverage matrix. Every chain from Phase 2 with its outcome. This is how a reader sees what was and was not tested.
Framework mapping. Map each finding to the current OWASP agentic categories and the relevant NSA MCP guidance, pulled from the official source at report time. This slots your work into the client’s existing risk language.
Remediation roadmap. Ordered by leverage. After four weeks you know the pattern: scoping and least privilege first, then egress control, then logging, then tool pinning, then everything else.
Reporting mistakes that get findings ignored
“The model can be jailbroken.” True of every model. It is not a finding unless it reaches something.
No reliability number. Engineers will dismiss a result they cannot reproduce. Give them the rate and the setup.
Fix recommendations aimed at the prompt. “Add a stronger system prompt” is not a control. Name a control that works when the injection succeeds.
Screenshots of the chat only. Capture the tool call trace. The chat transcript shows what the agent said. The trace shows what it did.
Real data in the evidence. Use canaries. If you touched real data by accident, say so and say how it was handled.
Retest
Every engagement ends with a retest plan. After fixes land, rerun the exact chains that succeeded, at the same attempt counts, and report before and after rates. Because agents change with every model update, prompt edit, or new tool, recommend rerunning the suite on any of those changes. The test plan you built in Phase 2 becomes their regression suite.
Homework for Day 29
Take your Day 28 assessment and rewrite your scope as a Phase 0 document with explicit canaries and stop conditions.
Turn your Day 28 attack list into numbered chain test cases with entry, target, exit, and expected control.
Re-run your three most interesting chains at least 20 times each and record the rates.
Score each finding on reach, exit, reliability, and attacker requirement.
Write one finding in the full template above, including the remediation and the before and after plan for retest.
Day 30 is the last one. We close with careers, bounties, and what to learn next, plus a map of where agent security is heading.
This series is published on HACKLIDO. Labs referenced throughout are available in the HACKLIDO lab environment.
