Designing an Agentic Security Harness with Tools, Memory and Evolving Skills
An agentic security harness should amplify a tester's judgment without turning model output into unreviewed authority. Its core architecture is an evidence pipeline: scope, tool execution, normalization, reasoning, verification and reusable knowledge.
Begin with scope and authority
Represent allowed targets, tools, credentials, time windows and action classes as machine-checkable policy. Agents should not infer authorization from a prompt. Risky or state-changing actions require an explicit approval boundary and an audit event.
Wrap tools behind typed contracts
Each security tool adapter should validate inputs, constrain command construction, capture version and configuration, and return normalized findings alongside raw evidence. Timeouts, output limits and cancellation are part of the contract, not operational afterthoughts.
Separate evidence from model interpretation
Store raw observations, parsed facts, hypotheses and conclusions as different records. A model may rank or connect findings, but every reported issue should link back to reproducible tool output. Conflicting evidence should remain visible.
Use memory deliberately
Short-term memory tracks the active assessment; durable memory stores approved patterns, environment facts and resolved false positives. Apply provenance, retention and tenant boundaries. Never promote credentials or unreviewed target data into reusable memory.
Turn reports into candidate skills
A completed assessment can suggest a reusable playbook: prerequisites, tool sequence, decision points, expected evidence and stop conditions. Treat generated skills as proposals that must pass review and tests before they can influence later assessments.
Evaluate the harness, not only the model
Measure reproducibility, evidence coverage, false-positive handling, policy violations, approval bypass attempts and recovery from failed tools. Use controlled vulnerable targets and regression fixtures. A persuasive narrative without traceable evidence is a system failure.