Security control assessment automation
Automate the preparation, not the determination. Evidence gathering, normalization, correlation and currency tracking are mechanical; deciding whether a control objective is satisfied is an authorization judgment with a person's name on it.
Security control assessment is the RMF step where a system’s controls are examined, tested and judged. It is labour-intensive, it happens under deadline, and it is the step most obviously begging for automation.
It is also the step where automating the wrong part creates a problem that surfaces years later, when somebody asks who decided something and the answer is nobody.
The line
A machine may prepare an assessment. Gather the evidence, normalize it, attach it to the objectives it bears on, flag what is missing, flag what is stale, and lay the whole thing out so an assessor can work through it efficiently.
A person makes the determination. Satisfied or not satisfied is a judgment about whether the evidence meets the objective. It carries an assessor’s name and a date, and it is the authoritative act of the whole step.
The line is not squeamishness about AI. It is that a determination is an input to an authorization decision, and authorization decisions have accountable humans attached by design. A machine that marks controls satisfied has made authorization judgments without authority, and every artifact downstream inherits that silently.
What automation should do, in order of value
Assemble the evidence per objective. The assessor’s most wasted hour is spent finding things. Every CCI should arrive with the scan results, checklist items, documents and prior determinations that bear on it already attached. This alone changes the character of the work.
Normalize across sources. ACAS and Nessus, SonarQube, ZAP, Grype, Trivy, Gitleaks, Semgrep, Dependency-Check, CycloneDX, STIG checklists — each speaks a different dialect, and an assessor should not be learning six formats to answer one question.
Preserve prior adjudication. A false positive judged once stays judged. Re-adjudicating four hundred findings you already dismissed, every cycle, is the specific experience that makes practitioners hate tooling, and it is entirely avoidable.
Track currency. Which determinations are older than your threshold? Which name an architecture version the system has moved past? Which rest on evidence that has expired? These are queries against a record and they turn staleness from a discovery into a report.
Compute reassessment scope. When the system changes, which objectives does that change actually bear on? Derived from a diff of the architecture rather than from a meeting.
Carry findings forward. A not-satisfied determination should produce a POA&M item that retains the link back to the observation and the evidence behind it, without anyone retyping anything.
Assess the objective, not the control
The structural point, and the one most non-DoD tooling gets wrong.
DoD assessment happens at the Control Correlation Identifier level. A control statement decomposes into individually testable assertions, and those assertions can have genuinely different outcomes under the same control.
Take AC-2. Automated account management may be working. Inactive account disabling may not be implemented. Account review may be happening but undocumented. Three different states under one control identifier.
A system that stores one status per control cannot express that. What happens in practice is that the assessor picks the worst status and writes the detail into a comment field — and that comment field is now load-bearing, unqueryable, and the reason the reporting is wrong.
Model the objective. Roll up to the control for reporting. Never invert it.
The three methods, and why the distinction survives
SP 800-53A gives three assessment methods and they are not interchangeable, however tempting it is to treat them that way.
EXAMINE — reviewing documents, configurations, records. Highly automatable. A machine can pull the configuration and check the value.
INTERVIEW — talking to people. Not automatable, and it is often where the real findings come from. A configuration file says the process exists. An interview reveals whether anyone follows it.
TEST — exercising the control. Partly automatable. A scanner tests some things well and cannot test a contingency plan.
Automation shifts effort toward EXAMINE because that is what is easy. Watch for that drift. An assessment that is entirely EXAMINE is an assessment of documentation, and a system can be perfectly documented and wrong.
Where AI genuinely helps
Two places where a model earns its place, both with the same shape.
Extraction from documents you already have. A program with a two-hundred-page SSP has content worth reusing. A model can propose which passage corresponds to which control statement, carrying the passage with the proposal so a reviewer checks it in seconds rather than searching. The practitioner accepts or rejects. The acceptance is the authoritative act and it is audited.
Applicability ranking. Which STIGs plausibly apply to this technology stack? Ranking hundreds of candidates is a good use of a model, and the output is a ranked list a person confirms — not a decision.
Both share the shape that makes them safe: the output is a proposal, it carries its own basis, and it stays a proposal until a person accepts it. Two additional guards matter in practice. Model- invented control, CCI and STIG identifiers get discarded server-side, because a reviewer will not catch a well-formed reference to a CCI that does not exist. And a proposal whose target changed between extraction and acceptance is refused rather than applied — somebody made a more recent decision and it should win.
What you should be suspicious of
- Automated determinations. Any product that marks controls satisfied without a person. Ask who the assessor of record is — the answer should be a named human, because a determination feeds an authorization decision.
- Confidence scores as determinations. “94% compliant” is not an RMF outcome. Controls are satisfied or they are not, and a percentage is a way of avoiding saying which.
- Generated narratives with no source. Fluent implementation statements a model wrote from nothing. They read well and describe a system that may not exist.
- Assessment without an architecture reference. If a result does not name what it was assessed against, you cannot tell later whether it still holds.
The realistic outcome
Done properly, assessment automation does not reduce the number of determinations anyone makes. It reduces everything around them: the searching, the format-wrangling, the re-adjudication, the reconstruction of what was decided last time and why.
An assessor whose evidence is already assembled per objective, whose prior decisions are preserved, and who can see exactly which objectives a recent change touched, is doing the job they were hired to do. That is the whole ambition, and it is a bigger change than it sounds.