What Makes Assessment Evidence Defensible?
The properties that separate evidence which survives challenge from a screenshot nobody can trace
"Defensible" means one thing: it holds up when someone who was not there, and who has reason to disagree, examines it later.
That someone might be an auditor, a system owner disputing a finding, or an authorizer a year from now asking why they signed. Evidence that only convinced the person who collected it is not evidence. It is a memory.
The test of evidence is not whether the assessor believed it. It is whether a stranger with a motive could not reasonably reject it.
What Evidence Has to Establish#
ITSP.10.033 defines the purpose of an assurance activity in one sentence, and it doubles as the standard evidence has to meet:
[A] collection of tasks that increases the confidence that a security or privacy control is appropriately designed and implemented and is operating as intended.
ITSP.10.033Three conditions.1 A policy establishes the first. A configuration export establishes the second. Only something that spans time — logs over a period, review records with dates, a ticket history — establishes the third.
Scroll sideways to see the full diagram →
Most disputed findings trace back to evidence that stopped at level one or two and was reported as if it reached three.
Seven Properties of a Defensible Artifact#
Scroll sideways to see the full diagram →
| Property | Means | Fails when |
|---|---|---|
| Attributable | You can tell who or what produced it, from which system | An unnamed console, an unsigned export |
| Current | It reflects the system as assessed, within a defined window | Last year's screenshot |
| Complete | It covers every lettered part of the control statement | Part A evidenced, parts B–D assumed |
| Reproducible | Another assessor following the same steps gets the same artifact | "I saw it on their screen" |
| Tamper-evident | Alteration would be detectable | An edited spreadsheet, a cropped image |
| In scope | It describes a component inside the authorization boundary | Evidence from a different environment |
| Independent | Its provenance does not rest solely on the party being assessed | A vendor's marketing page |
The complete row deserves a second look. ITSP.10.033 splits control statements into lettered parts,2 so a control with parts A through D is four things to evidence. "Partially met" without naming which part failed is a finding nobody can fix.
The Screenshot Problem#
Screenshots are often the only practical artifact, and this is not an argument against them. It is an argument for what a screenshot must carry:
- A visible hostname or tenant, so it is attributable and in scope
- A visible timestamp, so it is current
- Enough of the screen to show what setting, on what object, so it is complete
- A note of who captured it and how, so it is reproducible
An undated image of an unnamed console with a green tick in it is decoration.
Evidence Has a Shelf Life#
NIST's definition of the examine method ends on the phrase that matters:
… the results of which are used to support the determination of security control or privacy control effectiveness over time.
NIST SP 800-53A Rev. 5Over time.3 A configuration export proves a state at a moment. A control that operates continuously — access reviews, log monitoring, patching — needs evidence that spans a period, and evidence that was true in March says nothing about September.
Define a validity window per control type, and re-collect rather than re-use. It is the single cheapest improvement most programs can make.
What a Challenger Actually Attacks#
Write the finding from the other side of the table. Four attacks are reliable:
- "That is out of scope." Answered by the boundary — see Step 1.
- "That is out of date." Answered by the timestamp and the validity window.
- "That shows one instance, not the estate." Answered by the stated sample.
- "That shows the policy, not the practice." Answered by level-three evidence.
An assessor who has pre-answered all four has a defensible finding. One who has not will spend the closing meeting defending their own notes.
Where CtrlFort Fits#
Attribution and currency are the first properties to go when evidence is collected by hand. Files get renamed. Context is lost. Nobody remembers which tenant a screenshot came from, and by the time the finding is disputed the person who captured it has moved on.
Preserving provenance is a systems problem, and CtrlFort Assess treats it as structure. Evidence, controls, findings and risks are linked objects, so every artifact is attached to the control it evidences and linked forward to the finding and the risk that rest on it — the walk back from a challenged finding to its artifact is a click, not an archaeology project. Control Intelligence records, per control, what evidence is expected, and the deterministic engine's coverage analysis reports what was reviewed and what was missing. That is the "complete" property, checked rather than hoped for.
The assessment intelligence architecture explains why a verdict has to be explainable. This post is about why the input has to be defensible. The two are one argument: CtrlFort AI can only explain a result as well as the evidence beneath it, which is why the platform treats provenance as data rather than as good intentions.
Final Thoughts#
Defensible evidence is not more evidence. It is evidence that answers, in advance, the questions a reasonable sceptic would ask.4
Previous: Assessments Don't Reduce Risk. Decisions Do. · Next: Step 1 — Security Categorization and Assessment Scoping
Frequently Asked Questions#
Is a screenshot acceptable as assessment evidence?#
Yes, if it carries a hostname or tenant, a timestamp, enough context to show which setting on which object, and a note of who captured it. Without those it is an image, not evidence.
How old can evidence be before it needs re-collecting?#
It depends on the control. A configuration that rarely changes might be valid for the assessment period; a review cadence needs evidence from within that cadence; a log-based control needs a current extract. Set a validity window per control type rather than one blanket rule.
Can a vendor's own attestation count as independent evidence?#
A vendor's self-assessment cannot. An attestation from an independent third party can, once you have confirmed its scope covers the services and locations you use, and its period covers the time you are relying on.
Do we need evidence for every control, or is a sample enough?#
Every control needs evidence. Within a control that applies to many instances, a sample is normal — but the population, the method and the size must be stated, and one failure in the sample is treated as a finding about the whole.
References#
- ITSP.10.033 — Foreword, overview and introduction ↩Canadian Centre for Cyber Security · https://www.cyber.gc.ca/en/guidance/cyber-security-privacy-risk-management/itsp10033/foreword-overview-introduction
- ITSP.10.033 — Concepts and structure ↩Canadian Centre for Cyber Security · https://www.cyber.gc.ca/en/guidance/cyber-security-privacy-risk-management/itsp10033/concepts-structure
- Examine — glossary entry, citing NIST SP 800-53A Rev. 5 ↩National Institute of Standards and Technology · https://csrc.nist.gov/glossary/term/examine
- ITSG-33 Annex 2 — Information System Security Risk Management Activities ↩Canadian Centre for Cyber Security · https://www.cyber.gc.ca/en/guidance/annex-2-information-system-security-risk-management-activities-itsg-33
- ITSP.50.105 — Guidance on cloud security assessment and authorization ↩Canadian Centre for Cyber Security · https://www.cyber.gc.ca/en/guidance/guidance-cloud-security-assessment-and-authorization-itsp50105