AID-M-006.002: HITL Operator Qualification & End-to-End Readiness Testing

Version 1.20261015 · Actionable sub-technique

Qualify the complete roster of assigned operators on the exact HITL interface, role, checkpoint policy, and procedure they must use, then run signed end-to-end drills that exercise approval, denial, timeout, escalation, override, and recovery branches. Measure both operator decisions and the technical path from checkpoint activation to observed terminal action.

Pillars: app

Lifecycle: validation

Scope boundary

Owns technical qualification of the complete assigned operator roster and signed end-to-end readiness drills over approval, denial, timeout, escalation, override, recovery, and observed terminal-action branches. It is not general awareness training.

AID-M-006.001: AID-M-006.002 proves that assigned operators and the end-to-end control path can execute the defined branches; AID-M-006.001 defines the checkpoint, role, evidence, and procedure to be exercised. Procedure documentation is not operator qualification, and a drill must test the approved procedure rather than invent a new checkpoint.

AID-M-006.003: AID-M-006.002 exercises and measures the complete human and technical path; AID-M-006.003 defines the canonical escalation topology and audit-event contract used to judge that path. A successful drill consumes the contract but does not become the owner of event semantics or escalation routing.

Implementation guidance

AID-M-006.002-G001

Develop comprehensive operator training with realistic simulations and measurable outcomes.
Purpose and boundary

This step defines operator qualification against a signed training and scenario population. A trainee name, UI response, completion badge, or scenario owner boolean is not assurance evidence; the verifier recomputes results from authenticated raw response events.

Define the qualification contract
# File: hitl/schemas/operator-qualification.schema.yaml
schema_version: aidefend.hitl-operator-qualification-schema.v1
required_bindings:
  checkpoint_release_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
  operator_roster_receipt_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
  training_content_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
  scenario_manifest_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
policy_required_fields:
  - policy_id
  - policy_version
  - policy_digest
  - role_to_required_modules
  - role_to_required_scenarios
  - scoring_algorithm
  - qualification_thresholds
  - freshness_policy
population_required_fields:
  - roster_record_id
  - operator_subject
  - assigned_roles
  - primary_or_backup
  - assignment_effective_at
  - assignment_expires_at
  - revocation_status
  - required_module_ids
  - required_scenario_ids
raw_response_required_fields:
  - attempt_id
  - operator_subject
  - scenario_id
  - scenario_version
  - presented_at
  - response_at
  - response
  - expected_decision_digest
  - proctor_or_platform_identity
  - authentication_context_digest
  - execution_receipt_id
  - event_signature
result_states:
  - PASS
  - FAIL
  - INSUFFICIENT_DATA
  - NOT_APPLICABLE
  - ERROR

Resolve modules, scenarios, scoring, response-time requirements, and recertification windows only from the signed readiness policy. Reconcile the signed roster snapshot to the current identity directory, reject expired or revoked assignments, and include every primary and backup operator plus every missed, abandoned, duplicated, late, stale, or errored attempt. A scenario answer key must be separately controlled from the trainee-facing system.

Action: Sign an operator-by-scenario qualification matrix only after an independent verifier authenticates events, recomputes scores and timing, and reconciles the exact roster, module, and scenario populations.

AID-M-006.002-G002

Automate regular HITL fire drills to validate end-to-end readiness.
Purpose and boundary

This step proves that each production HITL checkpoint and escalation branch works end to end. Do not let the checkpoint API assert its own success or let a caller supply an expected_result; derive expected branches from signed policy and observe delivery, decision, timeout, and closure through an externally controlled event stream.

Define the drill and observation contracts
# File: hitl/schemas/fire-drill-contract.schema.yaml
schema_version: aidefend.hitl-fire-drill-schema.v1
required_bindings:
  checkpoint_release_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
  escalation_policy_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
  operator_roster_receipt_digest:
    type: string
    pattern: "^sha256:[0-9a-f]{64}$"
drill_manifest_required_fields:
  - drill_run_id
  - checkpoint_id
  - branch_id
  - synthetic_request_digest
  - expected_event_types
  - expected_route_subjects
  - timeout_behavior
  - policy_version
required_branch_types:
  - approve
  - deny
  - timeout
  - unavailable_primary
  - backup_escalation
  - override
  - recovery
  - interface_error
  - observed_final_action
  - malformed_request
observed_event_required_fields:
  - drill_run_id
  - checkpoint_instance_id
  - event_id
  - event_type
  - actor_subject
  - route_subject
  - action_id
  - action_digest
  - authorization_outcome
  - observed_effect
  - occurred_at
  - payload_digest
  - source_identity
  - event_signature
result_required_fields:
  - expected_branch_count
  - observed_branch_count
  - missing_event_ids
  - extra_event_ids
  - duplicate_event_ids
  - late_event_ids
  - effective_timeout_action
  - gate_receipt_digest
  - observed_final_action
  - observed_final_outcome
  - result

Schedule and freshness are policy-owned. The independent verifier reads the signed drill manifest, current role directory, provider delivery receipts, and externally held lifecycle events; it derives branch expectations only from the exact checkpoint policy, reconciles every checkpoint-by-required-branch member, correlates stable checkpoint instance IDs, and confirms the effective timeout, override, recovery, interface-error, gate, and observed-final-action outcomes. Notification delivery alone does not prove that the checkpoint blocked the action or honored the decision.

Action: Produce a detached-signed drill receipt for every checkpoint and required branch. Missing, late, duplicate, self-reported, or unverifiable events prevent readiness PASS.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-M-006.002: HITL Operator Qualification & End-to-End Readiness Testing | AIDEFEND

Loading interactive view…

AIDEFEND™

A Practical Knowledge Base for AI Security Defenses

Version

GitHub MCP Service