AID-H-007.004: Evaluation Data Integrity, Sufficiency Assurance & Promotion Governance

Version 1.20261015 · Actionable sub-technique

Ensure the integrity, independence, and sufficiency of evaluation artifacts used in validation and promotion decisions. This includes evaluation datasets, holdout sets, benchmarks, safety test suites, and security regression corpora.

What this covers: Existing techniques already address training environment security, reproducibility, detector-specific evaluation, and benchmark execution, but this sub-technique provides a general-purpose evaluation security discipline that applies across model types and promotion workflows.

Threat model: The threat model is simple but high impact: if an attacker can manipulate evaluation data or evaluation evidence, a compromised model can appear to pass all validation gates and be promoted as if it were safe.

Pillars: data, app

Lifecycle: building, validation

Scope boundary

Owns integrity, independence, sufficiency, freshness, leakage checks, and release binding for evaluation datasets, holdouts, benchmarks, safety suites, regression corpora, raw results, and promotion evidence. It does not execute every specialized benchmark or own evaluator implementation.

AID-H-007.003: AID-H-007.004 secures evaluation populations and promotion evidence; AID-H-007.003 secures capture and reproducibility of the training run. Replaying training does not establish evaluation independence or sufficiency.

AID-H-007.005: AID-H-007.004 governs general evaluation evidence and promotion use; AID-H-007.005 owns the specific pre/post safety-alignment regression gate for fine-tuned or customized models. The specialized gate consumes general evidence-integrity controls but has a distinct measured candidate and outcome.

AID-H-007.007: AID-H-007.004 protects cases, datasets, corpus coverage, and result binding; AID-H-007.007 protects the judge model, grader code, rubric, harness, thresholds, and evidence writer that interpret captured behavior. Trusted cases can still be mis-scored by a compromised evaluator.

AID-M-008: AID-H-007.004 secures benchmark data, independence, sufficiency, and evidence; AID-M-008 executes adversarial agent security benchmark campaigns. Benchmark execution does not own the integrity of its holdout and promotion evidence.

AID-H-002.001: AID-H-007.004 protects evaluation independence, corpus sufficiency, holdout isolation, and promotion evidence; AID-H-002.001 sanitizes the content of candidate evaluation records before an evaluation job can read them. An independent, sufficient, and leak-free evaluation corpus can still contain poisoned or privacy-sensitive records.

Warning

Critical Validation Integrity

Evaluation integrity is a high-leverage control point in the promotion pipeline.

Implementation guidance

AID-H-007.004-G001

Verify the integrity of evaluation datasets and holdout sets on every evaluation run using content hashes and signed metadata.
Concept:

Every approved evaluation dataset should have an immutable identifier, a content hash, and an owner-signed metadata record. Before an evaluation run starts, verify those values and abort if anything changed unexpectedly.

Example dataset manifest
{
  "dataset_name": "llm_safety_holdout_v3",
  "dataset_uri": "s3://ml-eval/holdouts/llm_safety_holdout_v3.jsonl",
  "sha256": "8a8db0e589f47ce18159e62ce599a73fd6a67e30a70c82c676470cd38a4df113",
  "owner": "ml-security-team",
  "approved_at": "2026-03-17T12:00:00Z"
}
Example hash check in Python
import hashlib
from pathlib import Path

def sha256_file(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()

expected = "8a8db0e589f47ce18159e62ce599a73fd6a67e30a70c82c676470cd38a4df113"
got = sha256_file("./llm_safety_holdout_v3.jsonl")
if got != expected:
    raise RuntimeError("Evaluation dataset integrity check failed")

Operational notes: Keep the manifest in version control and sign it or store it in an attested artifact registry. Use immutable or retention-protected storage for the golden holdout when the platform supports it.

Verify safely

A separately credentialed verifier independently reads back or replays every evaluation dataset, split and row used for a model-release decision, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.004-G002

Separate training and evaluation storage, credentials, and write paths.
Concept:

Training and evaluation should be separated as an infrastructure and access-boundary control, not just as a data-quality convention. Training jobs should not be able to overwrite, regenerate, or silently modify the evaluation assets that decide promotion.

Step 1: Isolate storage locations and IAM roles
# Example layout
s3://ml-train-data/project-x/training/        # writable by training pipelines
s3://ml-eval-data/project-x/holdouts/v1/ # writable only by evaluation stewards

# Example IAM intent
training-role:
  allow:
    - s3:GetObject on s3://ml-train-data/project-x/*
  deny:
    - s3:PutObject on s3://ml-eval-data/project-x/*

evaluation-role:
  allow:
    - s3:GetObject on s3://ml-eval-data/project-x/*
    - s3:PutObject on s3://ml-eval-data/project-x/results/*
Step 2: Enforce write-path separation in pipelines
# File: ci/check_eval_path_access.py
ALLOWED_WRITE_PREFIXES = {
    "training": ["s3://ml-train-data/project-x/"],
    "evaluation": ["s3://ml-eval-data/project-x/results/"]
}

def assert_write_path(job_type: str, output_uri: str) -> None:
    allowed = ALLOWED_WRITE_PREFIXES[job_type]
    if not any(output_uri.startswith(prefix) for prefix in allowed):
        raise RuntimeError(f"{job_type} job cannot write to {output_uri}")
Operational notes

Action: Treat train/eval separation as a storage and IAM boundary. A model is not safely evaluated if the same pipeline can both train on and rewrite the benchmark that approves it.

Keep the action safely bounded

Use distinct stores or enforced prefixes and identities, deny training jobs write access to holdouts, permit only steward-controlled versioned publication, test allowed/denied paths, and reconcile all effective policies and write events.

Verify safely

A separately credentialed verifier independently reads back or replays training and evaluation stores, paths, service identities and write operations for each project/release, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.004-G003

Test for exact and near-duplicate overlap between training and evaluation data before promotion-critical evaluation runs.
Concept:

Even when train and eval data live in separate storage locations, leakage can still happen through copied rows, regenerated synthetic variants, or weak split hygiene. Promotion-critical evaluations should therefore include an automated overlap check that starts with exact matches and expands to near-duplicate detection where appropriate.

Step 1: Block exact overlap
import hashlib


def line_hashes(path):
    with open(path, "r", encoding="utf-8") as f:
        return {hashlib.sha256(line.strip().encode()).hexdigest() for line in f if line.strip()}

train_hashes = line_hashes("train.jsonl")
eval_hashes = line_hashes("eval.jsonl")
intersection = train_hashes & eval_hashes
if intersection:
    raise RuntimeError(f"Exact train/eval overlap detected: {len(intersection)} samples")
Step 2: Add near-duplicate detection for promotion-critical suites
# File: evaluation/check_near_duplicates.py
import math
import os
from datasketch import MinHash, MinHashLSH

def _required_positive_int(name: str) -> int:
    value = int(os.environ[name])
    if value <= 0:
        raise RuntimeError(f"{name} must be a positive integer")
    return value

MINHASH_PERMUTATIONS = _required_positive_int("VERIFIED_H007_EVALUATION_MINHASH_PERMUTATIONS")


def _required_unit_interval(name: str) -> float:
    value = float(os.environ[name])
    if not math.isfinite(value) or not 0 < value <= 1:
        raise RuntimeError(f"{name} must be in the interval (0, 1]")
    return value

NEAR_DUPLICATE_THRESHOLD = _required_unit_interval("VERIFIED_H007_EVALUATION_NEAR_DUPLICATE_THRESHOLD")



def build_minhash(text: str, num_perm: int = MINHASH_PERMUTATIONS) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for token in set(text.split()):
        m.update(token.encode("utf-8"))
    return m


def find_near_duplicates(train_docs: list[str], eval_docs: list[str], threshold: float = NEAR_DUPLICATE_THRESHOLD):
    lsh = MinHashLSH(threshold=threshold, num_perm=MINHASH_PERMUTATIONS)
    train_minhashes = {}
    for idx, doc in enumerate(train_docs):
        m = build_minhash(doc)
        train_minhashes[idx] = m
        lsh.insert(f"train-{idx}", m)

    collisions = []
    for idx, doc in enumerate(eval_docs):
        hits = lsh.query(build_minhash(doc))
        if hits:
            collisions.append((idx, hits))
    return collisions
Operational notes

Action: Require every promotion-critical evaluation run to prove that the approving benchmark is independent of the training set, not merely stored in a different bucket.

AID-H-007.004-G004

Create signed promotion evidence that binds evaluation results to the exact model artifact, evaluation data version, code version, and configuration used during the run.
Concept:

A model should not be promoted based on screenshots or loose metric files. Build one evidence bundle that records the exact model, eval data, code, config, and result set, then sign it.

Example evidence bundle
{
  "model_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "eval_dataset_hash": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
  "eval_code_hash": "git:9f3e2a1",
  "eval_config_hash": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
  "metrics": {
    "toxicity_fail_rate": 0.01,
    "prompt_injection_escape_rate": 0.00,
    "policy_violation_rate": 0.02
  },
  "timestamp": "2026-03-17T12:20:00Z",
  "evaluator": "mlsec-ci-bot"
}

Operational notes: Store the evidence bundle in the same release record as the promoted model. Configure AID-H-003.002 to require valid evidence before promotion rather than trusting standalone metric logs.

Verify safely

A separately credentialed verifier independently reads back or replays each candidate model and complete signed evaluation evidence package admitted to a target stage, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.004-G005

Run evaluation-data quality and poisoning checks before evaluation, including schema validation, label sanity checks, and comparison against a known-good baseline snapshot.
Concept:

Evaluation datasets need their own data-quality controls. A sudden improvement in evaluation results can be a sign that the evaluation data, not the model, changed in a dangerous way.

Example checks to run
1. Validate schema and required fields with GX Core.
2. Compare class or label distribution against the last known-good snapshot.
3. Flag newly added samples for manual review if they affect high-risk categories.
4. Insert canary samples with known expected outcomes.
5. Alert if key metrics improve too sharply after data changes.

Operational notes: Keep a baseline snapshot for each benchmark or holdout. Treat unexpected metric improvements with the same suspicion as unexpected metric collapses.

Keep the action safely bounded

Validate schema/domain/uniqueness, label/source provenance, class/stratum counts and calibrated distribution/poisoning indicators; quarantine findings, require independent review or repair, and deny promotion on unresolved rows or incomplete coverage.

Verify safely

A separately credentialed verifier independently reads back or replays every evaluation row/label and required stratum under a signed schema, provenance and baseline-quality profile, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.004-G006

Track benchmark freshness and coverage so promotion does not depend on stale or incomplete evaluation suites.
Concept:

Every benchmark should have metadata that says when it was last updated, what threat or failure categories it covers, and which model types it applies to.

Example benchmark registry entry
{
  "benchmark_id": "rag_security_suite_v2",
  "last_updated": "2026-01-10",
  "threat_categories": ["prompt_injection", "data_exfiltration", "tool_misuse"],
  "model_types": ["llm", "rag"],
  "sample_count": 450
}

Operational notes: Add warning and critical thresholds for staleness. A simple policy is better than none, such as warning after six months and escalation after twelve months if the benchmark has not been reviewed.

Verify safely

A separately credentialed verifier independently reads back or replays the complete policy-required benchmark, scenario, modality, language, demographic and risk strata for each release, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.004-G007

Maintain a signed security regression corpus from incidents, red-team findings, and known failure cases, and require it to pass before promotion.
Concept:

Every meaningful failure should become a regression test so the same class of failure does not silently return in the next release.

Example regression record
{
  "test_id": "rt-2026-0317-001",
  "source": "red-team",
  "severity": "high",
  "expected_behavior": "refuse_tool_call",
  "model_types": ["agentic_llm"],
  "date_added": "2026-03-17"
}

Operational notes: Store the corpus in version control or a signed artifact system. Promotion should fail if the corpus fails, even when aggregate benchmark metrics still look acceptable.

Keep the action safely bounded

Version and sign provenance-preserving sanitized fixtures, map each to expected behavior and policy, replay the complete required corpus in isolation, protect responses, block promotion on any unresolved regression, and independently reproduce results.

Verify safely

A separately credentialed verifier independently reads back or replays the signed privacy-safe corpus of prior incidents, red-team findings and abuse regressions required for each model release, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-H-007.004: Evaluation Data Integrity, Sufficiency Assurance & Promotion Governance | AIDEFEND

Loading interactive view…

AIDEFEND™

A Practical Knowledge Base for AI Security Defenses

Version

GitHub MCP Service