AID-H-007.003: Training Process Reproducibility

Version 1.20261015 · Actionable sub-technique

For every training run, create an immutable manifest binding source commit, configuration, dependency lock, dataset/version digest, seed, builder and runtime image digest, hardware/runtime settings, and produced model digest. Before promotion, independently replay the exact manifest in an approved environment and compare declared outputs and policy-bounded tolerances; missing inputs, non-replayable state, or unexplained divergence fails the gate.

Pillars: infra, model

Lifecycle: building, validation, improvement

Scope boundary

Owns immutable training-run capture and independent replay of source, configuration, dependency lock, dataset, seed, images, hardware settings, runtime, and produced model, including policy-bounded divergence decisions. It does not prove each input is trustworthy or the model is safe.

AID-H-007.002: AID-H-007.003 proves whether the exact run can be replayed from its bound inputs; AID-H-007.002 monitors and audits the original running job. Reproducibility and runtime anomaly detection answer different integrity questions.

AID-H-007.004: AID-H-007.003 proves run reconstruction and output consistency; AID-H-007.004 proves that evaluation data, suites, independence, sufficiency, and promotion evidence are trustworthy. A reproducible model can still be evaluated on manipulated or inadequate evidence.

Implementation guidance

AID-H-007.003-G001

Strictly version control all training code, configurations, and dependencies using a single Git commit.
Concept:

Treat every training run as an atomic, versioned event. All the code, model configurations, and software dependencies required to run the training should be captured in a single Git commit. This creates a definitive, time-stamped snapshot of the logic used to produce a model.

Step 1: Ensure All Components are in Git

Before starting a training run, ensure that your training script, model definition, hyperparameters configuration (e.g., `params.yaml`), and the pinned dependency file (`requirements.txt`) are all committed to Git.

# Before running your training script:

# 1. Check the status to see what has changed
git status

# 2. Add all relevant files
git add src/train.py configs/params.yaml requirements.txt

# 3. Commit with a descriptive message
git commit -m "feat(fraud-model): Update hyperparameters for v2.1 training run"

# 4. Push the commit
git push

Action: Institute a mandatory policy that no training job can be run from a 'dirty' Git state. The first step of any automated training pipeline should be to verify that the Git working directory is clean and to record the current commit hash.

Verify safely

A separately credentialed verifier independently reads back or replays every training code, configuration, dependency lock and entry-point revision used by a model build, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.003-G002

Use Data Version Control (DVC) to version the dataset and link it to the code commit.
Concept:

Git is not designed for large data files. A tool like DVC allows you to version your dataset by storing a small 'pointer' file in Git. This pointer file contains the hash of the actual data, which is kept in a separate, remote storage location (like S3). This allows you to version datasets and models of any size.

Step 1: Track the Dataset with DVC

This command tells DVC to start tracking your data file. It creates a small \`.dvc\` file containing the data's hash.

# Assume you have already run 'dvc init' and configured remote storage

# Tell DVC to start tracking your dataset
dvc add data/processed/training_data_v3.csv

# This creates 'data/processed/training_data_v3.csv.dvc'
Step 2: Commit the DVC Pointer File to Git

You commit the small pointer file to Git alongside the code that uses it. This links the two together.

# Add the .dvc file to your Git commit
git add data/processed/training_data_v3.csv.dvc

# Now commit it with your code changes
git commit -m "feat(data): Add v3 of training data for fraud model"

# Finally, push the actual data to DVC remote storage
dvc push

Action: Use DVC to version all training and evaluation datasets. Commit the resulting \`.dvc\` files to the same Git repository as your training code.

Keep the action safely bounded

Publish content-addressed immutable manifests, pin remote object versions and split membership, verify hashes before read and after restore, deny mutable aliases or missing objects, and bind the complete manifest digest to run and model artifact.

Verify safely

A separately credentialed verifier independently reads back or replays every dataset, split, feature artifact and checkpoint referenced by an eligible training run, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.

AID-H-007.003-G003

Capture a signed digest-qualified training environment and independently replay the version-bound training run.
Delivery level: production CI image handoff

A Git-derived tag is useful for discovery but is mutable and is not the training-environment identity. Build and push once, capture BuildKit's registry digest, verify the signature against that digest, and pass only the digest-qualified reference to the training scheduler.

Build, push, and capture the OCI digest
# File: ci/publish-training-image.sh
#!/usr/bin/env bash
set -euo pipefail
umask 077

: "${IMAGE_REPOSITORY:?set the approved registry/repository}"
: "${GIT_COMMIT:?set the full expected source commit}"
: "${IMAGE_SIGNING_KEY_URI:?set the CI KMS signing-key URI}"
: "${IMAGE_VERIFY_KEY:?set the read-only image verification key}"

test "$(git rev-parse HEAD)" = "$GIT_COMMIT"
test -z "$(git status --porcelain --untracked-files=normal)"
if [[ ! "$GIT_COMMIT" =~ ^[0-9a-f]{40}$ ]]; then
  echo "GIT_COMMIT must be a full lowercase SHA-1 commit" >&2
  exit 2
fi

evidence_dir="evidence/training-image/$GIT_COMMIT"
mkdir -p "$evidence_dir"
metadata="$evidence_dir/build-metadata.json"

docker buildx build   --pull   --provenance=mode=max   --tag "$IMAGE_REPOSITORY:$GIT_COMMIT"   --metadata-file "$metadata"   --push   .

digest="$(python - "$metadata" <<'PY'
import json, re, sys
from pathlib import Path
value = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
digest = value.get("containerimage.digest")
if not isinstance(digest, str) or re.fullmatch(r"sha256:[0-9a-f]{64}", digest) is None:
    raise SystemExit("BuildKit metadata has no valid containerimage.digest")
print(digest)
PY
)"
readonly digest
image_ref="$IMAGE_REPOSITORY@$digest"
readonly image_ref

cosign sign --yes --key "$IMAGE_SIGNING_KEY_URI" "$image_ref"
cosign verify --key "$IMAGE_VERIFY_KEY" "$image_ref"   > "$evidence_dir/cosign-verification.json"
docker buildx imagetools inspect "$image_ref"   > "$evidence_dir/registry-readback.txt"

python - "$evidence_dir/release.json" "$GIT_COMMIT" "$image_ref" "$metadata" <<'PY'
import hashlib, json, sys
from pathlib import Path
output, commit, image_ref, metadata = sys.argv[1:]
metadata_bytes = Path(metadata).read_bytes()
Path(output).write_text(json.dumps({
    "schema_version": "aidefend.training-image-release.v1",
    "source_commit": commit,
    "image_reference": image_ref,
    "build_metadata_sha256": hashlib.sha256(metadata_bytes).hexdigest(),
}, indent=2, sort_keys=True) + "\n", encoding="utf-8")
PY

printf '%s\n' "$image_ref"
Use only the digest-qualified reference

The final output has the form registry.example/repository@sha256:.... Bind that exact reference, source commit, build metadata, signature verification, platform selection, and scheduler receipt to the training run. The deployment admission identity should repeat cosign verify and registry readback immediately before scheduling. A tag-only reference, dirty or mismatched commit, missing digest, signature failure, registry readback failure, or digest drift blocks training; the tag is never accepted as immutable evidence.

Replay the retained run independently

The prior steps capture inputs; they do not establish reproducibility until the training job actually runs again. Store one signed original-run manifest with image_reference, source_commit, platform, hardware_profile_sha256, exact dataset_files digests, dependency lock digest, seed, training command argv and original metrics. Use an independently provisioned replay worker that obtains the immutable dataset snapshot and verifies the manifest/signature/current replay policy before this module runs. The approved training program must consume TRAINING_SEED, seed every relevant framework RNG and write /output/metrics.json. This CPU container example is NOT_APPLICABLE as an execution method when the job needs a GPU/distributed runtime; use that runtime's admitted replay adapter with the same evidence bindings, not a false CPU replay.

# File: ci/replay_training.py
import hashlib
import json
import math
import re
import subprocess
import sys
import uuid
from pathlib import Path

def replay(manifest, policy, dataset_root, output_root):
    # Inputs are captured from verified original-run and replay-policy artifacts.
    image = manifest["image_reference"]
    if not re.fullmatch(r"[^\s]+@sha256:[a-f0-9]{64}", image):
        raise ValueError("immutable training image required")
    if manifest["platform"] != policy["platform"] or manifest["hardware_profile_sha256"] != policy["hardware_profile_sha256"]:
        raise ValueError("replay runtime differs from approved comparison profile")
    deadline = policy["timeout_seconds"]
    if not math.isfinite(deadline) or deadline <= 0 or type(manifest["seed"]) is not int:
        raise ValueError("invalid governed replay bounds")
    # Inspect the actual current worker hardware outside this process before
    # passing policy; a matching string copied from the manifest is not evidence.
    subprocess.run(["cosign", "verify", "--key", policy["image_verify_key"], image],
                   check=True, capture_output=True, timeout=deadline)
    subprocess.run(["docker", "pull", "--platform", manifest["platform"], image],
                   check=True, capture_output=True, timeout=deadline)
    root = Path(dataset_root).resolve(strict=True)
    files = list(root.rglob("*"))
    if any(path.is_symlink() for path in files):
        raise ValueError("replay dataset contains links")
    actual = {path.relative_to(root).as_posix() for path in files if path.is_file()}
    if actual != set(manifest["dataset_files"]):
        raise ValueError("replay dataset population differs")
    for name, expected in manifest["dataset_files"].items():
        digest = hashlib.sha256()
        with (root / name).open("rb") as source:
            for chunk in iter(lambda: source.read(1024 * 1024), b""):
                digest.update(chunk)
        if digest.hexdigest() != expected:
            raise ValueError("replay dataset bytes differ: " + name)
    command = manifest["command"]
    if not isinstance(command, list) or not command or any(not isinstance(x, str) or not x for x in command):
        raise ValueError("approved training argv required")
    output = Path(output_root).resolve()
    output.mkdir(exist_ok=False, parents=True)
    container_name = "training-replay-" + uuid.uuid4().hex
    argv = ["docker", "run", "--rm", "--name", container_name,
            "--platform", manifest["platform"], "--network", "none",
            "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
            "--user", policy["uid_gid"], "--cpus", str(policy["cpus"]),
            "--memory", policy["memory"], "--pids-limit", str(policy["pids_limit"]),
            "--tmpfs", policy["tmpfs"],
            "--mount", "type=bind,src=" + str(root) + ",dst=/data,readonly",
            "--mount", "type=bind,src=" + str(output) + ",dst=/output",
            "--env", "TRAINING_SEED=" + str(manifest["seed"]),
            "--entrypoint", command[0], image, *command[1:]]
    try:
        subprocess.run(argv, check=True, timeout=deadline)
    finally:
        # Stop the container as well as the client on timeout; the shared I001
        # scheduler additionally owns process/resource readback and teardown.
        subprocess.run(["docker", "rm", "--force", container_name],
                       check=False, capture_output=True, timeout=policy["cleanup_timeout_seconds"])
    result = json.loads((output / "metrics.json").read_bytes())
    comparisons = {}
    for name, baseline in manifest["metrics"].items():
        value, tolerance = result[name], policy["absolute_tolerances"][name]
        if any(type(x) not in (int, float) or not math.isfinite(x) for x in (value, baseline, tolerance)) or tolerance < 0:
            raise ValueError("invalid replay metric/comparator")
        comparisons[name] = {"baseline": baseline, "replayed": value,
                             "absolute_tolerance": tolerance, "matched": abs(value-baseline) <= tolerance}
    if not comparisons:
        raise ValueError("empty comparison population")
    return {"status": "PASS" if all(x["matched"] for x in comparisons.values()) else "FAIL",
            "image_reference": image, "source_commit": manifest["source_commit"],
            "seed": manifest["seed"], "comparisons": comparisons}

# The project training argv inside the admitted image must initialize every
# framework RNG from TRAINING_SEED and write its metrics.json to /output.
# Store source/data/dependency/image/seed/command/hardware and original metrics
# in the signed original-run manifest. Invoke replay(...) from an independently
# credentialed worker with verified captured inputs, then sign/read back its
# result. Never treat publishing an image or recording a seed as replay PASS.

Validate the observed worker hardware/driver profile against the signed policy before replay. AID-I-001 owns resource/isolation assurance and teardown readback; the Docker flags are bounded execution configuration, not an additional isolation PASS. Retain actual scheduler/container exit evidence and compare the complete declared metric population using policy-derived absolute tolerances. For deterministic builds additionally compare model artifact digests; for stochastic training retain the preregistered repeated-seed population and statistical comparator rather than demanding universal byte equality. Missing inputs, unsupported runtime, timeout, absent metrics or failed signature verification cannot be PASS. A metric mismatch is FAIL and remains visible even when task quality is acceptable.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-H-007.003: Training Process Reproducibility | AIDEFEND

Loading interactive view…

AIDEFEND™

A Practical Knowledge Base for AI Security Defenses

Version

GitHub MCP Service