AID-H-009.004: Accelerator Isolation & VRAM/KV-Cache Hygiene
torch.cuda.empty_cache() is not secure zeroization and must never be accepted as proof.Pillars: infra
Lifecycle: operation
Scope boundary
Owns physical accelerator assignment, partition, reset, teardown, and low-level device-lifecycle evidence before reuse across tenants or trust zones. Framework allocator cleanup is not accepted as secure zeroization.
AID-H-027: AID-H-009.004 protects physical accelerator and device-memory reuse; AID-H-027 protects logical inference caches through key binding, integrity, partitioning, expiry, and safe reuse policy. Application cache eviction is not proof of hardware reset, and device reset does not validate cache keys.
AID-H-032.002: AID-H-009.004 owns low-level device assignment and reset across trust zones; AID-H-032.002 owns broader cross-tenant serving-state isolation across batching, KV cache, allocators, buffers, and runtime workers. Device lifecycle evidence is one layer of the complete serving-state boundary.
Implementation guidance
- AID-H-009.004-G001: Verify a signed, device-bound reset or disposable-worker termination receipt before every cross-zone accelerator reassignment.
- AID-H-009.004-G002: Independently verify exclusive or partitioned accelerator ownership from a signed scheduler allocation snapshot.
AID-H-009.004-G001
Before you begin
Unsupported hardware reset, missing stable device identity, unavailable provider evidence, or stale evidence is insufficient_data and denies reuse.
# File: accelerator/verify_reuse_transition.py
from __future__ import annotations
import math
import os
import json
import subprocess
import tempfile
from datetime import datetime, timezone
from pathlib import Path
def _required_positive_seconds(name: str) -> float:
value = float(os.environ[name])
if not math.isfinite(value) or value <= 0:
raise RuntimeError(f"{name} must be a finite positive number of seconds")
return value
ACCELERATOR_RECEIPT_VERIFY_TIMEOUT_SECONDS = _required_positive_seconds("VERIFIED_H009_ACCELERATOR_RECEIPT_VERIFY_TIMEOUT_SECONDS")
def strict_json_object(raw: bytes, label: str) -> dict:
def unique_pairs(pairs):
value = {}
for key, item in pairs:
if key in value:
raise ValueError(f"duplicate {label} key: {key}")
value[key] = item
return value
def reject_constant(value):
raise ValueError(f"non-finite {label} number: {value}")
document = json.loads(
raw.decode("utf-8", errors="strict"),
object_pairs_hook=unique_pairs, parse_constant=reject_constant,
)
if not isinstance(document, dict):
raise ValueError(f"{label} root must be an object")
return document
def parse_time(value: str) -> datetime:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise ValueError("timestamp must include a timezone")
return parsed.astimezone(timezone.utc)
def verify_transition(
transition_path: Path,
receipt_path: Path,
receipt_bundle: Path,
policy: dict,
) -> dict:
receipt_raw = receipt_path.read_bytes()
snapshot: Path | None = None
try:
with tempfile.NamedTemporaryFile(
prefix="accelerator-boundary-receipt-", suffix=".json", delete=False
) as handle:
handle.write(receipt_raw)
handle.flush()
os.fsync(handle.fileno())
snapshot = Path(handle.name)
subprocess.run([
"cosign", "verify-blob",
"--bundle", str(receipt_bundle),
"--certificate-identity", policy["lifecycle_certificate_identity"],
"--certificate-oidc-issuer", policy["lifecycle_oidc_issuer"],
str(snapshot),
], check=True, timeout=ACCELERATOR_RECEIPT_VERIFY_TIMEOUT_SECONDS)
finally:
if snapshot is not None:
snapshot.unlink(missing_ok=True)
transition = strict_json_object(transition_path.read_bytes(), "transition")
receipt = strict_json_object(receipt_raw, "receipt")
required_transition = {
"schema_version", "device_uuid", "node_uid", "source_job_id",
"source_trust_zone", "destination_trust_zone", "released_at",
}
if set(transition) != required_transition:
raise ValueError("transition schema differs")
if transition["source_trust_zone"] == transition["destination_trust_zone"]:
return {"outcome": "not_applicable", "reason": "same_trust_zone"}
if receipt.get("schema_version") != "aidefend.accelerator-boundary-receipt.v1":
raise ValueError("boundary receipt schema differs")
for field in ("device_uuid", "node_uid", "source_job_id", "source_trust_zone"):
if receipt.get(field) != transition[field]:
raise ValueError(f"boundary receipt {field} binding differs")
if receipt.get("destination_trust_zone") != transition["destination_trust_zone"]:
raise ValueError("destination trust-zone binding differs")
completed = parse_time(receipt["completed_at"])
released = parse_time(transition["released_at"])
age = (datetime.now(timezone.utc) - completed).total_seconds()
if completed < released or age < 0 or age > int(policy["max_receipt_age_seconds"]):
raise ValueError("boundary receipt is stale or precedes release")
boundary = receipt.get("boundary_type")
if boundary == "disposable_worker_terminated":
evidence = receipt.get("termination_evidence")
if (
not isinstance(evidence, dict)
or evidence.get("instance_id") != receipt.get("instance_id")
or evidence.get("node_uid") != transition["node_uid"]
or evidence.get("state") != "terminated"
or not evidence.get("provider_event_id")
):
raise ValueError("worker termination evidence is incomplete")
elif boundary == "vendor_reset_with_verified_receipt":
evidence = receipt.get("reset_evidence")
if (
not isinstance(evidence, dict)
or evidence.get("device_uuid") != transition["device_uuid"]
or evidence.get("result") != "succeeded"
or not evidence.get("operation_id")
or evidence.get("adapter_version") not in policy["approved_reset_adapter_versions"]
):
raise ValueError("vendor reset evidence is incomplete or unsupported")
else:
raise ValueError("boundary type is not approved")
return {
"schema_version": "aidefend.accelerator-reuse-verification.v1",
"outcome": "pass",
"device_uuid": transition["device_uuid"],
"node_uid": transition["node_uid"],
"source_job_id": transition["source_job_id"],
"source_trust_zone": transition["source_trust_zone"],
"destination_trust_zone": transition["destination_trust_zone"],
"boundary_type": boundary,
"completed_at": receipt["completed_at"],
}
Verify safely
A separately credentialed verifier independently reads back or replays every cross-tenant, cross-zone or lower-to-higher-trust accelerator/worker reassignment, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.
AID-H-009.004-G002
# File: accelerator/verify_partition_snapshot.py
from __future__ import annotations
from collections import defaultdict
def verify_partition_snapshot(snapshot: dict, policy: dict) -> dict:
if snapshot.get("schema_version") != "aidefend.accelerator-allocation-snapshot.v1":
raise ValueError("allocation snapshot schema differs")
allocations = snapshot.get("allocations")
if not isinstance(allocations, list):
raise ValueError("allocation list is missing")
owners: dict[tuple[str, str], set[str]] = defaultdict(set)
for item in allocations:
required = {"device_uuid", "partition_id", "trust_zone", "workload_uid", "sharing_mode"}
if set(item) != required or not all(item.values()):
raise ValueError("allocation record is incomplete")
if item["sharing_mode"] not in policy["approved_sharing_modes"]:
raise PermissionError("accelerator sharing mode is not approved")
owners[(item["device_uuid"], item["partition_id"])].add(item["trust_zone"])
conflicts = [
{"device_uuid": key[0], "partition_id": key[1], "trust_zones": sorted(zones)}
for key, zones in owners.items()
if len(zones) > 1
]
return {
"schema_version": "aidefend.accelerator-partition-verification.v1",
"outcome": "fail" if conflicts else "pass",
"snapshot_id": snapshot["snapshot_id"],
"allocator_policy_version": snapshot["allocator_policy_version"],
"conflicts": conflicts,
"allocations_measured": len(allocations),
}
Keep the action safely bounded
Admit only signed supported partition profiles, bind device/partition IDs to one workload identity and trust zone, deny conflicting allocations/debug/peer paths, read back scheduler plus device/vendor state independently, and evict/quarantine on mismatch.
Verify safely
A separately credentialed verifier independently reads back or replays every active exclusive, MIG/vGPU or other supported partition allocation and its tenant/trust-zone ownership, exercises positive, negative, boundary and dependency-failure fixtures, and reconciles the complete eligible population.
Open-source tools
- NVIDIA GPU Operator
- NVIDIA k8s-device-plugin
- NVIDIA Data Center GPU Manager (DCGM; allocation and health readback only)
- vLLM (serving-engine prefix KV-cache reset only; does not prove accelerator-memory zeroization or cross-tenant isolation)
- Kata Containers (workload VM boundary only; independent accelerator partition and reset evidence required)
Commercial / hosted tools
- NVIDIA System Management Interface (nvidia-smi)
- NVIDIA vGPU Software (MIG-backed isolation)
Threat mappings
MITRE ATLAS
- AML.T0025 Exfiltration via Cyber Means
- AML.T0037 Data from Local System
- AML.T0044 Full AI Model Access (VRAM isolation prevents co-tenants from gaining full access to model weights in shared accelerator memory)
MAESTRO
- Data Exfiltration (L2)
- Data Leakage (Cross-Layer) (VRAM clearing prevents cross-tenant data leakage)
OWASP LLM Top 10 2026
- LLM02:2026 Sensitive Information Disclosure
OWASP ML Top 10 2023
- ML05:2023 Model Theft
OWASP Top 10 for Agentic Applications 2026
- N/A
NIST Adversarial Machine Learning 2025
- N/A
Cisco Integrated AI Security and Safety Framework
- AITech-14.1 Unauthorized Access (Enforces isolation of GPU allocations and released accelerator memory)
- AITech-8.2 Data Exfiltration / Exposure
- AITech-8.3 Information Disclosure
- AISubtech-10.1.2 Weight Reconstruction (prevents parameter recovery from residual accelerator state across trust zones)
- AISubtech-14.1.2 Insufficient Access Controls (independently verified accelerator allocation and cross-zone residual-state exclusion)
Google Secure AI Framework 2.0 - Risks
- MXF: Model Exfiltration (VRAM isolation prevents model data leakage between tenants)
Databricks AI Security Framework 3.0
- Model Serving - Inference requests 9.10: Accidental exposure of unauthorized data to models (VRAM hygiene prevents accidental data exposure)