AID-H-003.010: Deployed AI Software Vulnerability Remediation Lifecycle
Pillars: infra, app
Lifecycle: operation, improvement
Scope boundary
Owns continuous vulnerability discovery, prioritization, remediation decisions, compatible rebuild or replacement, staged deployment, rollback, compensating-control expiry, and complete-fleet verification for deployed AI software after initial build or deployment admission.
AID-H-003.001: AID-H-003.010 continuously remediates vulnerabilities disclosed in deployed software; AID-H-003.001 decides whether exact dependency bytes may enter a build or deployment. Admission is a point-in-time gate, while remediation maintains the approved posture throughout operation.
AID-H-003.002: AID-H-003.010 owns the need, deadline, staged rollout, rollback, and fleet closure for a software remediation; AID-H-003.002 owns final promotion, signing, and distribution of an approved release artifact. A remediation normally uses the release gate, but passing that gate does not prove the vulnerable deployed population was replaced.
AID-H-009.002: AID-H-003.010 covers AI application, framework, runtime, SDK, plugin, container, and library software; AID-H-009.002 covers accelerator firmware and drivers. The inventories, compatibility constraints, rollout mechanisms, and recovery evidence differ even when one change requires coordinated testing with the other.
AID-H-028.006: AID-H-003.010 decides whether deployed software must be upgraded, rebuilt, replaced, disabled, or temporarily isolated and verifies fleet completion; AID-H-028.006 verifies MCP client, plugin, and update authenticity and integrity. Update authenticity answers whether bytes are trusted, not whether remediation is timely, compatible, complete, or still required.
Warning
Remediation Is Broader Than Installing a Patch
A vendor patch may be unavailable, disputed, or incompatible with the deployed CUDA, driver, framework, model, or plugin matrix. A bounded remediation may instead rebuild on a fixed base image, upgrade or replace the component, disable the vulnerable feature, remove exposure, or isolate it under an approved compensating control. Every exception needs an owner, affected population, expiry, next review, replacement plan, and independent readback; an indefinite risk acceptance is not remediation.
Implementation guidance
- AID-H-003.010-G001: Continuously reconcile the exact deployed AI software population with versioned ecosystem advisories and issue signed, deadline-bound remediation decisions.
- AID-H-003.010-G002: Rebuild and redeploy fixed AI software through a drained canary and staged fleet rollout, with rollback and digest-level completion readback.
- AID-H-003.010-G003: Use expiring compensating controls when no compatible fix exists, and keep the vulnerability open until replacement or independently verified remediation.
AID-H-003.010-G001
Build the deployed population, not a repository-only list
Apply this method to every production and production-equivalent inference server, training or orchestration framework, agent runtime, MCP client or server, SDK, plugin, container, and supporting library. Export the exact running image or package digest and bind it to workload, cluster, tenant, owner, release, and first-seen time. A source lockfile is supporting evidence but cannot substitute for runtime readback.
Scan immutable bytes with pinned advisory snapshots
set -euo pipefail
umask 077
: "$DEPLOYED_COMPONENT_ID" "$DEPLOYED_IMAGE_DIGEST" "$SCANNER_IMAGE"
[[ "$DEPLOYED_IMAGE_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "exact immutable deployed digest required" >&2; exit 2; }
case "$SCANNER_IMAGE" in *@sha256:*) ;; *) echo "digest-pinned scanner image required" >&2; exit 2 ;; esac
test -s evidence/deployed-component.oci.tar
test -s policy/remediation-policy.json
test -s advisories/advisory-snapshot.manifest.json
mkdir -p evidence/remediation
docker run --rm --network none --read-only --tmpfs /tmp:rw,noexec,nosuid,size=256m -e GRYPE_DB_AUTO_UPDATE=false -e TRIVY_SKIP_DB_UPDATE=true -v "$PWD/evidence/deployed-component.oci.tar:/input/component.oci.tar:ro" -v "$PWD/advisories:/advisories:ro" -v "$PWD/evidence/remediation:/out" "$SCANNER_IMAGE" sh -ceu '
export GRYPE_DB_CACHE_DIR=/advisories/grype
syft oci-archive:/input/component.oci.tar -o cyclonedx-json=/out/sbom.cdx.json
grype sbom:/out/sbom.cdx.json -o json > /out/grype.json
trivy image --input /input/component.oci.tar --cache-dir /advisories/trivy --skip-db-update --scanners vuln --format json --output /out/trivy.json
syft version > /out/syft.version
grype version > /out/grype.version
trivy --version > /out/trivy.version
'
jq -e '.bomFormat == "CycloneDX" and (.components | type == "array")' evidence/remediation/sbom.cdx.json >/dev/null
sha256sum evidence/deployed-component.oci.tar policy/remediation-policy.json advisories/advisory-snapshot.manifest.json evidence/remediation/* > evidence/remediation/sha256sums.txtAction: Normalize OSV or ecosystem advisories, GitHub Security Advisories, vendor notices such as NVIDIA PSIRT, exploited-vulnerability intelligence, and scanner findings into one case keyed by component identity and deployed digest. The signature-verified policy supplies advisory freshness, exploitability rules, deadlines, and required feeds. Missing inventory, stale feeds, an unsupported ecosystem, scanner disagreement, or tool error is INSUFFICIENT_DATA or ERROR, never PASS.
Evidence and independent replay
Sign the inventory snapshot, advisory manifests and fetch receipts, SBOM, raw findings, policy and scanner digests, normalized case, owner, deadline, and decision. A separate read-only verifier must reproduce the component population and advisory join from immutable bytes and confirm that every affected deployed digest has exactly one open remediation, verified closure, or unexpired exception.
AID-H-003.010-G002
Patch by rebuilding immutable artifacts
Do not modify long-running inference or agent containers in place. Re-resolve fixed dependencies, rebuild the image, rerun AID-H-003.001 admission and AID-H-003.002 release gating, and deploy only the resulting signed digest. Validate the CUDA, driver, framework, model, adapter, and plugin compatibility matrix before production promotion.
Promote one bounded Kubernetes ring
set -euo pipefail
umask 077
: "$NAMESPACE" "$CANARY_DEPLOYMENT" "$CONTAINER_NAME" "$FIXED_IMAGE"
case "$FIXED_IMAGE" in *@sha256:*) ;; *) echo "fixed image must be digest pinned" >&2; exit 2 ;; esac
mkdir -p evidence/remediation-rollout
kubectl -n "$NAMESPACE" get deployment "$CANARY_DEPLOYMENT" -o json > evidence/remediation-rollout/before.json
kubectl -n "$NAMESPACE" patch deployment "$CANARY_DEPLOYMENT" --type merge -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}'
kubectl -n "$NAMESPACE" set image "deployment/$CANARY_DEPLOYMENT" "$CONTAINER_NAME=$FIXED_IMAGE"
kubectl -n "$NAMESPACE" rollout status "deployment/$CANARY_DEPLOYMENT" --timeout=10m
kubectl -n "$NAMESPACE" get deployment "$CANARY_DEPLOYMENT" -o json > evidence/remediation-rollout/after.json
jq -e --arg image "$FIXED_IMAGE" --arg container "$CONTAINER_NAME" '
[.spec.template.spec.containers[] | select(.name == $container) | .image] == [$image]
and .status.updatedReplicas == .spec.replicas
and .status.availableReplicas == .spec.replicas
and .status.unavailableReplicas == 0
' evidence/remediation-rollout/after.json >/dev/nullAction: Route only the policy-defined canary population to the fixed digest, drain in-flight inference, Tasks, subscriptions, queues, and GPU work before terminating old replicas, and run vulnerability, model-output, latency, resource, tool-call, and rollback probes. Promote each later ring only from a signed canary decision. On regression, restore the last approved digest while keeping the vulnerability case open and activating an approved compensating control.
Close from fleet evidence
After every ring, independently enumerate workloads, nodes, images, packages, and long-lived developer or MCP tooling in the affected population. A release receipt without runtime readback is insufficient. Retain old and new digests, compatibility results, drain and rollout events, probe outputs, rollback result, owner approvals, and an independent query proving that no affected digest remains outside an approved exception.
AID-H-003.010-G003
Make a no-patch path explicit and temporary
A missing vendor fix, disputed CVE, or broken CUDA and framework compatibility matrix does not permit silent deferral. The security owner records the exact vulnerable digests and deployments, exploit preconditions, exposure, selected controls, residual risk, owner, next review, hard expiry, and replacement plan. Controls may remove public exposure, disable the vulnerable endpoint or feature, restrict ingress and egress, isolate the workload, or replace the component.
Validate the exception contract before enforcement
# File: remediation/validate_exception.py
from __future__ import annotations
import json
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
def no_duplicate_keys(pairs: list[tuple[str, object]]) -> dict:
result: dict = {}
for key, value in pairs:
if key in result:
raise ValueError(f"duplicate JSON key: {key}")
result[key] = value
return result
def reject_constant(value: str) -> None:
raise ValueError(f"non-finite JSON number: {value}")
FIELDS = {
"schema_version", "exception_id", "vulnerability_ids", "affected_digests",
"affected_deployments", "owner", "approved_by", "issued_at", "next_review_at",
"expires_at", "compensating_controls", "verification_receipts", "replacement_plan"
}
def timestamp(value: object) -> datetime:
if not isinstance(value, str):
raise ValueError("timestamp must be a string")
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise ValueError("timestamp must include an offset")
return parsed.astimezone(timezone.utc)
raw = Path(sys.argv[1]).read_bytes()
if not raw or len(raw) > 1024 * 1024:
raise ValueError("exception document is empty or oversized")
record = json.loads(raw.decode("utf-8", errors="strict"), object_pairs_hook=no_duplicate_keys, parse_constant=reject_constant)
if not isinstance(record, dict) or set(record) != FIELDS:
raise ValueError("exception schema differs")
if record["schema_version"] != "aidefend.software-remediation-exception.v1":
raise ValueError("unsupported exception schema")
for field in ("vulnerability_ids", "affected_digests", "affected_deployments",
"compensating_controls"):
value = record[field]
if (not isinstance(value, list) or not value or
any(not isinstance(item, str) or not item for item in value) or
len(value) != len(set(value))):
raise ValueError(field + " must be a nonempty unique string list")
receipts = record["verification_receipts"]
if not isinstance(receipts, list) or not receipts:
raise ValueError("verification_receipts must be nonempty")
for receipt in receipts:
if (not isinstance(receipt, dict) or set(receipt) != {"control_id", "target_id", "receipt_sha256", "outcome"} or
receipt["outcome"] != "PASS" or receipt["target_id"] not in record["affected_deployments"] or
not isinstance(receipt["control_id"], str) or not receipt["control_id"] or
not isinstance(receipt["receipt_sha256"], str) or
re.fullmatch(r"[0-9a-f]{64}", receipt["receipt_sha256"]) is None):
raise ValueError("verification receipt is unbound or invalid")
if any(not isinstance(value, str) or not value for value in
(record["exception_id"], record["owner"], record["approved_by"],
record["replacement_plan"])):
raise ValueError("exception identity, ownership, and replacement plan are required")
issued = timestamp(record["issued_at"])
review = timestamp(record["next_review_at"])
expires = timestamp(record["expires_at"])
now = datetime.now(timezone.utc)
if not issued <= now < review <= expires:
raise PermissionError("exception is future-dated, expired, or overdue for review")
print(json.dumps({"outcome": "PASS", "exception_id": record["exception_id"],
"expires_at": record["expires_at"]}, sort_keys=True))Action: Verify the signed exception before applying it, fail closed when the policy, enforcement, or readback is unavailable, and automatically page the owner before review and expiry. Independently probe the blocked exposure or disabled feature and compare the complete affected population with enforcement targets. At expiry, remediate, replace, remove, or stop the affected component; do not auto-renew.
Outcome
Use PASS only for a fully remediated population. An active, verified exception is an explicitly bounded interim state, not closure. Missing population coverage is INSUFFICIENT_DATA; failed enforcement is FAIL; verifier or control-plane failure is ERROR.
Open-source tools
- Trivy
- Syft
- Grype
- pip-audit
- OSV-Scanner
- OWASP Dependency-Track
Commercial / hosted tools
- Snyk Open Source
- Mend SCA
- JFrog Xray
- Qualys VMDR
Threat mappings
MITRE ATLAS
- AML.T0010 AI Supply Chain Compromise
- AML.T0010.001 AI Supply Chain Compromise: AI Software
- AML.T0049 Exploit Public-Facing Application
- AML.T0122 Exploitation of Remote Services (patching reachable AI services removes the weaknesses lateral exploitation depends on)
MAESTRO
- Compromised Framework Components (L3)
- Compromised Container Images (L4)
- Supply Chain Attacks (Cross-Layer)
OWASP LLM Top 10 2026
- LLM04:2026 Supply Chain
OWASP ML Top 10 2023
- ML06:2023 AI Supply Chain Attacks
OWASP Top 10 for Agentic Applications 2026
- ASI04:2026 Agentic Supply Chain Vulnerabilities
- ASI05:2026 Unexpected Code Execution (RCE)
NIST Adversarial Machine Learning 2025
- N/A
Cisco Integrated AI Security and Safety Framework
- AITech-14.1 Unauthorized Access (Remediates exploitable weaknesses in the deployed AI hosting infrastructure)
- AITech-9.3 Dependency / Plugin Compromise
Google Secure AI Framework 2.0 - Risks
- IIC: Insecure Integrated Component
Databricks AI Security Framework 3.0
- Platform 12.1: Lack of vulnerability management
- Agents - Tools MCP Client 13.35: Update and Patch Management