AID-H-003.010: Deployed AI Software Vulnerability Remediation Lifecycle

Version 1.20261015 · Actionable sub-technique

Continuously identify and remediate known vulnerabilities in deployed AI software components, including inference servers, orchestration frameworks, agent and MCP runtimes or SDKs, plugins, containers, and supporting libraries after initial admission. Maintain a versioned deployed-component inventory, ecosystem-specific advisory coverage, policy-defined remediation deadlines, rebuild-and-redeploy updates, staged rollout and rollback, and time-bound compensating controls when a compatible fix is unavailable. Close a finding only after independent fleet readback proves the affected population is remediated or an approved exception remains effective.

Pillars: infra, app

Lifecycle: operation, improvement

Scope boundary

Owns continuous vulnerability discovery, prioritization, remediation decisions, compatible rebuild or replacement, staged deployment, rollback, compensating-control expiry, and complete-fleet verification for deployed AI software after initial build or deployment admission.

AID-H-003.001: AID-H-003.010 continuously remediates vulnerabilities disclosed in deployed software; AID-H-003.001 decides whether exact dependency bytes may enter a build or deployment. Admission is a point-in-time gate, while remediation maintains the approved posture throughout operation.

AID-H-003.002: AID-H-003.010 owns the need, deadline, staged rollout, rollback, and fleet closure for a software remediation; AID-H-003.002 owns final promotion, signing, and distribution of an approved release artifact. A remediation normally uses the release gate, but passing that gate does not prove the vulnerable deployed population was replaced.

AID-H-009.002: AID-H-003.010 covers AI application, framework, runtime, SDK, plugin, container, and library software; AID-H-009.002 covers accelerator firmware and drivers. The inventories, compatibility constraints, rollout mechanisms, and recovery evidence differ even when one change requires coordinated testing with the other.

AID-H-028.006: AID-H-003.010 decides whether deployed software must be upgraded, rebuilt, replaced, disabled, or temporarily isolated and verifies fleet completion; AID-H-028.006 verifies MCP client, plugin, and update authenticity and integrity. Update authenticity answers whether bytes are trusted, not whether remediation is timely, compatible, complete, or still required.

Warning

Remediation Is Broader Than Installing a Patch

A vendor patch may be unavailable, disputed, or incompatible with the deployed CUDA, driver, framework, model, or plugin matrix. A bounded remediation may instead rebuild on a fixed base image, upgrade or replace the component, disable the vulnerable feature, remove exposure, or isolate it under an approved compensating control. Every exception needs an owner, affected population, expiry, next review, replacement plan, and independent readback; an indefinite risk acceptance is not remediation.

Implementation guidance

AID-H-003.010-G001

Continuously reconcile the exact deployed AI software population with versioned ecosystem advisories and issue signed, deadline-bound remediation decisions.
Build the deployed population, not a repository-only list

Apply this method to every production and production-equivalent inference server, training or orchestration framework, agent runtime, MCP client or server, SDK, plugin, container, and supporting library. Export the exact running image or package digest and bind it to workload, cluster, tenant, owner, release, and first-seen time. A source lockfile is supporting evidence but cannot substitute for runtime readback.

Scan immutable bytes with pinned advisory snapshots
set -euo pipefail
umask 077
: "$DEPLOYED_COMPONENT_ID" "$DEPLOYED_IMAGE_DIGEST" "$SCANNER_IMAGE"
[[ "$DEPLOYED_IMAGE_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "exact immutable deployed digest required" >&2; exit 2; }
case "$SCANNER_IMAGE" in *@sha256:*) ;; *) echo "digest-pinned scanner image required" >&2; exit 2 ;; esac
test -s evidence/deployed-component.oci.tar
test -s policy/remediation-policy.json
test -s advisories/advisory-snapshot.manifest.json
mkdir -p evidence/remediation
docker run --rm --network none --read-only   --tmpfs /tmp:rw,noexec,nosuid,size=256m   -e GRYPE_DB_AUTO_UPDATE=false -e TRIVY_SKIP_DB_UPDATE=true   -v "$PWD/evidence/deployed-component.oci.tar:/input/component.oci.tar:ro"   -v "$PWD/advisories:/advisories:ro"   -v "$PWD/evidence/remediation:/out"   "$SCANNER_IMAGE" sh -ceu '
    export GRYPE_DB_CACHE_DIR=/advisories/grype
    syft oci-archive:/input/component.oci.tar -o cyclonedx-json=/out/sbom.cdx.json
    grype sbom:/out/sbom.cdx.json -o json > /out/grype.json
    trivy image --input /input/component.oci.tar --cache-dir /advisories/trivy       --skip-db-update --scanners vuln --format json --output /out/trivy.json
    syft version > /out/syft.version
    grype version > /out/grype.version
    trivy --version > /out/trivy.version
  '
jq -e '.bomFormat == "CycloneDX" and (.components | type == "array")' evidence/remediation/sbom.cdx.json >/dev/null
sha256sum evidence/deployed-component.oci.tar policy/remediation-policy.json   advisories/advisory-snapshot.manifest.json evidence/remediation/*   > evidence/remediation/sha256sums.txt

Action: Normalize OSV or ecosystem advisories, GitHub Security Advisories, vendor notices such as NVIDIA PSIRT, exploited-vulnerability intelligence, and scanner findings into one case keyed by component identity and deployed digest. The signature-verified policy supplies advisory freshness, exploitability rules, deadlines, and required feeds. Missing inventory, stale feeds, an unsupported ecosystem, scanner disagreement, or tool error is INSUFFICIENT_DATA or ERROR, never PASS.

Evidence and independent replay

Sign the inventory snapshot, advisory manifests and fetch receipts, SBOM, raw findings, policy and scanner digests, normalized case, owner, deadline, and decision. A separate read-only verifier must reproduce the component population and advisory join from immutable bytes and confirm that every affected deployed digest has exactly one open remediation, verified closure, or unexpired exception.

AID-H-003.010-G002

Rebuild and redeploy fixed AI software through a drained canary and staged fleet rollout, with rollback and digest-level completion readback.
Patch by rebuilding immutable artifacts

Do not modify long-running inference or agent containers in place. Re-resolve fixed dependencies, rebuild the image, rerun AID-H-003.001 admission and AID-H-003.002 release gating, and deploy only the resulting signed digest. Validate the CUDA, driver, framework, model, adapter, and plugin compatibility matrix before production promotion.

Promote one bounded Kubernetes ring
set -euo pipefail
umask 077
: "$NAMESPACE" "$CANARY_DEPLOYMENT" "$CONTAINER_NAME" "$FIXED_IMAGE"
case "$FIXED_IMAGE" in *@sha256:*) ;; *) echo "fixed image must be digest pinned" >&2; exit 2 ;; esac
mkdir -p evidence/remediation-rollout
kubectl -n "$NAMESPACE" get deployment "$CANARY_DEPLOYMENT" -o json   > evidence/remediation-rollout/before.json
kubectl -n "$NAMESPACE" patch deployment "$CANARY_DEPLOYMENT" --type merge -p   '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}'
kubectl -n "$NAMESPACE" set image   "deployment/$CANARY_DEPLOYMENT" "$CONTAINER_NAME=$FIXED_IMAGE"
kubectl -n "$NAMESPACE" rollout status   "deployment/$CANARY_DEPLOYMENT" --timeout=10m
kubectl -n "$NAMESPACE" get deployment "$CANARY_DEPLOYMENT" -o json   > evidence/remediation-rollout/after.json
jq -e --arg image "$FIXED_IMAGE" --arg container "$CONTAINER_NAME" '
  [.spec.template.spec.containers[] | select(.name == $container) | .image] == [$image]
  and .status.updatedReplicas == .spec.replicas
  and .status.availableReplicas == .spec.replicas
  and .status.unavailableReplicas == 0
' evidence/remediation-rollout/after.json >/dev/null

Action: Route only the policy-defined canary population to the fixed digest, drain in-flight inference, Tasks, subscriptions, queues, and GPU work before terminating old replicas, and run vulnerability, model-output, latency, resource, tool-call, and rollback probes. Promote each later ring only from a signed canary decision. On regression, restore the last approved digest while keeping the vulnerability case open and activating an approved compensating control.

Close from fleet evidence

After every ring, independently enumerate workloads, nodes, images, packages, and long-lived developer or MCP tooling in the affected population. A release receipt without runtime readback is insufficient. Retain old and new digests, compatibility results, drain and rollout events, probe outputs, rollback result, owner approvals, and an independent query proving that no affected digest remains outside an approved exception.

AID-H-003.010-G003

Use expiring compensating controls when no compatible fix exists, and keep the vulnerability open until replacement or independently verified remediation.
Make a no-patch path explicit and temporary

A missing vendor fix, disputed CVE, or broken CUDA and framework compatibility matrix does not permit silent deferral. The security owner records the exact vulnerable digests and deployments, exploit preconditions, exposure, selected controls, residual risk, owner, next review, hard expiry, and replacement plan. Controls may remove public exposure, disable the vulnerable endpoint or feature, restrict ingress and egress, isolate the workload, or replace the component.

Validate the exception contract before enforcement
# File: remediation/validate_exception.py
from __future__ import annotations
import json
import re
import sys
from datetime import datetime, timezone
from pathlib import Path

def no_duplicate_keys(pairs: list[tuple[str, object]]) -> dict:
    result: dict = {}
    for key, value in pairs:
        if key in result:
            raise ValueError(f"duplicate JSON key: {key}")
        result[key] = value
    return result

def reject_constant(value: str) -> None:
    raise ValueError(f"non-finite JSON number: {value}")

FIELDS = {
    "schema_version", "exception_id", "vulnerability_ids", "affected_digests",
    "affected_deployments", "owner", "approved_by", "issued_at", "next_review_at",
    "expires_at", "compensating_controls", "verification_receipts", "replacement_plan"
}

def timestamp(value: object) -> datetime:
    if not isinstance(value, str):
        raise ValueError("timestamp must be a string")
    parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
    if parsed.tzinfo is None:
        raise ValueError("timestamp must include an offset")
    return parsed.astimezone(timezone.utc)

raw = Path(sys.argv[1]).read_bytes()
if not raw or len(raw) > 1024 * 1024:
    raise ValueError("exception document is empty or oversized")
record = json.loads(raw.decode("utf-8", errors="strict"), object_pairs_hook=no_duplicate_keys, parse_constant=reject_constant)
if not isinstance(record, dict) or set(record) != FIELDS:
    raise ValueError("exception schema differs")
if record["schema_version"] != "aidefend.software-remediation-exception.v1":
    raise ValueError("unsupported exception schema")
for field in ("vulnerability_ids", "affected_digests", "affected_deployments",
              "compensating_controls"):
    value = record[field]
    if (not isinstance(value, list) or not value or
            any(not isinstance(item, str) or not item for item in value) or
            len(value) != len(set(value))):
        raise ValueError(field + " must be a nonempty unique string list")
receipts = record["verification_receipts"]
if not isinstance(receipts, list) or not receipts:
    raise ValueError("verification_receipts must be nonempty")
for receipt in receipts:
    if (not isinstance(receipt, dict) or set(receipt) != {"control_id", "target_id", "receipt_sha256", "outcome"} or
            receipt["outcome"] != "PASS" or receipt["target_id"] not in record["affected_deployments"] or
            not isinstance(receipt["control_id"], str) or not receipt["control_id"] or
            not isinstance(receipt["receipt_sha256"], str) or
            re.fullmatch(r"[0-9a-f]{64}", receipt["receipt_sha256"]) is None):
        raise ValueError("verification receipt is unbound or invalid")
if any(not isinstance(value, str) or not value for value in
       (record["exception_id"], record["owner"], record["approved_by"],
        record["replacement_plan"])):
    raise ValueError("exception identity, ownership, and replacement plan are required")
issued = timestamp(record["issued_at"])
review = timestamp(record["next_review_at"])
expires = timestamp(record["expires_at"])
now = datetime.now(timezone.utc)
if not issued <= now < review <= expires:
    raise PermissionError("exception is future-dated, expired, or overdue for review")
print(json.dumps({"outcome": "PASS", "exception_id": record["exception_id"],
                  "expires_at": record["expires_at"]}, sort_keys=True))

Action: Verify the signed exception before applying it, fail closed when the policy, enforcement, or readback is unavailable, and automatically page the owner before review and expiry. Independently probe the blocked exposure or disabled feature and compare the complete affected population with enforcement targets. At expiry, remediate, replace, remove, or stop the affected component; do not auto-renew.

Outcome

Use PASS only for a fully remediated population. An active, verified exception is an explicitly bounded interim state, not closure. Missing population coverage is INSUFFICIENT_DATA; failed enforcement is FAIL; verifier or control-plane failure is ERROR.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-H-003.010: Deployed AI Software Vulnerability Remediation Lifecycle | AIDEFEND

Loading interactive view…

A Practical Knowledge Base for AI Security Defenses

Version