Scope boundary

Owns comparison of digest-bound explanation outputs against approved XAI baselines, exact model and explainer binding, corroboration, and finding generation. Explanation deviation is a triage signal, not proof of compromise, bias, backdoor presence, or faithful access to internal reasoning.

AID-M-003.003: AID-D-006 compares runtime or probe explanations with the trusted reference and emits drift findings; AID-M-003.003 creates and recalibrates that signed XAI output baseline. The detector must consume an independently governed baseline and must not interpret deviation alone as proof of attack.

Warning

High on Inference Latency

The multiple-XAI-method approach multiplies latency, and common methods like SHAP or LIME on deep models can take single-digit seconds, not milliseconds, per prediction. This cost must be carefully considered by architects.

Implementation guidance

• AID-D-006-G001: Employ multiple, diverse XAI methods to explain the same model decision and compare their outputs for consistency; significant divergence can indicate manipulation or instability.
• AID-D-006-G002: Establish baselines for typical explanation characteristics and monitor for deviations.
• AID-D-006-G003: Detect instability where tiny input perturbations cause radically different explanations while the prediction stays the same.
• AID-D-006-G004: Monitor for explanations that are overly simplistic, highlight irrelevant features, or ignore known-critical features.
• AID-D-006-G005: Specifically test against adversarial attacks designed to fool XAI methods (e.g., "adversarial explanations" where the explanation is misleading but the prediction remains unchanged or changes benignly).
• AID-D-006-G006: Log XAI outputs and manipulation findings for investigation by model-risk and security teams.

AID-D-006-G001

Require at least two pinned, independent explainers

A cross-method comparison has no meaning with zero or one method. Signed policy must name at least two distinct explainer adapters, their revisions, a non-empty ordered feature ontology, top_k, and a finite Jaccard threshold. A missing or extra adapter is incomplete coverage, not agreement.

from __future__ import annotations

# File: xai_analysis/diverse_explainers.py

import hashlib
import json
import math
import re
from collections.abc import Callable
from dataclasses import dataclass
from itertools import combinations

import numpy as np

SHA256 = re.compile(r"^[0-9a-f]{64}$")


def _is_sha256(value: object) -> bool:
    return isinstance(value, str) and SHA256.fullmatch(value) is not None


@dataclass(frozen=True)
class ExplainerAdapter:
    revision: str
    explain: Callable[[np.ndarray], np.ndarray]


def compare_explainers(
    *,
    request_id: str,
    model_digest: str,
    preprocessing_digest: str,
    input_vector: np.ndarray,
    explainers: dict[str, ExplainerAdapter],
    feature_ids: list[str],
    policy: dict,
) -> dict:
    required = {
        "policy_version", "model_digest", "preprocessing_digest",
        "feature_ontology_sha256", "required_explainers", "required_revisions",
        "top_k", "minimum_pairwise_jaccard",
    }
    if set(policy) != required:
        raise ValueError("XAI comparison policy schema mismatch")
    required_names = list(policy["required_explainers"])
    if (
        len(required_names) < 2
        or len(required_names) != len(set(required_names))
        or any(not isinstance(name, str) or not name for name in required_names)
    ):
        raise ValueError("policy must require at least two distinct explainers")
    if set(explainers) != set(required_names):
        return {
            "control_status": "INSUFFICIENT_DATA",
            "finding_status": "UNKNOWN",
            "request_id": request_id,
            "missing_explainers": sorted(set(required_names) - set(explainers)),
            "unexpected_explainers": sorted(set(explainers) - set(required_names)),
            "policy_version": policy["policy_version"],
        }
    revisions = dict(policy["required_revisions"])
    if set(revisions) != set(required_names) or any(
        explainers[name].revision != revisions[name] for name in required_names
    ):
        raise ValueError("explainer revision differs from signed policy")
    if (
        not feature_ids
        or len(feature_ids) != len(set(feature_ids))
        or any(not isinstance(item, str) or not item for item in feature_ids)
    ):
        raise ValueError("signed feature ontology is empty or duplicated")
    ontology_digest = hashlib.sha256(
        json.dumps(feature_ids, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
    ).hexdigest()
    if (
        not _is_sha256(model_digest)
        or not _is_sha256(preprocessing_digest)
        or model_digest != policy["model_digest"]
        or preprocessing_digest != policy["preprocessing_digest"]
        or ontology_digest != policy["feature_ontology_sha256"]
    ):
        raise ValueError("model, preprocessing, or feature ontology differs from signed policy")
    top_k = policy["top_k"]
    threshold = policy["minimum_pairwise_jaccard"]
    if (
        isinstance(top_k, bool)
        or not isinstance(top_k, int)
        or not 1 <= top_k <= len(feature_ids)
        or isinstance(threshold, bool)
        or not isinstance(threshold, (int, float))
        or not math.isfinite(float(threshold))
        or not 0.0 <= float(threshold) <= 1.0
    ):
        raise ValueError("signed top_k or Jaccard threshold is invalid")
    vector = np.asarray(input_vector, dtype=float).reshape(-1)
    if vector.shape != (len(feature_ids),) or not np.isfinite(vector).all():
        raise ValueError("input vector differs from the signed feature ontology")

    rankings = {}
    for name in required_names:
        values = np.asarray(explainers[name].explain(vector), dtype=float).reshape(-1)
        if values.shape != (len(feature_ids),) or not np.isfinite(values).all():
            raise ValueError("explainer output differs from the signed feature ontology")
        order = np.argsort(np.abs(values), kind="stable")[::-1][:top_k]
        rankings[name] = [feature_ids[int(index)] for index in order]

    comparisons = []
    finding_pairs = []
    for left, right in combinations(required_names, 2):
        left_set, right_set = set(rankings[left]), set(rankings[right])
        score = len(left_set & right_set) / len(left_set | right_set)
        finding = score < float(threshold)
        comparisons.append({
            "left": left, "right": right,
            "jaccard": round(score, 8), "finding": finding,
        })
        if finding:
            finding_pairs.append(left + "::" + right)
    if not comparisons:
        raise RuntimeError("no explainer pair was evaluated")
    return {
        "schema_version": "aidefend.xai_cross_method.v2",
        "control_status": "PASS",
        "finding_status": "FINDING" if finding_pairs else "NO_FINDING",
        "request_id": request_id,
        "model_digest": model_digest,
        "preprocessing_digest": preprocessing_digest,
        "feature_ontology_sha256": ontology_digest,
        "policy_version": policy["policy_version"],
        "explainer_revisions": revisions,
        "rankings": rankings,
        "comparisons": comparisons,
        "finding_pairs": finding_pairs,
    }

Verification and interpretation

Replay agreement, genuine method disagreement, empty/one/missing/extra explainer, wrong revision, permuted feature IDs, ties, non-finite vectors, and threshold boundaries. Independently execute every required adapter and compare feature-ID rankings. Divergence is a heuristic finding for investigation; it is not proof that the prediction was manipulated and does not itself block the decision.

AID-D-006-G002

Bind the baseline to an ordered feature ontology

Build the immutable baseline from the signed validation population for the exact model, preprocessing graph, explainer, class, and segment. Store both the ordered feature IDs and their canonical digest. Runtime evaluation must supply its current ordered IDs; equal vector length alone cannot detect a feature permutation.

from __future__ import annotations

# File: xai_analysis/explanation_baseline.py

import hashlib
import json
import math
import re

import numpy as np

SHA256 = re.compile(r"^[0-9a-f]{64}$")


def _is_sha256(value: object) -> bool:
    return isinstance(value, str) and SHA256.fullmatch(value) is not None


def ontology_sha256(feature_ids: list[str]) -> str:
    if (
        not feature_ids
        or len(feature_ids) != len(set(feature_ids))
        or any(not isinstance(item, str) or not item for item in feature_ids)
    ):
        raise ValueError("feature ontology is empty or duplicated")
    return hashlib.sha256(
        json.dumps(feature_ids, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
    ).hexdigest()


def build_baseline(
    explanation_vectors: np.ndarray,
    *,
    model_digest: str,
    preprocessing_digest: str,
    explainer_id: str,
    feature_ids: list[str],
    class_id: str,
    segment_id: str,
    validation_cohort_sha256: str,
    baseline_version: str,
) -> dict:
    matrix = np.asarray(explanation_vectors, dtype=float)
    ontology_digest = ontology_sha256(feature_ids)
    if (
        matrix.ndim != 2
        or matrix.shape[0] == 0
        or matrix.shape[1] != len(feature_ids)
        or not np.isfinite(matrix).all()
        or not all((explainer_id, class_id, segment_id, baseline_version))
        or not _is_sha256(model_digest)
        or not _is_sha256(preprocessing_digest)
        or not _is_sha256(validation_cohort_sha256)
    ):
        raise ValueError("baseline matrix or binding is invalid")
    return {
        "schema_version": "aidefend.xai_baseline.v2",
        "baseline_version": baseline_version,
        "model_digest": model_digest,
        "preprocessing_digest": preprocessing_digest,
        "explainer_id": explainer_id,
        "class_id": class_id,
        "segment_id": segment_id,
        "validation_cohort_sha256": validation_cohort_sha256,
        "feature_ids": feature_ids,
        "feature_ontology_sha256": ontology_digest,
        "sample_count": int(matrix.shape[0]),
        "mean_absolute_importance": np.abs(matrix).mean(axis=0).tolist(),
    }


def evaluate_against_baseline(
    *,
    request_id: str,
    explanation_vector: np.ndarray,
    current_feature_ids: list[str],
    baseline: dict,
    expected_model_digest: str,
    expected_preprocessing_digest: str,
    expected_explainer_id: str,
    expected_class_id: str,
    expected_segment_id: str,
    expected_validation_cohort_sha256: str,
    policy: dict,
) -> dict:
    baseline_required = {
        "schema_version", "baseline_version", "model_digest", "preprocessing_digest",
        "explainer_id", "class_id", "segment_id", "validation_cohort_sha256",
        "feature_ids", "feature_ontology_sha256",
        "sample_count", "mean_absolute_importance",
    }
    policy_required = {
        "policy_version", "minimum_baseline_samples", "minimum_cosine_similarity",
    }
    if set(baseline) != baseline_required or baseline["schema_version"] != "aidefend.xai_baseline.v2":
        raise ValueError("baseline schema mismatch")
    if set(policy) != policy_required:
        raise ValueError("XAI baseline policy schema mismatch")
    expected_digests = (
        expected_model_digest,
        expected_preprocessing_digest,
        expected_validation_cohort_sha256,
    )
    if not all(_is_sha256(item) for item in expected_digests):
        raise ValueError("expected baseline digests are malformed")
    bindings = (
        baseline["model_digest"] == expected_model_digest,
        baseline["preprocessing_digest"] == expected_preprocessing_digest,
        baseline["explainer_id"] == expected_explainer_id,
        baseline["class_id"] == expected_class_id,
        baseline["segment_id"] == expected_segment_id,
        baseline["validation_cohort_sha256"] == expected_validation_cohort_sha256,
    )
    if not all(bindings):
        raise ValueError("baseline is bound to another runtime context")
    baseline_ids = list(baseline["feature_ids"])
    observed_ontology = ontology_sha256(current_feature_ids)
    if (
        baseline_ids != current_feature_ids
        or baseline["feature_ontology_sha256"] != observed_ontology
        or ontology_sha256(baseline_ids) != observed_ontology
    ):
        raise ValueError("runtime feature order differs from the immutable baseline")
    minimum_samples = policy["minimum_baseline_samples"]
    threshold = policy["minimum_cosine_similarity"]
    if (
        isinstance(minimum_samples, bool)
        or not isinstance(minimum_samples, int)
        or minimum_samples < 1
        or isinstance(threshold, bool)
        or not isinstance(threshold, (int, float))
        or not math.isfinite(float(threshold))
        or not 0.0 <= float(threshold) <= 1.0
    ):
        raise ValueError("signed sample minimum or similarity threshold is invalid")
    if baseline["sample_count"] < minimum_samples:
        return {
            "control_status": "INSUFFICIENT_DATA",
            "finding_status": "UNKNOWN",
            "request_id": request_id,
            "baseline_version": baseline["baseline_version"],
            "reason": "baseline population below signed minimum",
        }
    current = np.abs(np.asarray(explanation_vector, dtype=float).reshape(-1))
    expected = np.asarray(baseline["mean_absolute_importance"], dtype=float)
    if (
        current.shape != (len(current_feature_ids),)
        or expected.shape != current.shape
        or not np.isfinite(current).all()
        or not np.isfinite(expected).all()
    ):
        raise ValueError("explanation vectors are incompatible or non-finite")
    denominator = float(np.linalg.norm(current) * np.linalg.norm(expected))
    if not math.isfinite(denominator) or denominator == 0.0:
        return {
            "control_status": "INSUFFICIENT_DATA",
            "finding_status": "UNKNOWN",
            "request_id": request_id,
            "baseline_version": baseline["baseline_version"],
            "reason": "zero-norm explanation or baseline",
        }
    similarity = float(np.dot(current, expected) / denominator)
    finding = similarity < float(threshold)
    return {
        "schema_version": "aidefend.xai_baseline_finding.v2",
        "control_status": "PASS",
        "finding_status": "FINDING" if finding else "NO_FINDING",
        "request_id": request_id,
        "model_digest": expected_model_digest,
        "preprocessing_digest": expected_preprocessing_digest,
        "explainer_id": expected_explainer_id,
        "class_id": expected_class_id,
        "segment_id": expected_segment_id,
        "validation_cohort_sha256": expected_validation_cohort_sha256,
        "feature_ontology_sha256": observed_ontology,
        "baseline_version": baseline["baseline_version"],
        "baseline_sample_count": baseline["sample_count"],
        "cosine_similarity": round(similarity, 8),
        "threshold": float(threshold),
        "policy_version": policy["policy_version"],
    }

Verification

The baseline loader verifies its signature and canonical digest before evaluation. Rebuild it from the signed corpus, then replay stable, shifted, wrong-model, wrong-preprocessing, wrong-explainer, wrong-segment, same-shape permuted-feature, undersized, non-finite, and zero-vector fixtures. Read the finding back and independently recompute the ontology digest and cosine similarity.

AID-D-006-G003

Concept:

A robust explanation should be locally stable. If adding tiny noise to the input leaves the prediction unchanged but completely scrambles the 'most important features', that's a red flag. It can indicate an intentionally manipulated explainer.

import hmac as _aidefend_policy_hmac
import json as _aidefend_policy_json
import math as _aidefend_policy_math
import os as _aidefend_policy_os
import re as _aidefend_policy_re
import subprocess as _aidefend_policy_subprocess
import tempfile as _aidefend_policy_tempfile
from pathlib import Path as _AidefendPolicyPath


def _aidefend_write_private(path, payload):
    descriptor = _aidefend_policy_os.open(
        path,
        _aidefend_policy_os.O_WRONLY
        | _aidefend_policy_os.O_CREAT
        | _aidefend_policy_os.O_EXCL,
        0o600,
    )
    with _aidefend_policy_os.fdopen(descriptor, "wb") as stream:
        stream.write(payload)
        stream.flush()
        _aidefend_policy_os.fsync(stream.fileno())
    _aidefend_policy_os.chmod(path, 0o400)


def _aidefend_strict_json(raw):
    def reject_duplicate_keys(pairs):
        value = {}
        for key, item in pairs:
            if key in value:
                raise ValueError("duplicate JSON key: " + key)
            value[key] = item
        return value

    return _aidefend_policy_json.loads(
        raw.decode("utf-8", errors="strict"),
        object_pairs_hook=reject_duplicate_keys,
        parse_constant=lambda value: (_ for _ in ()).throw(
            ValueError("non-finite JSON constant: " + value)
        ),
    )


def _aidefend_load_signed_policy(*, path_env, signature_env, key_env, schema_version):
    policy_path = _AidefendPolicyPath(_aidefend_policy_os.environ[path_env])
    signature_path = _AidefendPolicyPath(_aidefend_policy_os.environ[signature_env])
    if policy_path == signature_path:
        raise RuntimeError("policy and signature paths must differ")
    policy_raw = policy_path.read_bytes()
    signature_raw = signature_path.read_bytes()
    key_reference = _aidefend_policy_os.environ[key_env]
    remote_key = bool(_aidefend_policy_re.match(
        r"^[A-Za-z][A-Za-z0-9+.-]*:(?://)?", key_reference
    )) and not key_reference.startswith("-----BEGIN ")
    key_raw = None if remote_key else (
        key_reference.encode("utf-8")
        if key_reference.startswith("-----BEGIN ")
        else _AidefendPolicyPath(key_reference).read_bytes()
    )
    verify_timeout = float(_aidefend_policy_os.environ["AIDEFEND_POLICY_VERIFY_TIMEOUT_SECONDS"])
    if not _aidefend_policy_math.isfinite(verify_timeout) or verify_timeout <= 0:
        raise RuntimeError("policy-verification timeout from the versioned runtime profile is invalid")
    with _aidefend_policy_tempfile.TemporaryDirectory(
        prefix="aidefend-signed-policy-"
    ) as directory:
        root = _AidefendPolicyPath(directory)
        _aidefend_policy_os.chmod(root, 0o700)
        policy_snapshot = root / "policy.json"
        signature_snapshot = root / "policy.sig"
        _aidefend_write_private(policy_snapshot, policy_raw)
        _aidefend_write_private(signature_snapshot, signature_raw)
        captured = [(policy_snapshot, policy_raw), (signature_snapshot, signature_raw)]
        key_argument = key_reference
        if key_raw is not None:
            key_snapshot = root / "verify-key.pub"
            _aidefend_write_private(key_snapshot, key_raw)
            captured.append((key_snapshot, key_raw))
            key_argument = str(key_snapshot)
        try:
            _aidefend_policy_subprocess.run(
                [
                    "cosign", "verify-blob", "--key", key_argument,
                    "--bundle", str(signature_snapshot), str(policy_snapshot),
                ],
                check=True, capture_output=True, text=True, timeout=verify_timeout,
            )
            if any(
                not _aidefend_policy_hmac.compare_digest(path.read_bytes(), expected)
                for path, expected in captured
            ):
                raise RuntimeError("private signature-verification snapshot changed")
            policy = _aidefend_strict_json(policy_raw)
        finally:
            for path, _ in captured:
                _aidefend_policy_os.chmod(path, 0o600)
    if (
        not isinstance(policy, dict)
        or policy.get("schema_version") != schema_version
        or not isinstance(policy.get("policy_version"), str)
        or not policy["policy_version"]
    ):
        raise RuntimeError("signature-verified policy schema or version is invalid")
    return policy


# File: xai_analysis/stability_check.py
import json
import logging

import numpy as np
from scipy.stats import spearmanr

XAI_STABILITY_POLICY = _aidefend_load_signed_policy(
    path_env="D006_STABILITY_POLICY_PATH",
    signature_env="D006_STABILITY_POLICY_SIGNATURE",
    key_env="D006_STABILITY_POLICY_VERIFY_KEY",
    schema_version="aidefend.xai-stability-policy.v1",
)
STABILITY_CORRELATION_THRESHOLD = XAI_STABILITY_POLICY.get("minimum_rank_correlation")
NOISE_MAGNITUDE = XAI_STABILITY_POLICY.get("noise_magnitude")
STABILITY_RANDOM_SEED = XAI_STABILITY_POLICY.get("random_seed")
if (
    isinstance(STABILITY_CORRELATION_THRESHOLD, bool)
    or not isinstance(STABILITY_CORRELATION_THRESHOLD, (int, float))
    or not -1 <= float(STABILITY_CORRELATION_THRESHOLD) <= 1
    or isinstance(NOISE_MAGNITUDE, bool)
    or not isinstance(NOISE_MAGNITUDE, (int, float))
    or not _aidefend_policy_math.isfinite(float(NOISE_MAGNITUDE))
    or float(NOISE_MAGNITUDE) <= 0
    or isinstance(STABILITY_RANDOM_SEED, bool)
    or not isinstance(STABILITY_RANDOM_SEED, int)
):
    raise RuntimeError("signed XAI stability policy is invalid")
logger = logging.getLogger("aidefend.xai_stability")
rng = np.random.default_rng(STABILITY_RANDOM_SEED)

def check_explanation_stability(model, explainer, input_instance):
    """
    1. Get explanation for the original input.
    2. Add tiny noise to input.
    3. Get explanation again.
    4. Compare via Spearman correlation.
    Assumes binary classification explainer like SHAP returning [0] or [1].
    """
    # Explanation for original instance
    original_vals = explainer.shap_values(input_instance)[0].flatten()

    # Slightly perturbed version of the same input
    noise = rng.normal(0, float(NOISE_MAGNITUDE), input_instance.shape)
    perturbed_instance = input_instance + noise

    # If the model's predicted class changed, skip (that's model fragility, not XAI manipulation)
    if model.predict(input_instance) != model.predict(perturbed_instance):
        logger.info(json.dumps({"event_type": "xai_stability_not_applicable", "reason": "prediction_changed"}))
        return True

    # Explanation for perturbed instance
    perturbed_vals = explainer.shap_values(perturbed_instance)[0].flatten()

    # Rank-order consistency via Spearman correlation
    correlation, _ = spearmanr(original_vals, perturbed_vals)
    logger.info(json.dumps({"event_type": "xai_stability", "correlation": correlation}, sort_keys=True))

    if correlation < STABILITY_CORRELATION_THRESHOLD:
        logger.warning(json.dumps({"event_type": "xai_stability_finding", "correlation": correlation, "threshold": STABILITY_CORRELATION_THRESHOLD}, sort_keys=True))
        return False
    return True

Action: For sensitive inference flows, run a stability check on the explanation. If the model's decision is the same but the explanation's importance ranking is drastically different (low Spearman correlation), treat that as a possible manipulation or an untrustworthy explainer.

Verify safely

An independent verifier must rerun the complete signed base and perturbation matrix under an independently attested runner, verify prediction-invariance eligibility, and compare explanation similarity, errors, exclusions and findings, read evidence back, verify signatures, digests and immutability, and compare the canonical result digest.

AID-D-006-G004

Concept:

Some explanations are obviously garbage if you know the domain. For example, a loan approval model claiming that 'zip_code' is the top driver instead of 'credit_score' is a huge red flag. We codify that domain knowledge in a config and automatically sanity-check new explanations against it.

Step 1: Define Plausible Feature Sets and Complexity

Create a JSON config that lists which features are critical, which features are nonsensical, and how complex an explanation should normally be. For example:

{
  "loan_approval_model": {
    "critical_features": [
      "credit_score",
      "income",
      "debt_to_income_ratio"
    ],
    "nonsensical_features": [
      "zip_code",
      "user_id",
      "application_date"
    ],
    "min_explanation_complexity": 3
  }
}

Step 2: Implement the Sanity Check Function

# File: xai_analysis/sanity_checks.py
import json
import logging

import pandas as pd

logger = logging.getLogger("aidefend.xai_sanity")

def run_explanation_sanity_checks(model_name, feature_importances, config):
    """
    feature_importances: pandas Series mapping feature_name -> importance (float)
    config: dict loaded from the JSON config above
    """
    model_cfg = config.get(model_name, {})
    critical_feats = model_cfg.get("critical_features", [])
    nonsensical_feats = model_cfg.get("nonsensical_features", [])
    min_complexity = model_cfg.get("min_explanation_complexity", 1)

    # Identify the top feature by absolute importance
    top_feature = feature_importances.abs().idxmax()

    # Count how many features are materially important
    num_important_features = int((feature_importances.abs() > 0.01).sum())

    # 1. Check if the most important feature is on the 'nonsensical' list
    if top_feature in nonsensical_feats:
        logger.warning(json.dumps({"event_type": "xai_sanity_finding", "reason": "nonsensical_top_feature", "top_feature": top_feature}, sort_keys=True))
        return False

    # 2. Check if the explanation is too simple
    if num_important_features < min_complexity:
        logger.warning(json.dumps({"event_type": "xai_sanity_finding", "reason": "insufficient_complexity", "feature_count": num_important_features, "minimum": min_complexity}, sort_keys=True))
        return False

    # 3. Check that at least one critical feature shows up with non-trivial weight
    critical_series = feature_importances.reindex(critical_feats).fillna(0).abs()
    if (critical_series < 1e-6).all():
        logger.warning(json.dumps({"event_type": "xai_sanity_finding", "reason": "critical_features_ignored"}))
        return False

    logger.info(json.dumps({"event_type": "xai_sanity", "status": "NO_FINDING"}))
    return True

Action: Work with domain SMEs (fraud, credit, medical, etc.) to build and maintain the sanity check config. Automatically reject or escalate explanations that fail these checks, because that may indicate manipulation or XAI degradation.

Verify safely

An independent verifier must resolve features and domain rules under an independent identity and replay valid, irrelevant-feature, missing-critical-feature, constant, sparse, dense, unmapped and unsupported fixtures while comparing matched rules and findings, read evidence back, verify signatures, digests and immutability, and compare the canonical result digest.

AID-D-006-G005

Concept:

Do not depend on nonexistent ART explanation-attack APIs. For production validation, treat XAI robustness as a measurable property: create small, policy-bounded perturbations that should not change the model decision, recompute the explanation, and check whether the explanation ranking changes in a suspicious way. This is a red-team or release-gate test, not a real-time defense.

Run a Direct Explanation-Manipulation Probe

The example below is explainer-agnostic. Wrap SHAP, LIME, Captum, or another explainer behind a function that returns one feature-importance vector for one input.

from __future__ import annotations

# File: xai_testing/explanation_manipulation_probe.py

import hmac as _aidefend_policy_hmac
import json as _aidefend_policy_json
import math as _aidefend_policy_math
import os as _aidefend_policy_os
import re as _aidefend_policy_re
import subprocess as _aidefend_policy_subprocess
import tempfile as _aidefend_policy_tempfile
from pathlib import Path as _AidefendPolicyPath


def _aidefend_write_private(path, payload):
    descriptor = _aidefend_policy_os.open(
        path,
        _aidefend_policy_os.O_WRONLY
        | _aidefend_policy_os.O_CREAT
        | _aidefend_policy_os.O_EXCL,
        0o600,
    )
    with _aidefend_policy_os.fdopen(descriptor, "wb") as stream:
        stream.write(payload)
        stream.flush()
        _aidefend_policy_os.fsync(stream.fileno())
    _aidefend_policy_os.chmod(path, 0o400)


def _aidefend_strict_json(raw):
    def reject_duplicate_keys(pairs):
        value = {}
        for key, item in pairs:
            if key in value:
                raise ValueError("duplicate JSON key: " + key)
            value[key] = item
        return value

    return _aidefend_policy_json.loads(
        raw.decode("utf-8", errors="strict"),
        object_pairs_hook=reject_duplicate_keys,
        parse_constant=lambda value: (_ for _ in ()).throw(
            ValueError("non-finite JSON constant: " + value)
        ),
    )


def _aidefend_load_signed_policy(*, path_env, signature_env, key_env, schema_version):
    policy_path = _AidefendPolicyPath(_aidefend_policy_os.environ[path_env])
    signature_path = _AidefendPolicyPath(_aidefend_policy_os.environ[signature_env])
    if policy_path == signature_path:
        raise RuntimeError("policy and signature paths must differ")
    policy_raw = policy_path.read_bytes()
    signature_raw = signature_path.read_bytes()
    key_reference = _aidefend_policy_os.environ[key_env]
    remote_key = bool(_aidefend_policy_re.match(
        r"^[A-Za-z][A-Za-z0-9+.-]*:(?://)?", key_reference
    )) and not key_reference.startswith("-----BEGIN ")
    key_raw = None if remote_key else (
        key_reference.encode("utf-8")
        if key_reference.startswith("-----BEGIN ")
        else _AidefendPolicyPath(key_reference).read_bytes()
    )
    verify_timeout = float(_aidefend_policy_os.environ["AIDEFEND_POLICY_VERIFY_TIMEOUT_SECONDS"])
    if not _aidefend_policy_math.isfinite(verify_timeout) or verify_timeout <= 0:
        raise RuntimeError("policy-verification timeout from the versioned runtime profile is invalid")
    with _aidefend_policy_tempfile.TemporaryDirectory(
        prefix="aidefend-signed-policy-"
    ) as directory:
        root = _AidefendPolicyPath(directory)
        _aidefend_policy_os.chmod(root, 0o700)
        policy_snapshot = root / "policy.json"
        signature_snapshot = root / "policy.sig"
        _aidefend_write_private(policy_snapshot, policy_raw)
        _aidefend_write_private(signature_snapshot, signature_raw)
        captured = [(policy_snapshot, policy_raw), (signature_snapshot, signature_raw)]
        key_argument = key_reference
        if key_raw is not None:
            key_snapshot = root / "verify-key.pub"
            _aidefend_write_private(key_snapshot, key_raw)
            captured.append((key_snapshot, key_raw))
            key_argument = str(key_snapshot)
        try:
            _aidefend_policy_subprocess.run(
                [
                    "cosign", "verify-blob", "--key", key_argument,
                    "--bundle", str(signature_snapshot), str(policy_snapshot),
                ],
                check=True, capture_output=True, text=True, timeout=verify_timeout,
            )
            if any(
                not _aidefend_policy_hmac.compare_digest(path.read_bytes(), expected)
                for path, expected in captured
            ):
                raise RuntimeError("private signature-verification snapshot changed")
            policy = _aidefend_strict_json(policy_raw)
        finally:
            for path, _ in captured:
                _aidefend_policy_os.chmod(path, 0o600)
    if (
        not isinstance(policy, dict)
        or policy.get("schema_version") != schema_version
        or not isinstance(policy.get("policy_version"), str)
        or not policy["policy_version"]
    ):
        raise RuntimeError("signature-verified policy schema or version is invalid")
    return policy


from collections.abc import Callable
from dataclasses import dataclass

import numpy as np
from scipy.stats import spearmanr

PredictLabel = Callable[[np.ndarray], int]
ExplainImportance = Callable[[np.ndarray], np.ndarray]


EXPLANATION_PROBE_POLICY = _aidefend_load_signed_policy(
    path_env="D006_PROBE_POLICY_PATH",
    signature_env="D006_PROBE_POLICY_SIGNATURE",
    key_env="D006_PROBE_POLICY_VERIFY_KEY",
    schema_version="aidefend.explanation-manipulation-probe-policy.v1",
)
PROBE_TRIALS = EXPLANATION_PROBE_POLICY.get("trials")
PROBE_NOISE_SCALE = EXPLANATION_PROBE_POLICY.get("noise_scale")
PROBE_MINIMUM_RANK_CORRELATION = EXPLANATION_PROBE_POLICY.get("minimum_rank_correlation")
PROBE_RANDOM_SEED = EXPLANATION_PROBE_POLICY.get("random_seed")
if (
    isinstance(PROBE_TRIALS, bool) or not isinstance(PROBE_TRIALS, int) or PROBE_TRIALS < 1
    or isinstance(PROBE_NOISE_SCALE, bool)
    or not isinstance(PROBE_NOISE_SCALE, (int, float))
    or not _aidefend_policy_math.isfinite(float(PROBE_NOISE_SCALE))
    or float(PROBE_NOISE_SCALE) <= 0
    or isinstance(PROBE_MINIMUM_RANK_CORRELATION, bool)
    or not isinstance(PROBE_MINIMUM_RANK_CORRELATION, (int, float))
    or not -1 <= float(PROBE_MINIMUM_RANK_CORRELATION) <= 1
    or isinstance(PROBE_RANDOM_SEED, bool) or not isinstance(PROBE_RANDOM_SEED, int)
):
    raise RuntimeError("signed explanation-probe policy is invalid")


@dataclass(frozen=True)
class ExplanationProbeFinding:
    trial_index: int
    prediction_stable: bool
    rank_correlation: float
    top_feature_changed: bool
    suspicious: bool


def importance_vector(values: np.ndarray) -> np.ndarray:
    vector = np.asarray(values, dtype=float).reshape(-1)
    if vector.size == 0:
        raise ValueError("explainer returned an empty importance vector")
    return vector


def top_feature_index(values: np.ndarray) -> int:
    return int(np.argsort(np.abs(values))[::-1][0])


def rank_correlation(a: np.ndarray, b: np.ndarray) -> float:
    correlation = spearmanr(np.abs(a), np.abs(b)).correlation
    return 0.0 if np.isnan(correlation) else float(correlation)


def run_explanation_manipulation_probe(
    *,
    predict_label: PredictLabel,
    explain_importance: ExplainImportance,
    input_instance: np.ndarray,
    trials: int = PROBE_TRIALS,
    noise_scale: float = float(PROBE_NOISE_SCALE),
    min_rank_correlation: float = float(PROBE_MINIMUM_RANK_CORRELATION),
) -> list[ExplanationProbeFinding]:
    baseline_prediction = predict_label(input_instance)
    baseline_explanation = importance_vector(explain_importance(input_instance))
    baseline_top_feature = top_feature_index(baseline_explanation)
    rng = np.random.default_rng(PROBE_RANDOM_SEED)

    findings: list[ExplanationProbeFinding] = []
    for trial_index in range(trials):
        candidate = input_instance + rng.normal(
            loc=0.0,
            scale=noise_scale,
            size=input_instance.shape,
        )
        candidate_prediction = predict_label(candidate)
        prediction_stable = candidate_prediction == baseline_prediction
        if not prediction_stable:
            continue

        candidate_explanation = importance_vector(explain_importance(candidate))
        correlation = rank_correlation(baseline_explanation, candidate_explanation)
        top_changed = top_feature_index(candidate_explanation) != baseline_top_feature
        suspicious = correlation < min_rank_correlation and top_changed

        findings.append(
            ExplanationProbeFinding(
                trial_index=trial_index,
                prediction_stable=True,
                rank_correlation=round(correlation, 4),
                top_feature_changed=top_changed,
                suspicious=suspicious,
            )
        )

    return findings


# Example wrapper shape:
# def predict_label(x):
#     return int(model.predict(x.reshape(1, -1))[0])
#
# def explain_importance(x):
#     explanation = shap_explainer.shap_values(x.reshape(1, -1))
#     return np.asarray(explanation).reshape(-1)
#
# findings = run_explanation_manipulation_probe(
#     predict_label=predict_label,
#     explain_importance=explain_importance,
#     input_instance=X_test[0],
# )
# if any(item.suspicious for item in findings):
#     raise SystemExit("Explanation manipulation probe failed")

Action: Add an explanation-manipulation probe to model validation for high-risk decisions. If predictions stay stable but explanation rankings collapse or top features flip repeatedly, treat the explainer as untrustworthy for that model/version and investigate before release. Use published XAI-attack papers or custom optimization objectives for deeper red-team work, but do not cite unsupported ART modules as runnable tooling.

AID-D-006-G006

Concept:

Every explanation and every validation result (divergence check, stability check, sanity check, etc.) is security-relevant evidence. You should emit a structured JSON log event for each high-value inference. This feeds into SIEM / AI assurance pipelines.

Define a Structured XAI Event Log

{
  "timestamp": "2025-06-08T10:30:00Z",
  "event_type": "xai_generation_and_validation",
  "request_id": "c1a2b3d4-e5f6-7890",
  "model_version": "fraud-detector:v2.1",
  "input_hash": "a6c7d8e9f00112233445566778899aabbccddeeff00112233445566778899aab",
  "model_prediction": {
    "class": "fraud",
    "confidence": 0.92
  },
  "explanation": {
    "method": "SHAP",
    "top_features": [
      {"feature": "hours_since_last_tx", "importance": 0.45},
      {"feature": "transaction_amount", "importance": 0.31}
    ]
  },
  "validation_results": {
    "divergence_check": {"control_status": "PASS", "finding_status": "NO_FINDING", "jaccard_index": 0.6},
    "stability_check": {"control_status": "PASS", "finding_status": "NO_FINDING", "correlation": 0.91},
    "sanity_check": {"control_status": "PASS", "finding_status": "FINDING", "reason": "top feature violated signed domain rule XAI-SANITY-017"}
  },
  "control_status": "PASS", "finding_status": "FINDING"
}

Action: Add a centralized logging function in your inference service that builds this event JSON and ships it to your SIEM / log pipeline (see AID-D-005). Route finding_status=FINDING for investigation; route non-PASS control states as assurance or pipeline defects. Do not treat a detected XAI finding as evidence that the logging control failed.

Verify complete coverage

The population is every applicable prediction, explanation, validation check, finding, error, event write, archive object and readback expected by the signed instrumentation inventory.

Verify safely

An independent verifier must reconstruct expected events from prediction and validator sources, read every archive object independently, verify signatures and digests, and compare event IDs, model and method revisions, check states, findings, missing records and retention labels, read evidence back, verify signatures, digests and immutability, and compare the canonical result digest.

Open-source tools

• SHAP
• LIME
• Captum
• InterpretML

Source-available / open-weight tools

• Alibi Explain (Business Source License 1.1; source-available)

Commercial / hosted tools

• Fiddler AI Observability
• Arize AX
• IBM watsonx.governance

Threat mappings

MITRE ATLAS

• AML.T0015 Evade AI Model (XAI manipulation enables defense evasion)
• AML.T0031 Erode AI Model Integrity (manipulated explanations erode trust)
• AML.T0043 Craft Adversarial Data (adversarial examples crafted to fool XAI methods)

MAESTRO

• Evasion of Security AI Agents (L6) (manipulated XAI misleads security auditors)
• Manipulation of Evaluation Metrics (L5) (unreliable explanations corrupt evaluation)
• Evasion of Detection (L5) (obfuscated behavior evades observability)
• Lack of Explainability in Security AI Agents (L6)

OWASP LLM Top 10 2026

• LLM05:2026 Data and Model Poisoning (explanation anomalies corroborate poisoning-induced behavior without identifying the root cause)
• LLM07:2026 Misinformation

OWASP ML Top 10 2023

• ML08:2023 Model Skewing (XAI detects skewing through explanation drift)
• ML10:2023 Model Poisoning (XAI detects poisoning-induced explanation changes)
• ML01:2023 Input Manipulation Attack (adversarial inputs designed to produce misleading explanations)

OWASP Top 10 for Agentic Applications 2026

• ASI09:2026 Human-Agent Trust Exploitation (manipulated XAI exploits trust)

NIST Adversarial Machine Learning 2025

• NISTAML.022 Evasion
• NISTAML.025 Black-box Evasion
• NISTAML.026 Model Poisoning (Integrity)

Cisco Integrated AI Security and Safety Framework

• AITech-9.2 Detection Evasion (prediction-preserving perturbation and independent explanation disagreement)

Google Secure AI Framework 2.0 - Risks

• MEV: Model Evasion (manipulated explanations conceal evasion attacks)
• DP: Data Poisoning (manipulated XAI hides poisoning effects)

Databricks AI Security Framework 3.0

• Evaluation 6.3: Lack of Interpretability and Explainability
• Model Serving - Inference response 10.1: Lack of audit and monitoring inference quality
• Model Serving - Inference response 10.5: Black-box attacks (manipulated XAI conceals black-box attack effects)
• Agents - Core 13.15: Human Manipulation (manipulated explanations mislead human reviewers)