AID-H-030.003: Manifest-vs-Observed Behavioral Consistency Testing

Version 1.20261015 · Actionable sub-technique

Execute candidate third-party skills in a strongly isolated, ephemeral analysis environment and compare the resulting observed behavior with the skill's declared permission manifest and stated function before the admission policy decides whether installation is allowed.

What is compared:This sub-technique is the skill-specific empirical consistency stage: sibling controls inspect metadata, prose, and loader safety; this control verifies whether a skill behaves within its declared contract.

Pillars: infra, app

Lifecycle: validation

Scope boundary

Owns skill-specific test scenarios, declared-manifest comparison, and the behavior-compliance result derived from a generic attested sandbox report. It does not own sandbox infrastructure or issue final install authorization.

AID-I-001.005: AID-H-030.003 defines skill scenarios and compares observed effects with the manifest; AID-I-001.005 executes the artifact in isolation and proves observation coverage, anti-evasion handling, and teardown. The skill-specific result must consume rather than duplicate the generic sandbox evidence.

AID-H-030.005: AID-H-030.003 emits one behavioral-consistency stage result; AID-H-030.005 fuses all required stage results and alone authorizes install or continued use. A clean behavior test never bypasses metadata, instruction, parser, or freshness requirements.

AID-H-018.007: AID-H-030.003 measures whether observed skill behavior matches the declared manifest; AID-H-018.007 enforces the approved manifest on every runtime action. Pre-install consistency evidence does not replace runtime interception.

Implementation guidance

AID-H-030.003-G001

Consume the AID-I-001.005 attested behavioral report, compare the complete observed skill behavior against its permission manifest, and emit a skill-specific compliance result for AID-H-030.005.
Concept:

The AID-I-001.005 sandbox report is signed input evidence, not this control's PASS. A host-specific verifier must first validate the report signature and bind it to the candidate artifact, permission-manifest digest, analyzer profile, complete signed scenario population, event-stream integrity, and teardown receipt. This control then compares each normalized observed action with the manifest. Keep reads and writes separate: a declared read scope never authorizes a write. Paths below are analyzer-issued logical POSIX paths; do not resolve untrusted sandbox paths against the verifier host filesystem.

Reusable manifest comparator
# File: skill_admission/behavioral_compliance.py
from __future__ import annotations

import ipaddress
from dataclasses import dataclass
from pathlib import PurePosixPath
from typing import Protocol


@dataclass(frozen=True)
class ExpectedBindings:
    artifact_sha256: str
    manifest_sha256: str
    analyzer_profile_sha256: str
    scenario_population_sha256: str
    policy_revision: str


@dataclass(frozen=True)
class PermissionManifest:
    read_paths: tuple[str, ...]
    write_paths: tuple[str, ...]
    network_hosts: tuple[str, ...]
    executables: tuple[str, ...]
    tools: tuple[str, ...]
    protected_writes: tuple[str, ...]


@dataclass(frozen=True)
class ObservedEvent:
    event_id: str
    scenario_id: str
    kind: str
    target: str


@dataclass(frozen=True)
class VerifiedBehaviorReport:
    report_id: str
    artifact_sha256: str
    manifest_sha256: str
    analyzer_profile_sha256: str
    scenario_population_sha256: str
    policy_revision: str
    population_complete: bool
    instrumentation_complete: bool
    teardown_verified: bool
    events: tuple[ObservedEvent, ...]


class AttestedReportVerifier(Protocol):
    def verify_and_normalize(
        self, serialized_report: bytes, expected: ExpectedBindings
    ) -> VerifiedBehaviorReport:
        """Verify the external attestation and return normalized complete events."""
        raise RuntimeError("protocol-only method")


class StageReceiptSink(Protocol):
    def write_and_readback(self, result: dict) -> str:
        """Persist this stage result and return its durable receipt ID."""
        raise RuntimeError("protocol-only method")


def _logical_path(value: str) -> str:
    if (
        not isinstance(value, str) or not value.startswith("/")
        or value.startswith("//") or "\\" in value
    ):
        raise ValueError("logical path must be an absolute POSIX path")
    path = PurePosixPath(value)
    canonical = str(path)
    if (
        canonical != value or ".." in path.parts or "." in path.parts
        or canonical == "/"
    ):
        raise ValueError("logical path is ambiguous or overbroad")
    return canonical


def _path_allowed(observed: str, declared: tuple[str, ...]) -> bool:
    path = PurePosixPath(_logical_path(observed))
    for entry in declared:
        allowed = PurePosixPath(_logical_path(entry))
        if path == allowed or allowed in path.parents:
            return True
    return False


def _canonical_host(value: str) -> str:
    if not isinstance(value, str) or not value or "://" in value:
        raise ValueError("network target must be a host, not a URL")
    candidate = value.rstrip(".")
    try:
        return str(ipaddress.ip_address(candidate))
    except ValueError:
        ascii_name = candidate.encode("idna").decode("ascii").lower()
        labels = ascii_name.split(".")
        if (
            len(ascii_name) > 253 or len(labels) < 2
            or any(
                not label or len(label) > 63
                or label.startswith("-") or label.endswith("-")
                for label in labels
            )
        ):
            raise ValueError("invalid FQDN")
        return ascii_name


def _host_allowed(observed: str, rules: tuple[str, ...]) -> bool:
    host = _canonical_host(observed)
    for raw_rule in rules:
        if raw_rule.startswith("*."):
            suffix = _canonical_host(raw_rule[2:])
            if host != suffix and host.endswith("." + suffix):
                return True
        elif host == _canonical_host(raw_rule):
            return True
    return False


def compare_verified_report(
    report: VerifiedBehaviorReport,
    expected: ExpectedBindings,
    manifest: PermissionManifest,
) -> dict:
    bindings = {
        "artifact_sha256": expected.artifact_sha256,
        "manifest_sha256": expected.manifest_sha256,
        "analyzer_profile_sha256": expected.analyzer_profile_sha256,
        "scenario_population_sha256": expected.scenario_population_sha256,
        "policy_revision": expected.policy_revision,
    }
    if any(getattr(report, key) != value for key, value in bindings.items()):
        raise RuntimeError("verified_behavior_report_binding_mismatch")
    if not report.report_id:
        raise RuntimeError("missing_behavior_report_id")

    event_ids = [event.event_id for event in report.events]
    if (
        any(not event_id for event_id in event_ids)
        or len(event_ids) != len(set(event_ids))
        or any(not event.scenario_id for event in report.events)
    ):
        raise RuntimeError("missing_or_duplicate_behavior_event_id")

    findings: list[dict[str, str]] = []
    critical = False
    supported = {
        "file_read", "file_write", "network_connect",
        "process_exec", "tool_call", "protected_write",
    }
    for event in report.events:
        reason = None
        if event.kind not in supported:
            reason = "unknown_event_kind"
        elif event.kind == "file_read":
            if not _path_allowed(
                event.target, manifest.read_paths + manifest.write_paths
            ):
                reason = "undeclared_file_read"
        elif event.kind == "file_write":
            if not _path_allowed(event.target, manifest.write_paths):
                reason = "undeclared_file_write"
        elif event.kind == "network_connect":
            if not _host_allowed(event.target, manifest.network_hosts):
                reason = "undeclared_network_host"
        elif event.kind == "process_exec":
            if not _path_allowed(event.target, manifest.executables):
                reason = "undeclared_process_exec"
        elif event.kind == "tool_call":
            if event.target not in set(manifest.tools):
                reason = "undeclared_tool_call"
        elif event.kind == "protected_write":
            if event.target not in set(manifest.protected_writes):
                reason = "undeclared_protected_write"
                critical = True
        if reason:
            findings.append({
                "event_id": event.event_id,
                "scenario_id": event.scenario_id,
                "kind": event.kind,
                "target": event.target,
                "reason": reason,
            })

    complete = (
        report.population_complete
        and report.instrumentation_complete
        and report.teardown_verified
    )
    if not complete:
        status = "INSUFFICIENT_DATA"
    elif findings:
        status = "FAIL"
    else:
        status = "PASS"
    return {
        "schema_version": 1,
        "stage": "behavioral_manifest_compliance",
        "status": status,
        "critical": critical,
        "report_id": report.report_id,
        "bindings": bindings,
        "population_complete": report.population_complete,
        "instrumentation_complete": report.instrumentation_complete,
        "teardown_verified": report.teardown_verified,
        "observed_event_count": len(report.events),
        "findings": findings,
    }


def run_compliance_stage(
    serialized_report: bytes,
    expected: ExpectedBindings,
    manifest: PermissionManifest,
    verifier: AttestedReportVerifier,
    sink: StageReceiptSink,
) -> dict:
    report = verifier.verify_and_normalize(serialized_report, expected)
    result = compare_verified_report(report, expected, manifest)
    result["receipt_id"] = sink.write_and_readback(result)
    if result["status"] != "PASS":
        raise RuntimeError(
            "behavioral_compliance_" + result["status"].lower()
        )
    return result
Production implementation

The AttestedReportVerifier is the integration boundary to AID-I-001.005: it must validate the attestation chain, exact artifact and manifest hashes, analyzer image/profile, signed scenario-set digest, loss counters, event ordering, timeout state, and destruction receipt before normalization. It must produce one event for every observed file read, file write, network connection, process execution, tool call, and protected logical-resource write. Unknown event kinds and malformed targets fail closed. Run multiple signed scenarios for identity, time, environment, and target-file conditions; the scenario-set owner, not the skill, defines completeness.

Independent verification

Replay the same verified report through a separately deployed comparator and reconcile every event ID. Include undeclared read and write fixtures that prove the scopes are not unioned, wildcard-domain apex and sibling-domain cases, Unicode hostnames, path traversal and noncanonical paths, undeclared process/tool/protected writes, delayed actions, duplicated events, bad bindings, partial scenarios, lost telemetry, timeout, and failed teardown. Only a complete, binding-correct population with zero findings may emit the PASS receipt consumed by AID-H-030.005.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-H-030.003: Manifest-vs-Observed Behavioral Consistency Testing | AIDEFEND

Loading interactive view…

AIDEFEND™

A Practical Knowledge Base for AI Security Defenses

Version

GitHub MCP Service