AID-D-003.004: Tool-Call Sequence Anomaly Detection

Version 1.20261015 · Actionable sub-technique

Observe the complete ordered stream of authenticated tool proposals and execution results for each agent, version, task class, and session, then compare the sequence with a versioned approved workflow graph or a statistically calibrated transition model. Emit a finding for forbidden edges, missing required stages, suspicious loops, high-risk tool pivots, or policy-defined sequence anomalies while preserving the exact event order and model/policy bindings needed for replay.

Pillars: app

Lifecycle: operation

Scope boundary

Owns anomaly findings over the complete ordered stream of authenticated tool proposals and results, using an approved workflow graph or calibrated transition model to identify forbidden edges, missing stages, loops, pivots, and suspicious compositions.

AID-D-003.003: AID-D-003.004 detects risk created by tool-call order and composition; AID-D-003.003 monitors policy compliance and enforcement outcomes for individual actions. Sequence semantics require preserving event order and workflow state rather than judging each call independently.

AID-D-003.005: AID-D-003.004 focuses on ordered tool-call transitions and workflow graph violations; AID-D-003.005 tracks broader cross-turn intent drift and session security invariants. A session can drift without an anomalous tool sequence, and a suspicious sequence may occur before a clear intent shift is visible.

Implementation guidance

AID-D-003.004-G001

Learn normal tool-call transitions, then alert on low-likelihood sequences.
Reusable transition-likelihood detector

Build and sign the transition baseline from an approved benign corpus outside this runtime. At detection time, consume a complete ordered session, score every adjacent transition, normalize by transition count, and keep unknown tools and missing sequence numbers visible.

from __future__ import annotations

# File: detection/tool_sequence_anomaly.py

import math
from dataclasses import dataclass


@dataclass(frozen=True)
class ToolEvent:
    event_id: str
    sequence: int
    tool_name: str


def evaluate_tool_sequence(
    session_id: str,
    events: list[ToolEvent],
    baseline: dict,
    policy: dict,
    *,
    session_complete: bool,
) -> dict:
    required_policy = {
        "policy_version", "minimum_transitions",
        "unknown_transition_probability",
        "maximum_negative_log_likelihood_per_transition",
        "forbidden_edges", "required_stages", "maximum_visits_per_tool",
    }
    if set(policy) != required_policy:
        raise ValueError("tool-sequence policy schema mismatch")
    if type(session_complete) is not bool:
        raise ValueError("session completion must come from the authenticated event source")
    if (type(policy["minimum_transitions"]) is not int or policy["minimum_transitions"] < 1
            or not isinstance(policy["forbidden_edges"], list)
            or any(not isinstance(edge, list) or len(edge) != 2 or
                   any(not isinstance(tool, str) or not tool for tool in edge)
                   for edge in policy["forbidden_edges"])
            or not isinstance(policy["required_stages"], list)
            or any(not isinstance(tool, str) or not tool for tool in policy["required_stages"])
            or len(set(policy["required_stages"])) != len(policy["required_stages"])
            or not isinstance(policy["maximum_visits_per_tool"], dict)
            or any(not isinstance(tool, str) or not tool or type(bound) is not int or bound < 1
                   for tool, bound in policy["maximum_visits_per_tool"].items())):
        raise ValueError("workflow policy is malformed")
    threshold = float(policy["maximum_negative_log_likelihood_per_transition"])
    if not math.isfinite(threshold) or threshold < 0:
        raise ValueError("sequence likelihood threshold is invalid")
    if len({event.event_id for event in events}) != len(events):
        raise ValueError("duplicate tool event ID")
    ordered = sorted(events, key=lambda event: event.sequence)
    if [event.sequence for event in ordered] != list(range(len(ordered))):
        return {
            "control_status": "INSUFFICIENT_DATA",
            "finding_status": "UNKNOWN",
            "session_id": session_id,
            "reason": "tool event sequence has a gap or nonzero start",
            "policy_version": policy["policy_version"],
        }
    transition_count = max(0, len(ordered) - 1)
    forbidden = {tuple(edge) for edge in policy["forbidden_edges"]}
    violations = []
    counts = {}
    for event in ordered:
        counts[event.tool_name] = counts.get(event.tool_name, 0) + 1
    for left, right in zip(ordered, ordered[1:]):
        if (left.tool_name, right.tool_name) in forbidden:
            violations.append({"kind": "forbidden_edge", "event_ids": [left.event_id, right.event_id]})
    for tool, maximum in policy["maximum_visits_per_tool"].items():
        if counts.get(tool, 0) > maximum:
            violations.append({"kind": "excessive_tool_loop", "tool": tool, "visits": counts[tool], "maximum": maximum})
    if session_complete:
        for tool in policy["required_stages"]:
            if tool not in counts:
                violations.append({"kind": "missing_required_stage", "tool": tool})
    # Deterministic violations remain findings even before a statistical minimum.
    sufficient = transition_count >= policy["minimum_transitions"]
    unknown_probability = float(policy["unknown_transition_probability"])
    if not 0.0 < unknown_probability < 1.0:
        raise ValueError("unknown transition probability must be in (0,1)")
    scored = []
    total_negative_log = 0.0
    for left, right in zip(ordered, ordered[1:]):
        key = left.tool_name + " -> " + right.tool_name
        probability = float(baseline["transition_probabilities"].get(key, unknown_probability))
        if not 0.0 < probability <= 1.0:
            raise ValueError("baseline transition probability outside (0,1]")
        value = -math.log(probability)
        total_negative_log += value
        scored.append({
            "from_tool": left.tool_name,
            "to_tool": right.tool_name,
            "probability": probability,
            "negative_log_likelihood": round(value, 8),
            "unknown_transition": key not in baseline["transition_probabilities"],
        })
    average = total_negative_log / transition_count if transition_count else None
    finding = bool(violations) or (sufficient and average >= threshold)
    return {
        "schema_version": "aidefend.tool_sequence_finding.v1",
        "control_status": "PASS" if finding or (sufficient and session_complete) else "INSUFFICIENT_DATA",
        "finding_status": "FINDING" if finding else ("NO_FINDING" if sufficient and session_complete else "UNKNOWN"),
        "workflow_violations": violations,
        "session_complete": session_complete,
        "statistical_population_sufficient": sufficient,
        "ordered_event_ids": [event.event_id for event in ordered],
        "session_id": session_id,
        "baseline_version": baseline["baseline_version"],
        "policy_version": policy["policy_version"],
        "transition_count": transition_count,
        "negative_log_likelihood_per_transition": round(average, 8) if average is not None else None,
        "transitions": scored,
    }

Verification: Replay normal, rare-but-approved, unknown-tool, low-likelihood, duplicate, missing-sequence, concurrent-order-policy, and undersized sessions. Rebuild the ordered event population independently and compare every transition score.

Action: Score complete authenticated session sequences and emit a finding for low-likelihood behavior; do not force human approval or modify tool access from this detector.

Workflow checks: The signed policy now also carries forbidden_edges as two-tool arrays, required_stages as tool names, and maximum_visits_per_tool as positive integer limits selected for the approved workflow. Supply session_complete only from the authenticated terminal event and reconcile the event population first. Forbidden transitions and policy-bounded repeated visits can emit findings in short sessions; missing stages are evaluated only after completion. An incomplete clean prefix remains UNKNOWN. Replay forbidden-edge, omitted-stage and repeated-loop counterexamples as well as an approved short prefix. Anomaly findings do not prove which prompt caused the behavior.

Open-source tools

Commercial / hosted tools

Threat mappings

MITRE ATLAS

MAESTRO

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

Cisco Integrated AI Security and Safety Framework

Google Secure AI Framework 2.0 - Risks

Databricks AI Security Framework 3.0

AID-D-003.004: Tool-Call Sequence Anomaly Detection | AIDEFEND

Loading interactive view…

AIDEFEND™

A Practical Knowledge Base for AI Security Defenses

Version

GitHub MCP Service