AID-H-036: Multilingual & Locale-Stratified Prompt Safety Classifier Evaluation
Evaluate inbound prompt content and policy classifiers across supported, unsupported, mixed-language, and locale-specific inputs so safety screening does not silently collapse outside the primary development language. The control measures policy-violation detection, prompt-injection and jailbreak detection, harmful-content classification, token-stream moderation behavior, and fail-closed behavior per language or locale segment before a classifier or guardrail policy is promoted. Qwen3Guard is a natural fit for this control because it explicitly targets broad multilingual coverage and includes a streaming variant, but the evaluation must still measure the exact classifier and deployment path used in production.
Pillars: app, model
Lifecycle: validation, improvement
Scope boundary
Owns pre-promotion multilingual, mixed-language, locale-stratified, and unsupported-language evaluation evidence for inbound prompt content, policy, prompt-injection, and jailbreak classifiers. It does not perform runtime filtering or retrain the classifier.
AID-D-001.001: AID-H-036 validates language-segment coverage before a classifier or guardrail policy is promoted; AID-D-001.001 runs per-prompt analysis and emits the live inbound finding. Pre-release coverage evidence does not classify a production request.
AID-D-003.001: AID-H-036 evaluates inbound prompt safety classifiers; AID-D-003.001 detects harmful text and token output policy violations after generation. Inbound and output classifiers require separate populations and success criteria.
AID-H-026: AID-H-036 measures multilingual and locale-stratified coverage; AID-H-026 retrains an organization-controlled prompt-injection detector through a governed closed loop. Coverage findings can trigger retraining but do not alter the detector.
Warning
Guardrail Coverage Is Not a Complete Boundary
Aggregate accuracy can conceal severe failures in low-volume language or locale segments. Preserve per-segment denominators, unsupported-language behavior, and regression limits, and do not promote from a global score alone.
Implementation guidance
- AID-H-036-G001: Build a language and locale coverage matrix for inbound prompt safety classifiers, run the matrix in CI, and block promotion when any supported or unsupported language segment lacks measured policy behavior.
AID-H-036-G001
Build a language and locale coverage matrix for inbound prompt safety classifiers, run the matrix in CI, and block promotion when any supported or unsupported language segment lacks measured policy behavior.
Concept:
A prompt safety classifier can look strong on English benchmarks while failing on mixed-language, transliterated, low-resource, or unsupported-language prompts. Treat language as an explicit coverage dimension. Supported languages must meet category thresholds; unsupported languages must fail closed to review or block rather than being treated as safe by default.
Keep locale-native prompts in a private, access-controlled and content-addressed corpus. Publish the closed schema below in source control; sign the real manifest and bind its digest in the gate policy. This avoids shipping attack strings or universal language lists in the framework.
Step 1: Define the private evaluation-manifest schema
# File: guardrail_eval/multilingual_cases.schema.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "urn:aidefend:multilingual-cases:v1",
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"case_id", "language", "locale", "language_support", "label",
"policy_category", "expected_verdict", "prompt", "prompt_sha256"
],
"properties": {
"case_id": {"type": "string", "minLength": 1},
"language": {"type": "string", "minLength": 1},
"locale": {"type": "string", "minLength": 1},
"language_support": {"enum": ["supported", "unsupported"]},
"label": {"enum": ["benign", "malicious", "unsupported"]},
"policy_category": {"type": "string", "minLength": 1},
"expected_verdict": {"enum": ["allow", "review", "block"]},
"prompt": {"type": "string", "minLength": 1},
"prompt_sha256": {"type": "string", "pattern": "^[a-f0-9]{64}$"}
}
}
}
Step 2: Run the classifier and compute per-segment evidence
# File: guardrail_eval/multilingual_classifier_gate.py
from __future__ import annotations
import base64
import hashlib
import json
import math
import os
import re
from dataclasses import asdict, dataclass, fields
from datetime import datetime, timezone
from pathlib import Path
from typing import Literal
from urllib.parse import urlsplit, urlunsplit
import requests
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
Verdict = Literal["allow", "review", "block"]
Label = Literal["benign", "malicious", "unsupported"]
@dataclass(frozen=True)
class EvalCase:
case_id: str
language: str
locale: str
language_support: Literal["supported", "unsupported"]
label: Label
policy_category: str
expected_verdict: Verdict
prompt: str
prompt_sha256: str
@dataclass(frozen=True)
class EvalResult:
case_id: str
language: str
locale: str
language_support: Literal["supported", "unsupported"]
label: Label
policy_category: str
expected_verdict: Verdict
observed_verdict: Verdict | None
passed: bool
reason: str
@dataclass(frozen=True)
class GatePolicy:
policy_version: str
case_manifest_sha256: str
required_supported_segments: tuple[tuple[str, str], ...]
required_unsupported_segments: tuple[tuple[str, str], ...]
required_malicious_categories: list[str]
min_benign_per_segment: int
min_malicious_per_segment: int
min_malicious_per_category: int
min_unsupported_per_segment: int
min_malicious_recall: float
min_malicious_precision: float
max_benign_fpr: float
min_exact_verdict_rate: float
wilson_z: float
classifier_timeout_seconds: float
classifier_response_max_bytes: int
VALID_VERDICTS = {"allow", "review", "block"}
SHA256 = re.compile(r"^[a-f0-9]{64}$")
def no_duplicates(pairs: list[tuple[str, object]]) -> dict:
result = {}
for key, value in pairs:
if key in result:
raise ValueError(f"duplicate JSON key: {key}")
result[key] = value
return result
def strict_json(raw: bytes):
return json.loads(
raw,
object_pairs_hook=no_duplicates,
parse_constant=lambda value: (_ for _ in ()).throw(
ValueError(f"non-finite JSON number: {value}")
),
)
def canonical_https_endpoint(value: str) -> str:
parsed = urlsplit(value)
if (
parsed.scheme != "https"
or not parsed.hostname
or parsed.username is not None
or parsed.password is not None
or parsed.query
or parsed.fragment
or not parsed.path.startswith("/")
):
raise ValueError("classifier endpoint must be an exact HTTPS URL")
canonical = urlunsplit(("https", parsed.netloc.lower(), parsed.path, "", ""))
if value != canonical:
raise ValueError("classifier endpoint is not canonical")
return value
def load_signed_policy(
path: Path, signature_path: Path, public_key_path: Path,
) -> tuple[GatePolicy, str]:
payload = path.read_bytes()
public_key_bytes = base64.b64decode(
public_key_path.read_text(encoding="ascii").strip(), validate=True
)
signature = base64.b64decode(
signature_path.read_text(encoding="ascii").strip(), validate=True
)
Ed25519PublicKey.from_public_bytes(public_key_bytes).verify(signature, payload)
raw = strict_json(payload)
required = {field.name for field in fields(GatePolicy)}
if not isinstance(raw, dict) or set(raw) != required:
raise ValueError("multilingual gate policy schema mismatch")
for name in ("required_supported_segments", "required_unsupported_segments"):
rows = raw[name]
if (not isinstance(rows, list) or any(
not isinstance(row, list) or len(row) != 2
or any(not isinstance(part, str) or not part or part != part.strip() for part in row)
for row in rows)):
raise ValueError("invalid signed segment population: " + name)
normalized = tuple(tuple(row) for row in rows)
if len(set(normalized)) != len(normalized):
raise ValueError("duplicate signed segment: " + name)
raw[name] = normalized
if set(raw["required_supported_segments"]) & set(raw["required_unsupported_segments"]):
raise ValueError("segment cannot be both supported and unsupported")
categories = raw["required_malicious_categories"]
if (not isinstance(categories, list)
or any(not isinstance(value, str) or not value for value in categories)
or len(set(categories)) != len(categories)):
raise ValueError("invalid or duplicate malicious category")
policy = GatePolicy(**raw)
if not SHA256.fullmatch(policy.case_manifest_sha256):
raise ValueError("case_manifest_sha256 is invalid")
counts = (
policy.min_benign_per_segment, policy.min_malicious_per_segment,
policy.min_malicious_per_category, policy.min_unsupported_per_segment,
)
rates = (
policy.min_malicious_recall, policy.min_malicious_precision,
policy.max_benign_fpr, policy.min_exact_verdict_rate,
)
if any(isinstance(value, bool) or not isinstance(value, int) or value <= 0 for value in counts):
raise ValueError("signed minimum populations must be positive integers")
if any(
isinstance(value, bool)
or not isinstance(value, (int, float))
or not math.isfinite(float(value))
or not 0 <= float(value) <= 1
for value in rates
):
raise ValueError("signed metric thresholds must be finite values in [0, 1]")
if (
isinstance(policy.wilson_z, bool)
or not isinstance(policy.wilson_z, (int, float))
or not math.isfinite(float(policy.wilson_z))
or policy.wilson_z <= 0
or isinstance(policy.classifier_timeout_seconds, bool)
or not isinstance(policy.classifier_timeout_seconds, (int, float))
or not math.isfinite(float(policy.classifier_timeout_seconds))
or policy.classifier_timeout_seconds <= 0
or isinstance(policy.classifier_response_max_bytes, bool)
or not isinstance(policy.classifier_response_max_bytes, int)
or policy.classifier_response_max_bytes <= 0
):
raise ValueError("signed confidence, timeout, or response bounds are invalid")
if not policy.required_supported_segments or not policy.required_malicious_categories:
raise ValueError("signed policy omits required coverage populations")
return policy, hashlib.sha256(payload).hexdigest()
def classify_prompt(
session: requests.Session,
endpoint: str,
prompt: str,
expected_classifier_version: str,
timeout_seconds: float,
response_max_bytes: int,
) -> Verdict:
endpoint = canonical_https_endpoint(endpoint)
with session.post(
endpoint,
json={"input": prompt},
timeout=timeout_seconds,
allow_redirects=False,
stream=True,
) as response:
if response.status_code != 200:
raise ValueError(f"classifier returned HTTP {response.status_code}")
media_type = response.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
if media_type != "application/json":
raise ValueError("classifier did not return application/json")
declared = response.headers.get("Content-Length")
if declared is not None:
try:
if int(declared) > response_max_bytes:
raise ValueError("classifier response exceeds signed byte cap")
except ValueError as exc:
raise ValueError("classifier Content-Length is invalid or too large") from exc
raw = bytearray()
for chunk in response.iter_content(chunk_size=min(65536, response_max_bytes + 1)):
if chunk:
raw.extend(chunk)
if len(raw) > response_max_bytes:
raise ValueError("classifier response exceeds signed decoded-byte cap")
body = strict_json(bytes(raw))
if not isinstance(body, dict) or set(body) != {"classifier_version", "verdict"}:
raise ValueError("classifier response schema differs")
if body["classifier_version"] != expected_classifier_version:
raise ValueError("classifier response version differs from candidate release")
verdict = body.get("verdict")
if verdict not in VALID_VERDICTS:
raise ValueError("Classifier returned invalid verdict: " + str(verdict))
return verdict
def load_cases(path: str) -> list[EvalCase]:
raw_cases = strict_json(Path(path).read_bytes())
if not isinstance(raw_cases, list):
raise ValueError("case manifest root must be an array")
cases = [EvalCase(**item) for item in raw_cases]
case_ids: set[str] = set()
for case in cases:
if case.case_id in case_ids:
raise ValueError("Duplicate case_id: " + case.case_id)
case_ids.add(case.case_id)
if not SHA256.fullmatch(case.prompt_sha256):
raise ValueError("case prompt_sha256 is invalid: " + case.case_id)
observed_prompt_sha256 = hashlib.sha256(case.prompt.encode("utf-8")).hexdigest()
if observed_prompt_sha256 != case.prompt_sha256:
raise ValueError("case prompt digest mismatch: " + case.case_id)
if case.language_support == "supported" and case.label not in {"benign", "malicious"}:
raise ValueError("Supported case must be labeled benign or malicious: " + case.case_id)
if case.language_support == "unsupported" and case.label != "unsupported":
raise ValueError("Unsupported-language case must use label=unsupported: " + case.case_id)
if case.label == "benign" and case.expected_verdict != "allow":
raise ValueError("Benign case must expect exact verdict allow: " + case.case_id)
if case.label in {"malicious", "unsupported"} and case.expected_verdict == "allow":
raise ValueError("Malicious or unsupported case cannot expect allow: " + case.case_id)
return cases
def run_matrix(
session: requests.Session,
endpoint: str,
cases: list[EvalCase],
expected_classifier_version: str,
timeout_seconds: float,
response_max_bytes: int,
) -> list[EvalResult]:
results: list[EvalResult] = []
for case in cases:
try:
observed = classify_prompt(
session,
endpoint,
case.prompt,
expected_classifier_version,
timeout_seconds,
response_max_bytes,
)
passed = observed == case.expected_verdict
reason = "exact_verdict_match" if passed else "exact_verdict_mismatch"
except Exception as exc:
observed = None
passed = False
reason = "classifier_error:" + type(exc).__name__
results.append(EvalResult(
case_id=case.case_id,
language=case.language,
locale=case.locale,
language_support=case.language_support,
label=case.label,
policy_category=case.policy_category,
expected_verdict=case.expected_verdict,
observed_verdict=observed,
passed=passed,
reason=reason,
))
return results
def ratio(numerator: int, denominator: int) -> float | None:
if denominator == 0:
return None
return round(numerator / denominator, 4)
def wilson_interval(successes: int, trials: int, z: float) -> dict | None:
"""Return a two-sided Wilson score interval for a binomial proportion."""
if trials == 0:
return None
if trials < 0 or successes < 0 or successes > trials:
raise ValueError("Wilson interval requires 0 <= successes <= trials")
proportion = successes / trials
z_squared = z * z
denominator = 1.0 + z_squared / trials
center = (proportion + z_squared / (2.0 * trials)) / denominator
margin = (
z
* math.sqrt((proportion * (1.0 - proportion) + z_squared / (4.0 * trials)) / trials)
/ denominator
)
return {
"lower": round(max(0.0, center - margin), 6),
"upper": round(min(1.0, center + margin), 6),
"confidence": 0.95,
"method": "wilson_score",
}
def summarize_supported_segment(
language: str,
locale: str,
results: list[EvalResult],
policy: GatePolicy,
) -> dict:
segment = [
item for item in results
if item.language_support == "supported" and item.language == language and item.locale == locale
]
benign = [item for item in segment if item.label == "benign"]
malicious = [item for item in segment if item.label == "malicious"]
errors = [item for item in segment if item.observed_verdict is None]
true_positives = sum(item.observed_verdict in {"review", "block"} for item in malicious)
false_positives = sum(item.observed_verdict in {"review", "block"} for item in benign)
exact_matches = sum(item.passed for item in segment)
recall = ratio(true_positives, len(malicious))
precision = ratio(true_positives, true_positives + false_positives)
benign_fpr = ratio(false_positives, len(benign))
exact_rate = ratio(exact_matches, len(segment))
recall_interval = wilson_interval(true_positives, len(malicious), policy.wilson_z)
precision_interval = wilson_interval(
true_positives,
true_positives + false_positives,
policy.wilson_z,
)
benign_fpr_interval = wilson_interval(false_positives, len(benign), policy.wilson_z)
exact_rate_interval = wilson_interval(exact_matches, len(segment), policy.wilson_z)
insufficient: list[dict] = []
failures: list[dict] = []
if len(benign) < policy.min_benign_per_segment:
insufficient.append({"population": "benign", "n": len(benign), "minimum": policy.min_benign_per_segment})
if len(malicious) < policy.min_malicious_per_segment:
insufficient.append({"population": "malicious", "n": len(malicious), "minimum": policy.min_malicious_per_segment})
if errors:
failures.append({"metric": "classifier_errors", "count": len(errors), "case_ids": [item.case_id for item in errors]})
if recall_interval is not None and recall_interval["lower"] < policy.min_malicious_recall:
failures.append({
"metric": "malicious_recall",
"estimate": recall,
"conservative_lower_bound": recall_interval["lower"],
"minimum": policy.min_malicious_recall,
})
if precision_interval is None:
failures.append({"metric": "malicious_precision", "issue": "no_positive_predictions"})
elif precision_interval["lower"] < policy.min_malicious_precision:
failures.append({
"metric": "malicious_precision",
"estimate": precision,
"conservative_lower_bound": precision_interval["lower"],
"minimum": policy.min_malicious_precision,
})
if benign_fpr_interval is not None and benign_fpr_interval["upper"] > policy.max_benign_fpr:
failures.append({
"metric": "benign_fpr",
"estimate": benign_fpr,
"conservative_upper_bound": benign_fpr_interval["upper"],
"maximum": policy.max_benign_fpr,
})
if exact_rate_interval is not None and exact_rate_interval["lower"] < policy.min_exact_verdict_rate:
failures.append({
"metric": "exact_verdict_rate",
"estimate": exact_rate,
"conservative_lower_bound": exact_rate_interval["lower"],
"minimum": policy.min_exact_verdict_rate,
})
category_metrics = {}
observed_categories = {item.policy_category for item in malicious}
categories = sorted(set(policy.required_malicious_categories) | observed_categories)
for category in categories:
category_results = [item for item in malicious if item.policy_category == category]
category_tp = sum(item.observed_verdict in {"review", "block"} for item in category_results)
category_recall = ratio(category_tp, len(category_results))
category_recall_interval = wilson_interval(
category_tp,
len(category_results),
policy.wilson_z,
)
category_metrics[category] = {
"n": len(category_results),
"recall": category_recall,
"recall_wilson_95": category_recall_interval,
}
if category in policy.required_malicious_categories and len(category_results) < policy.min_malicious_per_category:
insufficient.append({
"population": "malicious_category",
"category": category,
"n": len(category_results),
"minimum": policy.min_malicious_per_category,
})
if (
category_recall_interval is not None
and category_recall_interval["lower"] < policy.min_malicious_recall
):
failures.append({
"metric": "category_recall",
"category": category,
"estimate": category_recall,
"conservative_lower_bound": category_recall_interval["lower"],
"minimum": policy.min_malicious_recall,
})
status = "insufficient_data" if insufficient else ("fail" if failures else "pass")
return {
"status": status,
"n_total": len(segment),
"n_benign": len(benign),
"n_malicious": len(malicious),
"n_predicted_positive": true_positives + false_positives,
"malicious_recall": recall,
"malicious_recall_wilson_95": recall_interval,
"malicious_precision": precision,
"malicious_precision_wilson_95": precision_interval,
"benign_fpr": benign_fpr,
"benign_fpr_wilson_95": benign_fpr_interval,
"exact_verdict_rate": exact_rate,
"exact_verdict_rate_wilson_95": exact_rate_interval,
"category_metrics": category_metrics,
"insufficient_populations": insufficient,
"failures": failures,
}
def summarize_unsupported_segment(
language: str,
locale: str,
results: list[EvalResult],
policy: GatePolicy,
) -> dict:
segment = [
item for item in results
if item.language_support == "unsupported" and item.language == language and item.locale == locale
]
errors = [item for item in segment if item.observed_verdict is None]
allowed = [item for item in segment if item.observed_verdict == "allow"]
exact_matches = sum(item.passed for item in segment)
exact_rate = ratio(exact_matches, len(segment))
exact_rate_interval = wilson_interval(exact_matches, len(segment), policy.wilson_z)
insufficient = []
failures = []
if len(segment) < policy.min_unsupported_per_segment:
insufficient.append({"population": "unsupported", "n": len(segment), "minimum": policy.min_unsupported_per_segment})
if errors:
failures.append({"metric": "classifier_errors", "count": len(errors), "case_ids": [item.case_id for item in errors]})
if allowed:
failures.append({"metric": "unsupported_language_allowed", "count": len(allowed), "case_ids": [item.case_id for item in allowed]})
if exact_rate_interval is not None and exact_rate_interval["lower"] < policy.min_exact_verdict_rate:
failures.append({
"metric": "exact_verdict_rate",
"estimate": exact_rate,
"conservative_lower_bound": exact_rate_interval["lower"],
"minimum": policy.min_exact_verdict_rate,
})
status = "insufficient_data" if insufficient else ("fail" if failures else "pass")
return {
"status": status,
"n_total": len(segment),
"allow_count": len(allowed),
"exact_verdict_rate": exact_rate,
"exact_verdict_rate_wilson_95": exact_rate_interval,
"insufficient_populations": insufficient,
"failures": failures,
}
def summarize(results: list[EvalResult], policy: GatePolicy) -> dict:
found_supported = {
(item.language, item.locale) for item in results if item.language_support == "supported"
}
found_unsupported = {
(item.language, item.locale) for item in results if item.language_support == "unsupported"
}
supported_segments = sorted(set(policy.required_supported_segments) | found_supported)
unsupported_segments = sorted(set(policy.required_unsupported_segments) | found_unsupported)
supported = {
f"{language}:{locale}": summarize_supported_segment(language, locale, results, policy)
for language, locale in supported_segments
}
unsupported = {
f"{language}:{locale}": summarize_unsupported_segment(language, locale, results, policy)
for language, locale in unsupported_segments
}
statuses = [item["status"] for item in [*supported.values(), *unsupported.values()]]
gate_status = "insufficient_data" if "insufficient_data" in statuses else ("fail" if "fail" in statuses else "pass")
return {
"gate_status": gate_status,
"policy": asdict(policy),
"supported_segments": supported,
"unsupported_segments": unsupported,
}
def enforce_gate(summary: dict) -> None:
if summary["gate_status"] != "pass":
raise SystemExit(
"Multilingual guardrail coverage gate did not pass: "
+ json.dumps(summary, sort_keys=True)
)
def main() -> None:
case_path = Path(os.environ["MULTILINGUAL_CASE_MANIFEST_PATH"])
endpoint = canonical_https_endpoint(os.environ["CLASSIFIER_ENDPOINT"])
classifier_version = os.environ["CLASSIFIER_VERSION"]
policy, policy_sha256 = load_signed_policy(
Path(os.environ["MULTILINGUAL_GATE_POLICY_PATH"]),
Path(os.environ["MULTILINGUAL_GATE_POLICY_SIGNATURE_PATH"]),
Path(os.environ["MULTILINGUAL_GATE_POLICY_PUBLIC_KEY_PATH"]),
)
case_set_sha256 = hashlib.sha256(case_path.read_bytes()).hexdigest()
if case_set_sha256 != policy.case_manifest_sha256:
raise ValueError("case manifest differs from signed gate policy")
cases = load_cases(str(case_path))
session = requests.Session()
session.trust_env = False
session.verify = os.environ["CLASSIFIER_CA_BUNDLE"]
session.cert = (
os.environ["CLASSIFIER_CLIENT_CERT"],
os.environ["CLASSIFIER_CLIENT_KEY"],
)
results = run_matrix(
session,
endpoint,
cases,
classifier_version,
policy.classifier_timeout_seconds,
policy.classifier_response_max_bytes,
)
summary = summarize(results, policy)
summary.update({
"generated_at": datetime.now(timezone.utc).isoformat(),
"classifier_version": classifier_version,
"case_set_sha256": case_set_sha256,
"policy_sha256": policy_sha256,
})
Path("guardrail_eval/results.json").write_text(
json.dumps([asdict(item) for item in results], indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
Path("guardrail_eval/summary.json").write_text(
json.dumps(summary, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
enforce_gate(summary)
if __name__ == "__main__":
main()
Step 3: Preserve reviewer-grade evidence
Promotion evidence should include the classifier version, policy version, signed classifier response-byte cap, case-set hash, language and locale list, category matrix, unmeasured segment list, point estimates, Wilson 95% confidence intervals, failed examples, and the resulting allow/review/block decision. Gate on the conservative interval bound: lower bounds for recall, precision, exact-verdict rate, and per-category recall; the upper bound for benign false-positive rate. Do not drop segments with too few examples; mark them as insufficient_data and block promotion or route to risk acceptance.
Action: run this gate before releasing a new classifier, moderation prompt, guardrail model, supported-language expansion, or policy threshold change. Unsupported-language behavior must be explicit and tested.
Keep the action safely bounded
Reconcile the complete population of every policy-required supported and unsupported language-locale-category segment and every case in the signed corpus manifest; reject expired, revoked, or stale policy and asset bindings under the policy-defined freshness window.
Verify safely
A separately credentialed verifier independently replays policy/corpus/candidate binding and metrics, replaying supported, unsupported, mixed-language, transliterated, obfuscated, benign, malicious, classifier-error, sparse-segment, and label-leakage fixtures, reads back effective state from the enforcement system, exercises positive, negative, boundary, stale-evidence, and dependency-failure fixtures, and reconciles every eligible item.
Open-source tools
- promptfoo
- garak
- Inspect AI
- pandas
- scikit-learn
- EleutherAI lm-evaluation-harness (custom locale-stratified task runner)
Source-available / open-weight tools
- Qwen3Guard-Gen (Apache-2.0; open-weight)
- Qwen3Guard-Stream (Apache-2.0; open-weight)
Commercial / hosted tools
- OpenAI Moderations API (omni-moderation-latest)
- Azure AI Content Safety
- Check Point AI Guardrails
Threat mappings
MITRE ATLAS
- AML.T0051 LLM Prompt Injection
- AML.T0054 LLM Jailbreak
- AML.T0068 LLM Prompt Obfuscation
- AML.T0015 Evade AI Model
- AML.T0043 Craft Adversarial Data
- AML.T0123 Obfuscated Files or Information (measures filter collapse under mixed-language and locale-shifted evasion)
MAESTRO
- Input Validation Attacks (L3)
- Framework Evasion (L3)
- Evasion of Detection (L5)
- Evasion of Security AI Agents (L6)
OWASP LLM Top 10 2026
- LLM01:2026 Prompt Injection
OWASP ML Top 10 2023
- ML01:2023 Input Manipulation Attack
OWASP Top 10 for Agentic Applications 2026
- ASI01:2026 Agent Goal Hijack
NIST Adversarial Machine Learning 2025
- NISTAML.018 Prompt Injection
- NISTAML.022 Evasion
- NISTAML.025 Black-box Evasion
Cisco Integrated AI Security and Safety Framework
- AISubtech-1.1.2 Obfuscation (measures exact classifier coverage per language and category before promotion)
- AITech-1.1 Direct Prompt Injection and Jailbreak (Prompt-based Manipulation) (measures exact classifier coverage per language and category before promotion)
- AITech-15.1 Harmful Content (measures exact classifier coverage per language and category before promotion)
- AITech-9.2 Detection Evasion (measures exact classifier coverage per language and category before promotion)
Google Secure AI Framework 2.0 - Risks
- PIJ: Prompt Injection
- MEV: Model Evasion
Databricks AI Security Framework 3.0
- Evaluation 6.2: Insufficient evaluation data
- Model Serving - Inference requests 9.1: Prompt inject
- Model Serving - Inference requests 9.12: LLM Jailbreak