Scope boundary

Owns the versioned machine-readable schema and tags that express allowed AI stages, purposes, consent scope and expiry, jurisdiction, classification, and ownership for each data asset. It defines policy facts but does not make the stage-transition decision.

AID-H-029.002: AID-H-029.001 defines and attaches the data-use facts; AID-H-029.002 evaluates those facts at each lifecycle boundary and issues the allow or deny result. Tags without an enforcement gate are descriptive metadata only.

AID-M-002: AID-H-029.001 expresses authorized use and consent constraints; AID-M-002 records provenance, lineage, integrity, and risk characterization. The two metadata sets may be stored together but answer different questions.

Implementation guidance

• AID-H-029.001-G001: Define a versioned machine-readable schema for data-use tags and reject untagged or malformed assets at ingestion time.
• AID-H-029.001-G002: Propagate data-use tags through transformations and derived assets using a defined inheritance rule, and keep the tags queryable in the metadata or lineage system.
• AID-H-029.001-G003: Backfill legacy assets with suggested tags using automated scanning, but require human approval before those suggested tags become authoritative.

AID-H-029.001-G001

Concept:

Every dataset, document collection, embedding corpus, or interaction log that may reach an AI stage carries the same closed tag set. AID-H-029.002 reads these exact fields at every lifecycle-stage gate, so the stage vocabulary, per-stage purpose grants, asset and tag versions, withdrawal state and expiry defined here are the contract that gate enforces. A tag object that passes this schema but omits a field the gate reads would be denied at every stage.

Define the closed tag schema

Grant purposes per stage rather than as one flat list, so a purpose approved for evaluation cannot authorize RAG indexing. asset_version binds the tags to the exact asset bytes, and tag_version changes on every tag edit so a gate decision can be tied to the tag state it evaluated. consent_expiry is always present: write null only when the authoritative grant has no time limit. An empty authorized_stages list is valid and means the asset is denied at every stage.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "schema_version", "asset_id", "asset_version", "tag_version",
    "authorized_stages", "authorized_purposes", "classification_tier",
    "consent_type", "consent_expiry", "authorization_withdrawn",
    "jurisdiction", "data_owner"
  ],
  "$defs": {
    "stage": {
      "enum": [
        "training", "fine_tuning", "evaluation", "rag_indexing",
        "inference_context", "logging", "memory_write", "retraining"
      ]
    },
    "text": {"type": "string", "minLength": 1}
  },
  "properties": {
    "schema_version": {"const": "2.0"},
    "asset_id": {"$ref": "#/$defs/text"},
    "asset_version": {"$ref": "#/$defs/text"},
    "tag_version": {"$ref": "#/$defs/text"},
    "authorized_stages": {
      "type": "array",
      "items": {"$ref": "#/$defs/stage"},
      "uniqueItems": true
    },
    "authorized_purposes": {
      "type": "object",
      "propertyNames": {"$ref": "#/$defs/stage"},
      "additionalProperties": {
        "type": "array",
        "items": {"$ref": "#/$defs/text"},
        "minItems": 1,
        "uniqueItems": true
      }
    },
    "classification_tier": {"enum": ["public", "internal", "confidential", "restricted"]},
    "consent_type": {"type": ["string", "null"]},
    "consent_expiry": {"type": ["string", "null"]},
    "authorization_withdrawn": {"type": "boolean"},
    "jurisdiction": {"$ref": "#/$defs/text"},
    "data_owner": {"$ref": "#/$defs/text"}
  }
}

Validate the schema and the cross-field rules at ingestion

JSON Schema cannot require the stage list and the purpose-grant keys to match, and the jsonschema date-time format check is not enforced by default. Check both explicitly. The timestamp profile uses uppercase T/Z, an explicit Z or numeric offset (hours 00-23, minutes 00-59), and at most nine fractional digits. Reject invalid calendar times, leap seconds and instants outside the signed 64-bit nanosecond range supported by OPA time.parse_rfc3339_ns; never repair an invalid offset or round an expiry to make it pass.

# File: data_governance/validate_tags.py
from __future__ import annotations

from datetime import datetime, timezone
import json
from pathlib import Path
import re

from jsonschema import Draft202012Validator

TAG_SCHEMA = json.loads(Path("data_use_tag_schema.json").read_text(encoding="utf-8"))
Draft202012Validator.check_schema(TAG_SCHEMA)
VALIDATOR = Draft202012Validator(TAG_SCHEMA)
RFC3339 = re.compile(
    r"(?P<whole>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2})"
    r"(?:\.(?P<fraction>[0-9]{1,9}))?"
    r"(?P<offset>Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])"
)
UNIX_EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
NS_PER_SECOND = 1_000_000_000
MIN_OPA_NS = -(1 << 63)
MAX_OPA_NS = (1 << 63) - 1


class TagRejected(ValueError):
    pass


def parse_rfc3339_ns(value: str) -> int:
    """Parse the tag timestamp profile without offset normalization or rounding."""
    match = RFC3339.fullmatch(value) if isinstance(value, str) else None
    if match is None:
        raise TagRejected("consent_expiry requires RFC 3339 with a valid offset and at most 9 fractional digits")
    try:
        # Validate calendar/time fields separately; retain the fraction as an integer.
        whole = datetime.fromisoformat(match["whole"] + match["offset"].replace("Z", "+00:00"))
    except ValueError as exc:
        raise TagRejected("consent_expiry is not a real calendar time") from exc
    delta = whole - UNIX_EPOCH
    seconds = delta.days * 86400 + delta.seconds
    fraction_ns = int((match["fraction"] or "").ljust(9, "0"))
    timestamp_ns = seconds * NS_PER_SECOND + fraction_ns
    # OPA time.parse_rfc3339_ns requires a signed 64-bit nanosecond instant.
    if not MIN_OPA_NS <= timestamp_ns <= MAX_OPA_NS:
        raise TagRejected("consent_expiry is outside the OPA nanosecond range")
    return timestamp_ns


def validate_tags(tag_doc: dict) -> dict:
    errors = sorted(VALIDATOR.iter_errors(tag_doc), key=lambda e: list(e.absolute_path))
    if errors:
        raise TagRejected(f"invalid data-use tags: {errors[0].message}")
    # Each authorized stage carries its own purpose grant, with no extra grant keys.
    if set(tag_doc["authorized_stages"]) != set(tag_doc["authorized_purposes"]):
        raise TagRejected("authorized stages and per-stage purpose grants differ")
    if tag_doc["consent_expiry"] is not None:
        parse_rfc3339_ns(tag_doc["consent_expiry"])
    return tag_doc

Operational notes: Start fail-closed for new pipelines. For legacy data, allow a temporary quarantine state and route assets without tags into a remediation queue instead of silently admitting them.

Production implementation

At ingestion admission, validate the asset and data-use tag object against the pinned closed schema and signed vocabulary, require authoritative source and policy bindings, reject unknown/missing/expired tags and persist the canonical tag digest with the asset version.

Independent verification

A separately credentialed verifier independently reads back or replays source bytes, schema/vocabulary, canonical tag object and catalog readback with untagged, unknown enum, extra field, stale policy, duplicate, malformed, stage-without-purpose-grant, purpose-grant-for-unauthorized-stage, missing expiry and non-RFC-3339 expiry fixtures, runs positive, negative, boundary, stale-policy, partial-population and dependency-failure fixtures, and reconciles the complete eligible population without trusting a component-supplied verdict.

AID-H-029.001-G002

Concept:

Derived assets need derived authorization. A common safe default is most-restrictive-wins: the derived asset is authorized only for stages allowed by every upstream source.

Example propagation logic

Derive every field that AID-H-029.002 reads. Purposes intersect per stage, so a stage is dropped when the parents share no purpose for it. Withdrawal of any parent withdraws the child, and the earliest bounded parent expiry becomes the child expiry. Parents from different jurisdictions are not merged automatically. Run validate_tags from AID-H-029.001-G001 on the derived object before publishing it.

# File: data_governance/derive_tags.py
from __future__ import annotations

from datetime import datetime

TIER_ORDER = ["public", "internal", "confidential", "restricted"]


class DerivationRejected(ValueError):
    pass


def _expiry(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def derive_child_tags(parent_tags: list[dict], *, child_asset_id: str,
                      child_asset_version: str, tag_version: str,
                      data_owner: str) -> dict:
    """Most-restrictive-wins derivation over parent tags that passed validate_tags."""
    if not parent_tags:
        raise DerivationRejected("a derived asset needs at least one tagged parent")
    jurisdictions = {t["jurisdiction"] for t in parent_tags}
    if len(jurisdictions) != 1:
        raise DerivationRejected("parents span jurisdictions; route the join to review")

    common_stages = set.intersection(*(set(t["authorized_stages"]) for t in parent_tags))
    purposes = {}
    for stage in sorted(common_stages):
        shared = set.intersection(*(set(t["authorized_purposes"][stage]) for t in parent_tags))
        if shared:  # a stage survives only with at least one purpose every parent grants
            purposes[stage] = sorted(shared)
    consent_types = {t["consent_type"] for t in parent_tags}
    bounded = [t["consent_expiry"] for t in parent_tags if t["consent_expiry"] is not None]
    return {
        "schema_version": "2.0",
        "asset_id": child_asset_id,
        "asset_version": child_asset_version,
        "tag_version": tag_version,
        "authorized_stages": sorted(purposes),
        "authorized_purposes": purposes,
        "classification_tier": max((t["classification_tier"] for t in parent_tags),
                                   key=TIER_ORDER.index),
        "consent_type": consent_types.pop() if len(consent_types) == 1 else "mixed",
        # The earliest bounded parent wins; null only when every parent is unbounded.
        "consent_expiry": min(bounded, key=_expiry) if bounded else None,
        "authorization_withdrawn": any(t["authorization_withdrawn"] for t in parent_tags),
        "jurisdiction": jurisdictions.pop(),
        "data_owner": data_owner,
    }

Operational notes: Record both the upstream tag set and the derived tag decision in the lineage system so reviewers can explain why a downstream asset is or is not authorized for a given stage.

Production implementation

At transformation commit, read authoritative parent tags, apply the signed inheritance/join rule to the complete parent set, bind the derived tag digest to inputs/code/output and reject publication when lineage or a required tag cannot be reconciled.

Independent verification

A separately credentialed verifier independently reads back or replays the full parent-edge population, inheritance recomputation and catalog/output readback with missing parent, conflicting tags, multi-parent join, per-stage purpose intersection, withdrawn parent, mixed expiry, cross-jurisdiction join, drift and bypass fixtures, runs positive, negative, boundary, stale-policy, partial-population and dependency-failure fixtures, and reconciles the complete eligible population without trusting a component-supplied verdict.

AID-H-029.001-G003

Concept:

Legacy datasets often have no purpose metadata. Use automated heuristics to suggest tags, but do not auto-approve high-impact authorization decisions.

Example heuristic suggestion flow

Suggestions use the same per-stage purpose shape as the authoritative schema. The reviewer still supplies the asset and tag versions, consent type and expiry, withdrawal state, jurisdiction and owner, and the completed record must pass validate_tags before it becomes authoritative.

def suggest_tags(asset_profile: dict) -> dict:
    """Return review suggestions only; never write these as authoritative tags."""
    contains_pii = asset_profile.get("contains_pii") is True
    purpose = "hr_support" if asset_profile.get("source_system") == "hr" else "case_handling"
    stages = ["inference_context"] if contains_pii else ["evaluation", "inference_context"]
    return {
        "classification_tier": "restricted" if contains_pii else "internal",
        "authorized_stages": stages,
        "authorized_purposes": {stage: [purpose] for stage in stages},
    }

Operational notes: Keep the human-in-the-loop on final approval. Track classification coverage as an operational metric so you can see what percentage of your AI-reachable data estate has authoritative data-use tags.

Production implementation

At the backfill workflow, scan exact asset bytes/metadata with pinned rules and scanner release, store suggestions separately from authoritative tags, require a qualified human decision bound to the asset digest and deny lifecycle use until approval or explicit quarantine.

Independent verification

A separately credentialed verifier independently reads back or replays legacy inventory, scanner result, reviewer identity/decision and effective tag state with false-positive, changed-asset, stale-review, rejected and unreviewed fixtures, runs positive, negative, boundary, stale-policy, partial-population and dependency-failure fixtures, and reconciles the complete eligible population without trusting a component-supplied verdict.

Open-source tools

• Ajv
• python-jsonschema
• Apache Avro tooling (machine-readable policy-tag validation)
• fastavro (machine-readable policy-tag validation)
• OpenMetadata
• DataHub
• Apache Atlas
• GX Core (Great Expectations)
• dbt Core

Commercial / hosted tools

• Collibra Platform
• Alation Data Intelligence Platform
• OneTrust Data Use Governance
• BigID Data Security Platform
• Immuta Data Security Platform
• Veeam DataAI Command Platform
• Microsoft Purview Data Map (metadata scanning and classification)
• Microsoft Purview Unified Catalog (policy and classification metadata curation)

Threat mappings

MITRE ATLAS

• N/A

MAESTRO

• Regulatory Non-Compliance by AI Security Agents (L6) (missing or incorrect purpose and consent metadata)

OWASP LLM Top 10 2026

• N/A

OWASP ML Top 10 2023

• N/A

OWASP Top 10 for Agentic Applications 2026

• N/A

NIST Adversarial Machine Learning 2025

• N/A

Cisco Integrated AI Security and Safety Framework

• N/A

Google Secure AI Framework 2.0 - Risks

• N/A

Databricks AI Security Framework 3.0

• Raw Data 1.2: Missing data classification
• Raw Data 1.8: Legality of data
• Raw Data 1.6: Insufficient data lineage
• Governance 4.1: Lack of traceability and transparency of model assets