What is AI supply chain security?

AI supply chain security is the protection of every third-party component an AI system relies on, including foundation models, fine-tunes, datasets, ML libraries, MLOps tooling and external APIs, against tampering, poisoning and compromise. It extends software supply chain security to AI-specific artifacts such as model weights and training data.

What is an AI-BOM (AI bill of materials)?

An AI-BOM is a machine-readable inventory of the models, datasets, frameworks and dependencies that make up an AI system, along with their versions, licences and provenance. Standards such as OWASP CycloneDX 1.6 (ECMA-424) and SPDX 3.0 provide formats for producing one so teams can track and verify what they depend on.

Why is the pickle format a security risk for AI models?

Python's pickle module can execute arbitrary code during deserialization, so simply loading a pickle-based model file can run an attacker's payload. JFrog found around 100 malicious models on Hugging Face in early 2024 that abused pickle to open backdoors on load.

Is safetensors safer than pickle?

Yes. The safetensors format stores only numerical tensor data and cannot execute code when a model is loaded, which removes the arbitrary-code-execution risk that pickle carries. Preferring safetensors for model weights is a core AI supply chain control.

What is OWASP LLM03:2025 Supply Chain?

LLM03:2025 Supply Chain is the OWASP Top 10 for LLM Applications entry covering vulnerabilities introduced by third-party models, datasets, packages and plugins. It highlights that a single compromised dependency can execute with the full privileges of the AI application, giving it a large blast radius.

How do I verify the provenance of an AI model?

Use build-attestation frameworks such as SLSA and Sigstore to verify who produced a model and how, record model cards and datasheets for origin and intended use, and scan the file with a tool such as ModelScan before loading. Provenance standards like C2PA further help verify the origin of training data and outputs.

AI Supply Chain Security: Provenance, AI-BOM and Risks

AI supply chain security is the practice of protecting every third-party component that an AI system depends on, from foundation models and fine-tunes to training datasets, machine-learning libraries, MLOps tooling and external inference APIs, against tampering, poisoning and compromise. It extends traditional software supply chain security to the artifacts that are unique to AI: model weights, checkpoints and the data used to train them.

What is the AI supply chain

A modern AI application is assembled far more than it is written. Teams pull a base model from a public hub, layer a fine-tune or adapter on top, wire in vector databases and orchestration frameworks, and call third-party inference or embedding APIs at runtime. Each of those pieces arrives with its own provenance, its own maintainers and its own trust assumptions.

This is exactly why OWASP promoted supply chain to its own entry, LLM03:2025 Supply Chain, in the 2025 Top 10 for LLM Applications. The blast radius of one poisoned dependency is larger in an AI system than in a typical web app, because model-loading code and agent tooling routinely run with access to credentials, data stores and, increasingly, tool-calling privileges. We cover the full list in our guide to the OWASP Top 10 for LLM applications.

Foundation models and their fine-tunes, adapters and quantised variants
Training, fine-tuning and evaluation datasets, including scraped or synthetic data
ML libraries and frameworks such as PyTorch, TensorFlow and transformers
MLOps and serving tooling: model registries, pipelines, proxies and gateways
Third-party inference, embedding and vector-store APIs called at runtime
Serialization formats and the model files themselves, including pickle, safetensors and GGUF

Two things make this harder than a conventional dependency tree. First, model weights are opaque binary artifacts: unlike source code, we cannot read them line by line to see what they do, so we depend heavily on their provenance and on behavioural testing. Second, the chain reaches back into data we never see, because a base model has already absorbed whatever was in its training corpus long before we download it. Trust, in other words, is inherited from parties several steps removed from us.

The AI supply chain attack surface

Once we see the supply chain as a graph of untrusted artifacts, the risk classes become clear. They map to the questions any reviewer should ask before a model or package is allowed into production.

Poisoned or backdoored models that behave normally until a trigger input activates hidden behaviour
Unsafe serialization, where merely loading a model file executes attacker-controlled code
Dataset poisoning that corrupts model behaviour at training or fine-tuning time
Dependency compromise through typosquatted, abandoned or hijacked ML packages
License and provenance gaps, where a model or dataset carries unknown or non-compliant terms
Compromised MLOps tooling and gateways sitting between the model and its callers

Real risks and incidents

These are not hypothetical. The most instructive attacks of the past two years targeted precisely these weak points, and each one reached real users of trusted, popular components.

Pickle deserialization on Hugging Face: JFrog identified roughly 100 malicious models on the Hub in early 2024 that abused the pickle module's __reduce__ method to run code the moment a model was loaded (JFrog, 2024).
The nullifAI campaign: in February 2025 ReversingLabs found models that evaded Hugging Face's Picklescan by using 7z compression and a deliberately broken pickle stream, so the reverse-shell payload executed before the scanner reached the corrupt section (ReversingLabs, 2025).
The Ultralytics compromise: on 4 December 2024 attackers published trojanised versions (8.3.41, 8.3.42, 8.3.45 and 8.3.46) of the popular computer-vision library to PyPI, shipping the XMRig cryptominer. The initial breach came through a poisoned GitHub Actions cache and a stolen PyPI publishing token, not a flaw in PyPI itself (PyPI Blog, 2024).

The pattern across all three is that the malicious component looked entirely legitimate. It came from the right hub, under the right name, and in the Ultralytics case even carried cryptographic provenance. Attackers are no longer only exploiting the models themselves; they are exploiting the trust we place in the pipelines and registries that deliver them. That is the shift AI supply chain security has to answer.

The Ultralytics attack is a warning about build-pipeline trust: because the project used Trusted Publishing, the malicious releases carried valid, signed provenance even though the code had been tampered with during the build (PyPI Blog, 2024). Signing proves who published an artifact, not that its contents are safe.

How to secure it: AI-BOM and model provenance

Securing the AI supply chain is a matter of knowing what you depend on, proving where it came from, and refusing to load anything unverified. The following controls, applied in order, close most of the gap. They complement the runtime controls in our note on securing GenAI with defense in depth.

Build an AI-BOM. Inventory every model, dataset, framework and API in a machine-readable bill of materials. OWASP CycloneDX added AI/ML-BOM support in version 1.6, published as ECMA-424, and SPDX 3.0 offers an AI profile, so you can capture base-model lineage, training data and licences in a standard format.
Prefer safe formats. Adopt safetensors for weights wherever possible; it stores only tensor data and cannot execute code on load, unlike pickle-based formats.
Scan models before load. Run tools such as ModelScan over PyTorch, safetensors and GGUF files to detect embedded code and unsafe operators, and gate ingestion on a clean result.
Verify provenance and signing. Use SLSA and Sigstore to attest and verify the build lineage of models and packages, and record model cards and datasheets so every artifact has documented origin and intended use.
Pin and curate dependencies. Maintain an internal, vetted model and package registry, pin versions with hashes, and block direct pulls of unreviewed artifacts from public hubs.
Monitor continuously. Track new CVEs, revoked models, license changes and model drift against your AI-BOM so the inventory stays live rather than a one-time snapshot.

Provenance standards are maturing quickly: the C2PA specification reached version 2.3 in December 2025, extending tamper-evident content credentials to live video and unstructured text, which matters for verifying the origin of both training data and model outputs (C2PA, 2025).

Governance and vendor due diligence

Tooling alone will not hold. AI supply chain security is also a procurement and governance problem, and it belongs in the same due-diligence process you already apply to other suppliers. Our guide on supplier security beyond questionnaires argues for evidence over attestations, and model vendors are no exception: ask for model cards, training-data provenance, evaluation results and a signed bill of materials, not just a marketing claim of safety.

Require an AI-BOM and model card from model and dataset providers as a contractual deliverable
Assess the security of the vendor's own build and release pipeline, given how the Ultralytics breach originated there
Define acceptable licences and provenance up front, and reject artifacts that cannot demonstrate either
Assign clear ownership for monitoring third-party model and dependency risk after onboarding

Governance also means deciding, in advance, what a failure looks like and who acts on it. If a model in your registry is later found to be backdoored, or a library version is pulled from PyPI mid-week as Ultralytics was, you need a defined path to identify affected systems from the AI-BOM, quarantine the artifact, and roll back to a known-good version. An inventory that no one owns and no one queries during an incident is documentation, not a control.

The through-line is simple. Treat every model, dataset and dependency as untrusted until proven otherwise, make that proof machine-readable through an AI-BOM, and verify provenance before anything loads. Do that consistently and the AI supply chain becomes something we can reason about, rather than a chain of blind trust running with production privileges.