Enterprises are under rising pressure to put advanced AI solutions to work in order to stay competitive. But this pressure to adopt quickly comes with risks.

A new report by the Coalition for Secure AI (CoSAI) highlights the key AI security risks that enterprises need to be aware of and how they should think about addressing them. The report is among the most comprehensive studies on the state of AI supply-chain security risks and the strategies that enterprises can deploy to address them.

Among its key findings:

Organizations need to treat the entire AI supply chain as a dynamic, evolving attack surface that requires continuous innovation to defend
AI systems require specialized risk assessments beyond conventional security frameworks
Defense strategies should include end-to-end risk assessments, early engagement protocols to mitigate risks during development, and efforts to ensure data provenance and transparency
“Model signing” – where a model is cryptographically authenticated by the provider to verify its integrity and provenance – should be a priority to ensure models haven’t been tampered with
Security practitioners, AI developers, and executives should work together to integrate AI-specific and traditional controls

As enterprises struggle to get to grips with this new security paradigm, the CoSAI report provides a toolkit of strategies, empowering organizations to innovate with confidence and navigate the evolving landscape with resilience.

Addressing vital AI security issues

AI systems pose complex security challenges that are very different from those associated with traditional software. Spread across a complex landscape of data, models, applications, and infrastructure, the AI supply chain is vulnerable to a wide range of threats, including from malicious actors. This can expose organizations to serious repercussions from wary consumers and increasingly watchful regulators.

The following are four security issues highlighted in the report that are vital for enterprise leaders to understand.

Preventing tainted data

Data is the foundation of AI, providing the raw material for both training and using models.

The vulnerability

Bad actors can weaponize the public internet by introducing malicious or incorrect data, knowing that AI crawlers may eventually collect it for training.

Post-collection data is at risk too. Unlike conventional software where sensitive information is usually stored in specific locations that are easily tracked, information within an AI system may be distributed across millions or billions of weighted connections, creating a large and relatively unmapped surface of attack.

The solutions

Organizations need to monitor and test data sourced from the public internet. It can be a daunting task considering the amount of data used to train AI models, but achievable with the help of data validation pipelines and anomaly detection systems to screen the incoming information.

Organizations can then ensure this vetted data is kept secure by keeping meticulous records of where and how the data is stored and sourced. Cryptographic tools like content hashes and checksums can help verify data integrity, while URL logging can monitor web addresses accessed by the system to detect potential threats.

Securing model weights against tampering

Hackers may attempt to alter the numerical values that are used by AI models to provide responses, which can then result in harmful or incorrect outputs.

The vulnerability

Attackers may hack into a server or steal credentials to tamper with model weights, which can cause an AI model to misclassify data or generate harmful content. The effects aren’t necessarily immediate either. A hack could introduce hidden triggers that allow the model to act normally until a specific input causes it to fail or leak data.

The solution

Strong access controls and multi-factor authentication can limit access to the servers and systems where models are stored. Model signing allows you to verify the authenticity of a model artifact to be sure it hasn’t been tampered with.

Organizations should also establish a baseline for model behavior, then continuously monitor and audit the model to look for anomalies. Sigstore, for example, can be used to verify model artifacts (all the files generated in the training and packaging of a model) and help detect any subsequent tampering.

Identifying adversarial prompts

Prompts are used to instruct an AI model to find information and provide answers. But a sophisticated attacker may use an adversarial prompt to trick an AI model into providing sensitive, incorrect or harmful information.

The vulnerability

Adversarial prompts can be contained in reference materials, or introduced through the same interfaces employed by approved users, including chatbots, text, images and scripts that target APIs. By exploiting weaknesses such as flaws in the training data or inadequate content moderation, they can override a model’s guardrails and insert malicious instructions.

The solution

Models should be monitored for red flags such as repeated keywords, encoded characters, or suspicious phrasing, as well as responses that deviate from expected behavior, such as contradictions and a sudden switch to unsafe content.

Pattern matching and heuristic analysis can be applied to data sourced by retrieval-augmented generation (RAG) systems, ensuring that the retrieved context is free of adversarial prompts. Regular penetration testing, ethical red-teaming, and other adversarial testing can be used to assess a model’s performance during different attack scenarios.

Supply chain attacks

Malicious actors regularly try to compromise open-source libraries, a crucial foundation for AI model development. Their techniques include the insertion of backdoors (hidden pieces of code), taking over abandoned projects, and typo-squatting (where attackers create packages with names similar to popular libraries, relying on users mistaking the name.)

The vulnerability

Companies often use commonly available open-source resources to train their models. But malicious actors can compromise a dependency within an open-source framework, for instance by injecting code that could compromise sensitive data or, even worse, give them full remote access.

The solution

To protect themselves from this type of risk, organizations should adopt a comprehensive software supply chain security strategy.

This includes:

Dependency scanning: Regularly scanning all dependencies (an external software component or library) for known vulnerabilities using specialized tools.
Secure development practices: Encourage developers to follow secure coding practices and have a process of adopting, reviewing, and integrating third-party libraries securely.

Building a security-conscious AI culture

To leverage the power of AI effectively yet safely, business leaders need to create robust rules and a security-conscious culture. This includes clear governance around the use of AI, strict data integrity, and continuous risk mitigation to ensure compliance and trust.

Security professionals have a vital role in developing comprehensive AI-specific programs that oversee data, models, and the people who use them. For the researchers and engineers who work directly with AI, it’s important to integrate security at every step of development while continuously validating data and collaborating with their security colleagues to detect any vulnerabilities before they can be exploited.

Learn more about our approach to security.

Trust Center

Matt Maloney