As organizations race to adopt AI, they’re opening up novel vulnerabilities and an expanded attack surface — particularly as more agentic AI systems are deployed.

In one recent example, a cyber security company said it had discovered a critical vulnerability in Microsoft’s 365 Copilot AI assistant. The flaw, dubbed “EchoLeak,” enabled attackers to obtain sensitive internal data without any action by the user — a so-called “zero-click” attack.

AI agents are a growing source of anxiety for CTOs and CISOs. To do their work, agents need access to a broad range of internal data and tools. If they have access to private and sensitive data, exposure to untrusted content (such as that sent by an attacker), and the ability to communicate externally, it’s a recipe for potential breaches. Even with humans in the loop, their wide reach and autonomy could create serious concerns for security.

While the enormous potential of agentic AI is exciting, it’s important to highlight an uncompromising caveat: we can’t move forward unless these systems are built on a foundation of trust. – Aaron Shilts, CEO, NetSPI

To earn trust with AI, organizations must tackle security challenges, from data governance and model behavior to controlling how users and agents access systems and tools.

Top AI security questions to answer

Here are three critical questions CTOs and CISOs must ask to stay in control of AI.

1. How will you protect data privacy?

AI systems process vast amounts of sensitive data, from customer information to internal financial figures. Without strong privacy and governance policies, that data can be exposed by attackers, or even by well-intentioned employees.

Some 38% of employees report sharing sensitive company data with publicly available AI tools — a risky practice known as “shadow AI.” In addition, prompts can inadvertently leak sensitive information, and models can be deliberately coached into revealing private and proprietary data. AI's ability to infer personal information from seemingly anonymous data sets creates new vulnerabilities.

Our research shows that employees using AI applications without proper guardrails are unwittingly exposing sensitive company data at an alarming rate. – Ben van Enckevort, Co-Founder and CTO, Metomic

To mitigate data-leak risks with confidence, it’s crucial to embed privacy in AI solutions throughout their life cycle. That means integrating privacy-by-design principles at every stage, from model training and inference to deployment and decommissioning.

Running models privately, and particularly on companies’ own premises, is an effective way to insulate data, and it's growing in popularity: 21% of workloads and data are being moved back to private environments. This trend highlights a shift towards the security and control offered by private infrastructure.

2. How will you prevent misuse and adversarial attacks?

AI systems can be manipulated in ways that are hard to detect using traditional cybersecurity methods.

The No. 1 LLM security threat for 2025 is considered to be prompt injection attacks. A big challenge to AI security, harmful commands can be embedded in seemingly innocent inputs, or indirectly in external resources that are pulled into the model.

Security researchers have demonstrated how prompt injections could be used to compromise data (as with EchoLeak), steal or manipulate code, or take control of user accounts.

The fundamental risk … is that whoever provides input to the LLM has a high degree of influence over the output. – Rich Harang, AI Systems Security Architect, Nvidia

At Cohere, we approach these threats by combining red-teaming, vulnerability scanning, and adversarial testing to stress-test our models against a range of scenarios. For more, check out the Cohere Secure AI Model Framework or download our guide on how to securely deploy LLMs in private environments.

3. How can you ensure the integrity of AI supply chains?

AI models aren’t built in isolation. They rely on complex supply chains, including external data sources, open-source libraries, and third-party APIs, any of which can introduce security risks.

Bad actors can insert malicious elements into a model’s training data through "backdoors," or tamper with the weights that models use to provide responses. Even widely trusted repositories have been shown to contain backdoors that could be activated post-deployment to compromise an LLM and steal sensitive data.

AI supply chains are a growing target, with attackers manipulating data, training models, and software libraries. – Michael Adjei, Director of Systems Engineering, Illumio

We implement various security measures to protect the supply chain, including verifying the integrity of external components, ensuring secure data transmission, and maintaining a trusted network of partners and suppliers.

For AI to be truly enterprise-ready, it must be secure. We aim to help our customers move fast and innovate with confidence.

Mastering private AI deployments

Is your business ready for a secure, private AI deployment?

Download the eBook

Bringing secure AI to critical systems