Generative AI’s rapid evolution and boundless possibilities have been accompanied by a whole host of new security challenges. Tackling them will require a different approach to managing security and risk — one focuses on collaboration and knowledge sharing. Steve Wilson is ideally suited to help lead the charge. Currently the chief product officer for Exabeam, a security solutions provider, Steve has earned his bona fides over a career spanning more than three decades, with stints at Sun Microsystems, Oracle, and Citrix, among others.

Steve is at the forefront of efforts to understand the security risks associated with large language models (LLMs). He is instrumental in the development of the Open Worldwide Application Security Project’s (OWASP) Top 10 for Large Language Model Applications and the LLM AI Cybersecurity and Governance Checklist, resources that delve into common security risks of LLMs and recommendations to mitigate them. In his role as a project leader for the OWASP Foundation, he draws on the collective knowledge of hundreds of developers from around the world.

The response to date has been overwhelming: these have become the go-to resources for developers, CISOs, compliance officers, and anyone seeking to get their arms around the security issues associated with generative AI. Steve’s position at the epicenter of LLM security gives him a valuable perspective on emerging themes around security. He also weighs in on opportunities to shape the dialogue and increase understanding of LLM applications.

Hear Steve share key insights about LLM security in our interview.

0:00

/4:48

Steve Wilson on LLM Security

An edited version of our conversation follows:

How was the OWASP LLM Security Project started?

Last year, I started to see some patterns in LLMs that were unique security challenges. I proposed a project to the OWASP board and then put an announcement on my personal LinkedIn profile. I thought, “Maybe I'll find a dozen like-minded people in the world who are interested in this obscure corner of cybersecurity and AI.” It kind of blew up: two weeks later, we had 200 people in an expert group, and it has since expanded to 1,200 people at this point. We have no idea how many people have read the document, but it's probably north of 100,000.

It's been amazing to see vendors, governments, and standards bodies all adopting this document as a foundational guide for that next step in security.

What are the top security topics that product owners and managers need to be actively engaged in right now?

Product managers, product owners, and executives need to be thinking about the big picture, not getting mired in the details. Obviously, a development team needs to understand the ins and outs of prompt injection and output filtering.

They should seek to understand some of the risks we call out in the list, such as excessive agency and overreliance, where product requirements can really affect security. They need to think about how to design their organization, and even train their employees to use these new technologies in ways that are safe and responsible.

From the customer perspective, we get a set of questions: What are you doing with my data? What are you feeding it into? Are you training on it? What are the risks? Is my data going to wind up in a place where it's unsafe or fed back to other customers? These questions affect how product managers build their requirements and approach to LLM architecture. There are easy, clean answers, or it can be really complicated and messy.

Let’s talk about some of the specific LLM risks. What are some of the top risks?

Prompt injection is a significant risk where attackers manipulate the input prompts to a large language model (LLM) to elicit unintended or harmful outputs. By crafting specific input sequences, adversaries can exploit the model's interpretative capabilities, leading it to generate biased, misleading, or even malicious responses. This vulnerability is particularly concerning because it leverages the model's inherent complexity and the unpredictable nature of its responses to various prompts, making it a potent tool for disrupting or corrupting the outputs of LLM-based applications.

Data poisoning involves the deliberate insertion of malicious data into the training dataset of a large language model. Attackers aim to subtly influence the model's behavior by embedding harmful data, which can lead to inaccurate, biased, or compromised outputs. Given the vast and diverse datasets required for training LLMs, even a small amount of poisoned data can have significant impacts. This type of attack exploits the model's sensitivity to its training inputs, posing a substantial risk to the integrity and reliability of LLM-generated content.

What can developers do to protect sensitive information and mitigate the security risks inherent in LLMs?

Some of the issues early on involved people providing private data to public LLMs for training. As application developers, we've become very conscious that we need to be careful about what we teach the LLMs. We want to have the smartest possible AI, but anything introduced to LLMs is likely to be part of an output at some point. Developers need to be really thoughtful about what these systems truly need to know.

We can use traditional data loss prevention techniques, filtering the output and looking for Social Security numbers, phone numbers, addresses, and other information that shouldn’t be shared. More importantly, developers should look at the information they are adding to databases in the first place.

Anything that you use to train your model might be generated back to you. So the onus is on the developer to understand what's in those training data sets and make sure they’ve been scrubbed to the best degree possible.

Can you give us some methods used to thwart prompt injection attacks?

Prompt injection is going to be with us for a very long time. One reason is that it’s more like phishing than SQL injection. There are recognized, well-documented ways to stop it 100 percent of the time. However, there aren’t foolproof ways to solve fishing. It’s a constant war against hackers with increasing stakes.

People are now putting out large open-source projects, and a growing industry of commercial companies are building solutions focused on some of these problems, like screening the LLM’s input and output. These are starting to go by names like Guardrails frameworks or AI firewalls. Companies developing software that leverages LLMs should examine all available options and not assume that they can do it all themselves.

How do you view the risks of hallucinations?

Hallucinations are a fun topic in LLMs. They've gotten such bad press, and people worry about them a lot. They can be a major risk, or actually kind of wonderful. Hallucinations are at the heart of what people find really fascinating about generative AI. But the same underlying principle enabling an LLM to generate something that feels new and fresh also could produce an output that might be completely fabricated. You've got to take the good with the bad.

As developers, we want to make sure we're in control of using that characteristic for creativity and not letting it come out in places that can impact our business. A common example is where lawyers grabbed some information containing hallucinations from ChatGPT and submitted it to the court without verifying it. They got into a lot of trouble. Another one is a major airline that built a bot that gave out bad information to customers. The airline was held responsible for that. We want to make sure that the output of our bots is as accurate and verified as we can make it.

What’s next for OWASP?

There is so much interest in LLMs, generative AI, security, and safety, and a lot of standards bodies, government agencies, and corporations are engaging in this topic. Since OWASP is an open-source group, we were able to move quickly and build what became a foundational document.

In the past few months, we’ve spent a lot of time working with groups like NIST, MITRE, and some of the European regulatory agencies to try and promote adoption of the research and frameworks in a commercial and government context.

When we look at some of the coming legal and regulatory implications of LLMs, we’re getting into new territory. As an industry, we don't yet have good answers and responses. That has to be one of our priorities as an industry for this year: offering better guidance to developers on how to deal with regulations.

Is your organization building AI solutions? If not, now is the time to get started.

Get Started!

About Steve Wilson

Steve Wilson is Chief Product Officer at Exabeam. Wilson leads product strategy, product management, product marketing, and research at Exabeam. He is a leader and innovator in AI, cybersecurity, and cloud computing, with over 20 years of experience leading high-performance teams to build mission-critical enterprise software and high-leverage platforms.

He is the author of The Developer’s Playbook for Large Language Model Security.

Straight Talk on AI Security with Exabeam’s Steve Wilson