Every breakthrough technology brings new possibilities, but it’s also critical to understand the broader implications, particularly with respect to security and risk. That’s where Ken Huang, Chief AI Officer at DistributedApps.AI, excels. For nearly a quarter century, Ken has been on the front lines of emerging innovations — from blockchain to web3 and machine learning to generative AI. For each one, he’s helped business leaders understand the potential and perils of applications using these advanced technologies. Today’s proliferation of large language models (LLMs) has given him yet another landscape to deploy his expertise.

Ken came by his security knowledge on the job. After starting his career by designing systems for financial services institutions, he dove headlong into “the second wave of AI,” immersing himself in machine learning. For the past two decades, Ken has served as an application security engineer at companies like CGI and Huawei. These experiences have made Ken a leading thinker on all aspects of tech security.

Ken sat down with us to share his thoughts on how generative AI has expanded the attack surface for security; why adopting a systematic, principle-based approach is vital; and the need for CEOs to take responsibility for security

An edited version of our conversation follows:

You’ve promoted what you call a systematic, principle-based approach to securing LLMs. Tell us about that.

For legacy system security, you have tools. For AI, the tools are not yet mature, and innovations are always happening. That’s why I say that you should be principle-based. Fundamental principles of security do not change; they’re still relevant and will be relevant even in a dozen years.

The systematic approach is about viewing generative AI as a systematic architecture. With LLMs, the attack surface of the tech interface has increased, especially in retrieval-augmented generation (RAG). You have to secure the data, the model, and then the application. A more systematic approach involves first surveying the security landscape around generative AI. The next step focuses on how we can secure generative AI through data security, model security, and then application security.

How can security keep pace with all the advances?

I find that I cannot keep up with the vast number of innovations in technology, new models, new jargon, and now even many types of RAG, like so-called advanced RAG. You have new things like the LlamaIndex and Langchain. There’s always something new coming up. That's why security should focus on fundamental principles.

Can you talk a bit more about security concerns around RAG implementation?

With RAG, you have different touchpoints that create challenges. One is certainly the vector database, for which encryption is harder. And a vector database is based on similarity search, so if you need to do decryption, you have to decrypt the whole database. There's no column-based encryption commonly used in RDBMS. There may be some solutions in Pinecone, like using the namespace or a prefix in the index, but they aren’t mature yet. Developers still need to meet HIPAA, HITRUST, or other compliance requirements that require encryption on rest. PCI-DSS also requires encryption. And it’s not just whole disk encryption; it's on the embedding database, too. A research paper found that we can reverse an embedding to its original text. So that's why you need to encrypt the embedding, the index, and the metadata.

In addition to the vector database, RAG can call APIs, such as the Google Search API. And there are a lot of new APIs coming out, some good and some maybe not, so API security is also important.

Then you need to address access controls. There are no native access controls for vector databases. When developers build LLMs, they often pull all this data from HR, finance, and other functions. They put it into the RAG system which creates a problem: they violate access control, which is a critical principle to enforce.

Plus, you always need to validate the output from LLMs.

How do you change the enterprise’s mindset on LLM security?

Generative AI has altered the security landscape. You may have zero-day vulnerability and zero-day attacks because some vulnerabilities are much easier for hackers to leverage. Shadow AI amplifies this kind of problem through observability issues.

Enterprises should avoid paying attention to people talking about generative AI as the end of humanity. This is a prediction for the future. Nowadays, the greater risks are from threats like data poisoning and prompt injection.

For enterprise users who want to use RAG for LLM applications, the first approach is really to shift left on security. If you consider security at a later stage, it's always an issue.

This requires executive support. Especially with generative AI, security is not just the CISO’s responsibility. They follow the CEO’s lead, so that means that the CEO should make security a top priority. AI is important, but without security, AI will not be as successful.

Is your organization building AI solutions? If not, now is the time to get started.

Get Started!

About Ken Huang

Ken Huang is the author of eight books on AI and Web3. He is the Co-Chair of the AI Organizational Responsibility Working Group and AI Control Framework at the Cloud Security Alliance. He is a core contributor to OWASP's Top 10 Risks for LLM Applications and is heavily involved in the NIST Generative AI Public Working Group. He also provides feedback on publications like NIST SP 800-226.

A sought-after speaker, Ken has shared his insights at Davos WEF, ACM, IEEE, and CSA AI Summit, CSA AI Think Tank Day, and World Bank. His recent co-authorship of Blockchain and Web3: Building the Cryptocurrency, Privacy, and Security Foundations of the Metaverse is recognized as a must-read by TechTarget.

How generative AI has changed security