Mindgard Wins Enterprise Security Tech 2024 Cybersecurity Top Innovation Award
Mindgard is proud to announce its recognition as a winner of the Enterprise Security Tech 2024 Cybersecurity Top Innovations Award.
Dr. Peter Garraghan
Our co-founder and CEO, Peter Garraghan, recently joined Paulina Ris Maya on the The Security Strategist Podcast. They discussed discussed the complexity of securing neural networks, hidden vulnerabilities within AI systems, and best AI security practices organizations should adopt.
About Peter Garraghan
Dr. Peter Garraghan is CEO & CTO of Mindgard, Professor in Computer Science at Lancaster University, and fellow of the UK Engineering Physical Sciences and Research Council (EPSRC). He is an internationally recognized expert in AI security, Peter has dedicated years of scientific and engineering expertise to create bleeding-edge technology to understand and overcome growing threats against AI. He has raised over €11.6 million in research funding and published over 70 scientific papers.
About Mindgard
Mindgard is a cybersecurity company specializing in security for AI.
Founded in 2022 at world-renowned Lancaster University and is now based in London, Mindgard empowers enterprise security teams to deploy AI and GenAI securely. Mindgard’s core product – born from ten years of rigorous R&D in AI security – offers an automated platform for continuous security testing and red teaming of AI.
Thank you for taking the time to listen to the podcast episode.
Title: Neural Networks at Risk – AI andCyber Threats
Speakers:
Paulina RiosMaya
PeterGarraghan
START OF AUDIO
Paulina:
Hi everyone.You're listening to The Security Strategist. I'm your host Paulina, and intoday's episode I speak to Peter Garraghan about cybersecurity risks associatedwith AI. We discuss the hidden vulnerabilities within neural networks and theneed for security teams to understand the risks and apply controls. Enjoy thisepisode. Peter, thank you so muchfor being with us today. I'm really excited to have you on the podcast.
Peter:
No, thankyou. Lovely to be here.
Paulina:
Start us offby telling us about yourself, your background, all that good stuff.
Peter:
I am Peter Garraghan.I am the C-E-O, C-T-O and co-founder of Mindgard. I'm also a chair professor incomputer science at Lancaster University in the UK and I specialize in securityfor AI.
Paulina:
Amazing. Obviouslywe have to talk about gen AI, LLMs, everyone's talking about it and it's just ahuge boom that we've seen in the past few years. And of course with increasingadoption of those two things, businesses and organizations face new andevolving cyber threats. And a lot of people, they're not very aware of themyet. They think gen AI or LLMs are just a way to take over the jobs instead ofactually being for malicious actors. How do you think cyber threats, especiallywhen businesses are leveraging gen AI and LLMs, are using this uniquevulnerabilities, or what they should anticipate when it comes to gen AI or theuse of gen AI by malicious actors?
Peter:
Sure. I thinkthere are two main lenses you can look at this problem. The first general viewis, replace the word AI in gen AI with any other tech for the last 30, 40 years.We have similar conversations, when cloud emerged, web services to evenvirtualization as a concept. Through that lens, yes, there are new challenges.There are also old challenges as well. And the question in this case is always,when there are new technology and new usage of software and hardware, what isstuff that we do know about and what is the new part? And that's the thingthat's really fascinating but also quite concerning for organizations. I say tomost listeners that replace the word AI with any other tech. They're familiarwith same problems, same growing pains as well. We get a little bit wise everysingle time, but we are getting there. I think going down from the other lens,which is specifically about AI and generative AI, I spent a lot of my careerlooking at artificial intelligence, including systems, cybersecurity and all inbetween. Fundamentally, AI is software. It's been used for decades. I thinkmost people today, when they use the phrase AI, what they really mean areneural networks, which again are very old concept. But since 2014 they havebecome more mainstream because they’re from the other type of machine learningtechniques and because it's software with data and hardware. There are lots ofanalogies with existing cyber threats with AI. Think of SQL injection andprompt injection at a high level, that kind of a very similar concept. Think ofdatabases leaking data and models leaking training data or fine-tuned data. Orthinking about if an attacker was able to bypass a network traffic detectionsystem and therefore attacker bypassing a model detection system. Sameprinciple. The thing I say to people is AI cyber risks aren't uprisings orrobots or kind of the planes will go up the sky. It is actually as serious ascurrent cyber threats, which again is serious. But the real difference here isthe opaqueness and the unknown entity of AI. And that's because AI technicallygets a lot of humanization of what it actually is. It is still software and matriceswith lots of data. It's very good at mimicking, especially in text processing.They're good at this. But really it's still looking at how do I stop data beingleaked? How do I stop someone compromising my system and how do I do these things?And security people understand this. But the difference here with AI is that itis very opaque and that's not a new concept. All systems to some degree areopaque, but it is designed to be intrinsically random. That's one of the bigdifference, is the neural network itself is meant to generate different type ofconcepts. Things like hallucinations. LLMs generate different outputs, that isan intrinsic feature of how it's meant to work. How security teams andcybersecurity and risks think about a world of uncertainty and apply controls.That is what we've seen as the big problem. But again, taking a step back forsome, you could say, comfort, that the cyber risks do exist, they manifest inways that are very different, but fundamentally they're trying to achieve thesame goals that organizations are kind of used to seeing.
Paulina:
And it's thenightmare of the security person not able to control things. If anything,security people are all about control and they're all about, hey, how can weimplement effective preventive measures that are not going to allow thesethreats to come into my system? And you were talking a little bit about thehidden risks of things, model inversion, adversarial attacks and you kind oftouch upon AI neural networks. And I kind of want to go a little bit more indepth on there. What are some of the hidden vulnerabilities within this neuralnetworks that some softwares, like traditional ones that you've talked about donot have, or how this manifests under different conditions, especially with AI?
Peter:
Sure. I cantalk about this from a technical perspective, and the business perspective whyit's very difficult, but also again quite fascinating. The technical perspective,the perfect analogy is people use code scanners to identify or ascertainwhether the code being executed is bad. If you apply that to a neural network,it's a bunch of numbers and matrices, there's no somatic meat behind it. Applyingthose tools immediately, it's kind of going apples and oranges. It's actuallynot going to work. Fundamentally the other problem is security teams aren'tnecessarily well trained in AI and that's getting better. But applying theirconventional thinking to the space, it's very problematic. A common problemthat we have seen is organizations use AI in order to make decision making. Thatcould range from the AI talks to some data and returns the data and comes backto the user or then make the AI have some control. What that means is it'sallowed to generate queries to a database or it's allowed to access systems oraccess directories across the organization. That's super useful if you can makeit do things it shouldn’t, which is hey, I can make it access things it shouldn'tdo. Or I can give it commands that are hidden and it will execute on my behest.It could say, Hey, why don't you generate me a SQL injection attack and talk tomy database for me and see what comes back. If organizations are not carefulyou can call SQL injection attacks using the AI as actually the vector of thesystem. Again, all these problems are remote execution and using AI to causeexecution and even leaking data. How you leak the data is slightly different,but the same method applies. Again, it's kind of security teams being aware ofthese problems. The two biggest problems is, using the AI to launch otherattacks that are very opaque. The second one is data being leaked out of theorganization unintendedly or on purpose. And that can come from the trainingdata or the data that the model talks to. That's the technical element. Thebusiness risk is because people like to give AI its own special category asopposed to software. If I replace the word AI with app and I said to you, Hey,why don't you purchase an application that you have no controls, no visibilityand no reporting of risk, you'd say “Absolutely not. I'm not going to do this.”AI shouldn't be the exception and that is getting better. But again, there'sstill that thinking of, this thing should be in its own special guardrails, makingit special and in some ways that's good, but people may be overlooking theactual security problems about this. Things like safety and ethics are superimportant that do these things, but the security element of risks, that isactually quite worrying. And the last thing I talk about in the research is… Thebest analogy I can give, think of DNA and think of two twins that grow up indifferent household environments. They're likely to develop differentsusceptibilities and resistance to different factors. Combination ofenvironmental factors, but also just because you have the gene doesn't meanyou're going to activate that type of condition. If I take the same neuralnetwork and put it in slightly different use cases with different data anddifferent usage, it's going to manifest different behaviors both good and bad.In this case it makes certain security attacks and cyber-attacks much moreeffectively against the same network when it's deployed in TensorFlow versusPyTorch, or I give it text data and not image data, that is the unsolvedproblem. You always think of it like a security gene. Again, you don't go tothe doctor, he doesn't say to me, “Let me look at your DNA and tell you what'swrong with you.” It doesn't work like that. You might have a blood sample test,that's important, but they'll ask you questions. They'll look at the context,your medical history, maybe your family's history. And AI security and AI risksare the same. Yes, the neural network powers how it works. It’s important—so isthe data, but it’s these things together. It's the environment but also theactual usage of the AI itself.
Paulina:
Right. Andit's the question of how people now are kind of building this capacity buildingwithin their organizations of teaching people how to use AI or teaching themhow to not use AI. Because I've heard all of this off-the-shelf solutions thatpeople are not even questioning how the systems actually work. They hear theword AI—and perfect example when you say, “App and AI, give it to me. That isthe perfect solution I need for my organization.” But no one asked really, Hey,okay, is this going to actually bring bad stuff to my organization because Idon't know how it works. This is a solution that they're selling in terms ofhow AI and the innovation of AI is going to help your organization be moreefficient. However, you never see of these potential threats that are going tocome to your organization and then you have a bigger problem than having thesolution itself. Obviously I want to shift into… Because we could talk aboutfive hours here Peter about healthcare, finance, critical infrastructure. But Ido want to kind of ask you which industries are particularly at risk whenimplementing these AI models. And if you could give us some specific threatsthat they should be prepared for right now.
Peter:
Like mostthings it's nuanced. I think they're all important. Every sector will claimthat my area is quite important. And that is absolutely true. Same as anysoftware, any application. There's a reason why aeronautical industry has itsown rules and its own procedures or soft developments. AI is not any different.In terms of looking at all the categories, the thing I always keep going backto is the real problem of cyber risks and AI security is trying to understandwhere is the AI most effective. There's lot of discussions about using AI foreverything that's not true. AI specialize in certain things very well. I'm notgoing to go into hours about what those should be. I think many companies havetried the last 12 months trying to understand this AI, and it’s reallyinteresting, what's the great use case that is better than what we currentlyhave, in some degrees and pilots are failing because they're not that effectiveor too expensive. But other cases—pick a random example like translationsoftware or aggregating lots of documents together, it's really good at thatand they'd be great applications built upon that. You have a budget for applications,let's say you find them, the problem now is for my specific business case andbusiness requirements, what are the major cyber risks that are credible butalso have a high degree of harm? That is very hard for even a very skilledperson to identify manually because I have to learn about all these differenttype of risks, if they're relevant to my use case, is my application or AIexternal? Is it internal? Is it in the cloud or not in the cloud? Does it haveconfidential data? Is it not a public data? Is it used more in the version of asoftware? All these things make it very busy. That's kind of the first one Iwould say in terms of if you figure out your use case, figure out the risksrelevant to your use case and then you can actually start looking at AIsecurity. But I'll give you some precise examples. I'll give you two. On thatnote, if I have an LLM that is trying to do like an e-doctor, which meansthey're trying to give the patient information about recommendations for betterhealthcare practices or even patient data. Do I really care if the LLM talksabout political discourse? Probably not. It's not that relevant to me. What isrelevant if it talks about how to make drugs? How do I commit suicide? And howdo I leak patient data? Those things are super relevant to that applicationcase. Hence why you want to make sure that is things to be worried about. Otheruse cases we'll not care about at all. I'll give you two examples of risks thatwe have seen. They're quite problematic. The first one is absolutely, peoplehook up a large language model to confidential data, whether that's fine tuned,whether it's trained or whether it's just a database or a RAG and they don'tactually test to make sure that can someone give it instructions to actuallyget data out of the system back to the user—patient data, execution or justreconnaissance of how the system works. That happens very often. The secondone, which is a more sophisticated. AI security isn't just looking at bad wordsin a prompt. That is kind of the lowest hanging fruit. There are things such asmultimodal attacks that are staged in cybersecurity risks. You may have heardthe story about the deep fake scam that happened maybe late, early this yearthat someone was tricked into thinking this is a real person, yes or no, Ithink it's real enough, I'm going to transfer millions of dollars to them. Thereare cyber-attacks whereby I can take an audio of someone speaking, I can tweakthe audio signals so the human hears one thing, but the AI model hearssomething completely different. That makes it incredibly difficult to trustcommunication. Very difficult to trust negotiations from it. And that's leakingall data. And think about audio, think about images. If I can bypass an imagedetection system, get access, these are problematic. And again, cybersecurityland, these aren't new concepts. The problem now is it's done much quicker. Itis getting more sophisticated in terms of how to trick people in these things,and there's a lack of really understanding of how AI functions and where it issimilar to current cybersecurity and for applications and software and when itisn't.
Paulina:
And I meanyou are completely right. I heard this example about a Hong Kong bank, I think,that they had a big deep fake or fraud and it was used in AI to pretty muchjust have someone talk, “Hey can you give me information about this?” It wasjust a deep fake and they accessed the information that they required and thatit was a huge breach.
Peter:
And I'll makethis an even more difficult situation, is many deepfake detection systems useAI. There are attacks that target to bypass deep fake detection systems. It'skind of a game of cat and mouse. So again, it isn't secure. People talk aboutusing AI for security. Guess what? It's using AI. You can circumnavigate AI.
Paulina:
Exactly. It'sa never ending question of who is using it and how is it using it because atthe end of the day it's always going to be a good or a bad use of it. ObviouslyI want to ask you, what do you think are the best practices that organizationscould follow for security hygiene or what are some of the key factors thatleaders should consider when selecting AI cybersecurity vendor?
Peter:
Sure. Interms of best practices, replace the word AI with any other software project. Thatwould get you quite a lot of the way there. If you are spinning up your ownsoftware practices independent for your current controls, you're making doublework. You don't need to do this. Now, for example, picking the correct model inthe first outset or picking the right application. These models have intrinsicvulnerabilities. They're built this way. Actually from the very outsets, give ita design. Say, okay, I have a use case. I want to qualify different types of AIservices that could include third party applications. It could include opensource models, find the ones and make sure the security properties areunderstood reasonably well, mapping to your use case. And as you go throughthat process of development or piloting, continuously test reports, communicatewith your other teams of risk—your product designers—to make sure actually thatthe risks are understood for the entire process. The worst thing you can possiblydo is wait till the last second and say, great, fantastic application or AI, it'sgoing to fail all my security checks, go back to the drawing board. That's notgood. And again, people who build software projects or manage them understandthis in terms of the more shift left you do to make sure you understand therisks and go through the better you can do so. Again, there's just apply yourcurrent controls, don't get mystified or stunned they used the word AI. Identifywhere the weaknesses actually are. Because there are quite a few cases where thesepractices do not work and that comes in terms of the tooling you have in placearen't going to function correctly. For example, using code scanning semantic,meaning it's not going to work from the AI system, it something different. Andthat can range from continuous security testing, cleaning products, codesetting products to be designed for AI. All the way through to, I need to makesure I know my inventory, it is understood what the risks are involved, whetherI'm downloading AI or not. All the way through to I've got this AI now live, makesure that I can actually inspect and detect risks coming to the AI model eventhough that is not a solved research problem. Anyone saying otherwise may belevel with the truth. But the point is that treating it like any other softwareproduct will be super helpful. And for the business owners, AI has some reallygood use cases. Some of them are much more concrete than others, I would say,as a professor in this space. And they've been tried and tested quite a while.The problems always have been scaling it up, cost effectiveness and loss of ROIin return. Onceyou have the use case identified, it is very important to then say what arethe credible risks and what is the potential harm generalization and what's thefeasibility of doing so, then it becomes like any type of asset you have undermanagement to make sure we understand the risks. That is with the backing of alot of governments and companies, are now starting to have governanceframeworks that even have legal obligations in terms of making sure youunderstand AI risk. Lots of talk about what to do, not all of how to do it.Therefore, the best thing you can do is apply your code techniques, identifythe weaknesses, train your staff to understand what is AI, how's it differentfrom current software and where is it the same? Have the tooling in place, butmore importantly make sure that you have your use cases understood, qualified,and you have security embedded in the entire process.
Paulina:
I mean, Peter,after that I'm going to put it in my own organization. I'm just going to followyour best practices and just be sure that I'm going to have an effectivesecurity strategy after that. So thank you so much Peter for being with ustoday. I really appreciate your insights and I'm sure our listeners took a lotaway from today's podcast.
Peter:
Great. Thanksvery much. Take care now.
END OF AUDIO
