I’ve been asked a lot recently about the misuse and weaponization of widely available GenAI tools.
There exist two sides to how threat actors are using GenAI. The first is the use of GenAI to augment their current capabilities, such as generating high quality and targeted phishing attacks via scraping publicly available data, combined with learning what is or isn’t successful. The second is how the AI itself is susceptible to threats, whereby techniques such as jailbreaks or prompt injection can elicit responses that incur business, safety, or security risks such as data leakage or harmful outputs.
The first problem is an old-age issue within organizations, which requires a blend of defence in depth tools to detect and block attacks (which itself is increasingly using AI) combined with clear education and training to spot GenAI-enabled cyber attacks. The second problem are security risks against AI tools themselves, companies are tasked to perform AI red teaming to identify and surface issues to remediate, as well as various input/output filtering techniques to block prompt injection techniques, as well as tracing GenAI activity to detect anomalous activity. Both of these problems are evergreen.
GenAI is just the latest in a long line of technologies used by threat actors. These are old and established problems that security leaders are aware of, with the key difference being that it is now cheaper, quicker, and more targeted to misuse GenAI tools for nefarious purposes at scale. Enterprise leaders must recognize that AI is a force multiplier for both defenders and attackers. While GenAI tools primarily speed up reconnaissance, scripting, and social engineering, they also lower the barrier for less-skilled attackers and increase the scale of cyber threats. To stay ahead, enterprises must adopt AI-driven security measures, integrate AI threat intelligence, and use red teaming and blue teaming to test and strengthen defenses against GenAI-powered attacks.
I’m also seeing that GenAI companies are taking their software misuse extremely seriously, given the negative impact to society and ultimately their product brand. Companies are taking a multi-faceted approach to combat this issue. This includes investing and hiring in talent dedicated to addressing security and safety risks within AI tools, who perform activities such as model evaluation and AI red teaming to understand and detect the potential for AI models to generate harmful or high risk outputs. The problem is being studied even before the AI model is released via sanitizing the data used for training and/or fine-tuning.
GenAI is reshaping the cybersecurity landscape, amplifying both offensive and defensive capabilities. While attackers exploit AI for more sophisticated and scalable threats, defenders must evolve their strategies with AI-driven security measures. Organizations need a proactive approach, combining AI threat intelligence, red teaming, and robust detection techniques to stay ahead. GenAI companies are also taking significant steps to prevent misuse, investing in security research and model safeguards. As AI continues to evolve, the key to resilience lies in innovation, vigilance and a commitment to securing AI-powered systems.
At Mindgard, we specialize in helping organizations secure their AI systems through advanced red teaming practices. Our Dynamic Application Security Testing for AI (DAST-AI) solution protects AI systems from new threats that can only be detected in an instantiated model and that traditional application security tools cannot address.
Book a meeting with me to learn how Mindgard can help safeguard your AI systems.
