Generative AI and Offensive Security

The Artificial Intelligence industry has witnessed an exponential growth over the past decade, with its global market revenue soaring from around €5 billion in 2015 to an estimated €107 billion in 2024. This dramatic increase, driven by continuous advancements in this ground-breaking technology, demonstrates that many sectors are adopting some sort of AI-driven product to boost productivity and increase revenue.

However, as with any powerful tool threat actors began weaponizing it, especially generative AI, making them able to carry out highly advanced attacks more efficiently, from targeted phishing campaigns to malware development.

In this post, we're going to dive into a few real-world examples of how threat actors have successfully used generative AI to carry out such attacks.

What is Generative AI?

AI is a term that covers a huge variety of technologies, but we'll be focussing only a specific type: Generative.

Generative AI is able to create new content starting from an input, called a prompt, which is usually in the form of natural language. These AIs can be trained to generate text (think of Large Language Models like ChatGPT), images, videos, code and even audio.

All these things can come very handy to a cybercriminal, especially the ability to generate code and convincing human-like text, but not only that as we'll see in the next section.

Phishing

As most attacks always do, let's start with phishing.

The ability to use LLMs to automatically generate text that sound human in a convincing manner has been a powerful addition to the cybercriminal’s arsenal, as it makes phishing content quick to create, more believable and easier to automate. Studies demonstrate that the entire phishing process can be entirely automated using LLMs, delivering better quality phishing content at 95% of the cost compared to manual methods.

For a long time, one of the main tells that would give away a phishing attempt was the use of poor grammar and badly shaped phrases, which we'll see less and less with the increasing availability of LLMs capable of generating convincing text, like ChatGPT. This poses a huge problem, especially because these tools can be combined with techniques like OSINT, which make creating a targeted and automated phishing campaign almost trivial, by manually or automatically collecting as much public data on the target, feeding it to a LLM and generating content crafted specifically for that person or organization.

Deepfakes

The threat doesn't stop at emails, as we've seen successful phishing attempts using advanced techniques such as deepfakes to resemble the voice and image of another person.

A notable case of this advanced technique was seen in February 2024, where a Hong Kong firm fell victim to a digital recreation of their chief financial officer in an online video call, along other employees, who instructed a real employee to make 15 money transfers, totalling HK$200 million (around €25 million), to five different bank accounts. You can read more here Deepfake scammer walks off with $25 million in first-of-its-kind AI heist - Ars Technica.

This attack was the first of this kind that managed such a high payout, which I believe will be a sufficient motivation to inspire a rise in similar attack in the coming years.

Moving on from phishing, we're going to see how LLMs can do more than just writing natural language and focus on its code-writing capabilities.

Malware Development

Just a year after ChatGPT was made public, security professionals noted how AI-written malware use became widespread, with LLMs being capable of being completely integrated in the infection pipeline: from phishing, as we've just seen, to writing the payload that will be executed on the victim's machine.

In recent years products like GitHub Copilot and OpenAI Codex have revolutionised how software is being developed, allowing developers to write code faster boosting productivity. Of course, the same applies to malware development, with generative tools being used to simplify and enhance the process of developing malware, enabling them to create more sophisticated and adaptive malware with less effort.

One notable application that's made possible by generative AI is the development of polymorphic malware.

What is Polymorphic Malware?

Traditional defence against malware relies on identifying unique "fingerprints", such as the hash of the program, to flag known malware. However, the problem with this type of defence is that even a small change to the malware's source code is all that's needed to avoid detection. In order to exploit this type of defence, polymorphic malware can "change shape" by generating part of the code every time it's executed, making it virtually impossible to identify using signature-based malware protection.

This adaptability also makes incident response much more difficult, as polymorphic malware will take different forms, potentially changing its infection vector, location and behaviour with each iteration inside the network. Additionally, autonomous malware powered by AI can operate without communication with a Command and Control (C2) server to take the next step. This autonomous behaviour reduces the amount of work and capital investment that cybercriminals need to move laterally inside a complex system.

For a Proof of Concept on polymorphic malware you can read this: BlackMamba: Using AI to Generate Polymorphic Malware

Conclusion

The rise of generative AI has brought many opportunities for innovation, both in positive and negative terms. By enabling an easy and cost-effective way of creating sophisticated phishing campaigns and the creation of a whole new breed of malware, generative AI has fundamentally changed the landscape of offensive security.

Now the challenge will be for security professional to develop equally sophisticated defences able to counteract these new types of threats, but more importantly, to educate the end user on these new trends ensuring an up-to-date knowledge on the current threat landscape.