Rewards and Risks: What Generative AI Means for Security

From countless news articles and posts on your social media feed to all-new tools built into your favorite software, artificial intelligence is everywhere.

Although the technology is not new, generative AI has created a recent buzz with the November 2022 release of ChatGPT, a large language model (LLM) that uses prompt engineering to generate various outcomes for users.

Almost immediately after ChatGPT’s release, similar generative AI tools launched, such as Google’s Bard and Microsoft’s Copilot, and the adoption of using AI to generate content, videos, photography, code, and more, has spread like wildfire.

As AI has grown in popularity, concerns are being raised about the risks involved with using the technology. Cybercriminals have already found ways to exfiltrate data from different AI tools, including using platforms like WormGPT, an AI model trained on malware creation data and used for ill intent or to generate malicious code.

Artificial intelligence dates back to the 1960s with the creation of the first AI chatbot, ELIZA, developed by Joseph Weizenbaum. So why is generative AI so popular now, more than 50 years later?

The introduction of ChatGPT in late 2022 accelerated the development of generative AI and gave the world access to the powerful tool.

With many software corporations developing their own AI programs, security teams may be caught off guard when these tools are released and might not be aware of how to combat the risks they present.

Microsoft Copilot, which is currently in an early-access phase, has the benefit of learning your organization in addition to its LLM design. Some use cases include Copilot joining your Teams meeting and taking notes in real time, it helping triage emails in Outlook and create replies, and even analyzing raw data in Excel for you.

Copilot is being called the most powerful productivity tool on the planet, and if you’ve ever used gen AI tools, you probably can see why it’s being called that. Imagine having a little ChatGPT that’s built into all of your Office apps like Word, PowerPoint, Excel, and Microsoft Teams.

In addition to Copilot’s abilities, there are several aspects of gen AI tools that security teams can benefit from, including enhancing cybersecurity operations, threat detection, and defense mechanisms.

Other beneficial uses of generative AI include:

Blue team defenders: Just as a threat actor may use AI tools for harm, businesses can use them for good. Thomas shared how ChatGPT has simplified ways users can check malicious code, detect specific vulnerabilities, and summarize outputs almost instantly.

Malware analysis: Generative AI can assist in generating variants of known malware samples, aiding cybersecurity professionals in creating more comprehensive malware detection and analysis systems.

Deception and honeypots: Generative AI can help create realistic decoy systems or honeypots that appear enticing to attackers. This allows security teams to monitor and analyze attack techniques, gather threat intelligence, and divert attackers away from real assets.

Automated response generation: When an attack is detected, generative AI can assist in generating automated responses to mitigate the threat. This can include generating firewall rules, deploying countermeasures, and isolating compromised systems. It can help save time for analysts responding to the threats as well.

Adaptive security measures: Generative AI can aid in developing security mechanisms that adapt to evolving threats. By continuously learning from new attack techniques, these systems can evolve and improve their defense strategies over time.

Visualizing attacks: Generative AI can assist in visualizing complex attack patterns and behaviors, making it easier for security analysts to understand how attacks are executed and identify patterns that might not be immediately apparent.

There are two sides to every story. While generative AI offers many benefits in addition to those listed above, there are also challenges and risks associated with the tool.

Generative AI introduces several security risks that need to be carefully considered when implementing and using the technology.

According to research conducted by Forrester, security is a top hurdle for companies adopting AI, with 64% of respondents reporting they don’t know how to evaluate the security of generative AI tools.

One of the top concerns of Microsoft Copilot is how its security model uses permissions and can access all the files and information that a user can. The problem is most users in an organization already have too much access to information they shouldn’t.

As the adoption of AI tools develops, humans will get lazier and potentially over-trust AI to do security checks they should be doing. For example, an employee could ask Microsoft Copilot to generate a proposal using existing documents and meeting notes, eliminating hours of work for an employee. They might skim the result and think it’s fine, but sensitive information from the original documentation could sneak its way in if it isn’t thoroughly reviewed.

Aside from the internal security concerns, threat actors will use AI to write malicious code, find vulnerabilities, and launch large-scale campaigns.

Attackers will also use AI to generate fake data sets and use them to try to extort businesses (or at a minimum, waste their time).

Other security concerns and risks associated with generative AI include:

Cyberattack campaigns on demand: Attackers can harness generative AI to automate the creation of malware, phishing campaigns, or other cyber threats, making it easier to scale and launch attacks. In Thomas’ presentation on ChatGPT, he shared an example of how ChatGPT can personalize an email to appeal to Elon Musk about investing in X, formerly known as Twitter. Including information about the target, in this case Elon Musk, in the prompt can help threat actors write messages that are more appealing and more likely to result in users taking action. AI tools could also be prompted with information such as age, gender, education, company information, and more.

No tool-proofing: AI tools also run the risk of being manipulated to produce incorrect or malicious outputs. Some AI tools have ethical standards in place to help combat improper use of the product, but threat actors have found ways around them.

Leaking sensitive information: Generative AI models often learn from large datasets, which might contain sensitive data, depending on what information is shared. If not properly handled, there’s a risk of inadvertently revealing confidential information through generated outputs. AI models can store this information too, making your sensitive data accessible to anyone who accesses your user account with different AI tools.

Intellectual property theft: Generative models often pull in a massive amount of publicly available information, including exposed proprietary data. There’s a real risk that generative AI could infringe upon others’ intellectual property rights and be subject to lawsuits. For example, image-based AI tools have been implementing Getty’s watermark on images because the AI-generated photos are created based on using Getty’s multitude of public data. Additionally, there is also a risk that your intellectual property could end up being used in AI tools if it’s not secure.

Identity risk and deepfakes: Generative AI can be used to create convincing fake images, videos, or audio clips, leading to identity theft, impersonation, and the creation of deepfake content that can spread misinformation. The tools can also make phishing campaigns seem more human and appeal to their target. An image of the Pope wearing a Balenciaga jacket went viral before it was shared that the image was created using AI tools, proving the likelihood of AI imagery and deepfake videos being believable is at an all-time high.

ChatGPT, in particular, is designed to create a believable human interaction, making it the perfect tool for phishing campaigns. Threat actors also use the LLM to package malware into fake applications, which was popular during the rise of ChatGPT before parent company OpenAI had issued an iOS application.

If you wait until a data breach occurs to start implementing security measures around AI, you’ll be coming from behind.

One of the first steps leaders can take to address concerns around employees using generative AI is to properly train them on what is acceptable to share and what isn’t.

Some people may find it harmless to include customer data in ChatGPT prompts, for example, but this is exactly the type of action threat actors are hoping your employees take. All it takes is one employee accessing a fake ChatGPT site and entering sensitive information for your company to be at risk.

As new generative AI tools are released, companies must educate their teams on how to properly use them and stay aware of the security concerns as they are discovered.

Having a Data Security Platform (DSP) in place can also prevent employees from having access to sensitive data they shouldn’t have in the first place. DSPs can help security teams automatically discover, classify, and label sensitive data, enforce least privilege, and continuously remediate data exposure and misconfigurations.

There is no denying that AI has taken the world by storm, and the technology will continue to evolve in the years to come.

Understanding the benefits and risks involved with AI, training staff to use different AI tools properly, and setting parameters around what is acceptable and not acceptable to share is the starting point.