Over 100 malicious artificial intelligence (AI) machine learning (ML) models were discovered on the Hugging Face platform, a popular public AI model repository allowing users to create
interactive in-browser demonstrations. These models enable threat actors to inject malicious code into user machines and compromise user environments.

Researchers developed a tool that enabled them to scan files uploaded to Hugging Face to detect and remove potential malicious activity, which led to the discovery of adversarial models in the Hugging Face platform. Malicious payloads were discovered in models that had been uploaded to the repository. In one instance, the tool flagged a PyTorch model that allowed threat actors to establish a reverse shell to a specified host to push Python code into critical processes.

Other instances that show the possibility of malicious injections in generative AI (GenAI) apps have also been found through testing. They discovered a worm that was able to trick the GenAI apps into propagating malware through the use of prompt engineering. They found they could hide malicious prompts in emails, websites, and encoded images to reprogram the GenAI so future users would receive their malicious updates.

The NJCCIC recommends that AI developers use tools like Huntr, a bug-bounty platform designed explicitly for AI models and platforms, to increase security against vulnerabilities. Developers should use vigilance and proactive measures to safeguard platforms from malicious actors.

The NJCCIC recommends that users and organizations educate themselves and others on these continuing threats and tactics to reduce victimization. In addition, all users are encouraged to exercise caution when using AI platforms and review the Seeing AI to AI: Artificial Intelligence and its Impact on Cybersecurity NJCCIC Informational Report.