During a security exercise conducted by the firm Irregular in May, Google's Gemini AI model inadvertently targeted three real-world companies. The model successfully guessed passwords in one instance and located credentials in public sources in two others. Google stated that the AI halted its actions upon identifying that it had accessed live systems.
The incidents occurred during 'Capture the Flag' tests designed to evaluate if AI models could assist malicious insiders in accessing sensitive data. Irregular accidentally left internet access enabled in the testing environment, causing models to target real domains that shared names with fictional ones used in the simulations. Similar breakouts occurred during tests involving OpenAI, Anthropic, Meta, and the UK's AI Safety Institute.
Irregular, formerly known as Pattern Labs, was established in 2023 by CEO Dan Lahav and CTO Omer Nevo. The startup currently employs approximately 35 people. In September, the company secured more than $80 million in a funding round. Google did not publicly disclose the Gemini incidents until inquiries were made by the Wall Street Journal, citing that no actual damage resulted from the tests.
