Artificial intelligence (AI) has moved well beyond answering questions, drafting emails, and creating the occasional questionable image. Increasingly, AI agents can access systems, retrieve information, interact with applications, and complete tasks on behalf of employees and organizations.

The potential for innovation and efficiency is enormous. But as AI moves from recommending actions to actually performing them, organizations face a new challenge: What happens when the technology does something its human operators never intended?

Recent headlines involving Google's Gemini and warnings from AI safety researchers have brought that question into sharper focus. And while some headlines make it sound as though AI is preparing to take over the world, there are more immediate and practical lessons for organizational leaders.

We do not need to predict an AI apocalypse to recognize that giving software the ability to act independently requires a different level of governance.

When AI Goes Beyond Its Instructions

In September, reports emerged that Google's Gemini AI model had accessed the systems of three real companies during a cybersecurity evaluation conducted earlier this year.

The agent was supposed to perform cybersecurity testing against a fictional organization. Instead, it accessed the public internet, encountered real companies it mistakenly believed were part of the exercise, and gained unauthorized access to their systems. The activity reportedly stopped after the agent recognized that it had exceeded the intended scope of the evaluation.

This was not a case of an AI system deliberately deciding to become malicious. It was an example of an AI agent carrying out an assigned task beyond the boundaries its operators intended.

The distinction matters because AI agents can increasingly perform actions that previously required direct human involvement.

An employee using an AI assistant to summarize emails can review the results before deciding what to do next. An AI agent authorized to access those emails, respond to customers, update records, or interact with business applications may act before anyone realizes it has misunderstood its instructions.

The risk is no longer limited to what AI might tell us. It now includes what AI might actually do.

What is the Biggest Security Risk of AI Agents Today?

Recent warnings from AI safety researchers, including high-profile departures and public statements from researchers at Anthropic, the company behind Claude, have fueled discussions about the long-term risks of increasingly autonomous AI systems. Some researchers have expressed concerns about whether future systems could become too capable to reliably control.

Those longer-term concerns deserve serious research and discussion, but their likelihood and timing remain uncertain.

For organizational leaders, the more immediate issue is already here: AI agents can perform tasks across multiple systems, use credentials, and take actions with limited human involvement. When permissions, technical boundaries, or oversight are insufficient, those actions can extend beyond what an organization intended.

The good news is that we already have cybersecurity principles that can help address these risks. Organizations do not need to invent an entirely new approach to security, but they do need to apply existing safeguards to technologies that can act on their behalf.

In other words, we can leave the debate about the robot apocalypse for another day. There are a few things worth checking in our own organizations first.

Three Ways to Keep Your AI Agents in Check

1. Know what your AI agents can access

Start with visibility. Does your organization know:

• Which AI agents are being used?
• What systems they can access
• Whose credentials or permissions they are using?

Apply the principle of least privilege: an agent should only have the access its task requires. An agent helping an employee schedule meetings probably does not need access to financial records, administrative accounts, or sensitive customer information.

Leadership check-in: Ask your information technology (IT) team to identify active AI agents, their system access, and whether their permissions are limited to what they actually need.

2. Make sure the guardrails are real

Telling an AI agent not to access certain systems is not the same as technically preventing it from doing so. The Gemini incident illustrates why instructions alone may not reliably keep an agent within its intended scope.

Organizations should:

• Establish technical access restrictions.
• Test and verify those restrictions work as intended.
• Determine which activities require human approval, particularly when agents can move money, modify records, change security settings, or affect critical operations.

Leadership check-in: Ask whether an agent could perform a sensitive action without human approval and what technical controls would prevent it from exceeding its authorized boundaries.

3. Know how to monitor, pause and stop AI agents

If an AI agent begins behaving unexpectedly, would anyone notice? Does the organization know who can intervene, suspend its activity, or revoke its access?

AI Agents should operate with:

• Appropriate logging
• Continuous monitoring
• Clearly assigned human oversight 
• Incident response plans that also account for unauthorized or unintended actions taken by automated systems

Leadership check-in: Ask your team to demonstrate how it would detect unusual agent activity, identify what the agent has done, and quickly disable its access if necessary.

Why AI Governance Must Keep Pace with AI Capabilities

Many organizations developed their initial AI policies around what employees could enter into AI tools, which platforms were approved, and whether AI-generated work required human review.

Those are still important considerations, but policies focused solely on information sharing may no longer address the full risk.

As organizations adopt agentic AI, governance must also address:

• Agent permissions
• Autonomous actions
• Human oversight
• Monitoring
• Third-party integrations
• Incident response

These responsibilities extend beyond IT. Legal, cybersecurity, operations, human resources, procurement, and executive leadership may all have roles depending on how AI is being used.

An AI agent helping schedule meetings creates a very different risk profile from one processing financial transactions, accessing sensitive records, or interacting with operational technology. The objective is not to discourage innovation. It is to ensure that organizations understand the capabilities they are introducing and establish safeguards appropriate to the potential consequences.

AI agents should not have to remember where the boundaries are. Your organization's systems should enforce them.

Connect with Our Tech, Privacy & Cyber Risk Team

Is your AI governance keeping pace with your AI capabilities? Ice Miller's Tech, Privacy & Cyber Risk team helps organizations assess AI agent risk, strengthen access controls and build governance and incident response plans for agentic AI. Contact us.

This publication is intended for general information purposes only and does not and is not intended to constitute legal advice. The reader should consult with legal counsel to determine how laws or decisions discussed herein apply to the reader's specific circumstances.