Artificial intelligence has moved quickly from emerging technology to everyday business tool. But while many organizations are focused on what employees may be putting into AI systems, another question is receiving far less attention:

What information is AI taking from your organization?

AI developers may use automated crawlers and other data sources to collect publicly available content from across the internet, including webpages, blogs, posts, images and other materials, for model development.

For organizations, that creates a growing set of questions around data governance, privacy, intellectual property and even website performance.

How AI Models Gather Information from Your Website?

AI systems are developed by analyzing enormous amounts of data and learning patterns from that information. Some of that data comes from automated web crawling. These crawlers move across websites and collect publicly accessible content that may later be used to train or improve AI systems.

Web crawlers themselves are not new. Googlebot has been wandering around websites for decades, politely indexing the internet one page at a time. AI crawlers have arrived with a somewhat bigger appetite.

That content may include information your organization intentionally publishes, such as articles, reports, research, images, presentations, marketing materials and other public-facing information.

The challenge is that publishing something for people to read is not necessarily the same thing as deliberately deciding that it should be collected at scale and used for AI development. That distinction is now driving significant debate.

Public Does Not Necessarily Mean Free for Any Use

The legal landscape surrounding AI training is still evolving.

The U.S. Copyright Office has devoted an entire portion of its ongoing artificial intelligence study to generative AI training, including questions surrounding the use of copyrighted works and whether particular uses may qualify as fair use.

For leaders, the takeaway is simpler than the legal debate – Publicly accessible content is not necessarily free of copyright, contractual, privacy or other legal restrictions.

Organizations may publish information online because they want customers, residents, researchers or other audiences to see it. That does not eliminate the need to consider how the information may be collected, reused or incorporated into other technologies.

The issue becomes particularly important when public websites contain proprietary research, copyrighted materials, photographs, personal information or other content that may carry legal or business value.

Do You Actually Know What Your Website Is Sharing?

For most organizations, the immediate concern is probably not that every webpage contains highly sensitive information. It is whether anyone has recently stopped to look at everything the organization is making available.

A public website may contain years of reports, staff biographies, presentations, technical documents, research, images, policies and archived materials accumulated over time. Much of it may be completely appropriate to publish. Some of it may have intellectual-property value. Other information may have been appropriate to publish for one purpose without anyone considering how automated systems could collect and reuse it later.

The leadership question should therefore be broader than, “Do we have confidential information on our website?”

It should be:

Do we know what information we are exposing, who may be collecting it and whether that aligns with how we expect our data to be used?

AI Crawlers Can Also Create an Operational Issue

This is not only a privacy or intellectual-property question.

AI-related automated traffic has become a significant part of the modern internet. Cloudflare reported that, as of June 2026, 52 percent of the crawler requests it observed were associated with AI training, up substantially from the prior year.

For many organizations, that additional activity may cause little noticeable impact. For others, particularly websites with limited infrastructure or large amounts of valuable content, aggressive crawling can consume resources, increase costs or affect site performance.

That places AI crawling at an unusual intersection of data governance, cybersecurity and ordinary website operations.

What Organizations Can Do About AI Crawlers?

Organizations do not necessarily need to block every AI crawler. They do need to make an intentional decision about what they are willing to allow.

• Start with visibility. Understand what information is publicly available and whether there are categories of content the organization would prefer not to have collected for AI-related purposes.
• Use established technical controls. A robots.txt file, for example, allows a website to communicate access instructions to recognized automated crawlers. OpenAI allows site owners to disallow GPTBot through robots.txt, while Google provides its Google-Extended control for determining whether crawled content may be used for future Gemini training and certain AI functions.
• Add server-side protections where warranted. Organizations may also consider firewall rules, IP blocking and rate limiting when automated traffic becomes excessive or when additional protections are warranted.
• Provide private content properly. robots.txt cannot hide or secure private material; anything sensitive should sit behind authentication, not just a crawler instruction.

But technical controls should follow a governance decision, not substitute for one. Legal, privacy, communications, cybersecurity and web teams may all have a role in deciding what content should be exposed, what automated activity should be permitted and who is responsible for monitoring it.

For some organizations, the answer may be to allow most crawling. For others, it may mean restricting certain bots, protecting particular areas of a site or reconsidering what information belongs online in the first place.

The important part is knowing which decision you have made.

The Bigger Data-Governance Question

Much of the current conversation around AI governance focuses on what employees put into AI tools. That remains important. But organizations also need to consider the other direction: What data is AI taking from us?

AI is changing how information that has traditionally been considered “public” can be collected, analyzed and reused at extraordinary scale. Organizations do not need to panic every time an automated crawler visits their website. They do need visibility into their public data footprint and a deliberate approach to how that information may be accessed and used.

Understanding what your organization makes available, how automated systems may interact with it and whether existing controls reflect your expectations is increasingly becoming part of modern data governance.

The internet has always made information accessible. What AI does is just change what accessibility can mean.

Contact Our Tech, Privacy & Cyber Risk Team

Do you know what your website is sharing with AI? Ice Miller's Tech, Privacy & Cyber Risk team helps organizations map their public data footprint and build a deliberate governance approach to AI crawling. Contact us.

This publication is intended for general information purposes only and does not and is not intended to constitute legal advice. The reader should consult with legal counsel to determine how laws or decisions discussed herein apply to the reader's specific circumstances.