Josh Blackburn SEO Study Finds 353 Government Websites Block ChatGPT Search While Letting Google In

Two shared robots.txt files account for two-thirds of the cases, and almost none of the sites block AI search by name.

Houston, TX, Oct. 29, 2026 (GLOBE NEWSWIRE) -- Josh Blackburn SEO today published the AI Crawler Access Study, a check of the robots.txt file on every state and local .gov domain in the federal government's official domain list. The study found 353 sites that block OAI-SearchBot, the crawler behind ChatGPT search, while allowing Googlebot. That is 2.9% of the 12,298 government sites that responded, and 5.2% of the 6,765 that publish a valid robots.txt, about 1 in 19.

353 Government Sites Block ChatGPT Search

Most of the blocking is not a decision to block AI. Of the 423 government sites that block OAI-SearchBot, only 11 name it. The other 412 catch it with a catch-all rule at the bottom of a list of approved crawlers. Two shared robots.txt files account for 242 of the 353 sites, about two-thirds. The larger file appears to be the default on a website platform used by small local governments and is served by 183 sites in 27 states. An older template written for 2000s-era search engines covers another 59 sites in 26 states.

"These sites didn't choose to block ChatGPT. Their robots.txt was written for the search engines of its day, so any crawler it doesn't list is shut out, including crawlers that didn't exist when the file was written," said Josh Blackburn.

The effect is practical. OpenAI's documentation says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. When a resident asks ChatGPT search about trash pickup, permits or the next council meeting, the city's own website cannot be a cited source, while Google keeps indexing it.

Key findings

• 80% of the 353 sites (283) let in both Googlebot and Bingbot but block OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• Minnesota has the highest rate: 21.9% of sites with a robots.txt (72 of 329), about four times the national rate. 63 of its 72 cases use the platform template.
• 20 election-office websites are affected, 5.4% of the 369 with a robots.txt, about the same as every other government site.
• The rate matches the 10,000 most popular websites (5.2%, 301 of 5,820), but for the opposite reason. Large commercial sites mostly block AI crawlers by name, while government sites block by omission.
• The platform template lets in CCBot, the Common Crawl crawler widely used to collect AI training data, while shutting out the AI search crawlers that would send residents back to the site.

The fix is one small edit. Site operators can add a group to their robots.txt that allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, and keep blocking training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot if they choose. Letting AI search in does not mean letting AI training in. Sites hosted by a platform vendor may need to ask the vendor to change the file.

About Josh Blackburn

Josh Blackburn is an SEO lead with over a decade in search who builds AI tools for SEO, AEO, and GEO. He researches how AI search engines crawl, cite and rank websites and publishes the findings and tools at joshblackburn.co.

About the study

The crawl covered 15,331 state, county, city, special district, school district and interstate domains on October 24, 2026. Files were read under RFC 9309 using the same group-matching rules as Google's parser, and only the homepage was checked. The study is a one-day snapshot. Individual sites are not named, and state figures are suppressed where fewer than 10 sites publish a robots.txt. The crawler identified itself in server logs, and any site can ask to be excluded from future runs.

Availability

The full study and the state-by-state results (CSV) are available now. Interviews and aggregate tables are available on request.

Highest Rate By State

About Josh Blackburn SEO

Josh Blackburn SEO researches how AI search engines crawl, cite and rank websites, and builds AI tools for SEO, AEO and GEO. Its studies and free tools are published at joshblackburn.co.

Press Inquiries

Josh Blackburn
josh [at] joshblackburn.co
https://joshblackburn.co/