Data piece: AI-readiness scan of 97 large sites

We scanned the home pages of 97 large publishers, tech companies, retailers and reference sites on October 4, 2026. The clearest pattern is a split by site type. Most news publishers we could read block at least one AI crawler in robots.txt. Almost no tech company does. For llms.txt it runs the other way: it is common on SEO, marketing and developer sites and nearly absent from news sites, and none of the blocking sites we could check had a valid one. FAQ schema on home pages was rare everywhere.

What we scanned

The list had 97 well-known sites in five groups: news and publishers (30), tech and developer companies (30), SEO, marketing and content sites (12), ecommerce and consumer (15), and reference and education (10). For each site we fetched three things: the home page, /robots.txt and /llms.txt. We used Half Dev's own checker logic, which we describe in how we built the free AI visibility checker, and checked 13 known AI crawler names. You can see every name in our AI crawler list for robots.txt.

This is one page per site, not a full crawl. It is a snapshot of big, well-known brands from a hand-picked list and does not represent the wider web.

Result 1: Who blocks AI crawlers

Of the 87 sites with a readable robots.txt, 36 (41%) block at least one of the 13 AI crawlers we checked, and 24 (28%) block GPTBot.

The split by group is sharp:

• News and publishers: 23 of 27 block at least one AI crawler.
• Ecommerce and consumer: 6 of 13.
• Reference and education: 4 of 8.
• Tech and developer companies: 2 of 28.
• SEO, marketing and content sites: 1 of 11.

Share of sites blocking at least one of 13 known AI crawlers, by site type. Sites with a readable robots.txt only (87).

Among the 36 that block, the median blocks 9 or 10 of the 13 crawlers (9.5), 6 block all 13, and 7 block only 1 to 5. Most use named rules for each bot. Four block only through a wildcard rule that covers every crawler without its own entry. A wildcard is easy to get wrong, and Fix: robots.txt blocking AI bots by accident covers what to do if a rule catches more than you meant.

A further 12 of the 36 block other AI crawlers but leave GPTBot allowed, so counting GPTBot alone (28%) understates how many sites restrict AI crawlers. Nine of those 12 are news or publishing sites, two are reference sites and one is a tech company. We cannot tell from robots.txt why a site treats one crawler differently. If you want to set GPTBot's access yourself, see how to allow or block GPTBot in robots.txt.

Our results record how many of the 13 crawlers each site blocks, not which ones, so we only report per-crawler totals for GPTBot.

Our list includes control tokens such as Google-Extended and Applebot-Extended, which opt content out of model training but do not affect normal search indexing. A site that blocks only those still counts as blocking "at least one" crawler here. We cannot tell from robots.txt whether any block is deliberate.

Result 2: Who has an llms.txt

Of the 75 sites where we could read the file, 24 (32%) have an llms.txt that starts with a title heading, which is the format the proposal expects. One more had a text file without a title. If the idea is new to you, what is llms.txt, and do you need one explains it.

Again the groups differ:

• SEO, marketing and content sites: 8 of 11.
• Tech and developer companies: 12 of 25.
• Ecommerce and consumer: 2 of 13.
• Reference and education: 1 of 9.
• News and publishers: 1 of 17.

Share of sites with a valid llms.txt (title heading), by site type. Sites with a readable llms.txt only (75).

"Valid" here means the format check only. We did not read the content or test whether any AI tool uses the file. Remember that llms.txt is a proposal, not a standard, and no major search engine requires it. If you decide to publish one, how to write an llms.txt file, with template walks through it, and llms.txt examples and what to put in yours shows what belongs inside.

Result 3: Sites that block AI crawlers do not publish llms.txt

We could read both robots.txt and llms.txt on 71 sites: 25 that block at least one AI crawler and 46 that block none. None of the 25 blockers has a valid llms.txt. Among the 46 sites that block nothing, 24 (52%) do. Every one of the 24 valid files in our scan sits on a site that blocks no AI crawler we checked.

The two choices pull in opposite directions. One asks AI tools to stay out, and the other helps them read the site. So the gap is not surprising, but the scan cannot show whether any site chose either on purpose. It also cannot speak for the 11 blocking sites whose llms.txt we could not read.

Share of sites with a valid llms.txt, by whether robots.txt blocks at least one AI crawler. Sites with both files readable only (71).

Result 4: FAQ schema is rare on home pages

Of the 67 home pages we could read, 2 (3%) include FAQPage structured data: one SEO and marketing site and one reference site. None of the 14 readable news home pages had it, and neither did any of the 25 readable tech home pages. This undercounts real usage, because FAQ markup usually sits on inner pages rather than the home page, and we did not scan those. Markup that a page adds with JavaScript after it loads is also missed, because our scan reads the page as the server sends it.

It is also worth knowing that Google stopped showing FAQ rich results in search on May 7, 2026, although FAQPage is still a valid schema type. What FAQ schema does and does not do covers what the markup is still good for.

What could not be read

We could not read robots.txt on 10 of 97 sites, llms.txt on 22, and the home page on 30. The reasons differ. For the home pages, 21 sites answered our request with an error such as 401, 402, 403 or 405, 7 were larger than our 1.5 MB page limit, and 2 timed out.

The unreadable home pages are not spread evenly:

• News and publishers: 16 of 30, including 10 refusals.
• Tech and developer companies: 5 of 30.
• Ecommerce and consumer: 5 of 15.
• Reference and education: 2 of 10.
• SEO, marketing and content sites: 2 of 12.

News sites account for more than half of the unreadable home pages. Large publishers often use bot protection, so this fits, although we cannot tell why each site refused. The size limit also removes seven large pages from the FAQ count.

We left unreadable sites out of every percentage instead of counting them as passes or fails. If those sites behave differently from the rest, the figures could shift. Note that the robots.txt of several sites we could not load as a home page was still readable, so the AI crawler numbers cover more sites than the FAQ numbers do.

What this means for your site

Your choices depend on what you want. A publisher selling access to its content may block AI crawlers on purpose. A business that wants to be found and cited in AI answers probably does not want a crawler blocked by accident. The numbers show that blocking is mainly a news-industry habit and that most tech companies leave their sites open.

Whichever you want, check what your robots.txt really says. The AI crawler checker lists which crawlers your site blocks, by name or by wildcard. To see whether you have an llms.txt, use the llms.txt checker. For a wider view of answer-engine readiness, run the AEO checker. If you want to see which basic checks fail most often, read our follow-up, which basic checks fail most often.

Method

Scanned October 4, 2026, from a single location, using a bot user agent that identified itself. One request each for the home page, /robots.txt and /llms.txt. Only whole-site Disallow: / rules count as blocking, which matches the live checker, so sites that restrict AI crawlers on parts of the site only are not counted. We publish totals only and do not name individual sites.

Related posts

• AI crawler list for robots.txt
• How to allow or block GPTBot in robots.txt
• What is llms.txt, and do you need one
• How to write an llms.txt file, with template
• llms.txt examples and what to put in yours
• What FAQ schema does and does not do
• Fix: robots.txt blocking AI bots by accident
• How we built the free AI visibility checker
• Which basic checks fail most often (follow-up)