Which basic checks fail most often (follow-up)

In our first scan of 97 large sites, 41% of readable sites blocked at least one AI crawler, 32% had a valid llms.txt, and 2 of 67 readable home pages had FAQ schema. This follow-up looks at all 12 basic checks: which ones do large sites miss most often?

How we ran the scan

• Sites: the same 97 large sites as the first scan, in the same five groups: news and publishers (30), tech and developer companies (30), ecommerce and consumer (15), SEO, marketing and content sites (12), and reference, education and other (10).
• Date: October 7, 2026. The first scan ran on October 4, 2026.
• Scope: the home page of each site, plus its robots.txt, sitemap.xml and llms.txt files. Nothing else.
• Checks: the 12 checks in our SEO, AEO and GEO checkers, run with the same code the live checkers use. Our sample report shows what one scan returns.
• Method: rule-based, run from a personal computer, not a data center. Four requests at a time, a short pause between them, a 10 second timeout and one retry.
• Readable sites: 67 home pages, 85 robots.txt files, 71 sitemaps and 77 llms.txt files could be read. A page or file we could not read is left out of that check, never counted as a pass or a fail. So each row below has its own count.

How each result is decided:

• Pass is a full pass. Warning is a partial result. Fail is a miss. Warnings are shown separately.
• Image alt text: pass when at least 90% of images have a description, warning from 50% to 90%.
• Title length: pass at 50 to 60 characters, warning from 30 to 49 or 61 to 70.
• Heading structure: pass for one H1 and no skipped levels.
• Internal links: pass at 3 or more.
• Freshness: a Last-Modified header, a modified or published meta tag, or a JSON-LD date field.
• Meta and Open Graph tags: warning when the title, description or any of the three Open Graph tags is missing, or the title is outside 10 to 70 characters.
• AI crawler access: pass when none of 13 known AI crawlers is blocked, warning when some are, fail when all 13 are.
• llms.txt: pass for a plain-text file that starts with a title heading.
• FAQ schema: pass for FAQPage markup with at least one question.

Results by check

Figure 1. Pass, warning and fail share for each check, among sites we could read.

Percentages are rounded, so a row may not add to exactly 100%. For AI crawler access, 41% of sites block at least one crawler (35 of 85), the same share as our first scan. Check any crawler result with the AI crawler checker, and any llms.txt with the llms.txt checker.

61 of the 67 readable home pages (91%) had at least one warning.

What stood out

1. FAQ schema was missing on 65 of 67 pages. It is the same 2 pages as in our first scan, so nothing has changed in the days between.
2. No home page passed all 12 checks. The best three passed 10, the middle page passed 6, and only 8 of 67 passed 9 or more.
3. Freshness failed on 47 of 67 pages (70%). It is the second most common failure, behind FAQ schema.
4. News sites and tech sites split on AI crawlers and headings. Of the sites with a readable robots.txt, 23 of 27 news sites block at least one AI crawler, against 1 of 27 tech sites. Only 2 of 14 news home pages pass heading structure, against 14 of 25 tech pages. Only 14 of 30 news home pages could be read, so treat the news figures as a small sample.
5. Title length passed on only 14 of 67 pages (21%). Another 48% were close, in a warning range of 30 to 70 characters. The 50 to 60 range is a rule of thumb, not a rule set by Google.

Figure 2. Checks passed per home page. Pages with 9 or more passes are in green.

Figure 3. Share passing, news and publishers against tech and developer companies. Only 14 of 30 news home pages could be read, so the news figures are a small sample.

What these numbers don't tell you

• Home page only. Inner pages may differ.
• Large sites only. The results may not describe small or new sites.
• Blocked sites are excluded from the percentages, so each check has its own count of readable sites. Sites that block AI crawlers were also more likely to refuse our scanner: 16 of 35 had an unreadable home page, against 5 of 50 that block none. The news figures lean on that.
• Markup added by JavaScript is missed. A page that adds structured data or FAQ schema after loading can look like it has none.
• Empty alt text counts as missing. An empty alt attribute is correct for decorative images, but the check treats it as a miss. 1,213 of the 3,578 images we counted (34%) had one, so the image alt text failures are overstated.
• The sitemap check reads /sitemap.xml only. A sitemap at another address, or one listed only in robots.txt, shows as failing.
• A pass is not a ranking. These are basics, and a site that passes all of them can still rank poorly. Our guide to what your score means covers this.
• A failure is not always a mistake. Blocking a crawler on purpose is a choice. 3 of the 35 sites that block AI crawlers do it only through a wildcard rule that shuts out every bot, not just AI.

What to do with it

Scan your own site with the SEO checker, the AEO checker and the GEO checker, and compare your result with the table. Start with the checks that most sites pass but yours doesn't: canonical tag, internal links and structured data. If you are setting up a new site, work through the SEO audit checklist first.

Related posts

• Data piece: AI-readiness scan of 97 large sites
• What your Half Dev score means
• Half Dev Report: What You Get From Free Page Scan
• SEO audit checklist for a new site
• How to allow or block GPTBot in robots.txt
• How to fix missing image alt text
• How to add internal links to a page
• How to create and submit an XML sitemap
• How to fix your heading structure
• How to add a publish and updated date to your pages
• How to add meta description and Open Graph tags

Terms used here: Answer Engine Optimization, Generative Engine Optimization, AI crawler.