How We Built a Free AI Visibility Checker

What we built

We built Half Dev's free checkers to answer one question quickly: can search engines and AI tools reach your site, and does the page give them something clear to use? You enter a URL, the tool scans one page, and you get three scores with specific fixes. This post covers how the tools are organized, how scans and share links work, what runs underneath, the known limits of the current version, and what we chose not to build.

One scan, three scores, five pages. Every scan runs all three checkers, whichever page you start from.

• The SEO checker has five checks: XML sitemap, canonical tag, image alt text, internal links and title length. Each is worth 20 points.

• The AEO checker has five checks: structured data, meta and Open Graph tags, AI crawler access, heading structure and freshness signals. Each is worth 20 points.

• The GEO checker has two checks: an llms.txt file and FAQ schema. Each is worth 50 points.

Two more pages are built around a single question each. The llms.txt checker opens the GEO view, and the AI crawler checker opens the AEO view. We made them separate pages because people search for them separately, and a page that answers one question directly serves that visitor better than a general tool page.

Why three scores and not one

SEO, AEO and GEO overlap a lot, and most of the work counts toward all of them. We still keep them separate because they look at different signals: ranking basics, answer-friendly pages and crawler access, and newer extras. A site can be strong in one and weak in another. One site we scanned scored 86 on SEO and 0 on GEO.

The report does show an overall number, which is a plain average of the three scores. It is a convenient summary, and the three separate scores are more useful. Because GEO has only two checks, each one is worth half of its score, so a missing llms.txt moves a score far more than a missing alt text does. That is a property of the checklist, not a judgment about which matters more. What your Half Dev score means lists every point value and how partial credit works.

Each result lists what passed, what needs attention, and a concrete fix, ordered by the points at stake. What you get from a free page scan shows a sample report, and how we built the Half Dev report covers how the report was made.

How a scan works

A scan fetches your page, your robots.txt, your sitemap.xml and your llms.txt, checks them against a fixed set of rules, and returns the findings. The rules are plain fetching and pattern matching. There is no AI model and no paid API, so the direct cost of each scan is close to zero, and the same page normally gives the same result.

What one scan reads, checks and returns.

We built it to run inside the time limits of our free hosting: the page fetch times out after 8 seconds and the root files after 5. The checker identifies itself as HalfDevAEOChecker. Scans are rate limited to 8 per visitor every 10 minutes, so one visitor or script cannot overload the service. A small database table counts recent requests.

Three rules show how simple the logic is:

• AI crawler access reads robots.txt and checks 13 known AI crawlers by name. Only a whole-site Disallow: / counts as a block, whether set by name or by a wildcard. It scores 20 if none are blocked, 10 if some are, and 0 if all 13 are.

• Title length scores 20 for 50 to 60 characters, 12 for 30 to 49 or 61 to 70, and 4 for anything else. The 50 to 60 range is a rule of thumb, not a rule set by Google.

• llms.txt scores 50 for a plain-text file that starts with a title heading, 25 for a text file without one, and 0 if it is missing, empty or a web page.

A pass means a signal is present, not that it is correct. The scan reads only the HTML your server returns, so markup added by JavaScript after the page loads can be missed. It also reads robots.txt only, so a firewall or CDN that refuses AI crawlers is invisible to it. We would rather show a smaller result and say what it cannot see than imply the tool knows more than it can.

Known limits of version 1.0

The current checker is version 1.0. We found these problems by testing it on our own sites and on the 97-site research. Each has a fix planned for version 1.1, listed under "What is next". Until then, read results with these in mind.

• A failed fetch can read as "not found." If robots.txt returns an error or times out, the AI crawler check can pass because no rules were read, while a failed sitemap or llms.txt fetch shows as missing.

• Bot protection can fool it. Many sites answer a bot with a 403, and the checker does not retry or label the page as unreadable. In our 97-site research we could not read 30 home pages, and 21 of those answered with an error such as 401, 402, 403 or 405. The research left unreadable sites out of every percentage, but the live checker does not yet do the same.

• Empty alt text counts as missing. An empty alt attribute is correct for a decorative image, and version 1.0 still scores it as a failure. In our October study scans, 1,213 of 3,578 images (34%) had an empty alt, so alt-text failures are overstated.

• One flaw can cost points twice. A title outside 10 to 70 characters lowers the meta check and is judged again by the title-length check. Special characters such as & are also counted raw, so a title can read a few characters longer than it looks.

• No JavaScript. This is the biggest source of false fails.

• A narrow sitemap and robots.txt reading. The sitemap is read only at /sitemap.xml, and Sitemap: lines in robots.txt are ignored. Partial blocks and Allow lines under a block are not counted.

• Judgment-based scoring. The partial credit values and the weights are our own choices. Nothing shows that llms.txt or FAQ schema change whether an AI tool cites a page.

The share loop

Every result can be shared, because people who run a scan often want to show a colleague or a client. We built three pieces:

• A signed result link. A shared result carries the domain, the three scores and the scan time inside the link itself, followed by an HMAC-SHA256 signature made with a server-side secret. Opening a link checks the signature first, then confirms the host and the scores are valid. Edited, forged, truncated, oversized or wrongly signed links are rejected, and a test script covers 14 cases including those. We store no results. Only the host goes in the link, never the path or query, because those can contain private tokens.

• A score badge, an SVG image served from the signed link, that a site can display.

• An image preview for every tool page, so links look right when posted in chat or social feeds.

How a shared result link is built and checked.

The signature proves a link was not edited. It does not prove who owns the site: anyone can scan any public page and share the result. Links also do not expire. The scan time is signed into every link, and showing the scan date beside each score on the result page is next on our list. Until then, a shared score is best read as what one scan found at one moment.

Connecting the articles and the tools

Our guides and the tools reference each other on purpose. Glossary entries link to the tool that checks the term, such as the llms.txt glossary entry. Posts end with a short tool box. Tool pages link back to related terms and guides. The principle is that each page keeps its own purpose: an article answers "what is it", a tool page answers "check mine", and they point at each other.

We track one thing from this: which links people click before running a scan, so we can see which pages send useful visitors. Our analytics are cookieless by default, and the more detailed Google Analytics events only fire after a visitor accepts the consent banner, so detailed counts run lower than real traffic.

The stack

The site runs on Next.js, deployed on Vercel, with Supabase for the database and file storage. The scanner uses the database for the rate-limit counts, and shared results need no table at all. We kept the stack small.

What we chose not to build

• A JavaScript-rendering scanner or a whole-site crawl. A scan reads one page and a few root files. That keeps it fast and free, and it is the reason for the limits above.

• AI-based scoring or paid services. Every score comes from a visible rule you can read.

• Stored results and accounts for sharing. The signed link carries what the result page needs, so there is nothing to store and nothing to log in to.

• One score as the only number. The overall average is a summary. The three scores are the result.

• Named results for other people's sites. Our 97-site research reports totals only, and the limits are stated there. The named scans we publish are of sites we run ourselves.

What we chose not to claim

We do not promise rankings. Search traffic takes weeks to months, and no checker can guarantee a position. The tools describe what is in place and what is missing, nothing more.

We treat llms.txt as foundation, not a ranking lever. We publish one on halfdev.com because it takes minutes and costs nothing. It does not allow or block anything, because robots.txt does that. Google's Search documentation says llms.txt is not needed to appear in Google Search, including its AI features, and that the file creates neither a positive nor a negative visibility effect there. Chrome's Lighthouse tool has an experimental check for it, so Google's guidance depends on the product. The checker scores llms.txt as an extra. What is llms.txt, and do you need one explains it in full.

We check vendor claims before we repeat them. Guidance in this space changes fast. For example, Google stopped showing FAQ rich results in Search on May 7, 2026, so any advice from before then needs rechecking. FAQ markup is still valid and still scored, as machine-readable questions and answers. What FAQ schema does and does not do has the detail. We label what we have verified and what we have not.

What is next

Version 1.1 is planned. The main changes:

• Failed fetches and bot-protection pages will be reported as "could not be read" instead of scored.

• A check for noindex directives, which can keep a page out of search results. Their effect on AI tools depends on the platform.

• A warning when a page is mostly JavaScript and its content may not reach crawlers that do not run it.

• Sitemap discovery from robots.txt.

• Empty alt text treated as decorative instead of missing.

• The scan date shown beside every score on shared results.

• A version and date on every report and published statistic, a change log, and a methodology page listing every rule, threshold and weight.

We plan to run the old and new versions on the same 97-site data and publish what changed. Alongside that, we are adding more explainers and quick fixes for each tool and building out the glossary. If a check is wrong or missing, tell us and we will fix the tool or the copy.

Where to start

• Site owner: run the SEO checker, the AEO checker or the GEO checker, then read what your Half Dev score means and look at a sample report.

• Developer or curious reader: read how we built the Half Dev report and the 97-site data piece.

Tools: SEO checker · AEO checker · GEO checker · llms.txt checker · AI crawler checker

Other posts: AI-readiness scan of 97 large sites · What is llms.txt, and do you need one · What FAQ schema does and does not do · What your Half Dev score means · Sample report · How we built the Half Dev report