Survey of 10,099 Shopify stores finds almost no AI crawler rules and a category filter that doesn't filter

What was measured

Two related measurements published on dev.to, both drawing on a corpus of 10,099 known Shopify storefronts, converge on one finding: almost no merchant has made an active decision about AI agents reaching their store. One study read robots.txt files across the whole corpus; the other probed Shopify's new agent-commerce endpoint on a sample of 200 stores. Both write-ups were originally published at shelfglance.com, and both ship their raw readings as CSV files with one row per store.

98.7 percent of stores block nothing

Between 29 August and 2 September 2026, the robots.txt study parsed every storefront's file using the conventions major crawlers document — the most specific user-agent group applies, the longest matching path rule wins, and Allow beats Disallow on ties — and asked twelve AI crawler names a single question: may they fetch /products/?

Only 133 of 10,099 stores (1.32 percent) block even one crawler. The blocked names are dominated by training and grounding crawlers: CCBot on 81 stores, GPTBot on 77, Bytespider on 72, Amazonbot on 60, Google-Extended on 58 and ClaudeBot on 53. The four crawlers that fetch pages at answer time — ChatGPT-User, PerplexityBot, OAI-SearchBot and Perplexity-User — are blocked by 11, 10, 6 and 3 stores respectively. Just 13 stores in the entire corpus block any answer-time crawler.

The blocking also looks copied rather than decided. According to the dev.to post, 130 of the 133 use a whole-site Disallow: /, and 28 block the identical list of eight training crawlers verbatim, which reads like a shared snippet that contains none of the answer-time names. The sharpest signal is OpenAI's pair: 77 stores block GPTBot, which trains models, but only six block OAI-SearchBot, which feeds ChatGPT's shopping results — and all six also block GPTBot. No store blocks the shopping crawler while allowing the training one. The author's conclusion is that no store in the corpus has deliberately opted out of AI shopping answers; the ones excluded got there because a copied training opt-out happened to include the name.

Eight stores carry a noai or noimageai meta robots tag; six of them block nothing in robots.txt, and six of the eight look like musicians' merchandise stores sharing one template. The study stresses that every count is a floor, since edge or firewall blocks are invisible to the scan.

The category filter that doesn't filter

The second study examines the Universal Commerce Protocol endpoint that, since 2026, every Shopify storefront answers at POST /api/ucp/mcp and advertises at /.well-known/ucp. Merchants did not enable it, and it does not appear in their admin. Its search_catalog tool hands AI shopping agents structured product data — prices in minor units, variants, SKUs, canonical URLs and a taxonomy category per product — and the schema declares a catalog.filters.categories parameter documented as combining category values with OR logic.

The author sampled 200 stores deterministically from the corpus and ran five identical calls per store on 2 September 2026; 190 could be read. The outcome: 186 stores ignored the category filter outright, so that even an impossible category ID returned the full unfiltered catalogue; four rejected every value, including categories their own products carry; and zero filtered correctly. On 177 of the 190, returned products do carry taxonomy categories, so this is not missing data — the filter simply does not use it.

A control call makes the null result credible. A price filter set to one cent correctly returned nothing on 150 of the 190 stores, and on 148 of those the very same request ignored the category parameter. One field in the same envelope moves the result; the other does not. The author also discloses a bug in an early run — sending category objects where the schema wants strings — which briefly produced a false finding of mass refusals, and added a bare-identifier variant to close off the "you sent it wrong" explanation.

Two consequences follow, per the post: an uncategorised product is not hidden from agents, because nothing is being filtered at all; and no agent can narrow a catalogue search by category on any of the readable stores today, whatever the schema promises. The author frames this as ordinary for a young protocol rather than a complaint about Shopify, with the lesson being to test what an endpoint does rather than trust what it declares.

Why it matters

Commerce is being wired for AI agents by platform default, not merchant choice. Shopify switched on a structured agent endpoint across every storefront without a merchant-facing setting, and the one control merchants do own — robots.txt — is unused by 98.7 percent of stores and, where used, mostly reflects a copied training opt-out written before answer-time shopping crawlers existed. The effective default is full openness to AI shopping agents, set incidentally.

For agent developers, the second study is a caution: a protocol's declared capabilities, like category filtering, may not behave as documented yet, so clients should treat catalogue responses as unfiltered and narrow results themselves. Because both studies publish per-store rows naming checkable domains, anyone can re-run the numbers rather than take them on trust.