LLM Crawler Access for Shopify: 3 Checks

LLM crawler access on Shopify passes through three checks. They cover permission in robots.txt, request delivery by Shopify, and product details in rendered HTML.

On stores I've run, theme code could look like the whole answer when a bot failed to reach a page.

The crawler rule, Shopify's network, and the rendered page can each stop the request. These checks help you find which one needs work before you change a theme file or pay someone to troubleshoot.

Key takeaways

Which AI crawlers should a store allow?

Allow search crawlers for public-page eligibility, choose training access separately, and treat user-request bots as a third case. To support visibility in ChatGPT search, allow OAI-SearchBot on public product pages. When you want to withhold future training access, GPTBot can follow a different rule. OpenAI's crawler guide explains this split.

Anthropic divides its bots by task too. Its crawler guide assigns search, training, and user-request tasks to different user agents. Decide which uses you want, then set each bot's rule.

Use this table as a policy aid. Confirm the current bot name with its operator before you publish a rule:

‍

OpenAI says a robots.txt update can take about 24 hours to affect its search systems. Use that timing only for OpenAI. Even with permission, the operator decides when to visit or index a page.

The Three-Gate LLM Crawler Access Check

Check the public rule first, Shopify delivery second, and the rendered page third. Each check tells you who can fix the problem and when to stop:

1. Public rule: You own the crawler-specific direction in robots.txt.
2. Shopify delivery: Shopify runs the network for the store.
3. Rendered page: You own the theme output and product information.

‍

Finish each gate before moving on. Fix the access rule before testing the page.

1. Inspect robots.txt before you edit it

Open your root-domain /robots.txt file and find the exact user agent before changing robots.txt.liquid. Proceed only when you can name the crawler and affected public path. A targeted change keeps ordinary search rules intact.

Shopify's robots.txt guide says to keep the default groups when you edit Liquid. These groups allow public pages and block private or repeated paths. A full replacement may go stale when Shopify changes its defaults.

The theme editor shows the template, but the live file shows what crawlers receive. An old customization or unexpected live theme can make them differ, so save both before editing.

Run this two-minute check before requesting code:

1. Open https://your-store.com/robots.txt in a private window.
2. Search for the exact user agent, such as OAI-SearchBot.
3. Read the matching Allow and Disallow rules for your product path.
4. Save the current result before changing the template.
5. Proceed only if no user agent receives both Allow and Disallow for the same product path.

I'd give the developer one crawler name and one affected path to keep the request specific. Shopify warns that bad customizations can cost traffic and suggests a Shopify Partner if you lack code or SEO experience.

The person editing the live theme owns this gate, and the public file confirms the result. Stop if the new group is missing, a default group disappears, or any unintended rule changes. Put the saved version back before another try.

2. Leave Shopify's network layer to Shopify

Shopify manages bot access at its network layer, so a normal hosted store leaves that layer to Shopify. Your control is the public robots.txt output. Shopify says store owners can leave its network controls unchanged and avoid putting a proxy in front of the platform.

Use these ownership rules:

• Leave network bot controls to Shopify on a hosted store.
• Ask the proxy or headless owner to record request handling.
• Review each Catalog channel in Shopify, outside robots.txt.

Paying a developer to add a proxy creates the wrong fix when Shopify owns the setting. A proxy handles the request first, and Shopify handles requests that pass through. That extra layer makes a crawler rule harder to trace.

Shopify Catalog sends product data to Agentic Storefronts apart from open-web crawlers. My limit here is private apps, headless storefronts, and custom edge setups. They need an account review before anyone changes a web application firewall.

On a normal hosted store, close this check when Shopify serves the Online Store. Confirm that your setup has no proxy, headless storefront, or custom edge service. A missing visit may mean the operator hasn't requested the URL. Move to Gate 3 once the public rule permits the request and no separate service owns delivery.

3. Confirm the product page is readable

A useful product page loads and puts its key product facts in rendered HTML. Fix a failed URL or missing fields after crawler permission passes.

Google's JavaScript SEO guide treats crawling, rendering, and indexing as different jobs. Googlebot can render JavaScript, though Google warns that other bots may lack that ability.

Use rendered HTML to find missing fields. Test each crawler on its own because bots render pages differently.

Check the response first. An error, redirect loop, login screen, or challenge stops the test before content loads. Record the final URL and response so a developer knows whether to check delivery, routing, or theme output.

Compare the page in three views before changing more code:

1. Load the product URL and confirm it returns the right page.
2. Open Chrome DevTools, use the Elements panel, and search for the visible title, selected price, and description.
3. Use the DevTools command menu to disable JavaScript, reload, and note what disappears.

Treat the JavaScript-disabled view as a warning because a capable renderer may still see facts added after scripts run.

When rendered HTML omits a field, send its name and product URL to the theme developer. That gives them a precise problem to trace.

The theme owner handles this gate, while the product manager confirms the facts. Pass when the public page and rendered HTML show the correct title, selected price, and description.

Stop when a variant shows a different price, an image holds the description, or a challenge page replaces the product.

Make the smallest safe change, then re-check

Change one justified crawler rule in robots.txt.liquid, publish it, and compare the public output with your saved copy. Keep the default Liquid loop intact. The custom rule belongs after that loop so your theme continues to print Shopify's managed groups.

This pattern preserves the default groups and allows OAI-SearchBot on product paths. It leaves GPTBot to a separate store-policy decision:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

# Search access decision
User-agent: OAI-SearchBot
Allow: /products/

# Add a GPTBot rule only after choosing your training policy.

Shopify's docs allow custom rules after the default-group loop. Use the current theme output as your baseline, and merge the rules already there instead of pasting a second full template.

Re-check the change in this order:

1. Open the public /robots.txt file and find the new user-agent group.
2. Confirm Shopify's default groups still appear as expected.
3. Reopen one public product URL and repeat the rendered-content check.
4. Save the before and after output with the date and person responsible.

Changes to the public file appear at once, but each crawler still decides when to react. Follow the named operator's update period before you call the visit a failure.

If the output breaks, remove the change and restore the saved template. Shopify also lets you delete a custom robots.txt.liquid template and return to its generated default. Use that rollback only after saving any rules your store still needs.

You're done when the intended rule is the only change. Default groups should remain intact, and the test product should be readable.

What crawler access can't do for a dropshipping store

Crawler access clears one block, while each AI system still chooses what to crawl, index, cite, or recommend. The system may still skip a page with copied text, stale product data, or little buyer demand.

Supplier-fed stores make that distinction important. An accessible page can still carry copied text, a stale price, or the wrong stock state.

After access passes, our LLM product-page checks help you fix those page problems.

Use robots.txt for access rules. The proposed llms.txt convention is only a content map, so skip it until the operator you care about documents support.

Match the observed problem to the next job:

• Use AI citation work when the page lacks evidence for its claims.
• Read AI referral analytics after visits show usable source data.
• Add product markup when machine-readable offers differ from visible facts.
• Use a product feed when a supported channel needs catalog data.

I recommend finishing the three gates and documenting the result before moving to the next observed problem. Once all three checks pass, stop changing crawler rules. Further edits add risk, while the AI system still controls selection. Record the owner and evidence before you hand the next problem off.

FAQ

Should I block GPTBot for ChatGPT search visibility?

You can block GPTBot and still allow OAI-SearchBot for ChatGPT search eligibility. GPTBot handles potential training use, while OAI-SearchBot controls automatic search crawling.

Does robots.txt control ChatGPT-User?

OpenAI says robots.txt rules might not apply to ChatGPT-User because a person triggers the request. Use OAI-SearchBot rules for the automatic ChatGPT search decision.

Does llms.txt allow or block AI crawlers?

Use llms.txt as a content map and robots.txt for allow and block directions.

Does crawler blocking stop Shopify Catalog sharing?

Shopify Catalog uses a separate channel to send data to the storefronts you've enabled.