New researchTechnical SEO & AI Search · 2027Explore the framework

Merchant Center crawler blocked: diagnose product access

Find why Google cannot fetch Merchant Center product pages or images, from robots.txt and WAF rules to mobile redirects and server errors.

Part of the E-commerce SEO series. For the full picture, start with E-commerce technical SEO: the complete guide for online stores.

“Merchant Center crawler blocked” means Google cannot reliably fetch a submitted product landing page, image or another URL needed for a merchant check. A product may look fine in your own browser while a crawler gets a robots block, challenge, timeout or error. Google’s Merchant Center troubleshooting guidance says inaccessible landing pages can stop affected products from showing in ads and listings. This article is about access, not the feed-versus-structured-data mismatches that happen after a page can be read.

Which URL and crawler should you test first?

Open the affected item in Merchant Center and copy its exact submitted landing URL. Include its variant parameters, selected country and mobile path. Also record any image URL named in the issue. Google’s product-page access help identifies Googlebot for landing pages and Googlebot-Image for images. For some local inventory or checkout checks, StoreBot-Google is relevant too.

Symptom First evidence to collect Likely owner
Page disallowed Effective robots.txt rule for the exact URL SEO or platform
Image unavailable Image URL status and Googlebot-Image access Media/CDN
403 or challenge CDN and WAF event for verified Google request Security team
5xx or timeout Origin logs and response time for item URL Hosting team
Wrong mobile destination Redirect chain and final mobile response Storefront team

Do not fix a product-page error by changing all products’ feed data. First establish where the request fails.

How do robots.txt and WAF rules block Merchant Center?

Check robots.txt for the relevant user agent and path. A broad disallow, a blocked parameter path or an image-directory rule can make otherwise public products unreachable. Google’s Merchant Center help recommends allowing Googlebot and Googlebot-Image to access product landing pages and assets. Make a narrow, deliberate rule change and retest the exact URL; do not assume the homepage test covers products.

A CDN or WAF may allow the page in a normal browser but challenge or block automated requests. Google’s StoreBot access guidance warns that a rule built only around ordinary Google Search crawler IPs may not cover StoreBot. It recommends verifying Google requests rather than maintaining a static allowlist of guessed addresses.

A Googlebot-looking user agent is not proof of Google origin. Anyone can send that header. Use Google’s verification method: check the published IP ranges or perform reverse DNS and a matching forward DNS lookup on the request IP. Then adjust the WAF exception for verified crawlers and only the routes they need.

What if robots.txt allows the URL but the item still fails?

Follow the full path from submitted URL to final response. A mobile redirect may send Google to a missing variant; a regional rule may block US-origin requests; an origin may time out during a price update; an image CDN may reject fetches without browser cookies. Google’s Merchant Center launch guidance notes that unstable landing pages and crawler errors can affect item approval.

Run these checks in order:

  1. Inspect the item issue and save the exact page and image URLs.
  2. Check robots.txt and the final URL after every redirect, including mobile behavior.
  3. Use Search Console URL Inspection for the landing page when relevant; compare with the server response.
  4. Search WAF and origin logs for verified crawler requests and the returned status.
  5. Correct the specific block or availability failure, then retest page and image.
  6. Ask Merchant Center to recheck the affected item and watch diagnostics.

If the page is accessible but its price or availability disagrees with the feed, switch to the Merchant Center feed and markup guide. If the product is genuinely unavailable, use the out-of-stock product guide rather than hiding a live URL from crawlers.

Crawler-access audit checklist

  • Record the Merchant Center issue and exact affected URLs.
  • Check Googlebot, Googlebot-Image and StoreBot only where the issue calls for them.
  • Test robots.txt, redirects, mobile response, page and image status.
  • Verify genuine Google requests before changing WAF rules.
  • Fix the access problem before feed resubmission or review.
  • Recheck Merchant Center diagnostics after Google revisits the item.

Frequently asked questions

Which Google crawler does Merchant Center use?

It depends on the check. Google's Merchant Center guidance names Googlebot for product landing pages, Googlebot-Image for images, and StoreBot-Google for some store and checkout checks. Diagnose the exact issue before changing crawler rules.

Why can I open a product page while Merchant Center cannot?

Your browser and Google's crawler may receive different responses because of robots rules, a firewall, bot challenge, mobile routing, geography or server load. Test the submitted URL and inspect CDN and origin logs for verified Google requests.

Should I allowlist a user agent string to fix the issue?

A user agent string can be spoofed. Use Google's documented crawler IP ranges or reverse and forward DNS checks to verify requests, then adjust the specific WAF rule. Avoid turning off all bot protection without evidence.

Will resubmitting the feed fix a crawler block?

No. Reprocessing is useful after access is restored, but it does not remove a robots.txt disallow, a CDN challenge or a server error. Fix the retrieval problem first, then ask Merchant Center to recheck affected items.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows