It Says Your Store Blocked Our Crawler
What a blocked crawl means, why we stop instead of working around it, and how to let our crawler in through robots.txt or a bot protection app.
Short Answer
Your store, its firewall or a bot protection app refused one of our requests, or kept saying we asked too often. We stopped at once and did not try to get around it. Add the rule shown under How to let us in to your robots.txt, allow the same name in any bot protection app, then press Try Again.
The crawl screen's heading says "Your store blocked our crawler", the pill says Stopped and the platform line says Platform Not Checked. The sentence under it names the address where it happened and, when your store sent one, the answer code.
What Counts As A Block
We stop the whole crawl when your store does any of these:
- answers a page with "not allowed" (the codes 401, 403 or 451)
- shows a challenge page, the kind that asks a visitor to prove they are a person
- keeps answering that we are asking too often (code 429), even after we slowed down and waited
A store that asks us to slow down once is not a block. We double the wait between requests and try again. Only when it keeps refusing do we stop.
Why We Stop
We never try to get around a block: no other addresses, no disguised requests, nothing that answers a challenge for us. A block is your store, or a service in front of it, saying no. Only you can change that answer.
How To Let Us In
The box called How to let us in shows the rule to add. It is:
User-agent: CollectionOptimizerBot
Allow: /
- Press Copy Rule to copy it.
- Add it to your store's robots.txt file. On Shopify that means editing the robots.txt.liquid template in your theme. The steps are in Canonical Tags, Filters And Duplicate Collection Pages On Shopify, under When To Change Robots.txt. Add the two lines as their own group and change nothing else.
- If you use a firewall or a bot protection app, allow the name CollectionOptimizerBot there too. robots.txt alone does not get past a service that refuses requests before your store sees them.
- Press Try Again on the crawl screen.
If your robots.txt already names our crawler, the box shows those lines as we found them, so you can see what to change.
Pages Your Robots.txt Keeps Us Out Of
This is different from a block. If your robots.txt asks every crawler to stay out of some addresses, we skip those and carry on. They are listed under "We skipped" with the line "Your robots.txt asks crawlers not to open these pages, so we did not open them." The rule above lets our crawler into them too, if you want them crawled.
Still Stuck?
When a screen in the app shows Tell Us What Went Wrong, use it: the note it opens carries the page you were on, so we can look at the same thing. Otherwise, write to us at help@collectionoptimizer.com, and say which screen you were on and what it said.