Start Free TrialFree Trial

About Our Crawler

If you saw CollectionOptimizerBot in your server logs, this page says who it is, what it crawls and how to stop it.

Its User-Agent: Mozilla/5.0 (compatible; CollectionOptimizerBot/1.0; +https://collectionoptimizer.com/bot)

Questions? Email hello@collectionoptimizer.com

The short version

Who we are

Collection Optimizer helps online stores improve their collection and product pages. Store owners create an account, connect their own store and choose the searches they care about. Our crawler is the part of the product that fetches web pages for them.

The crawler is run by Right Thing Agency LLC. You can reach a person at hello@collectionoptimizer.com.

What it crawls

  • Our customers' own stores. When a store owner connects their store, the crawler crawls its collection and product pages so the owner can see how they are built.
  • Competing pages for a search an owner chose. When an owner names a search term, the crawler crawls the competing store pages that show up for it once, to compare them with the owner's page.
  • Your robots file. Before it crawls any page on a site it does not belong to, it reads that site's robots.txt.

It crawls only public pages, the way they first arrive. It does not log in, fill in forms, add items to a cart or follow links into checkout. It never crawls a page your robots rules ask crawlers to skip.

Through Cloudflare's browser

Our crawler asks Cloudflare's browser service to load each store page, with CollectionOptimizerBot in its User-Agent as always. It does not load images, video or fonts. Your robots.txt, sitemaps and product feeds are fetched directly, without the browser, under the same name.

Cloudflare marks these requests as bot traffic and adds headers of its own, so your logs show a bot, not a visitor. We never use another crawler's name, a browser's name or a proxy service.

Your robots rules apply to the browser as they do to a direct request: the crawler reads your robots.txt first and never asks the browser for a page your rules ask crawlers to skip. If your site answers with a challenge or a refusal, we tell our customer the page could not be read, and the crawler does not try another way in. If your site answers 429 and asks it to wait, it waits, asks again a few times at most, and then stops.

If a page you allow redirects to one your rules forbid, the browser will already have asked your server for it once by the time we see the redirect; we then drop that page without reading it and never ask for it again.

To stop it, add a group for CollectionOptimizerBot to your robots.txt, as shown below. You can also block Cloudflare's browser service in your own bot settings, and our crawler will take that as final.

How often

The crawler asks your site for one page at a time and waits at least two seconds between requests. If your robots rules set a crawl delay, it waits that long instead, up to 30 seconds between pages.

A competing page it has crawled is kept for seven days, so the same page is not fetched again for other searches in that time.

How to stop it

Add a group for our crawler to your robots.txt. To keep it off your whole site:

User-agent: CollectionOptimizerBot
Disallow: /

To keep it out of one part of your site only:

User-agent: CollectionOptimizerBot
Disallow: /private/

When there is no group for CollectionOptimizerBot, the crawler follows your group for every crawler (User-agent: *). A refusal from your server, such as a 403 answer, also stops it: the page is reported to our customer as could not be read, and nothing more is asked of your site in that run.

How to reach us

If the crawler is causing you trouble, or you want it to stop and robots rules are not an option, email hello@collectionoptimizer.com with your site's address. We answer every message.