The Crawl Is Slow Or Looks Stuck
Why crawling a store takes time, how to tell a slow crawl from a stopped one, and what you can do while it runs.
Short Answer
A crawl is slow on purpose: we wait at least a second between requests to your store so we never put a load on it. If the stage counts are still moving, it is working. If the pill says Stopped, read the sentence under it and press Try Again. You can pick what to track before every product is crawled.
Crawling Your Store runs four stages in order: Finding Pages, Crawling Collections, Checking What Your Product Cards Show and Crawling Products. The pill at the top says which stage is running, such as "Running · stage 2 of 4", and the counts under each stage move as pages come in.
Why It Takes A While
We crawl the way a careful search engine does. Requests to your store start at least one second apart, a few at a time. If your robots.txt sets a crawl delay, we wait longer. If your store answers that we are asking too often, we slow down further and try again later rather than push.
Each collection page is fetched as plain HTML, and some are also loaded in a browser so we can see what scripts add. A store with thousands of products can take a long time to crawl in full. Once a stage has a measured pace, the line under the heading says about how many minutes are left.
You Do Not Have To Wait For Every Product
See What We Found opens when two things are done: every collection is crawled, and we have checked what your product cards show on your largest collections. The rest of the products are crawled after that, with the products of the collections you track first. You are told when you can move on, in the line under the heading.
You can also close the page. As the screen says, "You can leave this page. The crawl keeps going without it."
How To Tell It Has Stopped
A crawl in motion has a pill that says Running and counts that change. A crawl that ended without finishing has a pill that says Stopped and a sentence saying why, such as "The crawl stopped before it finished."
- Read the sentence. If it says your store blocked our crawler, see It Says Your Store Blocked Our Crawler.
- Otherwise press Try Again. It picks the crawl up again from what is already stored.
- If it stops a second time, press Tell Us What Went Wrong under it. The note carries a reference to the stopped crawl, so we can find it.
What We Cannot Do
We do not speed a crawl up by sending more requests at once. A crawl that hammers a store can slow it down for real shoppers, and many stores would block it. Pages Crawled may also stop short of your full catalog: when the crawl reaches your plan's page limit, the rest are listed under "We skipped" with the reason.
Still Stuck?
When a screen in the app shows Tell Us What Went Wrong, use it: the note it opens carries the page you were on, so we can look at the same thing. Otherwise, write to us at help@collectionoptimizer.com, and say which screen you were on and what it said.