About our crawler

ProofsheetBot

ProofsheetBot reads public Shopify catalogs for DPA Plus. A store's team reviews the ads before anything runs. This is its user agent:

ProofsheetBot/1.0 (+https://proofsheet-web.vercel.app/bot)

When it reads a store

When the store's owner, or someone who manages it for them, connects the store to DPA Plus. They confirm that they own or manage it first.
When someone asks DPA Plus for a preview of the store. Previews follow the store's robots.txt, for User-agent: * and for ProofsheetBot. They stay unlisted and are never published.
It doesn't crawl the web looking for stores.

How often

A connected store's catalog is checked for changes every 5 or 10 minutes, and read in full every 6 hours, or sooner when a check falls behind. Its collections and product ratings are read once a day.
When someone pastes the store's address into DPA Plus, it makes a few requests at once to check that the store can be read.

How it reads

Only public pages and Shopify's public catalog endpoints, such as /products.json. No logins, no cookies, no carts or checkouts.
One request at a time per store, at least half a second apart. Product images come from Shopify's image CDN.
When a store asks it to slow down (429), it waits as long as Retry-After says, in seconds or as a date, up to 30 seconds, before trying again, and retries at most three times.

Asking it to stop

If you connected your store, disconnect it in DPA Plus, or ask whoever connected it to.
Every preview page has a link to remove that preview right away, with no questions. From the same place, an address at your store's domain can block future previews of it.
To stop ProofsheetBot reading your store at all, email us at bot@proofsheet.ai from an address at your store's domain. We'll stop reading it and block future previews.