DistriovaBot

DistriovaBot is the crawler of Distriova. It looks at a small number of pages on behalf of our customers: their own homepages, and the directory pages where their products are listed.

How to recognize it

Every request carries this user agent:

DistriovaBot/1.0 (+https://distriova.com/bot)

What it fetches

A customer's homepage, to read it once when they add their product (for the listing draft) and to check that their badge wall is in place.
Directory listing pages of our customers' products, to see whether a listing is up and how it links to the product.

It doesn't follow links across a site, submit forms, run scripts or download media.

How often

After a submission, until the listing appears: every 3 hours for two days, every 6 hours until day 7, daily until day 30, weekly until day 90, then never.
After a listing goes live: on days 1, 7, 14, 30, 60 and 90. A missing or changed link is looked at again the next day.
Badge walls: every few hours while a customer's directories are being submitted, daily while listings are in review, weekly for 90 days after the last one goes live.

Politeness

At least 6 seconds between two requests to the same host, across all our servers.
It reads robots.txt (cached for a day) and follows Disallow rules for DistriovaBot or *.
It honors Crawl-delay up to 60 seconds.
Requests time out after 15 seconds; pages over a few megabytes are cut off.

To keep it out of part of your site:

User-agent: DistriovaBot
Disallow: /private/

A blocked homepage means we can't confirm a customer's badge wall automatically; an operator checks it by hand instead.

Contact

Questions or a problem with our crawler: [email protected].