For website owners
ValidShopBot
ValidShopBot is the crawler behind ValidShop. It only visits a website when a person asks us to check that web shop, and it only reads public pages. It does not log in, submit forms, add products to a cart or crawl the whole site.
What it requests
- The homepage of the shop that a visitor asked us to check
- robots.txt and sitemap.xml
- The public returns, shipping, privacy, terms, contact and about pages that the homepage links to
- A TLS handshake to read the SSL certificate, and a public domain registration lookup (RDAP)
A check makes a few dozen requests at most, and the result is reused for 24 hours.
How to recognise it
Every request identifies itself with this user agent:
ValidShopBot/1.0 (+https://validshop.org/bot)How to block it
ValidShopBot follows your robots.txt. If you do not want your shop to be checked, add this to https://your-shop.com/robots.txt and we will not scan it:
User-agent: ValidShopBot
Disallow: /Bot protection and firewalls
We do not try to get around bot protection such as Cloudflare. If your site shows us a challenge page, we say that we could not read it, and we do not publish a score based on that page. To let ValidShopBot through, allow the user agent above in your firewall rules.