Information for Providers

ServerDB crawler bot

ServerDB.cc runs an automated bot that visits provider websites on a regular basis to read pricing, availability and specifications. Most offers are refreshed roughly once every 24 hours, though the exact interval depends on how often your catalogue changes. Between requests the bot waits 1 second by default. You can raise or lower that pause with Crawl-delay, as shown further down. We only ever crawl with your permission: the bot uses fixed IP addresses, sends an identifying user agent and honours your robots.txt. Questions are welcome at any time.

Permitting access

In most cases the bot reaches your site without any changes on your side. If something blocks it, check the following:

1. If you operate a firewall or use a proxy such as Cloudflare, allowlist the bot's source IP addresses.

If you maintain a robots.txt, permit our bot by adding:

User-agent: ServerDBBot
Allow: /

Limiting the request rate

The bot waits 1 second between requests by default. To request a different interval, add the following to your robots.txt, where 10 is the delay in seconds:

User-agent: ServerDBBot
Crawl-delay: 10

Stopping the bot

Before blocking us entirely, we would love to hear from you so we can fix the underlying issue. If the bot is causing problems right now and you must stop it, add this to your robots.txt:

User-agent: ServerDBBot
Disallow: /

The bot caches robots.txt for 1 hour, so changes are not applied instantly.

Bot identity