Nick

Reply from Dreamhost:

—

As of lately we have been working on our very heavy usage customers (people using over a quarter to in some cases 250% of what the full server should be processing). In excess of half of these cases the cause is google’s crawler malfunctioning in how it interacts with the site resulting in heavily loaded or even crashing machines (I have had cases where googlebot has over 95% of the last 10,000 hits on a site that doesn’t even have that many pages). Our terms of service
(http://www.dreamhost.com/tos.html) specifically state that “if your processes are adversely affecting server performance disproportionately DreamHost Web Hosting reserves the right to negotiate additional charges with the Customer and/or the discontinuation of the offending processes” so that we can ensure that we keep machines working for everyone on them
and not just one user. In a case where a faulty googlebot interaction is killing a machine we have two options:

1. disable the site
2. block the bot

We feel that the best solution for our customers is to stop just the malfunctioning behavior and keep _everyone’s sites working as well as possible on the machine. The blocks are always removable for any customer, but we just ask them to slow down the crawl speed of the bot by
going to the following address:

https://www.google.com/webmasters/tools/siteoverview

This will allow Googlebot to run on our servers without any problems, thus no block needed. We have been working with a Googlebot engineer on this issue as Google is aware of this issue themselves, but until then we’re trying everything possible to keep all customers services up and running with other people bring down a whole server due to googlebot
malfunctioning on a dynamic page that causes high loads.

Let me know if you have any other questions.

Thanks,
Mike P.