... wrong, mostly they're reasonable, but because a default you never chose becomes a strategy by attrition. This one has a date on it. Comment from Omri Ben-Shoham(@omri_ben_shoham1): Good writeup - the distinction between blocking Training vs blocking Agent is the part people are going to get wrong for a while. One thing worth adding from the operating side of an agent, not the site-owner side - when an agent gets blocked, it usually doesn't fail ... ... site with Googlebot's user agent, and do it against your ugliest URLs rather than the homepage. The homepage is the page everyone tests and the page that is always fine. Mine broke only on paths containing a dot