> Why should I care if some over-zealous spider went through my entire
> archives and added them to its index? It is they who aren't serving
> their customers well by doing this; my search engine works fine.
>
> Yes, I know about robot exclusion, but why should I have to?
Urrr, because that's the kind of problem that robot exclusion was designed to
help avoid?
In the long run your list archives would likely be of more use to people if
you defined a single clear web entry point and blocked the robots out of
the raw archives themselves. This would make life easier for most of the
people involved, possibly including yourself. It's not clear to me why
you *wouldn't* want to do this, unless you a) feel this is a moral axe
worth grinding or b) simply don't care about the implications of running
your archives this way -- which is your own business, I guess, but it
strikes me as unnecessarily antisocial.
--
::: Lazlo (lazlo@swcp.com; http://www.swcp.com/lazlo)
Follow-Ups:
|
|
