>>>>> "LN" == Lazlo Nibble <lazlo@swcp.com> writes:
LN> Urrr, because that's the kind of problem that robot exclusion was
LN> designed to help avoid?
I have always been under the impression that robot exclusion was meant to
keep spiders from exposing things that I don't want exposed except to
people who come to my site looking. I don't mind if my archives are
indexed, in fact I welcome it. But it the indexers go overboard (as judged
by the users of those indices), why should I try to fix things for them?
LN> In the long run your list archives would likely be of more use to
LN> people if you defined a single clear web entry point and blocked the
LN> robots out of the raw archives themselves.
Actually, the indexes themselves would be of more use to people. My
archives (with their own targeted search engines) remain useful regardless
of who indexes them.
LN> It's not clear to me why you *wouldn't* want to do this,
Because I actually like having at least a piece of my archives indexed by
others? Because it's extra work? Yes, it is _only_ one file, but consider
that it is to my advantage to have some of my archives indexed (especially
recent ones). I actually tried to do this by excluding sections of the
archives, then realized that the indexes just aren't expiring the old
stuff. So I end up back where I've started minus all of the time I've
wasted.
LN> feel this is a moral axe worth grinding
Well, it would help if the indices were designed to avoid it themselves.
After all, why should I be concerned about their quality? Indexers who
strive to maintain quality will eventually learn to take care of these
things.
LN> simply don't care about the implications of running your archives this
LN> way -- which is your own business, I guess, but it strikes me as
LN> unnecessarily antisocial.
I leave my archives open for all to come and browse. Being open is
antisocial? I just don't see it.
And even if I turned on robot exclusion today, the old stuff would still be
in there. I turned on exclusion for one of my lists many months ago, yet a
search for a troubling topic today turns up my site as the sixth best hit.
- J<
References:
|
|
