(just joined and have tried to pick the thread up off the archives)
My first reaction to the first message in this thread was much like others - set
up a robots.txt file. But then it occurred to me that this really isn't ideal -
until perhaps robots.txt files are tuned to allow you to indictate things like
list archive ???
The real problem is that whilst search engines seem to be improving their
features - one of the most annoying things they don't seem at all good at is
owrking out that those 1000s good matches with really similar URLs are probably
not all usefully provided at the top of the "best matches" list.
--
On a related note:
Is there a list anywhere of examples of web based archiving? We've spent (are
spending) a fair while putting together a fairly sophisticated/customised system
on top of majordomo (it makes the list a web conference area too) and I'd like
to see if there's anything else there to compare it with we haven't seen yet.
Manar
Follow-Ups:
|
|
