Webcrawlers, again
Submitted by xqus on Mon, 10/10/2005 - 11:50
I can't help it. I like, or disslike crawlers. I have written one, I have tracked them, and I have traped them. It's fun. It's challanging.
What's so fun about them? There are good spiders, there are bad spiders, there are spamming spiders, there are email harversters, and there are probably alot of useless spiders, like mine.
The challange is: what to allow, what to block?
I have started a project that aims to gather information about the behaviour of webspiders. Good and bad.
Oh.. and here is the link: SpiderWatch.
Have fun!
Tags:
Post new comment