Crawler
Home
Nov13
Pictures
Sending out a crawler today to search Google for things I've been talking about then to request blog entries that other people are talking about... I'm setting this blog entry as the referer, so if you see this page in your logs, that's why!
If you have a comment about this, then click on reply / comment, duh... if you just want me never to crawl your site again, put:
User-agent: *stokeybot*
Disallow: /
in your robots.txt file... I think that'll work, can't find any reliable PERL robots.txt code that I can install on this box...
2003 :: Comment / reply
Pictures
and I got one today from www.clarkeology.com
My site is utle.com
airplay :: 25 Nov :: 2003 :: Comment / reply
Pictures
This link:
www.clarkeology.com/stokey/51/9/
...was in my referrers for my site (www.LotusGeek.com).
Curiouser and curiouser ;)
www.LotusGeek.com
rock :: 16 Nov :: 2003 :: Comment / reply
Pictures
I'm using the Google API's to query Google for pages related to my pages...
13 Nov :: 2003 :: Comment / reply
Pictures
I actually got a hit with referer URL
www.clarkeology.com/stokey/48/8/
so you might want to check your crawler.
(which seems to work, as I did mention concert tickets as well lately)
capzilla :: 13 Nov :: 2003 :: Comment / reply
Pictures
worked quicker than I thought, thanks for the feedback. Should have passed in both pages as a refererer, this one once (when looking for robots.txt) and then the page of mine that's related to the topic of yours... which site were you then?
