Blog ≫ 2003 ≫ Crawler

Sending out a crawler today to search Google for things I've been talking about then to request blog entries that other people are talking about... I'm setting this blog entry as the referer, so if you see this page in your logs, that's why!

If you have a comment about this, then click on reply / comment1, duh... if you just want me never to crawl your site again, put:

User-agent: *[stokey](/wiki/#stoke/newington)bot*
Disallow: /

in your robots.txt file... I think that'll work, can't find any reliable PERL robots.txt code2 that I can install on this box...

💬 RE: Crawler - 514

💬 Google API's

💬 Here's the link that is showing up in my referrers...

💬 RE: Crawler - 560

⬅️ :: ➡️
Thu Nov 13 2003

Paul Clarke's weblog - I live in Hythe, Kent. Wed to Clare + dad to two, I am a full-stack web engineer, + I do javascript / Node, some ruby, other languages ect ect. I like pubs, running, eating, home automation + other diy stuff, history, family tree stuff, TV, squirrels, pirates, lego, and TIME TRAVEL.