RE: Crawler
Home
Sending out a crawler today to search Google for things I've been talking about then to request blog entries that other people are talking about... I'm setting this blog entry as the referer, so if you see this page in your logs, that's why!
If you have a comment about this, then click on reply / comment, duh... if you just want me never to crawl your site again, put:
User-agent: *stokeybot*
Disallow: /
in your robots.txt file... I think that'll work, can't find any reliable PERL robots.txt code that I can install on this box...
2003 :: Comment / reply
and I got one today from clarkeology.com
My site is utle.com
airplay :: 25 Nov :: 2003 :: Comment / reply
This link:
clarkeology.com/stokey/51/9/
...was in my referrers for my site (LotusGeek.com).
Curiouser and curiouser ;)
LotusGeek.com
rock :: 16 Nov :: 2003 :: Comment / reply
I'm using the Google API's to query Google for pages related to my pages...
13 Nov :: 2003 :: Comment / reply
I actually got a hit with referer URL
clarkeology.com/stokey/48/8/
so you might want to check your crawler.
(which seems to work, as I did mention concert tickets as well lately)
capzilla :: 13 Nov :: 2003 :: Comment / reply
worked quicker than I thought, thanks for the feedback. Should have passed in both pages as a refererer, this one once (when looking for robots.txt) and then the page of mine that's related to the topic of yours... which site were you then?
13 Nov :: 2003 :: Comment / reply
Paul Clarke's blog - I live and work near Folkestone. Married to Clare and father to Harry, I am a web dev, and I like javascript, PHP, Folkestone Gerald, pubs, restaurants, history, genealogy, TV, music, popbitch, squirrels, pirates ☠, time travel, stained glass...
Also from the same week
Some more posts from the same week as .
David Devant [Saturday 22 November 2003]
stellastarr* [Monday 24 November 2003]
Achoo [Monday 24 November 2003]
Baby, baby, where did the month go? [Monday 24 November 2003]
RE: stellastarr* [Monday 24 November 2003]
RE: stellastarr* [Monday 24 November 2003]
Property prices falling in Stokey [Monday 24 November 2003]
Todays tasks [Monday 24 November 2003]
A public holiday to celebrate us winning the world cup? [Monday 24 November 2003]
stellastarr* [Tuesday 25 November 2003]
Stellastarr* THEY'RE ACE. [Tuesday 25 November 2003]
Perl / Excel [Tuesday 25 November 2003]
Perl and Excel [Tuesday 25 November 2003]
RE: Crawler [Tuesday 25 November 2003]
Daily productivity [Tuesday 25 November 2003]
Maybe Stokey prices aren't falling... [Tuesday 25 November 2003]
Where does all this water come from? [Wednesday 26 November 2003]
Want some shoes? Oh, and tickets to a cheeky ESP gig... [Wednesday 26 November 2003]
Darkness download! [Wednesday 26 November 2003]
Leaky roof [Wednesday 26 November 2003]
RE: stellastarr* [Thursday 27 November 2003]
Ongoing Excel / PERL stuff [Thursday 27 November 2003]
Siegfried and Roy [Thursday 27 November 2003]
Hands off our 73! [Thursday 27 November 2003]
Bus route 106 [Thursday 27 November 2003]
Too many fields appear to be blank [Thursday 27 November 2003]
How to convert an Excel date to a UNIX date [Thursday 27 November 2003]
Very popular with Shoreditch types And another competition for you! You have all entered the win some shoes and tickets to a gig one haven't you? [Friday 28 November 2003]
IPOD's Dirty Secret [Friday 28 November 2003]
Popdex [Friday 28 November 2003]
Crawler [Thursday 13 November 2003]
RE: Crawler [Thursday 13 November 2003]
Lordy [Thursday 13 November 2003]
Google API's [Thursday 13 November 2003]
David Devant cancelled [Sunday 16 November 2003]
Here's the link that is showing up in my referrers... [Sunday 16 November 2003]
War On 106 [Wednesday 19 November 2003]
Doh! [Friday 28 November 2003]
Quicktime [Sunday 30 November 2003]
Good work [Monday 1 December 2003]
RE: stellastarr* [Wednesday 28 January 2004]
Hi Moker [Thursday 29 January 2004]
RE: Hi Moker [Friday 30 January 2004]
Stellastarr new tour [Saturday 31 January 2004]
RE: stellastarr* [Tuesday 2 March 2004]
Upload a file [Tuesday 8 November 2005]
