Thought I'd better revisit my old script that crawls websites looking for mentions of bands, so as to affect their divis on popex, as the script is very old, and wasn't looking at robots.txt, oops... will do more on it today, but in the mean time it's interesting to note that only TOTP of all the sites I visit actually has a robots file... Also checking this:
Paul Clarke's blog - I live in A small town, Kent. Wed to Clare + dad to 2, I am a full-stack web developr, and I do js / Node, some ruby, other languages ect ect. I like pubbing, running, eating, home-automation + other diy stuff, history, genealogy, Television, squirrels, pirates, lego, and TIME TRAVEL.