robots.txt

robots.txt

Thought I'd better revisit my old script that crawls websites looking for mentions of bands, so as to affect their divis on popex, as the script is very old, and wasn't looking at robots.txt, oops... will do more on it today, but in the mean time it's interesting to note that only TOTP of all the sites I visit actually has a robots file... Also checking this:

+Dotmusic +nme.com +The Fly +Barfly +nme.com reviews +TOTP +Billboard +Bizarre +3AM +BBC +Ananova +Yahoo Entertainment

💬 NME.com news

⬅️ :: Ed Harcourt ➡️
Mon Sep 01 2003

Paul Clarke's blog - I live in Hythe, Kent. Wed to Clare + dad to 2, I am a full-stack web developr, and I do js / Node, some ruby, other languages ect ect. I like pubbing, running, eating, home-automation + other diy stuff, history, genealogy, Television, squirrels, pirates, lego, and TIME TRAVEL.