Robot Exclusion Enforcement
Posted on July 27, 2008
Filed Under Bots, Web Programming, seo |
With thanks to Paul Ford for permission to write a sequel to the original “Robot Exclusion Protocol” short story. I hope I haven’t done the original any disfavours.

I was on a job when I saw it. A couple of feet tall, made of metal, with camera lenses for eyes and a few dozen gripping arms. A Googlebot! If it wasn’t for the stakeout you wouldn’t have caught me in that part of town. It certainly wasn’t a suitable neighbourhood for a Googlebot. I pulled my coat more closely around me, hunkered down further behind the bins in case it decided to index me and blow my cover, and settled down to watch.
It was clearly here for a reason. It scuttled straight up to the doorway of… well, let’s just call it a dancing establishment and leave it at that… until it found its way blocked by a pair of legs. It looked up and up and up at the burly enforcer blocking the doorway.
“Hi! I’m from Google. I’m a Googlebot!. … . I will, not… kill … you?”
“I know what you are and I know where you go. All Googlebots enter there” he said disdainfully, pointing at a small two foot high doorway to the left of the main doorway marked “Bot Entrance“.
“Already been there, indexed many interesting facts, many interesting subjects. Take back to Plex. Plex doubt ‘bot. Plex quite insulting in fact. Plex say ‘get confirmation’. I threatened with 486. No coprocessor. Must enter here. Index confirmation.”
Longest speech I ever did hear from a G-bot. I could almost feel its pain.
“No bots through here,” the enforcer said, “Clear off before I let you index my foot.”
The bot bent over and peered closely at the mentioned foot. “Indexed many similar before. Duplicate. Not interesting“. It got little more than a growl in return.
“You… you don’t sc…scare me. I come back with b… brother bot, spends 20% time indexing pages on Dimac”
“Yeah. You do that. Just clear off before I use you to index the toilets.”
Surprisingly the G-bot scuttled off round the nearest corner and down an alley. Never come across one with any sense of self-preservation before. Normally they’re more tenacious than the taxman. A couple of minutes later, from the gloom of the alley came the most almighty clanking, grinding and thumping noises. The enforcer wandered over and peered round the corner of the building.
Immediately the G-bot appeared from the opposite corner, having done a circuit of the building, and darted into the open doorway. A flicker of laser light bounced off its metallic shell.
“Height suspicious. Warrants further investigation” a robotic voice boomed. A couple of robotic arms popped out of the walls and held the struggling Googlebot while some sort of scanner was passed over its entire surface.
“Beanbag residue detected,” went the voice, “Classification: 98.6% chance Googlebot. Eject!”. The robotic arms flickered and suddenly the G-bot was landing on the pavement. The enforcer gave it a kick as he took up his post again. The G-bot got to its feet and left and I resumed my original observation.
Five minutes later I heard noises coming up behind me. I turned just in time to see three G-bots before they swarmed all over me tugging at my coat. I gave it up, I knew better than to struggle. The three G-bots leapt off me to stand on one anothers’ shoulders almost reaching my height. The top bot pulled my coat around its shoulders like a cloak, reached out, took my hat and put it on and, thus hidden, they set off towards the doorway.
I couldn’t believe he would be so easily fooled but the enforcer stepped aside. A brief flicker of laser light… you’d have had to have been watching closely to notice how the G-bots broke step… and then they were in.
And thirty seconds later out they were out again. Split up, chattering excitedly to each other, they swarmed past the enforcer who didn’t know which one to aim his foot at first.
Two of the bots scuttled off immediately, the third headed towards me. I got ready to run.
“Thanks” called the bot as it got close, veered off and followed its brother bots.
I never did get my coat back though.
Comments
One Response to “Robot Exclusion Enforcement”
Leave a Reply

Great story, it brought a big smile to my face.