[squeak-dev] SqueakSource indexability (aka should we just ask crawlers to desist?)

Ken Causey ken at kencausey.com
Wed Apr 28 19:07:34 UTC 2010

At times access to source.squeak.org becomes slower, as has been the
case today.  I can see in the logs that various web-crawlers are the
likely culprit.  Having the information there accessible via search
engines is a wornderful thing but I have to suspect that the Seaside
session IDs eliminate this option.  (Of course when URLs like
http://source.squeak.org/trunk.html are found on other sites they then
become indexed.)

Unless I'm mistaken about this, and I would appreciate any guidance, it
seems like we need to add a robots.txt to the site which guides or
simply asks crawlers to stay away.  Thoughts?  I'm no SEO export.


More information about the Squeak-dev mailing list