[squeak-dev] SqueakSource indexability (aka should we just ask
crawlers to desist?)
Ken Causey
ken at kencausey.com
Wed Apr 28 19:07:34 UTC 2010
At times access to source.squeak.org becomes slower, as has been the
case today. I can see in the logs that various web-crawlers are the
likely culprit. Having the information there accessible via search
engines is a wornderful thing but I have to suspect that the Seaside
session IDs eliminate this option. (Of course when URLs like
http://source.squeak.org/trunk.html are found on other sites they then
become indexed.)
Unless I'm mistaken about this, and I would appreciate any guidance, it
seems like we need to add a robots.txt to the site which guides or
simply asks crawlers to stay away. Thoughts? I'm no SEO export.
Ken
More information about the Squeak-dev
mailing list
|