Lee Kelleher’s Weblog

Last night I released version 2 of the Robots.txt Editor for Umbraco.

Changes & new features:

• Restructured the package files, they all live in a folder called “robots-txt” (still under the /umbraco folder – but self-contained)
• Errors are now displayed using the Feedback control (as opposed to the Speech Bubble in the bottom-right corner).
• Robots.txt editor has buttons for adding new User-Agent and Disallow rules, as well as comment/uncomment functionality.
• Changed the Robot icon… courtesy of Mozilla Firefox’s “about:robots” favicon.

Hope you all enjoy the update… if you come across any bugs or ideas for a future release, please let me know via the our.umbraco forum.

Following up on my recent post of using Robots.txt with Umbraco, I decided that it would be nice to be able to edit the robots.txt directly from the Umbraco back-end.  (Also I wanted to play a bit more with the BaseTree/ITree classes).

This afternoon I had a few hours to spare – actually I was procrastinating on another job, (don’t tell my client – I’ll finish it off later tonight) – so I got down to some coding.

The source-code is available on the Umbraco Extensions project (on CodePlex) and created a project page on the new Our Umbraco community website. (Don’t forget to give it some karma points! ;-p)

A direct download to the package installer (zip) is available here.

Very special thanks to Dave Kinsella for providing the Robot icon – although I know Dave, it was great proof that Twitter really does work! (Thanks to Dan too for his Johnny 5 attempt! – Here was my attempt too!  )

If you have any bugs, comments, feedback or suggestions – please feel free to get in touch with me via the Our Umbraco forums.

I originally posted this over at the Our Umbraco community wiki. [Robots.txt for use with Umbraco] I am only posting it on my blog as a cross-reference. The Our Umbraco wiki version will evolve with the community’s experience and knowledge.

The Robots Exclusion Protocol has been around for many years, yet there are a lot of web-developers who are unaware of the reasons for having a robots.txt file in the root of their websites.

There have been many rumours around whether the bigger search engine crwalers (i.e. Googlebot) consider your website amateurish if you didn’t have a robots.txt – and if handled badly, could lead to your site being invisible on SERPs.

If you are happy for a crawler to crawl/index all of your website’s content, then you can use the following:

User-agent: *
Disallow:

However, when using Umbraco to power my websites, it is preferable to define which folders are accessible by the crawler. Personally, I would not like to see the contents of my /umbraco/ folder to be returned in Google’s SERPs.

Here is an example of the robots.txt that I have used on several Umbraco-powered websites.

# robots.txt for Umbraco
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /css/
Disallow: /data/
Disallow: /scripts/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/

From my perspective, there is no reason for a search engine crawler to be crawling/indexing files from any of the above folders – you may have a different perspective, to which you can amend your robots.txt accordingly.

For more information about the robots.txt standard, please refer to the official website: http://www.robotstxt.org/robotstxt.html