Archive for the ‘ Sitecore ’ Category


I’ve recently been asked to provide the ability for our Sitecore users to manage the robots.txt file on our site. The way our environment is configured, modifying a file on the file system requires a formal deployment procedure, and it seemed like too much overhead involved to make the occasional file edits.

I thought about a couple possibilities: use an Alias or Proxy pointing to the file stored in the Media Library, or create an Item in the content tree named “robots.txt”. I tried several variations of Aliases and Proxies, but couldn’t get any of them to work.

Note: this solution was implemented in Sitecore 6.2, therefore, your mileage may vary.

It seemed like the easiest solution would be to allow the users to upload their text file to the media library and create an Item in the tree to serve the contents of the file. This solution not only solves the robots.txt issue, but also provides an easy way to serve the contents of plain text files in a managed fashion.

The first thing that needs to be done is to tell Sitecore to allow requests for .txt files to be processed in the pipeline. In web.config, add “txt” to the list of allowed extensions that will be handled by the PreProcessRequestProcessor:

<processor type="Sitecore.Pipelines.PreprocessRequest.FilterUrlExtensions, Sitecore.Kernel">
 <param desc="Allowed extensions (comma separated)">aspx,ashx,asmx,svc,txt</param>

We will need to create a simple Layout and Template to represent our plain text files in the content tree.

Create a new Template and give it a single field of type “File”. I called the field “Text Content” and saved the template as “Plain Text Page Template”.

Next, create a simple aspx-based layout in the Layouts folder named “Plain Text Layout”. I chose not to create a code-behind, since the markup and code is dead-simple.

<%@ Page language="c#" AutoEventWireup="true" %>
<%@ Import Namespace="Extensions" %>
<script runat="server">
 void Page_Load(object sender, EventArgs e) {
     Response.ContentType = "text/plain";
     var item = Sitecore.Context.Item.AsMediaItem("Text Content");
     if (item == null)

You may have noticed that I’m importing an “Extensions” namespace. This namespace contains some Sitecore Item extensions to make life easier when dealing with the content tree. In the code above, I’m using the AsMediaItem extension function

public static MediaItem AsMediaItem(this Item item, string fieldName)
    FileField file = item.Fields[fieldName];
    if (file == null || file.MediaItem == null)
        return null;
    var mediaItem = new MediaItem(file.MediaItem);
    return mediaItem;

Now we need to bring the template and the layout together in the content tree. Create a new item in the content tree, named “robots”. Use the “Plain Text Page Template” and the “Plain Text Layout”. In the “Content” section, browse to the actual robots.txt file in the media library, and save the item.

You should now be able to browse to /robots.txt and get the plain text contents of the robots.txt file that lives in the Media Library.