Robots and Sitemap

By Naji El Kotob.

Information Technology Consultant

_______________________________

naji [@] DotNETHeroes.com

Review 1.0.1 - 13

.

Business Intelligence Group BIG SARL

by Naji El Kotob

Outlines

SEO

Robots.txt

Sitemap.xml

Q&A

by Naji El Kotob

SEO

by Naji El Kotob

What is SEO

SEO is short for search engine optimization or search engine optimizer.

Search engine optimization is a methodology of strategies, techniques and tactics used to increase the amount of visitors to a website by obtaining a high-ranking placement in the search results page of a search engine (SERP)

-- including Google, Bing, Yahoo and other search engines.

Source | http://www.webopedia.com/TERM/S/SEO.html

by Naji El Kotob

Robots.txt

by Naji El Kotob

Robots.txt

Robots.txt is the file that Search Engines use to see what they should index.

This file and site maps help make your site easier to navigate by the bots and let them know what is legal and what you would rather not have the published in their engines.

by Naji El Kotob

Robots.txt

by Naji El Kotob

Robots.txt Syntax

User-agent: *

Disallow:

Sitemap: http://dotnetheroes.com/sitemap.xml

by Naji El Kotob

Robots.txt Generator

http://tools.seobook.com/robots-txt/generator/

by Naji El Kotob

Interactive Demo

by Naji El Kotob

Demo

Design and Create a Dynamic RobotsSource | http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/

Adding a Dynamic Robots.txt file to an ASP.NET MVC siteRobots.txt is required to allow search engines to properly index your site, and more importantly not index it.

If you have a public-facing staging or preliminary site that you don’t want to show up in Google results, you need to make sure that it returns the correct robots.txt with the

Disallow: /

line to prevent indexing. However, manually adding robots.txt files to staging and production environments as a manual process can be improved with the process below – the same code can serve up a locked down robots.txt in staging or internal URLs, and allow indexing in production.

by Naji El Kotob

Demo (cont'd)

You’ll also need to make sure that runAllManagedModulesForAllRequests is true in web.config as normally text files bypass the ASP.NET pipeline:

<system.webServer>

<modules runAllManagedModulesForAllRequests="true"></modules>

...

</system.webServer>

by Naji El Kotob

Demo (cont'd)

routes.MapRoute("Robots.txt","robots.txt",

new{

controller = "Robots",action = "RobotsText"

});

by Naji El Kotob

Demo (cont'd)

The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will go here:

public class RobotsController : Controller

{

public FileContentResult RobotsText()

{

var contentBuilder = new StringBuilder();

contentBuilder.AppendLine("User-agent: *");

// change this to however you want to detect a production URL

var isProductionUrl = Request.Url != null && !Request.Url.ToString().ToLowerInvariant().Contains("elasticbeanstalk");

if (isProductionUrl)

{

contentBuilder.AppendLine("Disallow: /elmah.axd");

contentBuilder.AppendLine("Disallow: /admin");

contentBuilder.AppendLine("Disallow: /Admin");

contentBuilder.AppendLine("Sitemap: http://www.mysite.com/sitemap.xml");

}

else

{

contentBuilder.AppendLine("Disallow: /");

}

return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");

}

}

by Naji El Kotob

Sitemap

by Naji El Kotob

Sitemap

Sitemap (XML) is a list of pages on your Website. It is easiest way to give Google, and other search engines, information about your site and may speed up Google’s crawlers finding you.

There are many advantages of submitting a Sitemap to Google, especially if your site is new and doesn’t have many backlinks.

Source | http://diywebjem.com/webdefinitions.html

by Naji El Kotob

Sitemap Format

<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<url>

<loc>http://www.example.com/</loc>

<lastmod>2005-01-01</lastmod>

<changefreq>monthly</changefreq>

<priority>0.8</priority>

</url>

</urlset>

Source | http://www.sitemaps.org/protocol.html

by Naji El Kotob

Sitemap Generators

There are hundreds of online tools that can be used to create an XML sitemap that can be submitted to Google, Bing, Yahoo and other◦ E.g.

http://www.xml-sitemaps.com/

http://www.sitemapdoc.com/

And more…

by Naji El Kotob

References

http://geekswithblogs.net/jjulian/archive/2012/04/25/adding-robots.txt-to-your-asp.net-mvc-3-applications.aspx

http://tools.seobook.com/robots-txt/

http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/

by Naji El Kotob

QnA?

by Naji El Kotob

Thank You

Please send your feedback and suggestions to:◦ naji [@] DotNETHeroes.com

Join us at www.facebook.com/DotNETHeroes