Optimize Your On-Page – Robots and Sitemaps (Part II of II) virginiaseo.org /blog/optimize-your-on-page-robots-and-sitemaps-part-2/ Chris Dill T his post is part two in a series, please read part one: Optimize Your On-Page – Meta Tags, Technical Analysis (Part I of II)
On-Site Content We have already discussed on-site content in our previous video series sessions, so I will not dive deep into this. When I analyze a website, I look at the website f rom its highest page (normally the home page) and then take a look at the content. I then navigate through the site, and absorb the content as I go. Af ter doing this f or 10 or so minutes, I ask myself a couple of questions: Is the content consistent throughout the site? Is all of the content on-topic? Does the content match the meta tags and keyword the site is using? Is the content hard to understand? By answering these questions, we can get a f eel f or the overall content of the website, and develop or modif y a strategy f rom there. If all of your meta tags are aimed at optimizing your business f or local traf f ic, but all of your content is optimized f or national traf f ic- there is going to be a conf lict of interest, so to speak.
Sit emap.xml A sitemap is literally a map of your site. It contains all pages and is organized in a logical structure. When search engines crawl your site, they look at your sitemap f or inf ormation and to ensure all of your pages get crawled. Sitemaps can also contain inf ormation about the f requency in which your site is updated. T his tells Google that they should crawl pages that change a lot (like your blog roll) more of ten than static pages (like your about page). T here are a lot of sitemap generators online, and we will discuss one f ree website based one, and then we will look at a f ew WordPress Addons.
XML-Sit emaps.com- ht t p://www.xml-sit emaps.com/ T his site is self -explanatory- plug in your website and hit start and it will generate the sitemap you need. T his tool is great f or a one time look or creation, but you will have to manually update this sitemap as you add content, which is a little bit tedious. So I pref er the use of addons- and every major and minor CMS has one or more. Once you generate your sitemap with this tool, it needs to be located at the root of your website. Speak with your web hosting company or web
designer if you’re not comf ortable with this, but uploading the XML f ile using FT P to the site root is pretty quick and painless. If you need us to help you out with this we will do it f ree of charge, just send us a note! T he ending result needs to be domain.com/sitemap.xml T he best tool f or WordPress sitemaps is of course Yoast. Install the Yoast WordPress plugin and go to the sitemap tab. Conf igure the f ew options there and you are of f running. T he generated sitemap can then be viewed by clicking the link at the top of the setting page: Another tool if you are looking ONLY f or sitemaps (let’s say you already use another SEO suite addon) is Google XML Sitemaps, by Arnee . Install this plugin and go to the plugin settings page and you will be walked through how to set it up. So by using these addons, you generate a sitemap which is automatically placed at the root of your site. Now in order to make sure the major search engines (in this case Google and Bing) crawl your sitemap, you need to submit it. Go to Google Webmaster Tools > Crawl > Sitemaps. To add your sitemap, you click the Add/Test Sitemap button and then tell Google where your sitemap is. Give Google 2 days to crawl it and come back and you will see metrics about how many pages have been submitted to the Google Index, and how many Google actually has in the index. T hese values should be the same. If we look at Colonial Diving Schools sitemap, we see that there are many more submitted than are in the index. T his is due to the event system which Colonial uses, and should be solved with some exclusions f or the sitemap.xml and also robots.txt, which we will discuss now.
Robot s.t xt Robots.txt is a special f ile that Google and other search engines read and respect. If you use this document to say “Do not crawl pages A, B, and C” Google makes a very good attempt at f ollowing those rules. T his is important to keep certain pages f rom showing up in search. Some great examples of pages that you might not want to show up in search are: Login Pages Registration Pages Cart Pages Certain Forms Pages that are necessary f or your site but not accessed by visitor T hese pages might be needed f or your site- f or instance my site needs a login page, in order f or customers to log in. Without it the UX is severely hampered. But I do not want Google to put my login page into search results- so I ask nicely using my robots.txt. Every website should have one of these, even if it is not used to restrict anything. You can f ind an awesome post about robots.txt on Yoast.com. Google has a help post on this, and they also of f er you some pointers on making a “def ault” robots.txt. Designing a robots.txt is done on a per site basis, but here is the sample that I use f or Colonial Driving School:
# robots.txt
User-agent: * Disallow: /wp-content/plugins/* Disallow: /login/ Disallow: /login/* Sitemap: http://colonialdrivingschool.com/sitemap_index.xml
As you can see I am blocking crawlers f rom reaching the login page, I am blocking plugins f rom presenting web pages to the crawler, and I am also declaring where the sitemap is. Let’s say that I do some research and I notice that a certain page of the site is being crawled but coming up with errors. Typically you see this with e-Commerce sites which have dynamic cart and product creation- you might see a bunch of 404 errors f or a product sorting query page. T his sort of issue can be solved by adding that page to robots.txt so Google knows not to crawl them.
SEO Expert at Virginia SEO Christopher Dill is a Christian entrepreneur who loves web design, marketing, and anything on a computer. He is the creator and author of T he Dill Design, a local Virginia web design company. He also runs Virginia SEO, which is a SEO and inbound marketing company. Chris is currently f inishing up his Master of Inf ormation Systems at University of Phoenix, and works by day as a Senior Network Engineer. 44
Lat est post s by Chris Dill (see all) A Fight To T he Death: Inbound Vs. Outbound Marketing - April 13, 2014 Optimize Your On-Page – Robots and Sitemaps (Part II of II) - April 7, 2014 SEO Video Training – Session 3: On-Page, Meta Tags, Robots, and Sitemap.xml - March 31, 2014