what is a robots.txt?
a robots.txt file is a file that tells search engines weather what pages to crawl and what not to crawl or block access to some specific pages and directory.this is very helpful for websites that want to block access to some important pages on thier websites.simply you are telling a search engine not to index those pages.
default robots.txt file for blogpsot blogs:
the default robots.txt file for blogspot blog looks just like this one;
Sitemap: http://www.handmadereviews.net/feeds/posts/default?orderby=UPDATED
this file has been causing errors as reported by some bloggers.so google introduced a new custom robots.txt feature for blogspot blog where you can set the robots.txt file confiqured by yourself.
how to set custom robots.txt for blogspot blogs.
there are three main components of blogger robots.txt file.
1-google adsense crawler
2-sitemap file
3-user agent(search engines)
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
this is the complete and best confiquaration file for custom robots.txt for blogspot blogs.it allows all pages to be index expect search labels.you will have better crawling and fast indexing for blogspot blogs.
Jillur Rahman
I'm Jillur Rahman. A full time web designer. I enjoy to make modern template. I love create blogger template and write about web design, blogger. Now I'm working with Themeforest. You can buy our templates from Themeforest.
No Comment to " HOW TO SET CUSTOM ROBOT.TXT FOR BLOGGER BLOG "