blogger custom robots.txt file has allowed us to block or diallow any folders or pages on our website or blog by our own selves.we just have to edit custom robots.txt file and rest of work will do crawlers.
by default,custom settings for robots.txt for blogger is as follows.
User-agent: Mediapartners-Google
Disallow:
user-agent: *
disallow: /search
allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
where as the the terms are explained as;
user-agent is for all robots.do not change this code or you will have troubles and troubles.
google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.
disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.
latest update:
after google pagination update,we use the following configuration file to block google pagination page.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
by default,custom settings for robots.txt for blogger is as follows.
User-agent: Mediapartners-Google
Disallow:
user-agent: *
disallow: /search
allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
where as the the terms are explained as;
user-agent is for all robots.do not change this code or you will have troubles and troubles.
google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.
disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.
latest update:
after google pagination update,we use the following configuration file to block google pagination page.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
Jillur Rahman
I'm Jillur Rahman. A full time web designer. I enjoy to make modern template. I love create blogger template and write about web design, blogger. Now I'm working with Themeforest. You can buy our templates from Themeforest.
No Comment to " UPDATED ROBOTS.TXT FOR BLOGGER. "