Robots.txt file is one of the most important part of website configuration.Minor mistake in this file can block search engines from crawling your website.So care should be taken while editing this file.In the title of the post I warned that /s or /s/ should not be added in robot.txt file.I made this mistake myself.I had one subdirectory names s.udinra.com to serve my static contents.So I wanted to exclude that directory from being crawled and added Disallow rule to /s in robots.txt file.
I made this change and forgot.After few days I logged into Google webmaster and Bing webmaster.I saw that there is big red cross in my sitemap status.This never happened to me as I am using two best plugins to create my sitemaps (Google XML Sitemap) and Udinra All Image Sitemap (for image sitemaps).I resubmitted the sitemap and received same message.
My robot.txt file was restricted the access to both the files (sitemap.xml and sitemap-image.xml).You can know the reason by clicking on the sitemap.
I then deleted the line Disallow /s from my robots.txt file and resubmitted the sitemaps again.It worked fine and there was big green sign.Having faced the issue I thought of sharing this so that others can avoid the issue.Do share the post it may be helpful to someone else.
You may also like:
