Google not crawling indexing my site reasons solutions

Google is the main traffic generator for most of the websites.After launching a new website or blog website owners want their content to get indexed by Google.Lets understand the terms Indexing and Crawling before diving into details of Google not crawling and indexing your site issues.

Google is has to know about your website so that it can list your website pages on it.To know about your website content Google have to visit your website.This process is known as Crawling.After visiting your site if it feels that your content is good then it will add your website pages into its list.The process of adding your content into its list is known as Indexing.

So crawling and indexing is closely related.Your first attempt is to make sure your website is easily discoverable by Google.There are too many websites out there so making it easy for Google to discover your website is must.Bloggers facing issues with Google crawling their website should make sure they do not have below mistakes on their website.Also they should follow the tips mentioned next in the post.

Blocking Google Bot

This is one of the major reasons Google not been able to crawl your site.If Google is not able to crawl your site then you do not stand a chance of appearing in Google Search.So the first step is to check your Robots.txt file.This simple file is used to hide your site from Search Engine bots or other bots.

Many a times we have wrong robots.txt in place.This blocks search engine bots.In case you are using WordPress then you can use below text in your robots.txt file.In below text just mention link to your sitemap file.

User-agent: *

Allow: /
Disallow: /wp-content
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-include

Sitemap: your sitemap file link here

We are instructing bots not to see content inside wp-content,cgi,wp-admin and wp-include.In case you are using WordPress upload method and your images are uploaded to default directory (wp-content/uploads) then you should remove /wp-content line from above.

But I would recommend to have your uploads folder outside wp-content.

Wrong Configuration

Most content management system have option of hiding your website from search engines.You might have this option enabled making your site invisible to search engines.I would cover WordPress and Blogger here.These two are most popular blogging platform.


Blogger user should check their search preferences settings.You can find the search preference option under Settings option of Blogger dashboard.Search Preference should be to allow your website visible to search engines.This is the default option.In case you are just starting with Blogger then read this guide to get started with Blogger.


WordPress users should go to Reading option under Settings menu of WordPress dashboard.You will see the option Search Engine Visibility.Check whether this check box is not enabled.If it is enabled then disable the checkbox and save the settings.

Selecting WWW version or Non WWW version

Your website can be accessed using two URL

It is better to select one of the URL as preferred one.It can be easily achieved using .htaccess file.This file is available in your root directory of hosting account.You can paste below code in your .htaccess file to redirect www version to non www version.I prefer redirecting www to non www version. You can paste below piece of code in your .htaccess file at the end

RewriteCond %{HTTP_HOST} ^www.(.*)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

After making above changes it is time to verify your Content Management System settings.Every content management system offers an option to set your website address and URL.I would give example from WordPress.From WordPress dashboard go to General option under Settings.There you can set below two options

1.WordPress Address
2.Site Address

Make these two options without www.This clearly indicates Search Engine about your preference.Consistency about selection of the address should be uniform.

Wrong configuration of SEO plugins

SEO plugins if not used properly might backfire.It is important to configure your SEO plugins properly.Most of the plugins do come with default working configuration.But to make most out of them it is important to configure them properly.I have written detailed tutorial on popular SEO plugins of WordPress.In case you are using WordPress then you can read these tutorials.

Using Gzip version of Sitemap file

Gzipping a sitemap file decreases size of sitemap file.This feature was introduced to make it easier for Search Engine bots to consume the sitemap files.Google officially recognizes .gz version of the file and promotes it as well.But recently I have faced issues.

I had added gzip version of the sitemap file in Google Webmasters.After one week or so Google was not able to read content of sitemap.I was able to see number of URL submitted but nothing under number of URL indexed.I was clueless about the problem.

After digging into details I added normal version of sitemap file and waited for couple of days.To my surprise Google was back reading the file as well as indexing it.

Duplicate URLs in Sitemap file

Google does not like duplicate content.You should avoid URLs in sitemap pointing to same content.For example you might have two separate URL pointing to similar content in sitemap.If you have such URL in sitemap then you should have rel=’canonical’ pointing to main content.

However I prefer excluding these kind of URLs in sitemap file.Including these URLS in sitemaps does increase Google search Engine bot time crawling your website.If number of such URLs are too much then Google might give up reading or indexing your website.

SEO plugins does take care of these issues if configured properly.So take care of above mentioned problem you should read the configure SEO plugin section.In case you are using different SEO plugin then it is better to read the help files provided with the plugin.


Above post discusses different issues with your website which might hamper Google crawl rate.But above points are also applicable for other search engines like Bing,Yahoo! and Yandex.I would like to hear from you.You can add other issues which decreases indexing rate of your website.

Consider sharing this post in case you found this useful.

Leave a Comment