Google Webmaster Tools Issue


Dev

Forum Member
Joined
Dec 29, 2014
Messages
21
Likes
3
Hello everyone,

I hope this is in the correct place. I'm having an issue with GWT and it has me pretty worried..

A few days back I logged in to see that a warning message had appeared saying Googlebot cannot access certain pages on my site when it tries to crawl them.. I used Wordpress and the Google XML Sitemap plug in, as well as Yoast. I've dug and dug and dug and I cannot find any reason why this is happening. My goal for the new year is to completely clean up my website from back end to front. Currently I'm running way too many plug ins and my sitemap is a mess, but that's beside the point..

I also found that about a week ago (roughly 1 week before GBot couldnt access parts of site) my index status doubled.. I haven't added any new content in a couple months... I do have a staging server set up but its noindexed so it can't be that (I hope)...

Here are my 2 sitemaps..
http://thebestguitarlessons.com/sitemap.xml
http://thebestguitarlessons.com/sitemap_index.xml

I don't have 2 intentionally.. I've had these plug ins for so long I can't even remember why I added the XML plug in if yoast generates one, too..

GWT message:
Over the last 24 hours, Googlebot encountered 14 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 36.8%
Index status:


There are also pages that I don't see in my sitemaps that GBot is trying to crawl and returning 404s to me for...


I know this isn't related to local SEO but any help would be GREATLY appreciated!! I'm pretty nervous this is going to result in a ding... :eek:
 
Last edited by a moderator:
Joined
Aug 7, 2013
Messages
781
Likes
412
Hey Dev,

You should definitely clean up the site map. You might have conflicts between the plugins. Download this plugin called Hobo Bot. It shows you what Google is crawling. It might show you some trends, like Google only crawling sitemap pages. Putting aside that Google is confused on the sitemap, I would focus on guiding Google to crawl the pages you want by re-organizing the robots.txt. help Google crawl the pages you want. Also, Noticed you have a bunch of 307 re-directs. Change those to 301's.
 

Dev

Forum Member
Joined
Dec 29, 2014
Messages
21
Likes
3
Hi Dusty,

Thank you for your response! Truth be told I have very little experience with setting up any redirects and haven't ever done any on my site- including the 307's that you've found. I'm not saying they aren't there, but I didn't put them there- at least not intentionally. Could you tell me where you found that and how I may go about changing them to 301s?

I'm thinking of deleting the XML sitemaps plug in and just using the Yoast sitemap feature. Will also definitely look into Hobo Bot.

Thanks again!
 

Dev

Forum Member
Joined
Dec 29, 2014
Messages
21
Likes
3
Thanks for the reply. I recvently deleted the XML sitemaps plugin and only use the Yoast one now. I also deleted the old maps from GWT and submitted the Yoast sitemap. I'm hoping this will help to solve the index status issue. Traffic for my site also saw a huge bump right after the new year, going from an avg 250 hits a day to 350 fairly consistently so far. I'm hoping this is as a result of something good I've done and not just a 'mistake' that will end in a 'correction' that brings me to less avg traffic than I used to get. Anyway, I'm enjoying the traffic spike no-less, temporary or not...

I actually just had a thought that I can't believe I didn't think of earlier...

I use the plug in Pretty Links on my site to create natural looking affiliate links.. Maybe those are the redirects you're seeing? For a while the links were normal but a few weeks ago I finally got around to nofollowing all of them..
 

Similar threads


Local Search Forum


Weekly Digest
Subscribe/Unsubscribe


Google Product Exert

@LocalSearchLink

Join Our Facebook Group

Top