Crawl Anomaly - help please!

lancemoore22

Member
Joined
Mar 26, 2015
Messages
23
Hi!

I have a client's website that isn't indexed in Google, however it has previously been indexed. And on occasion, the homepage might be indexed for a day then disappear.

Google search console said there is a crawl anomaly. I've done the research on that and here's what all I've troubleshot.

  • I tested it through many crawl tools and had no issue with it being crawled. (i.e. Fetch & Render | TechnicalSEO.com, SEMrush, etc.)
  • It's not on a bad server. All of the other websites on the same server are indexed.
  • DNS is functioning properly.
What other possible reason could this website not be indexed?
 

pony

Member
Joined
Jun 26, 2019
Messages
102
Have you tried inspecting one of the URLs using Google Search Console to determine why Google is telling you the page(s) aren't indexable? I would start there.

They should provide more specifics about the crawl anomaly. If you have some more details come back here but can't help much with what you have provided thus far.
 

lancemoore22

Member
Joined
Mar 26, 2015
Messages
23
I've used it a few times. This is the error message.

Crawl anomaly: An unspecified anomaly occurred when fetching this URL. This could mean a 4xx- or 5xx-level response code; try fetching the page using the URL Inspection tool to see if it encounters any fetch issues. The page was not indexed.

 

pony

Member
Joined
Jun 26, 2019
Messages
102
Strange! I wonder if the server was having issues on 10/29 when Google last crawled the page? Or, do they see those crawl anomalies every time they crawl no matter what?
 

lancemoore22

Member
Joined
Mar 26, 2015
Messages
23
I believe they are having issues every time they crawl it. When it's indexed it's only the homepage and only for a day. It's bizarre.
 

djbaxter

Administrator
Administrator
Joined
Jun 28, 2012
Messages
2,913
I've used it a few times. This is the error message.

Crawl anomaly: An unspecified anomaly occurred when fetching this URL. This could mean a 4xx- or 5xx-level response code; try fetching the page using the URL Inspection tool to see if it encounters any fetch issues. The page was not indexed.
I'm a bit confused: Is that the error message you see generally or an error message when you try to use the URL Inspection Tool?

Have you tried running it through this utility?

 

lancemoore22

Member
Joined
Mar 26, 2015
Messages
23
I have triple checked it. I see there's a crawl delay. That wasn't there before. Maybe that's the root issue. Going to remove that delay.
 
Last edited:

Suresh

Member
Joined
Jul 16, 2019
Messages
17
You can add website RSS feed in sitemap submission, Text files with links etc too. See if that can help a bit.
 

vivekrpatel

Member
Joined
Nov 25, 2016
Messages
88
I believe you don’t have an issue with robots.txt, htaccess as well as meta robots. All look good right?

Can you check your sitemap again sitemap.xml or sitemap_index.xml? Once you got your sitemap indexed your pages start getting indexed in Google.

What is your referring page under your URL Inspection when you retrieve data from Google index?
 

lancemoore22

Member
Joined
Mar 26, 2015
Messages
23
Hi Suresh and Vivekpatel,

Thanks for your input.

Since my last post, I have created a new domain for Google to index. Google has successfully indexed the new domain. Still working to get it the visibility that it needs though.
 

  Weekly Digest

Weekly Digest
Subscribe/Unsubscribe

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...

  Local University Guide


Google Product Exert

Top