SEMRush Bot seen on results page

Status
Not open for further replies.

Ana

Member
Joined
Apr 11, 2015
Messages
11
Hi Guys

Recently, I've a SEMRush BOT appear on my server search results, instantly placed a #SEO block then after a number of days unblocked it. Question is, why are they here?

Thanks
 
Joined
Oct 25, 2013
Messages
752
When you say you saw it in the server search results, you mean in your log files on your host? Is your question, just wondering why SEMRush was crawling your site?

There's a huge number of bots out there, apparently over half of all internet traffic is from bots. Most friendly bots crawl your site for the same general reason humans do... to find information. I don't know about SEMRush, but a common reason for bots to crawl might be to index pages, or find the link structure connecting your site with other sites. SEMRush likely isn't hurting anything, but (unless you're using their service) they probably aren't helping anything either. Your data's just being used to help others make informed marketing decisions, if you care enough to block, there's no downside for you at least. Unless you have reason to believe the bots are using up more than their fair share of server load though, there's not really any reason to take the time to block them.
 

Ana

Member
Joined
Apr 11, 2015
Messages
11
Re: SEMRush Bot seen on results LOG files

I think I know very good SEO. I actually taught myself via the white hat method, but have poor CSS, so I won't show my website. Yes in the log files. I did some reading on their
HTML:
https://www.semrush.com/bot/
and others then decided afterwards to re-block them in my /robots.txt as basically they are criminals pinching other people's hard work for their Adsense benefit. Someone stated a long while ago: Thou shalt not steal.

i believe the reason they are here is because of my high ranking for some top quality keywords i an using.

Thanks
 

Eric Rohrback

Moderator
Joined
Oct 3, 2012
Messages
998
Re: SEMRush Bot seen on results LOG files

They are visiting your site to document for their tools, not adsense. SEMrush is a reporting software that helps monitor keyword performance for organic ranking and paid search. They are one of the better companies out there to help understand site performance.

I've used both SEMrush and SpyFu, and never had a problem with them. If you block all bots from your site, then hopefully you have developed some internal/proprietary software for your own use. Otherwise you're going to have a tough time understanding the full market landscape.
 

Ana

Member
Joined
Apr 11, 2015
Messages
11
Eric, they are here like you say and because I have mastered (this area) how to fully optimize rank on page 1 of Google and Bing for some seriously juicy keywords and they want all the cake, but I have blocked the Semrushbot and SemRushbot-SA plus a few others just in case.
 
Last edited:

Eric Rohrback

Moderator
Joined
Oct 3, 2012
Messages
998
@Ana - Nice trolling. We'll consider this a case closed subject since you obviously are set on what you want to do and do not want to engage in any real discussion.

Please contribute in a meaningful way to threads in the future.
 
Status
Not open for further replies.

  Weekly Digest

Weekly Digest
Subscribe/Unsubscribe

Trending: Most Viewed

  Promoted Posts

New advertising option: A review of your product or service posted by a Sterling Sky employee. This will also be shared on the Sterling Sky & LSF Twitter accounts, our Facebook group, LinkedIn, and both newsletters. More...

  Local University Guide


Google Product Exert

Top