Semalt: How Does Google Know When (And Why) To Stop Crawling Your Website?

23.05.2018

Semalt: How Does Google Know When (And Why) To Stop Crawling Your Website?

Google spiders are as important as the SEO campaign itself when marketing a website. They crawl websites and index content from all the pages it can nd. It also performs re-indexing on updated pages on the site. It does so on a regular basis, based on several factors. They include, but not limited to, PageRank, crawling constraints, and links found on the page. The number of times the Google spiders crawl a site will depend on one or more of these factors.

A website should be easily navigable by the visitors as well as the Google crawl spiders. It is the reason why having a crawl-friendly website is an added advantage to one's SEO campaign. Otherwise, Google will be unable to gain access to the content, consequently reducing the site's ranking on the search engine ranking page.

Ross Barber, the Customer Success Manager of Semalt, de nes that two of the most important indexing factors that Google relies on to in uence its decision to either slow or stop crawling your site are the connect time and HTTP status code. Others include the disavow command, "no-follow" tags, and robots.txt.

Connect Time and HTTP Status Codes The connect time factor relates to the amount of time that the Google crawl bot takes to reach the site server and web pages. Speed is greatly valued by Google since it is highly indicative of good user experience. If the webpage is https://rankexperience.com/articles/article1675.html

1/2

23.05.2018

not speed-optimized, the site will then rank poorly. Google spiders will make attempts to reach the website, and if the time taken to create a connection is longer, they back off and crawl it less frequently.

Furthermore, if Google pushes to index the website with the current speed, then it might interrupt the user experience as it might signi cantly slow down its server.

The second indexing factor is the HTTP status codes which refer to how well the server responds to a request to crawl the site. If the status codes are within the 5xx range, then Google takes it upon itself to stop or delay the rate at which they crawl the current site. Anything within the 5xx range is an indicator of possible issues with the server and that responding to the request might be problematic. Due to the risk of causing additional problems, Google bots will step aside and conduct indexing when the server is more reachable.

When Does Google Resume Crawling the Site?

Google believes in providing users with the best experience and will rank sites that optimize their SEO elements towards these objectives high. However, if the website currently exhibits the problems mentioned above, it will command its Googlebot to try crawling it at a later time. If the problems persist, the owner will lose on a great opportunity to have Google go through its content and assign it a well-deserved rank on the search results. In addition to these problems, any sign of spam will have the site blocked from ever appearing in the search results.

Like all the other algorithms that Google uses, its spiders are also automatic. They are developed to nd, crawl, and index content based on certain parameters. If the site does not conform to certain best practices, indexing will not happen. There are many other factors involved, but always remember to pay close attention to the connect time and HTTP status codes of your site.

https://rankexperience.com/articles/article1675.html

2/2