The “bot” check used to block bad traffic can sometimes cause pages to be pulled from Google. In a recent episode of Search Off the Record, Google’s John Mueller explained why this happens, noting that it can cause Google to consider your page a duplicate of another site.
The problem
If the site’s security flags the visitor as suspicious, it displays a “are you a bot” page instead of the actual content. Sometimes, this page is returned to Google as normal, leading Google to index it. This can cause the original content of the site to drop from the index or be replaced by this space.
Given that the ‘are you a bot’ screen appears on other sites, Google encounters several similar pages. When it finds pages that look like duplicates, Google chooses one as the canonical version and considers the others as duplicates.
In addition to affecting indexing, this issue can also affect which page Google views as the primary version of your content. It may select a page from another website, causing your page to be marked as a duplicate.
Mueller explained that it is difficult to trace because it requires checking the page selected by Google and working backwards to find the cause. Additionally, since the average visitor may never encounter the “are you a bot” prompt, there seems to be nothing wrong with reviewing the site yourself.
Why It’s Hard to See
Regular browser checks may not allow you to see the “are you a bot” screen, because it is only visible to visitors marked as suspicious. For you, the page probably loads normally.
You will need to go to the Search Console to confirm that there is a problem. The page index report flags pages as duplicates or as being displayed elsewhere.
Additionally, Search Console’s URL Checker tool reveals which address Google has chosen as the primary version. If that address is for a site that isn’t yours, it’s a good idea to check it out.
Why This Matters
Sometimes, problems can appear without any obvious signs on the site itself because the “are you a bot” screen may be triggered by the CDN, host, or bot-protection layer, instead of the pages directly. Mueller said this type of protection can be effective when crawling increases.
The request goes through successfully, so the usual feeling of looking for a broken page is not helpful here. Since Google is able to access the site and find a valid response, the problem is that it is finding incorrect content. That’s why failures can be caught by checks that don’t see what Googlebot actually sees.
I ran into a similar issue when Mueller described the “Index Page Without Content” error. In that case, the website’s security settings silently blocked Googlebot but allowed regular visitors, causing Google to load the page without finding any content.
Looking Forward
If you run into this issue, contact whoever manages your security service, CDN, or web host to look for a solution. After fixing it, ask Google to re-crawl with Search Consoles Validate Fix. Google may also notice the correction during its next crawl.
Featured Image: Rokas Tennis/Shutterstock
