Google Says Why It May Ignore Robots.txt And It Has A Negative SEO Impact

John Mueller of Google answered a question about robots.txt and explained an easy mistake that can affect your SEO and website indexing goals. The particular issue was related to search box spam being identified by Google, but this error can happen to anyone in general under any context.

Website Search Box Spam

Someone who asked a question on Reddit was suffering from a search bar spam attack. What spammers do is search with a query that indicates their spammy niche, and add a link or website name. What happens next is that the search bar generates a URL that can be indexed to generate a spam search result.

And that happened to the person who asked the question. Their response was to add a line to the robots.txt file to prevent Google from indexing the file. But Google was indexing those URLs generated by spammy searches anyway.

Pages with Google Links Blocked by Robots.txt

Someone posted on Reddit that their Shopify client’s search box was generating spammy web pages in response to spammers’ queries and that Google was indexing them despite a robots.txt file preventing Google from indexing those pages. What the client did was redirect those spam URLs to another web page. The person asking the question didn’t ask how to set up pages to be indexed (which they should be asking); ask them if those redirect URLs should be marked with a 404 instead.

Someone asked:

“Working on a Shopify client store where we added /search as disallowed in robots.txt, however these search results are still being indexed within Google.

However, when I try to open one of these pages, they have a redirect set and redirect to another collection page in the store. Should we show a 404 page instead? What is the easiest way to fix this?”

Why Robots.txt File Caused Spam to be Detected

Google’s John Mueller took the extra step to identify and review the client’s robots.txt file and identified a bug that was causing Google to ignore the directive that prevents Googlebot from indexing search results pages.

The answer comes from Mueller:

“Also, I’m not sure if it’s your site, but the one I found with URLs with the same index had sections for “user-agent: Googlebot” (in the block “START: Custom Rules” in the comments) and more in the “user-agent: *” section further down. With robots.txt, more specific rules prevail, so if you’re going to use that *agent* Google section. apply all the rules in the section “user-agent: *”, you need to copy them and, if it’s your site, you can just write all the user agents you want to have shared rules together, eg:

user agent: googlebot

user agent: otherbot

user agent: imgsrc

user agent: somethingpt

don’t allow: /fish

disallow: /orange-cats

… etc…”

User Agent Specific Instructions Come First

What happened was that the client was relying on Google to follow instructions on the line directed to all user agents, “user-agent: *”, but because there is another section of the robots.txt file that is specific to Googlebot, Google ignored the “user-agent: *” instructions and listened to the one directed directly to Googlebot.

That may sound like a strange way of how robots.txt works, but it actually makes sense because this enables users to target specific browsers with unique rules and target everyone else with a different set of rules.

How to Stay Safe from Search Box Spam

WordPress and Shopify both have ways to reduce search box spam.

WordPress and Shopify both have ways to reduce search box spam.

Shopify Search Box to Reduce Spam

The Shopify website has a tutorial on how to automatically add noindex instructions to all search results. This will effectively prevent the indexing of all search results pages. However, you need to unblock search pages with robots.txt for this to work.

Using the noindex Directive works better than using robots.txt because robots.txt does not control indexing; it only controls crawling.

Shopify mandates:

“You can hide excluded pages in your robots.txt.liquid file by customizing it

part of your theme store.liquid layout file. You need to add metatag code to stop indexing of certain pages.

From your Shopify admin, go to Online Store > Themes.

Find the theme you want to edit, click the … button to open the action menu, and click Edit code.

In the theme folder, click on the theme.liquid file.

To remove the search template, paste the following code into the empty line at

part:

{% if template contains ‘search’ %}

{% endif %}”

Reducing WordPress Search Box Spam

It is very easy to reduce search box spam with WordPress. Users of Yoast, Rank Math, and AIOSEO SEO plugins have their search results pages automatically set to noindex by default. Additionally, some page builders and themes like Divi (and its Extra theme) will not automatically generate spam words and URLs in response to searches and will instead inject a few sentences saying that the search is not producing results.

Robots.txt Information

Effective SEO requires extensive knowledge. Robots.txt contains some issues that can cause it to work less than intended, so it’s helpful to read the official specifications to stay up to date.

Featured image by Shutterstock/Stockinq