• Home
  • Blog
  • What is robots.txt? Beginner’s Guide

What is robots.txt? Beginner’s Guide

The robots.txt file makes sure that Google’s indexing algorithms properly index the website. It lets you provide the search engine information about the website’s pages and prevents the indexing of particular pages. The site’s position might be worsened by faulty indexing brought on by wrong robots.txt file settings.

Main Directives 

The main directives for working with the robots.txt file are:

  • Disallow. This command restricts access to files. To prevent bots from ignoring the directive, the path must be specified after it.
  • Allow. The directive allows page crawling. Enables access to a certain document in a locked section.
  •  A sitemap. Provides the sitemap’s URL. An absolute path is marked when the directive is used.
  • Noindex denies page indexing. As of 2019, the directive is no longer supported.

Blocking in the robots.txt File

blocking for robots

The following issues might arise when Googlebot scans a URL:

– Blocking crawling. To check for blocking, use the robots.txt checker in the Google Search Console (GSC). Go to domain.com/robots.txt to access the file if you are unable to access the GSC. The Disallow directive needs to be deleted.

– Recurrent blocking. To find blockage, check the robots.txt history. Based on the exact issue, the blockage may need to be fixed. As an illustration, consider a shared cache between a test and a real environment. The cache might have to be deleted.

– User-agent blocking. User-agent blocking is possible when a website is blocking a specific user-agent. cURL may be used to check for blocking. We advise contacting your hosting company for assistance if this is the situation or if you are blocked by IP address.

How Should I Set Up the robots Meta Tag and Why Do I Need It?

Both the robots.txt file and the robots meta tag serve similar purposes. However, the meta tag offers greater indexing fine-tuning (e.g., content is closed and links are left open). Almost the entire page is impacted by the robots.txt file. Meta tag is used when there is no access to the resource root directory. The meta tag enables you to maintain access to certain directory pages while the robots.txt file prevents the full directory from indexing. Indexing may be done concurrently using both the robots meta tag and the robots.txt file. The search bot will prioritize the file over the meta tag when examining sites.

The robots.txt file is configured in the following steps:

  1. Entries for “User-agent” are made. They specify which search bots must abide by these regulations. “User-agent: *” is an example that applies to all bots.
  2. Pages are not permitted to be indexed (e.g., duplicates) are specified. Access must be available for pagination sites.
  3. Directives Allow and Disallow are employed. The Disallow function prohibits files for indexing, while Allow permits it.

The sitemap.xml file is given and verified to be valid in the last stage.

Disallow Indexing

The robots.txt file must prevent specific pages from indexing. The Disallow command is used to impose a prohibition on indexing. Most typically, integrated sites, product and service pages, administrator pages, and duplicate content pages are prohibited. Disallow is frequently used by e-commerce companies to conceal their pages. The Disallow instruction, however, could be partially or wholly disregarded by some search engines. You should thus determine if the page includes sensitive information before executing this directive.

Incorrectly Formatted robots.txt File

Incorrect configuration of the robots.txt file can lead to the following problems: 

– The site’s adaptive layout cannot be developed because the search engine cannot access the template files.

– Content that rendered by JS with access restrictions is not always crawled.

– Search bots could disregard the restriction on indexing pages.

– “Garbage pages” may be included in the index.

Working Guidelines for robots.txt

working with robots txt

When using the robots.txt file, the following guidelines should be followed:

– The file must follow a specified structure and be produced correctly. As an illustration, consider User-agent → Disallow → Allow → Host → Sitemap. You can choose the order in which pages are scanned using such a structure.

– Each URL must be listed on a separate line in Allow and Disallow.

– Symbols are not allowed, excluding “*” and “$”. Bots are unable to identify other characters.

– For each subdomain, a unique robots.txt file is created.

– The “#” sign should be used when leaving comments. If you put this character at the beginning of a line, its content will be ignored.

– When a page is disallowed for indexing, its SEO-weight is not taken into account.

– Robots.txt cannot be used to block sensitive information.

– When creating a robots.txt, you shouldn’t make blunders like adding an empty line to the Disallow section, writing Robots or ROBOTS in capital letters, listing several directories in a single Disallow line, or listing all files in a directory.


engine robots

Configuring the robots.txt file correctly enables indexing of legitimate pages while blocking garbage pages. Please be aware that any robots.txt directives are suggestions only and may be disregarded by search engine algorithms.

    Get your FREE, personalized SEO strategy

    Related articles

    Alexander Tarakhovich

    SEO expert with extensive experience. Helping you understand SEO. He takes the guesswork out of SEO, his advice in optimizations are based on what actually works. Alexander is currently working to make SEO known and accessible worldwide. Read Blog