, ,

What Is Robots.Txt and How to Verify Robots.Txt File

Most of the new bloggers and marketers always feel robots.txt as the “word of confusion”. Because they just don’t about it.

Do you know about robots.txt?


Have you heard the word before?

If you are a blog or website owner, or if you are in any sort of online marketing field then you’ll know about the robots.txt or at least hear the word robots.txt

robots.txt tutorial

Now, robots.txt is the important file, and it’s the crucial factor of seo as well, because once if you implement your robots.txt in a wrong way then your site will lose all the organic traffic and seo advantage of your site.

Even after consuming lots of efforts in writing the best seo friendly article you will not have desired results because if the robots.txt file is incorrect, then you can harm your site by blocking pages and resources.

Have you ever imagine trying to rank for a keyword on a page that googles can’t access. Now after listing this conversation, you may get doubt what is robots.txt.

Let me tell you,


What is robots.txt?

Robots.txt is a particular file with a set of proper instruction which says Google bots or other search bots to scan your web page or to block your web page. Robots.txt is the crucial file for seo because it can cause so much trouble if you placed a wrong robots.txt in your site.

Now, it actually works like this: Before any search bots crawl your site and take information from your site, they will check whether there are any restricted pages and limitations provided in your robots.txt.

After checking your robots.txt file, then google bot will access your site only through the instruction provided in your robots.txt.


For suppose:“if you block home page in your robots.txt, then your home page will not be indexed by Google or other search engines.”


How to see robots.txt?

The robots.txt file is located at “http://www.website.com/robots.txt.” you can see any websites robots.txt file with this command, All you have to do is just replace the domain name with you own and see the results.

This command lets you see the robots.txt file where you can find your website instruction for web crawlers.

It is the first place or location where search engines will visit your site before crawling and indexing your posts.


Robots.txt clear view:

Robots.txt file contains three main parts, and you should understand them carefully there are User-agent, Disallow, Sitemap.



The user agent is the term which is used to specify the rules and regulations that web crawlers must follow.

Ex: User-agent: *

This command instructs the crawlers to crawl your website, many websites mostly use * the user’s agents because it refers to “all user agents”.



Disallow is the command which simply says crawlers you must not see or index this part of the site.


User-agent: *

Disallow: /wp-admin/

Sitemap: https://www.seocompanynoieda.com/sitemap.xml

This command lets you all the web crawlers that disallow these files which are in wp-admin. Most probably the mistakes will happen because of the disallow command.


Allow:Allow means giving permission to access the particular file or folder.


Sitemap: Sitemap is another crucial part of seo, so you must include your websites sitemap in the robots.txt file. Because “sitemap” is the next part where search engines are going to crawl. After seeing robots.txt.

Ex: If you want to show your sitemap then you should use the sitemap

In robots.txt as follow:

Sitemap: http://www.website.com/sitemap.xml

Sitemap: http://www.website.com/sitemap_index.xml



How do you verify that robotst.txt is set properly?

Just go to google search console, and you’ll find a tool to test your robots.txt file. If you are having any problems and errors in your robots.txt, then it will show those specific URLs to make sure Google and other search engines indexes them.


Google search console dashboard — crawl option — robots.txt tester.


Generally, Search engines role is to crawl and index your website and provide the results.

The robots.txt file is more than seo it’s a technical seo aspect, and it can be even confusing for most of the bloggers.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *