What Is Robots.txt File And How To Use It For Your Blog?
September 27th, 2014 | by Ravi Chahar || 66 Comments |
There are many factors of SEO for your WordPress blog. While walking on the path of the blogging journey you will come to know about Robots.txt file which is just a simple file created by robots.txt file generator or you can create it yourself by using Notepad.
Are you wondering about this file? Do you know what is Robots.txt file? Many bloggers try to know about this simple file.
Today I am going to let you know about Robots.txt and how can you create it? Do you wonder why Google is indexing your admin folder? Do you think why Google is indexing your wp-folders/plugins and many other folders which shouldn’t get indexed by Google? The answers are in this post.
What is the use of Robots.txt file for your website?
Many people have confusion regarding Robots.txt. I must tell you that there are two different aspects i.e crawling and indexing.
Most of the bloggers think that the file is used to stop Google for indexing any particular place of your blog which is not true. Crawling and indexing are two different activities to know.
When you create Robots.txt file then you are telling Google bots, Bing bots or any other third party bots what to crawl and what not to crawl from your blog.
This file doesn’t have any relation to indexing and no-indexing. If you don’t want Google to index any particular part of your blog then you can add no-index tag to them.
If you don’t have Robots.txt file for your blog then Google will not be able to crawl all the inner-links of your website which will signal that your website is not search engine friendly.
There are many folders you don’t want to get crawled by Google bots for which you need to create Robots.txt file.
Sometimes you may get messages in your Google Webmaster Tools about the error in your Robots.txt file. It is because you have configured it wrong. Maybe you haven’t followed the right format.
Can You check Robots.txt file using GWT?
This file exists in the root directory of your domain. You can check this file by typing the URL of your website followed by /robots.txt.
e.g. = www.your website.com/robots.txt
You can check it using GWT (Google Webmaster Tools). You can see many options after logging in to your GWT account from which you have to go to ->crawl section where you will find ->robots.txt Tester. If you have any error then you will see in the current status but still, you should check it by using the tester.
After going to the tester you have to write Robots.txt after the URL given to your website. After adding the URL you need click at “TEST” and if you get the result as “ALLOWED” then there is nothing to worry about. Robots.txt file for your website is working fine.
The common structure of Robots.txt file is here.
User-agent: *
Disallow: /cgi – bin/
Here you are telling Google bots not to crawl cgi – bin folder of your website. Similarly, you can add many other folders which you don’t want to get crawled by Google bots.
Conclusion :
Have you created your Robots.txt file for your website? Do you get any error message from Google Webmaster Tools? Do you even know what is Robots.txt file?
There are many frequently asked questions by beginners about this file. As you all know that SEO is the major concern for bloggers and your Robots.txt file plays the main role in it.
Hi Ravi,
Nice post and very simple topic too.
But, this is good that you are taking care of newbies and posting about such small,simple but, most important things.
A proper robots.txt file is very necessary for better search engine crawling and blocking any bot if we want.
Thanks for sharing with us.
Have a great weekend!
Hi Nisha,
You know beginners face many problems while dealing with their blog and its good to help them.
I have faced a lot of difficulties while setting up my blog so somehow its my duty to take care of other newbies who don’t have any knowledge about these things related to SEO.
Thanks for sharing your thoughts.
Have a great weekend.:)
~Ravi
Hey Ravi,
Superb article,
even i had faced many problem editing robot.txt file for my WordPress blog initially for better search engine optimization but when i got how to edit it exactly. Now i know the importance of Robot.txt file for Search Engine traffic,
Thanks
Regards
Suprabhat
Hi Suprabhat,
I am glad that you liked this article.:)
SEO is so important for your website and robots.txt file plays a great role in it.
Thanks for taking time to visit.
Have a great weekend.:)
~Ravi
Hi Ravi
After reading this very interesting post, it is clear that I am still in the newbie blogging lane. ? There are so many terms that I am yet to learn and this is just one of them
I have heard the terminology before, but have never taken the time to know more about it. Usually when I have technicalities as it relates to my blog, I consult a good friend of mine who is a software developer and know a lot about these things. Matter of fact, that person set up the website, got it registered with Google and so on.
Still it is good to know these things for myself. The tutorial was very helpful. Thanks for sharing.
Hi Yvonne,
There are many things you need to learn about WordPress, SEO and a lot more. Like I said at Harleena’s blog that we all learn during our life and its good to know about new things.
Robots.txt file plays its major role in the SEO of your blog. You must check it using Google Webmaster Tools.
Thanks for stopping by and sharing your thoughts.
Have a good day.:)
~Ravi
Hi Ravi,
This is a pretty good explanation for a newbie! ?
Yes, I have my robot.txt and I’m doing fine. Guess Google is not complaining crawling into my site coz I’ve got that sitemap.xml in place. But for those who doesn’t know what robot.txt can do for your website, kudos to you for posting this. Have a cool Saturday!
Angela
Hi Angela,
I am glad that you liked this post.:)
You are right beginners should know about robots.txt because if they don’t it right then errors will boost up their blog.
Google bots will not crawl their website which will not good for them.
Learning is always better.
Thanks for dropping your valuable input.
You too have a good sleep.:)
~Ravi
Hi Ravi,
Your headline pulled me in because I just check SEO Quake and saw a problem with my Robot.txt file. It was fine a few days ago, so don’t know what happened.
I happen to have an SEO expert to handle this for me, so I am going to bookmark this post so I can learn all about it should I need to take care of something myself some day – you never know! It’s always a great idea to know anyway in order to not get taken advantage of, right? ?
I will be sharing this with my friends because it is definitely valuable information.
Thanks so much for providing such great tutorials with screen shots.
Have a wonderful weekend.
Talk soon,
– Carol
Hi Carol,
Robots.txt is just a text file and sometimes it shows error when you check it in Google Webmaster Tools. There is not any particular reason for the errors. Its just sometime may be you add something irrelevant in your text file.
As few days ago I tried to add “crawl delay 10” which is used when you have high rate of crawling for your website. If you want to tell Google bots to crawl your websites in an interval of time then you should add this message. I also tried it but got error after two days.
Experiments should be done and you should check your robots.txt file so that GWT doesn’t show any error while crawling your website.
Thanks for taking time to visit.
Have a nice weekend.:)
~Ravi
Hi Ravi, This is a fantastic topic. You chose a very interesting topic which many online users are not very knowledgeable about. I am sure this will help a lot of people. Even I learned a lot about this. I knew the difference between indexing and crawling, but you have mentioned some more points too.
Great going! ?
Hi Kaustav,
I am glad to know that you liked this post.:)
Crawling and indexing are two different things but still there are many bloggers who don’t know it.
Thanks for stopping by and sharing your thoughts.
Have a nice weekend.:)
~Ravi
Hi Ravi,
Very nice and informative post about Robots.txt File, specially for those guys who recently stepped in blogging and are going make their blogs. Its the best for them to make a search engine friendly website or blog because through this Google can easily crawl the inner-links of their blogs which sometimes left and they get less reward of their effort.
Thanks for sharing such a comprehensive post.
Hi Mairaj,
Newbies need to know about such things if they want to handle their blog themselves.
These things are simple can be learnt very easily.
Thanks for taking time to comment.
Have a wonderful weekend.:)
~Ravi
Hey Ravi,
You did a great job in explaining this. Robot.txt is obviously very important and I recently checked mine and it showed an error but according to what you told Carol, that’s normal and it shows that sometimes.
Also, in your meaage to Carol, you said, “Experiments should be done and you should check your robots.txt file so that GWT doesn’t show any error while crawling your website.” I’ll definitely have to do this more often.
Great post. Have a great day.
– Andrew
Hi Andrew,
Yeah! I personally think that we all should try to do different things. I mean I have met many bloggers who get so afraid about touching any techie stuff.
But if we don’t try anything then its not possible to learn more. Sometimes we do mistakes so that we can learn.
You can easily correct the errors you get in your robots.txt file. You can even replace it with new one.
Thanks for having keen observation at this post.
Have a nice weekend.:)
~Ravi
Hi Ravi,
Informative post indeed ?
It reminds me of the time when I was a new blogger and had NO idea about such a term and so many others, related to blogging. I wish there were more places or sites I knew, like yours, to head over and read such things.
Yes, new comers are very lost, and even the seasoned ones who don’t manage the blogs themselves and pass on the work to other experts, about such files, which is not difficult to learn about, provided you know how to – just as your post explained here. Yes, Google Webmaster Tools is the best place to check the errors you have in it, which can be corrected too.
Good work indeed, and I am happy you are doing so much to create such simple and helpful posts for budding bloggers, keep going!
Thanks for sharing. Happy weekend ?
Hi Harleena,
When I started my blogging career at that I was not aware about anything and like you said that in starting we all wish to find any websites from which we can learn but I found Wpbeginner and some more. The main source for me was YouTube from where I used to see videos about WordPress a lot more.
There are many easy and simple things to do but newbies get afraid and they don’t do these things their own. I mean how would they learn if they wouldn’t even try? They should try to learn about many such terms.
Its always good to hear from you.
You too enjoy your weekend.:)
~Ravi
Hi Ravi,
You’re writing superb post for the wordpress bloggers. I checked the Robots.txt file in Google Webmaster tool as you said. It shows no error and “allowed” and I feel good. Thanks for writing the helpful post for us.
Hi Nirmala,
You should check your robots.txt file in regular interval of time. Its very important from SEO point of view.
Thanks for stopping by dropping your input.
Enjoy your weekend.:)
~Ravi
Hello ravi
Hey this is well described article for robots file. I never checked my file in webmaster account. Thanks for adding such points here..
Hi Sahil,
Welcome to my blog.:)
Its very important to check robots.txt file for your blog. Check if there is any error or its error free.
Thanks for visiting.
Hope to see you again.
Have a good day.
~Ravi
Hello Ravi,
As above Nisha said it was simple topic but very important topic for webmasters especially for newbies ? because it tells search engines to ignore the contents of certain file folders.The importance of the robots.txt file has always been important in SEO so it is important to check or create a robot.txt file for your blog ?
Thanks you for sharing this useful topic ? . Have a great day !
Regards,
Minakshi Srivastava
Hey Minakshi,
SEO is not an easy thing to learn about and there are many aspects of it.
Robots.txt is an essential part of a website which guides Google bots to go through your website.
Its good to see you again.
You too enjoy your weekend.:)
~Ravi
Hey Ravi,
Good idea for a post. Knowing what your robots.txt file is can be useful. Your post is good for the newbie blogger or site owner. It’s probably enough for the beginner to know. A little in depth knowledge is good too but I always have to be careful. I can sometimes know enough to be dangerous LOL! ?
Thanks Ravi. Have a wonderful week ahead.
Liz
Hi Liz,
I am happy that you liked my post.:)
You are right its good to know about robots.txt file for beginners. They may get more knowledge but in starting they should know a little bit about everything.
Appreciate you for taking time to be here.
Have a good day.:)
~Ravi
Great post. Just today I had to create a robot txt to disallow some links that were causing issues with my site. Your post is simple for a newbie and you have made me to easily understand the term. Thanks
Hi Ikechi,
Welcome to my blog.:)
I am glad that this post could help you in having some knowledge about your robots.txt file.
Thanks for taking time to comment.
Have a nice weekend.:)
~Ravi
Hey Ravi,
Well that was a great explanation and very clear and simple to follow.
I guess I learned about this some years back and all is good with mine. At the moment that is. I’ve found that from time to time some of the files I have set up will all of a sudden stop working or some of the code I have in them magically disappear. I still wonder to this day how that even happens and I find that so very strange. I think every time my hosting service checks something for me they must take some thing out if they don’t think it needs to be there because I sure don’t know who else could be doing that. It’s certainly not a ghost.
Great share and I’ll be sure to share this too, thank you and enjoy your week.
~Adrienne
Hi Adrienne,
Good to see you after weekend.:)
Still there are many bloggers who don’t even try to handle this file and many other things about SEO.
I remember when I added some codes in my robots.txt and then Google bots stopped to crawl my websites and finally I got a message from GWT. After that time I figured that we all need to check this file regularly.
Our hosting service is not the solution every time. These type of simple problems can be solved by our own.
Thanks for your effective words.
Have an amazing week ahead.:)
~Ravi
Hi Ravi,
Yes.. Robot.txt is absolutely necessary from SEO point of view… For advance Robot.txt features you can add certain files and folders too..
Thanks for sharing this with us…
Regards..
Hi Karmakar,
There are many folders present in the directory which shouldn’t be crawled by Google bots. Robots.txt file is the solution for all of these to hide.
It helps you in the SEO of your website.
Thanks for taking time to visit.
Have a wonderful day.:)
~Ravi
Hi Ravi,
Very much useful post. Ofcourse if someone is new to blogging or SEO they obviously will not have any idea on Robots file and related things. Am glad that your post will be a perfect tutorial to teach enough about the terms. Well written post!!!
Hi Vino,
In the beginning of any business people don’t know about these techie things. With the passing of time our knowledge level increases.
Creating a robots.txt file is a necessary stuff to do if you don’t to ruin the SEO of your blog.
I am glad that you liked this post.
Thanks for dropping your input.
Have a wonderful week ahead.:)
~Ravi
Hi Ravi,
This is bran new to me. I knew nothing about Robots.txt file and what it’s for. Thank you for the clear and easy to understand tutorial here, and not too overwhelming even for someone like me.
Great post ?
Hey Sylviane,
I am glad that you liked my post.:)
This file is an important aspect of any website. You need to know about its management.
Thanks for taking time to share your thoughts.
Have a nice day.:)
~Ravi
Hey Ravi,
I always learn so much from your posts. I have to admit, even though I spent 20 years in IT from being a developer to a project manager, I never learned how to create and maintain a web site. I have my website and I post twice a week. I have no idea how to enrich my SEO and I don’t pay attention to most technical points for my site. Bad, I know. And when I read your posts, I realize that I’m probably not maximizing search engines and many other things. Makes me say hmmmm. Lisa
Hi Lisa,
You know I was like you when I started my blog. I was not even aware about anything related to SEO and site management. If we want to boost up our website then we need to learn a lot.
Time passes and gives us chances to know more about many techie things.
Its always good to hear from you.:)
Enjoy your week ahead.
~Ravi
Hey Ravi,
Quite clear and simple to follow ?
I think if I hadn’t known anything about robots.txt before, your tutorial will be a perfect one to follow.
Thanks for helping your readers on this bro.
Hi Enstine,
I am glad to know that this post will help you about knowing more about robots.txt file.
Thanks for being here.
Have a great week ahead.:)
~Ravi
Hi Ravi
For quite a long time I was not aware of robot-text file and did not alter it for my green blog to unindex admin and other admin related data. As a result I observed a little slow increase of traffic at my blog as Google has to bot everything to pick the relevant stuff from it.
Every blogger needs to change its setup to help Google just index the contents which you created for your community and want them to appear at the top position.
Thanks a lot for sharing yet another useful and informative post.
Hi Muba,
Its necessary to configure your robots.txt file because you need to give direction to Google bots about your websites as what to crawl.
There are many inner links in crawling which Google takes so much time which may result is an issue for your websites as GWT may show time delay message to you.
Bloggers should maintain their file properly.
Thanks for taking time to visit.
Have a peaceful day.:)
~Ravi
Mr Ravi,
Thanks for the screen shot, that explains it all for those who are less familiar with blogging terms.
Its very important to check your webmaster to correct any Robot.txt file error.
Great post….
Hi Van,
I am glad to see you back here at my blog.:)
Of course you need to check your robots.txt file in regular interval of time.
Sometime some error may occur due to which Google bots may stop crawling your website.
Thanks for your input.
Have an amazing day.:)
~Ravi
Hi Ravi,
You are completely right that Robot.txt files are really necessary. If you are using WordPress as your CMS then it should be placed in your Wp-Content directory so that Search Engines could not crawl that folder.
Because if they do all of our precious Plugins and themes could be used by others as well.
Hi Rohit,
You are right. Bloggers should add wp-folder and many other folders into robots.txt file so that Google bots will not crawl them.
There are chances that if hackers come to know about your website details then it can be hacked. For security purpose you should use robots.txt file.
Thanks for taking time to say your words.
Have a great day.:)
~Ravi
I’ve had a blog for almost 8 years Ravi, in one form or another, and until I read your post I was never quite clear what the robots.txt really was!
So Thank You! I look at GWT often and had never known to check the robots.txt tester, so I did it tonight! All clear. Approved for images, videos, mobile, etc…. Very cool.
I love when techie gets me excited, since it’s something I must understand yet not my main focus.
I hope you’ve been having a great week Ravi!
Warmly,
Deborah
Hey Deborah,
I am glad that liked my post.:)
I always say that techie stuffs are interesting to learn. If you have interest then its easy to do those things.
Robots.txt is an important part for every blog.
Thanks for taking time to say your words.
You too have fun this weekend.:)
~Ravi
Hi Ravi,
I seems you are covering all the basics steps of blogging, although they are basic but are most imporstant things that every blogger should be aware of.
Thanks for taking care of them ?
Robots.txt are I think most important thing that should be known but hardly newbies and even senior bloggers have no idea about that.
Your post has purely justifies it.
Thank you for sharing.
Hey Vishnu,
There are many things to learn related to SEO but robots.txt file is one the most important thing to know about.
Bloggers should know whats the use of this file? They should know how to tell Google Bots about what to crawl at your website.
Thanks for taking time to drop your great input.
Have an amazing day.:)
~Ravi
Hi Ravi,
Wonderful article on robots.txt file. This is the most crucial file of any website installtion and proper knowledge of robots.txt file, is in this case is must.
Keep posting such informative articles.
Hi Rakesh,
You are right.
Bloggers should take care of their robots.txt file so that Google bots can crawl their website properly.
Thanks for taking time to comment.
Have an awesome week.:)
~Ravi
Nicely summarised.
Often the content management systems include robots.txt with their installation, which always helps a lot with configuration. If you already have a robots.txt in your site it’s pretty easy (or at least easier) to modify it in case you need to add some pages to it.
Of course it’s good to remember robots.txt doesn’t necessarily stop the crawlers from crawling your site, but at least Google usually honours it as intended.
Very good for you to bring this subject up. I don’t think many bloggers even know about the existence of robots.txt.
Hi Mervi,
There are many bloggers who don’t even know about this file because most of bloggers don’t handle their techie side their own.
I have noticed that most of famous bloggers hire someone to take care of these things.
Knowing about robots.txt is very important and you should know what to add in it?
Thanks for taking time for your amazing comment.
Have a peaceful day.:)
~Ravi
Very nice and informative post about Robots.txt File.
Thanks for sharing.
Hey Parag,
Thanks for stopping by and dropping your comment.:)
~Ravi
It’s necessay thing bro.
I always use my custom robot.txt on blogger platform . and when i run on wordpress then every work will automatically done by plugins.
By the way nice post , nice to see the all about this phenomenon.
Hi Siddharth,
Though I haven’t used Blogger platform yet but for sure it would be great to use it. Robots.txt file may be same.
But let me tell you that robots.txt is just a simple text file and there is nothing to do with plugins regarding it.
Thanks for visiting.:)
~Ravi
Hi Ravi I think this is my forst visit to your blog and what a great topic to read. Its simple but only a webmaster knows its importance. You know wht is more useful about robots.txt is if you don’t stops bots to read your blog at certain sections then Google will think your blog has copied content like blogginglove.com and https://www.blogginglove.com are two different places for bots so your one content is seen as two different content by bots. Correct me if I am wrong.
Hi Swadhin,
Yeah! I agree that Google bots will consider any website having URL with “www” like “www.blogginglove.com” and another URL without “www” like “blogginglove.com” different but only if you don’t redirect your non-www URL to other.
Most of users type just the name of the website without “www’ so its necessary to redirect the URL so that Google consider both as one. There is no case of copy here.
Robots.txt file tells Google only about what not to crawl in your website.
Thanks for asking this question. I am glad that you know about these things.
Have a great week ahead.:)
~Ravi
Hi Ravi,
First This is a fantastic topic. You chose a very interesting topic which many online users are not very knowledgeable about. I am sure this will help a lot of people. Even I learned a lot about this. I knew the difference between indexing and crawling, but you have mentioned some more points too. Thanks a lot Great Work My dear i like it ,
Regard
Leena Shah
Hey Leena,
I am glad to know that you liked this post.:)
You may see many people who don’t know about this simple file because most of them don’t handle their techie stuff their own.
I have noticed that many bloggers hire someone to look upon their website.
Crawling is a different part for your blog and you must know what to get crawled by Google bots and what not to.
Thanks for your wonderful comment.
Enjoy your day.:)
~Ravi
robots.txt is the important file which tells Search Engine’s bots not to crawl the file given in the instructions. happy to see this post Ravi.
Hey Shane,
Welcome to my blog.:)
This simple file plays a great role in SEO of your blog.
Thanks for visiting.
Have fun around.:)
~Ravi
Hi Ravi,
Great explanation on the importance of the robots.txt file. It really is an important tool in your seo and it helps Google tremendously.
Another thing it’s really good for is telling Google you don’t want certain pages indexed. For example, I have “thank you” pages on my site to let people know I appreciate them and any next steps that may be applicable. People are directed to these pages after they fill out a form. I don’t want them indexed and I don’t want them found on the web so this one way to keep them from being found.
It’s like a “STOP, Don’t proceed past this point” message, lol…
I’m sharing your post Ravi it’s very informative.
I hope you’re having a great week!
~ Don Purdum
Hi Don,
Google webmaster tools is the best tool to look after your website and robots.txt is the way through which you guide Google bots about your website.
You can let them know about certain pages and folders which you don’t want to get crawled.
Like you many bloggers add their pages to robots.txt so that Google bots won’t crawl those pages.
Bloggers should know about this simple file which matters a lot from SEO point of view.
Thanks for your comment and shares.
Have a remarkable week ahead:)
~Ravi