Add a robots.txt file
A robots.txt file is an important part of the search engine optimization of your website. In this article we explain how to add a robots.txt file and what function it has.
Table of contents
Basic knowledge
Search engines have two tasks on the web: Searching for content on the web and indexing it so that other searchers can find it in an orderly fashion. When arriving at a website, the first thing a search engine crawler (such as a Google or Bing bot) usually looks for is a robots.txt file. If it finds one, it reads the contents before retrieving other pages.
There are many good reasons why you should use a robots.txt file. For example, you can specify here what the crawlers index or what content you might want to hide. We also recommend storing the sitemap.xml in the robots.txt file. This way you can be sure that a crawler will call it up at all and that your page will be indexed correctly. In the sitemap, you determine which pages are indexed and which are not.
For more information on the sitemap, we have a separate article for you.
It is important to mention that the robots.txt file is only a help for the crawlers. It is not guaranteed that the websites will not be crawled or vice versa. Google and Bing, for example, both state that they respect the robots.txt file, but they are not obliged to do so and we have seen cases, where the bots have ignored the requests. On the internet you will find many instructions on how to create a robots.txt and which instructions you should enter. The syntax is very simple. Every search engine has its own help articles in its database, like
Google for example (external link). There are even generators that do the work for you. Further information on possible specifications and regulations have been defined in the
Robots Exclusion Standard Protocol (external link).
You can test your robots.txt file under this link (external link).
Note: For more tips and tools, we recommend our article on frequently asked questions about search engine optimization. Please note that neither SEO strategy nor problems with submitting the Robots.txt file to the search engines are our responsibility. We recommend our community, where you can exchange ideas with other professionals.
Add a robots.txt file to your project
In order for the search engines to find the file, it must be located in the main directory of the domain.
If the file is hidden somewhere else, the crawlers assume that there is none. We can achieve this with a simple trick.
1. Upload your robots.txt into the media manager of your website.
2. From the File Manager, open the file in a new browser tab. This way, you will get to know the URL that you need. You could also click on "Get Link". This way, the link is directly copied into your clipboard.
3. From the new browser tab, copy the whole URL.
4. Go back to the Website Editor tab and click on the burger menu in the top left corner.
5. Click the "Redirections" menu item
6. Click "+ Add rule"
7. Select either "301" or "302" in the redirect "Type" field. We recommend a 301. Paste the copied URL as described.
Your Robots.txt file can now be found by search engines at its expected location.
Become a part of our Community!
Exchange ideas with other web designers about current developments, tips, and tricks and show your favorite sites. Get advice and talk to us about possible features you would like to see on Sitejet. You can join the Sitejetters community here.
Back to the top
