Add a robots.txt file
A robots.txt file is an important part of the search engine optimization of your website. In this article we explain how to add a robots.txt file and what function it has.
Table of contents
You can test your robots.txt file under this link (external link).
Add a robots.txt file to your project
In order for the search engines to find the file, it must be located in the main directory of the domain.
If the file is hidden somewhere else, the crawlers assume that there is none. We can achieve this with a simple trick.
1. Upload your robots.txt into the media manager of your website.
2. From the File Manager, open the file in a new browser tab. This way, you will get to know the URL that you need. You could also click on "Get Link". This way, the link is directly copied into your clipboard.
3. From the new browser tab, copy the part of the URL highlighted in the image below.
4. Go back to the Website Editor tab and click on the burger menu in the top left corner.
5. Click the "Redirections" menu item
6. Click "+ Add rule"
7. Select either "301" or "302" in the redirect "Type" field. We recommend a 301. Paste the copied URL as described.
Your Robots.txt file can now be found by search engines at its expected location.
Become a part of our Community!
Exchange ideas with other web designers about current developments, tips, and tricks and show your favorite sites. Get advice and talk to us about possible features you would like to see on Sitejet. You can join the Sitejetters community here.