Add a robots.txt file

Having a robots.txt file is important for SEO. You can read more about search engine optimization with Sitejet in our SEO article. In this article, we will show you how to set up your project's robots.txt file with Sitejet.

Basic knowledge

A robots.txt file contains information for search engines that specifies which pages and subpages are to be indexed and at which time intervals. Further information on the possible specifications and regulations are defined in the Robots Exclusion Standard Protocol.

Add a robots.txt file to your project

1. From the Website Builder navigate to "More" in the top right corner and select "Files", then upload your robots.txt file in the File Manager

2. From the File Manager, click the open icon to open the file in a new tab of your browser

3. From the new browser tab, copy the part of the URL highlighted in the image below.

4. Go back to the Website Builder and click on the burger menu in the top left corner

5. Click the "Redirections" menu item

6. Click "+ Add rule"

7. Select either "301" or "302" in the redirect "Type" field

8. Enter now "/robots.txt" in the "From" field

9. Paste the part of the URL you copied in step 3 into the "To" field

Your Robots.txt file can now be found by search engines at its expected location.

Test your robots.txt file

You can test your robots.txt file here.

Become a part of our Community!

Exchange ideas with other web designers about current developments, tips, and tricks and show your favorite sites. Get advice and talk to us about possible features you would like to see on Sitejet. You can join the Sitejetters community here.

Back to the top