Add a robots.txt file

Having a robots.txt file is important for SEO. You can read more about search engine optimization with Sitejet in our SEO article. In this article, we will show you how to set up your project's robots.txt file with Sitejet.

Table of contents

Basic knowledge

Add a robots.txt file to your project

Basic knowledge

A robots.txt file contains information for search engines that specifies which pages and subpages are to be indexed and at which time intervals. Further information on the possible specifications and regulations are defined in the Robots Exclusion Standard Protocol.

Add a robots.txt file to your project

1. From the Website Builder (CMS), navigate to "Website" in the top menu and click "Media", then upload your robots.txt file in the Media Manager

2. From the Media Manager, click the open icon to open the file in a new tab of your browser

3. From the new browser tab, copy the part of the URL highlighted in the image below.

4. Go back to the Website Builder (CMS) and hover over "Website"

5. Click the "Redirections" menu item

6. Click "+ Add rule"

7. Enter now "/robots.txt" in the "From" field

8. Paste the part of the URL you copied in step 3 into the "To" field

9. Select either "301" or "302" in the redirect "Type" field

Your Robots.txt file can now be found by search engines at its expected location.

Back to the top

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us