Add a robots.txt file
Having a robots.txt file is important for SEO. You can read more about search engine optimization with Sitejet in our SEO article. In this article, we will show you how to set up your project's robots.txt file with Sitejet.
Table of contents
A robots.txt file contains information for search engines that specifies which pages and subpages are to be indexed and at which time intervals. Further information on the possible specifications and regulations are defined in the Robots Exclusion Standard Protocol.
Add a robots.txt file to your project
1. From the Website Builder (CMS), navigate to "Website" in the top menu and click "Media", then upload your robots.txt file in the Media Manager
2. From the Media Manager, click the open icon to open the file in a new tab of your browser
3. From the new browser tab, copy the part of the URL highlighted in the image below.
4. Go back to the Website Builder (CMS) and hover over "Website"
5. Click the "Redirections" menu item
6. Click "+ Add rule"
7. Enter now "/robots.txt" in the "From" field
8. Paste the part of the URL you copied in step 3 into the "To" field
9. Select either "301" or "302" in the redirect "Type" field
Your Robots.txt file can now be found by search engines at its expected location.
Test your robots.txt file
You can test your robots.txt file here.