Website owners use a robots instruction file (also referred to as a “robots.txt” file) to give instructions about their site to search engine robots.
There are two important considerations when using robots instruction file:
- Robots can ignore your instruction file. In particular malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- Your robots instruction file is publically available; therefore anybody can see which sections of your website that you do not want robots to access.
Note Due to the nature of a robots instruction file, it is important that the file is not used with the sole intention of hiding information from visitors.
To edit your Robots.txt file
- From your Divendo admin, select Optimise.
- From the Site optimisation menu, select Edit robots file. You will be presented with the Robots file editor.
- Enter your desired robots configuration, and click Save.