Edit Robots.txt File

  • Small Business
  • Enterprise

Website owners use a robots instruction file (also referred to as a “robots.txt” file) to give instructions about their site to search engine robots.

There are two important considerations when using robots instruction file:

  • Robots can ignore your instruction file. In particular malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  • Your robots instruction file is publically available; therefore anybody can see which sections of your website that you do not want robots to access.

Note Due to the nature of a robots instruction file, it is important that the file is not used with the sole intention of hiding information from visitors.

To edit your Robots.txt file

  1. From your Divendo admin, select Optimise.
  2. From the Site optimisation menu, select Edit robots file. You will be presented with the Robots file editor.
  3. Enter your desired robots configuration, and click Save.