You shall then need to ensure that your particular robots.txt file is conserved. Be sure that you perform some after in order for internet crawlers will be able to find and recognize your file.
- Save the robots.txt file as being a text file.
- Put the file inside the directory that is highest-level of site ( or in the source associated with the domain).
- The file needs to be known as robots.txt.
- Example: a robots.txt file that is conserved in the cause of test.com aided by the Address of http://www.sample.com/robots.txt is discoverable by internet crawlers, if the robots.txt file is found at a URL like http://www.sample.com/not_root/robots.txt it will probably maybe perhaps not manage to be located by internet crawlers.
There is certainly a tool that is testing for robots.txt, and it surely will demonstrate in the event that file is effectively blocking Bing’s internet crawlers from accessing certain links on your website. The device has the capacity to run the same as Googlebot does, and verifies that all things are working correctly.
To try the file, follow these directions: