What is Robots.txt?
Robots.txt is a text file in which site operators and administrators can determine which pages may and may not be searched and listed by the search engine crawler. Robots.txt is essential for pages that are aiming for a good ranking on Google or other search engines. The search engine crawlers always check whether a Robots.txt file exists in the directories of a website before continuing their search in other directories. It is therefore always worth creating the text file and storing it in the main directory.
In Robots.txt, individual areas, directories and pages can also be blocked for a search engine crawler. This procedure can be helpful if you do not want some information or subpages to be listed on the respective search engine. You can also define differences for the search engines. For example, you can specify that the Google search engine crawler may search a certain directory or page, but the Bing or Yahoo crawler may not. This allows you to define exactly how the bots should behave when searching the website.
Where must a Robots.txt file be stored?
The search engines try to check the existence of a Robots.txt and also take it into account. To do this, however, they must also find the Robots.txt. In order for it to be taken into account, it must therefore always be stored in the main directory of a page. If the Robots.txt is stored in a different directory, it cannot be taken into account by the search engine crawlers.
How many Robots.txt files can be uploaded per page?
In order to ensure that the search engine bots also find and respect the Robots.txt file, it is important that there is only a single text file with such a name. Although there may be several Robots.txt files in the main directory for test purposes at the beginning, the page should not yet be listed with search engines. The crawler can only follow the commands if it finds exactly one text file in the main directory.
How secure is the creation of a Robots.txt file?
There is no guarantee that the pages in the Robots.txt file will not be taken into account by search engines. The small text file in the main directory only serves as an aid and is generally taken into account by every search engine crawler. Robots.txt cannot usually provide sufficient protection against other types of unauthorized access. For this purpose, the directory in the web server should always be provided with a password. However, the page can then no longer be viewed by normal users.
How is a Robots.txt file structured?
There are always two related elements in the Robots.txt file. First, you must define which crawler you want to address with a particular command. The respective command must then be named in the next element. The command to be executed must always contain the directory and the information as to whether the specified page is to be read by the crawler or not. When naming the crawlers, the name of the search engines must be taken into account. A list helps when creating the Robots.txt file:
-
MSN Crawler – Msnbot
-
Yahoo Crawler – Slurp
-
Bing Crawler – Bingbot
-
Google Crawler – Googlebot
