Meta robots tag
The meta robots tag is a meta tag in the header of the respective source code of a website. The behavior of spiders (see spider), which scan the web for search engines, is controlled by this tag. There are, for example, settings such as follow, nofollow, index or noindex. This specifies whether the spider should follow the hyperlinks on the website or not or whether the entire website should be indexed or not. A robots.txt file must be used to prevent entire directories being checked by a spider.