To stop unwanted content during the look for indexes, website owners can instruct spiders to not crawl selected files or directories throughout the standard robots.txt file in the basis Listing with the domain. On top of that, a site can be explicitly excluded from a online search engine's databases through the use of a meta tag specific to robots