Robots.txt

What is robots.txt?

Search engine spiders are given specific pages or sections of a website to not index via a file called Robots.txt. The majority of significant search engines, including Google, Bing, and Yahoo, are aware of and abide by Robots.txt instructions.