Before you go, check out these stories!

0
Hackernoon logoGlossary of Security Terms: Robots.txt by@mozilla

Glossary of Security Terms: Robots.txt

;
Author profile picture

@mozillaMozilla Contributors

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

Robots.txt is a file which is usually placed in the root of any website. It decides whether crawlers are permitted or forbidden access to the web site.

For example, the site admin can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines.

Learn more

General knowledge

View Previous Terms:

Author profile picture

@mozillaMozilla Contributors

Read my stories

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.