Hello friends,
I would like to know that What is the use of robert.txt file ? tell me guys..
If you already have an account with us, please use the login panel below to access your account.
Hello friends,
I would like to know that What is the use of robert.txt file ? tell me guys..
Robots.TXT helps in preventing the CRAWLERS to crawl the private and confidential information like Email address, contact number, credit card details.
Use:
Block one folder
Block one file
Allow full access
Block all access
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Bookmarks