How do I disallow specific page from robots.txt

4 Points
2 Posts

I have few pages in website and don't want to crawled by any crawlers. How do I disallow specific page, directory using robot.txt.

Views: 8134
Total Answered: 3
Total Marked As Answer: 0
Posted On: 28-Feb-2019 23:06

Share:   fb twitter linkedin
294 Points
82 Posts

You can also add a specific page or directory with extension in robots.txt file. Use Disallow keyword as follow:

Disallow: /index_test.html
Disallow: /products
Posted On: 01-Mar-2019 05:59
224 Points
52 Posts

Following are methods to disallowed index:

  • using either robots.txt or
  • the meta robots tag
Disallow: /index.html
Disallow: /prod
<meta name="robots" content="noindex, nofollow" />

The next time your site is indexed this will make your content drop out of the Google index.

You can also you the noarchive value – this will block caching of your page. This is Google specific:

<meta name="robots" content="noarchive" />
Posted On: 04-Mar-2019 08:24
Deven Malik
Deven M...
4 Points
2 Posts

This tag will work <meta name="robots" content="noindex, nofollow" />

Posted On: 09-Feb-2022 02:40
 Log In to Chat