How do I disallow specific page from robots.txt

kd
kd
4 Points
2 Posts

I have few pages in website and don't want to crawled by any crawlers. How do I disallow specific page, directory using robot.txt.

Views: 7120
Total Answered: 2
Total Marked As Answer: 0
Posted On: 28-Feb-2019 23:06

Share:   fb twitter linkedin
Answers
beginer
beginer
72 Points
31 Posts
     

You can also add a specific page or directory with extension in robots.txt file. Use Disallow keyword as follow:

Disallow: /index_test.html
Disallow: /products
Posted On: 01-Mar-2019 05:59
Priya
Priya
Member
62 Points
26 Posts
     

Following are methods to disallowed index:

  • using either robots.txt or
  • the meta robots tag
Disallow: /index.html
Disallow: /prod
<meta name="robots" content="noindex, nofollow" />

The next time your site is indexed this will make your content drop out of the Google index.

You can also you the noarchive value – this will block caching of your page. This is Google specific:

<meta name="robots" content="noarchive" />
Posted On: 04-Mar-2019 08:24
banner

Blog

Active User (1)

 Log In to Chat