How to block indexing of a parent page but not child pages by robots.txt or meta tag?

If x=y

Joined
Oct 29, 2020
Messages
13
Likes
4
Degree
0
Hello buso, i want to ask about blocking indexing for this kinda situation.

i want to block pages like this : https://www.domain.com/travel

but i don't want to block any pages under those pages, just because this pages ( /travel ) is already depricated, maybe getting update or not, i don't know yet.

any suggestion? Thank you.
 
Last edited by a moderator:
You can't block indexing from robots.txt, only crawling. A page can be indexed without being crawled. If you try this, you will be unhappy with the results as you start getting blank pages indexed with no description in the SERPs and whatever anchor text you linked to it from as the page title. Your Panda quality score will drop until you're basically penalized by the algorithm.

Your choices come down to:
  • sending HTTP headers that say to noindex before the page even loads
  • setting meta tags in the <head> with noindex directives in the source code
The way you'll achieve that will depend on if you're using an Apache (.htaccess file) or Nginx (.conf file) server for the first option, or what CMS you're using if any for the 2nd option.

Generally, with the first option, you'll be able to target those parent pages themselves without extending down to the child pages based on how you do the URL matching (with Regex probably).

With the 2nd option, CMS's like Wordpress offer easy pre-made functions to help you target what you want using if statements in PHP. Alternatively, a plugin like Yoast can help you do this if you already use one. If you're using something like Laravel or Drupal, you can check their documentation to see what functions they have available for this type of targeting and injecting into the <head>.
 
Thank you @Ryuzaki for the responses,

What about redirecting, domain.com/hotels to domain.com/hotels/all, it's all about remove hotels from search engine, and the valid pages is /all.

if i want to use meta-tag no-index on hotels, is not possible rights? i'am using laravel.

any advice guys?

thank you
 
If x=y said:
if i want to use meta-tag no-index on hotels, is not possible rights?
Click to expand...
No, it is very possible to use a meta noindex directive in the <head>.

Another option you could consider is setting the canonical URL on the /hotels page to /hotels/all. That would cause it to drop from the index, without any redirects, while still being available for users and Google. Google will assign it's "power" to the /hotels/all page.

Doing a redirect makes less sense in my mind for Google and your users.
 
Ryuzaki said:
No, it is very possible to use a meta noindex directive in the <head>.

Another option you could consider is setting the canonical URL on the /hotels page to /hotels/all. That would cause it to drop from the index, without any redirects, while still being available for users and Google. Google will assign it's "power" to the /hotels/all page.

Doing a redirect makes less sense in my mind for Google and your users.
Click to expand...
thank you ryu.
The main purposes is to no-index nor removed /hotels from indexing.

after i set canonical, how long does it take for google to removed the pages ( also if i implement no-index to the pages that already indexed, and i want to remove) ? should i request removal or just wait?

-----

sorry ryu, maybe for the best example is when you use wordpress but the categorize is not gonna being index, so you can setup in Yoast Setting, confusing :(
 
Last edited by a moderator:
If x=y said:
after i set canonical, how long does it take for google to removed the pages ( also if i implement no-index to the pages that already indexed, and i want to remove) ? should i request removal or just wait?
Click to expand...
It takes as long to be removed from the index as it takes for Google to recrawl the page and see the canonical tag. Same with setting up a noindex tag. You can either wait or you can request a crawl in Google Search Console for each page. Alternatively you can create a temporary sitemap that includes just the pages you want re-crawled and upload that to Search Console. Then once you've confirmed they've crawled the pages, you can remove the sitemap from your server and from Search Console.
 
You must log in or register to reply here.
Back