robots.txt ignore only slug only

Question

I want to achieve that behavior:

Allow: /plans and Disallow: /plans/*

crawl: www.example.com/plans

Do not crawl:

• www.example.com/plans/abcd-edfg-3fdv-3455
• www.example.com/plans/3fdv-3455-edfg-avhn

Answer 1

It would be:

Allow: /plans$
Disallow: /plans/

Entries are assumed to have a trailing wildcard so /plans/ and /plans/* are the same thing. However, this also means that /plans will also match /plansandstuff. This can be dealt with by using $ which matches "end of path".

See also: Robots.txt Specification

Keep in mind that the robots.txt file is advisory and not all crawlers pay attention to it.