robots.txt controls crawling, not indexing.
A URL blocked in robots.txt can still appear in search results if other pages link to it; the engine just cannot see what is on it. If the goal is to keep a page out of the index, it has to be crawlable with a noindex, or removed.
How often do you still find this mixed up on client sites?
#TechnicalSEO
Comments (0)
Sign in to comment.