Separate crawling from indexing
Review the availability of robots.txt and any page-level indexing directives found in the response. A crawl rule controls fetching, while noindex signals that a page should be excluded from search results. The audit respects its applicable crawl rules, so a blocked page may not produce an HTML report. Use your search-engine webmaster tools to confirm actual index coverage.