Start with intended pages and reusable evidence
List the public pages you want searchers to find: key products, articles, categories, and moved URLs. For each, record the preferred URL, expected response, indexability, template, owner, and priority. Keep legitimate exclusions separate.
Capture an unauthenticated response, headers, redirect path, page source, rendered view, internal-link state, and sitemap state for representative URLs. Sample every template and URL pattern. Evidence is dated: a CDN setting, deployment, consent layer, or personalization can change it later.
A passing homepage does not prove that listings, product variants, or localized routes work.
Keep dated headers and rendered evidence so fixes and later regressions can be compared.
Make indexability a deliberate decision
Indexability means a page may be included in a search index; it is not the same as crawling or ranking. Check meta robots directives in HTML and X-Robots-Tag response headers. The noindex directive tells search engines not to show a page; X-Robots-Tag also applies to non-HTML files such as PDFs.
Review robots.txt separately. A crawler must access a page to find its noindex directive, so blocking that URL can prevent the directive being seen. Check the deployed, public response because CMS settings and environment flags can add rules. Noindex is not access control: protect sensitive material with authentication and appropriate security review.
Flag noindex, login responses, staging references, and blocked essential assets on intended-index pages.
An HTTP response header can carry a directive even where no HTML head exists.
Make one preferred URL consistent
Canonicalization identifies the preferred URL among duplicate or very similar versions, such as HTTP and HTTPS, hosts, tracking parameters, filters, or print pages. Compare the canonical element, internal links, sitemap URL, and redirect destination. For an indexable page, they should normally identify the same absolute preferred URL.
Use a permanent server-side redirect when retiring a duplicate. Google describes permanent redirects and rel=canonical as strong signals, while sitemap inclusion is weaker; aligned signals reinforce the preference. Do not use robots.txt to canonicalize or give one page competing canonicals. A self-referencing canonical helps state the preferred version, but search-engine selection remains outside your control.
Document variants, the preferred target, the required control, and an owner.
Test old-to-new mappings before a migration and keep a reversible redirect change ready.
Check status codes and redirect behavior
Status codes tell browsers and crawlers what happened. Important live pages should return a meaningful 2xx response with intended content. A page with no relevant replacement should return a genuine 404 or 410; moved content should reach its closest relevant replacement. A 200 page that only shows an error may be treated as a soft 404.
Record every redirect hop. Google generally follows up to 10 hops for general web crawling, but chains add latency and fragility. Google says 5xx and 429 responses can temporarily slow crawling and persistent server errors can ultimately remove indexed URLs. Investigate recurring failures with the engineering owner across application, CDN, origin, cache, dependency, and deployment layers.
Save original URL, hops, codes, final URL and expected state.
Remove overlapping CMS, CDN, host, locale, or slash rules instead of adding another hop.
Validate JavaScript rendering and routes
Inspect both the initial HTML and a rendered view for key templates. Confirm the main content, headings, links, canonical, robots directives, and state exist after rendering. Google describes separate crawl, render, and index phases and says content not visible in rendered HTML cannot be indexed.
Use anchor elements with href values for navigation. Fragment-only routes that load distinct content are not reliable for discovery. Directly request nonexistent products, articles, and filtered URLs. Client-side applications need a real error state; Google documents routing to a server 404 or adding noindex to avoid soft 404s. Server rendering or prerendering can help, but verify the deployed response.
Check response HTML, rendered HTML, and the human browser view for material differences.
Prefer source HTML; if JavaScript sets it, ensure there is one non-conflicting canonical.
Keep sitemaps and internal links clean
A sitemap is a discovery map, not an indexing command. Include successful, intended, preferred canonical URLs; exclude redirects, errors, noindex pages, parameter duplicates, and protected URLs. Google recommends fully qualified absolute URLs and treats sitemap inclusion as a canonical hint, not a guarantee.
Validate the sitemap and compare it with the intended-index list and navigation. Google documents UTF-8 encoding and a limit of 50 MB uncompressed or 50,000 URLs per sitemap. For larger sites, split into useful groups and use an index. Update lastmod only for significant, verifiable page changes.
Investigate meaningful differences between intended URLs, sitemap entries, and internal links.
Check the generated sitemap and a sample of listed URLs after routing or CMS changes.
Measure performance and institutionalize the audit
Use field data where available and controlled lab testing to investigate Core Web Vitals: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for visual stability. Segment by page type and device because averages can hide a broken high-value template.
Trace slow or unstable delivery through images, CSS and JavaScript, third-party tags, fonts, embeds, and caching. Make one measured change, then recheck rendering and user flows. Field data reflects real users; lab data aids reproduction. Neither predicts every future crawler or visitor. Repeat a focused regression check after risky releases and a fuller audit after migrations or platform changes.
Repair failures on revenue-critical or heavily used templates before small score gains elsewhere.
Recheck response, canonical, rendered content, sitemap state, and relevant performance evidence before closing.