Crawl → render → index
Search engines crawl (discover URLs), render (run the page like a browser), and index (store it for ranking). A page can't rank if it never reaches the index — so technical SEO starts here.
Common blockers
- robots.txt disallowing important paths
- noindex tags left on by mistake (a classic post-launch disaster)
- Pages with no internal links (orphans) that crawlers never find
- Infinite spaces (faceted filters, calendars) that waste crawl budget
- Slow or error-prone responses that cut crawling short
Your toolkit
- Google Search Console → Pages report: see what's indexed and why pages are excluded.
- URL Inspection: test how Google sees a specific URL and request indexing.
- Crawl your own site (GrowMyWebsite's crawler / a desktop crawler) to find broken links, redirect chains, and orphans.
Exact steps: check why pages aren't indexed
- Open Search Console and select your property.
- In the left menu, click Indexing → Pages.
- Scroll to Why pages aren't indexed and read the reasons — common ones are "Excluded by 'noindex' tag", "Crawled – currently not indexed", and "Duplicate without user-selected canonical".
- Click a reason to see the exact affected URLs, fix the cause, then press Validate fix.
A concrete example: a team launches a redesign but forgets to remove the site-wide noindex tag that protected the staging site. Weeks later, traffic has quietly cratered. The Pages report shows the real pages sitting under "Excluded by 'noindex' tag" — a five-minute fix that was invisible without opening the report.
The mindset
Most technical SEO is removing friction so engines can reliably find, render, and store your best pages. Fix the blockers first; optimization comes after the page can actually be indexed.