This technical SEO guide exists to answer one question: why does a site with good content still fail to rank, and what do you fix first? The short version is that search engines and AI answer engines have to crawl, render, index and trust your pages before content or links matter at all. Technical SEO is the work that makes your site eligible to appear. Everything else is a bet placed on top of that eligibility.
We run audits for businesses in India and the UK, and the same pattern shows up on nearly every site: the content is fine, sometimes excellent, but the plumbing underneath is leaking ranking signals. Pages Google never crawls. Pages it crawls but refuses to index. JavaScript that Googlebot renders but ChatGPT and Claude never see. This guide is the order we actually work in, and the reasons behind each step, so you can diagnose your own site instead of guessing.
Key factors before you start
Technical SEO breaks into five layers, and they have to be fixed in order because a problem in an early layer hides problems in the later ones. The layers are crawlability, indexing, rendering, performance, and structured data. Fix crawling before you touch page speed. A blazing-fast page that Google cannot reach earns you nothing.
The 2026 shift is worth knowing: AI answer engines like ChatGPT, Claude and Perplexity read your raw HTML and do not execute JavaScript. Independent crawler analysis through mid-2026 found no JavaScript execution from the major AI crawlers (SearchOptimo). If your content only appears after a script runs, those engines see an empty shell. Technical SEO is now the gatekeeper for AI citations, not only Google rankings.
What is technical SEO, and why does it come before content?
Technical SEO is the practice of optimising your website's infrastructure so search engines and AI systems can discover, crawl, render, index and cite your pages. It covers site architecture, crawl controls, page speed, mobile behaviour, canonicalisation and structured data, everything except the words on the page and the links pointing to it.
It comes first because search is a pipeline with hard gates. A page has to be crawled to be seen, rendered to be understood, and indexed to be eligible for ranking. Miss any gate and the content behind it is invisible, no matter how good it is. Google's own documentation frames the fundamentals around exactly this sequence of crawling, rendering and indexing (Google Search Central). Most site owners spend months polishing content that sits behind a closed gate. An hour spent finding the gate is worth more than another blog post.
There is a limit to how far technical work takes you, and honest advice includes the ceiling. Once your pages are crawlable, indexable and reasonably fast, additional technical tweaks give diminishing returns. At that point content quality and links do the heavy lifting. Technical SEO removes the handbrake. It does not replace the engine.
Layer 1: Can search engines crawl your site?
Crawling is how search engines follow links from pages they already know to find pages they have not seen. If a crawler cannot reach a page, nothing downstream happens. This is where any real technical SEO audit begins.
Start with robots.txt, and understand what it does not do
Your robots.txt file sits at the root of your domain and tells crawlers which paths they may request. The trap that catches people: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still be indexed if other sites link to it, and Google will show it with no description because it was never allowed to read the page. Worse, if you block a page in robots.txt, Google cannot see the noindex tag or canonical tag sitting on that page, so those instructions are silently ignored.
The practical rule we give clients: use robots.txt to keep crawlers out of infinite spaces like faceted-search parameter URLs and internal search results. Use a noindex meta tag, not robots.txt, when you want a page kept out of the index. The two tools solve different problems, and swapping them is one of the most common self-inflicted wounds we find.
Then look at your XML sitemap
An XML sitemap is a machine-readable list of the URLs you want indexed, each ideally carrying a lastmod date so crawlers know what changed. A clean sitemap should contain only canonical, indexable, 200-status URLs. No redirects, no noindex pages, no 404s, no parameter duplicates. Keep each file under 50,000 URLs and 50 MB, and submit it in Google Search Console.
The mistake we see most often is a sitemap generated automatically by a plugin that dumps every URL on the site, including tag archives, paginated pages and thin category pages. That teaches Google that your sitemap is noise and dilutes crawl attention across pages you do not care about.
Watch your crawl budget on larger sites
Crawl budget is the number of URLs a search engine will fetch from your site in a given window. For a 50-page brochure site it is irrelevant. For an e-commerce catalogue with tens of thousands of parameter combinations, it decides whether your new products get discovered this week or next month. Duplicate URLs, conflicting canonicals and dead ends waste that budget on pages that will never rank.
The way to see the truth here is log file analysis. Your server logs record every Googlebot request, so you can compare what Google actually crawls against what you want crawled. The gap is usually surprising: crawlers burning requests on old parameter URLs while fresh product pages wait. No third-party tool estimates this. The logs are the ground truth.
Layer 2: Is your site actually being indexed?
A crawled page is not an indexed page. Indexing is the step where Google decides a page is worth storing and showing. Plenty of pages get crawled and then dropped, and the Page Indexing report in Search Console tells you which, with reasons like "Crawled - currently not indexed" or "Duplicate without user-selected canonical."
Canonical tags: telling Google which version is the master
A canonical tag tells search engines which URL is the primary version when several URLs hold similar or identical content. This matters for HTTP versus HTTPS, www versus non-www, trailing-slash variants, parameter URLs from filters and tracking, and product variants. Without a clear canonical signal, Google picks one for you, and it may not pick the version you want ranking.
Over 60% of enterprise sites carry conflicting canonical signals or broken hreflang, splitting PageRank across duplicate URLs (DigitalApplied). The fix is boring and effective: every page should declare a self-referencing canonical unless it is a genuine duplicate, in which case it points to the master. Then make sure your internal links, sitemap and canonical all agree on the same URL. Mixed signals are worse than none.
The "Crawled - currently not indexed" problem
This status is Google's polite way of saying it looked at your page and decided it was not worth keeping. On the sites we audit, the cause is usually thin or near-duplicate content, weak internal linking that leaves the page orphaned, or a template that produces hundreds of nearly identical pages. The fix is rarely technical alone. It is consolidation: merge thin pages, strengthen internal links to the ones that matter, and remove the ones that serve no searcher.
Layer 3: Rendering, JavaScript, and the AI-crawler blind spot
Rendering is where the modern web breaks quietly. Many sites deliver a near-empty HTML shell and build the real content with JavaScript in the browser. Google can handle this. It renders JavaScript in a second wave of processing, though that adds a delay of days for updates on client-side-rendered sites.
Here is the part most guides published before 2026 miss. AI answer engines do not render JavaScript at all. GPTBot, ClaudeBot and PerplexityBot read the raw HTML your server sends and stop there (SearchOptimo). If your product descriptions, prices, or article body only exist after a script runs, those engines see nothing to cite. As AI Overviews and chat assistants take a growing share of how people find businesses, this is no longer a niche concern.
The reliable answer is server-side rendering or static generation, so the meaningful content exists in the HTML before any JavaScript executes. If a full re-architecture is not on the table, prerendering or dynamic rendering for bots buys time. The test is simple and free: open your page, view source, and search the raw HTML for a sentence from your main content. If it is not there, neither Googlebot's first pass nor any AI crawler sees it.
Layer 4: Core Web Vitals and real page speed
Once a page can be crawled, indexed and read, speed becomes a ranking and conversion factor. Google measures three Core Web Vitals from real Chrome users, reported at the 75th percentile over a rolling 28-day window (Google Search Central).
The "good" thresholds you are aiming for:
Largest Contentful Paint (LCP) under 2.5 seconds. This is how fast the main content appears.
Interaction to Next Paint (INP) under 200 milliseconds. This is how quickly the page responds when someone taps or clicks.
Cumulative Layout Shift (CLS) under 0.1. This is how much the layout jumps around while loading.
One update that catches people out: INP replaced First Input Delay in March 2024 (web.dev). Any guide still coaching you to optimise FID is out of date, and INP is a harder metric to pass because it measures every interaction, not just the first. Heavy third-party scripts, chat widgets and analytics tags are the usual culprits behind a poor INP.
For businesses serving both India and the UK, physical distance to your server adds latency that no amount of code-splitting fixes. A UK-hosted site serving Mumbai visitors, or the reverse, pays a round-trip tax on every request. A content delivery network that caches assets close to each audience is the single highest-leverage speed fix for cross-border sites, and it is the one we recommend before micro-optimising images. Field data from the Chrome User Experience Report, not lab scores from a one-off test, is what Google actually grades, so optimise for real users in both regions rather than a green score on a fast office connection.
Layer 5: Structured data that earns rich results and AI citations
Structured data is code, usually in JSON-LD format, that labels what your content means: this is a product, this is its price, this is an FAQ, this is the author. It does not change what visitors see. It changes what machines understand.
Two payoffs. First, valid structured data makes you eligible for rich results in Google, the star ratings, FAQ dropdowns and product details that can meaningfully lift click-through. Second, and increasingly important, clean schema helps AI systems parse, quote and correctly attribute your content. Article, Organization, FAQ and HowTo schema are the ones that pull the most weight for AI visibility.
The Schema.org vocabulary defines what is available, and Google's structured data documentation defines what actually earns features in search (Google Search Central). Validate everything in the Rich Results Test before you ship it. Invalid markup earns you nothing and, if it misrepresents the page, can invite a manual action. Mark up what is genuinely on the page, and no more.
Mobile-first indexing and HTTPS: the non-negotiables
Google indexes the mobile version of your site, not the desktop one. If content, headings or structured data are stripped out on mobile to make the layout tidy, Google indexes the stripped-down version. Parity between mobile and desktop content is the rule. What is not on your mobile page effectively does not exist for ranking.
HTTPS is a baseline expectation, not an upgrade. Serve every page over a valid certificate, redirect HTTP to HTTPS with 301s, and make sure you are not mixing insecure assets into secure pages. This is table stakes, and the only reason it is worth stating is that we still find sites in 2026 leaking ranking equity through a half-finished HTTPS migration with the old HTTP URLs still crawlable.
How to run a technical SEO audit in the right order
A technical SEO audit works when you follow the pipeline, because early problems mask later ones. If you measure page speed on pages Google cannot even crawl, you are optimising the wrong thing.
The order we use on every engagement: confirm crawlability first (robots.txt, sitemap, internal links, log files), then indexing (Search Console coverage, canonicals, duplicates), then rendering (view-source test, server-side rendering for critical content), then performance (Core Web Vitals from field data), and finally structured data. Fixing them out of order wastes effort, because a canonical fix is pointless on a page that is blocked from crawling in the first place.
If you would rather have a specialist run this end to end on your site, our technical SEO audit service applies exactly this sequence and hands you a prioritised fix list rather than a 200-line report you cannot action. You can also work through our broader SEO services or get in touch with the specific symptom you are seeing, and we will tell you which layer it lives in.
Frequently asked questions
How long does technical SEO take to show results?
Crawling and indexing fixes can show up within days once Google recrawls the affected pages, because you are removing a block rather than building authority. A fixed canonical or an unblocked page can re-enter the index in a week or two. Core Web Vitals improvements take longer to reflect because Google grades them on a rolling 28-day window of real user data, so budget roughly a month before the field scores update.
Do I need technical SEO if my site is small?
Yes, though the scope shrinks. A small site does not need crawl-budget management, but it still needs a clean sitemap, correct canonicals, a mobile-parity check, HTTPS done properly, and content that exists in the raw HTML. The difference is that a small site can usually complete a full technical SEO audit in an afternoon, while a large catalogue site needs ongoing monitoring.
What is the difference between technical SEO and on-page SEO?
Technical SEO makes your pages accessible and understandable to machines: crawling, indexing, rendering, speed and structured data. On-page SEO optimises the content itself: titles, headings, keyword coverage and internal linking for relevance. Technical SEO decides whether a page is eligible to rank. On-page SEO influences how well it ranks once it is eligible.
Will technical SEO help me show up in ChatGPT and AI Overviews?
It is now a prerequisite. AI crawlers read raw HTML and do not execute JavaScript, so content that only appears after a script runs is invisible to them. Server-side rendering, clean structured data and crawlable pages are what let AI engines find, parse and cite your site, on top of the content quality that decides whether they choose to.
Where to start this week
If you do one thing after reading this technical SEO guide, open Search Console and read the Page Indexing report, then view-source on your three most important pages and confirm the main content is in the raw HTML. Those two checks surface the highest-impact problems on most sites, the pages Google is quietly dropping and the content AI engines cannot see. Fix the gate before you polish what is behind it.
