How to Diagnose Soft 404s With AI

A soft 404 occurs when a URL appears to be an error page even though the server returns a successful status code—usually 200 OK. It may show “page not found,” contain almost no useful content, or fail to render its main content.

Google may classify such a URL as a soft 404 and exclude it from Search. This classification is based on the page’s content, not on an actual 404 response from the server, according to Google Search Central’s crawling-error documentation.

AI can make diagnosis faster by grouping affected URLs, comparing page content, identifying recurring templates, and recommending likely actions. It cannot determine the correct fix by itself. You still need server responses, rendered-page evidence, business context, and human review.

The short diagnostic workflow

Use this sequence:

Export soft 404 examples from Google Search Console.
Crawl each URL and record its final status, redirect path, title, headings, body text, canonical, robots directives, and content length.
Compare raw HTML with rendered content for JavaScript-dependent pages.
Ask AI to cluster URLs by page type and likely failure pattern.
Review representative URLs from every cluster.
Assign each URL one of four actions: restore, improve, redirect, or return a real 404/410.
test the fixes and start validation in Search Console.

The important principle is simple: let AI organize the evidence, but do not let it invent the evidence or make irreversible redirect decisions.

Start with Google Search Console

Open the Page indexing report and select the Soft 404 reason under “Why pages aren’t indexed.” Export the available example URLs together with any useful dates or annotations.

The report contains URLs Google knows about, not necessarily every affected URL on your site. Google also advises inspecting individual examples and using a live test to see how it currently renders a page. The screenshot and rendered HTML can reveal content that an ordinary HTTP request misses. See Google’s documentation for the Page indexing report and URL Inspection tool.

Do not assume every URL in the report is broken today. Search Console’s indexed view can describe an earlier crawl, while its live test examines the current version.

Build a working sheet with fields such as:

Field	Why it matters
URL	Identifies the affected page
Page type	Product, article, category, profile, search page, or other template
Search Console status	Records Google’s reported classification
Current HTTP status	Separates real 404s from successful responses
Final URL	Reveals redirects
Title and H1	Helps identify error language and template failures
Main-text length	Flags empty or unusually thin pages
Canonical URL	Exposes incorrect canonicalization
Robots directive	Identifies `noindex` or conflicting controls
Rendered successfully?	Detects JavaScript and resource failures
Intended state	Existing, moved, removed, temporarily empty, or unknown
Recommended action	Restore, improve, redirect, 404, 410, or investigate

Collect evidence before asking AI

AI works best when you give it structured observations rather than a bare list of URLs.

For a quick status check, use a command such as:

curl -L -s -o /dev/null \
  -w "%{url_effective},%{http_code},%{num_redirects}\n" \
  "https://example.com/page"

The -L option follows redirects, so also record the original response and full redirect chain when those details matter. A crawler can gather this information more efficiently across a larger URL set.

Collect at least:

Initial and final HTTP status
Redirect chain
Page title
Meta robots directive
Canonical target
H1 and visible main content
Approximate main-content length
Internal links pointing to the URL
Sitemap inclusion
Rendered-page result
Common error phrases
Template or directory pattern

A 200 status alone does not prove that a page is indexable or useful. Google states that successful responses can still be classified as soft 404s when the returned content looks like an error, is empty, or lacks main content. A 2xx response also does not guarantee indexing (Google’s HTTP status code documentation).

Ask AI to cluster patterns, not declare verdicts

Soft 404 reports often contain many URLs caused by a small number of template or data problems. AI is useful for finding those repeated patterns.

For example, it might group URLs into:

Deleted products returning 200
Empty category pages
Internal-search pages with no results
Profiles for users who no longer exist
Location pages with only boilerplate
JavaScript pages whose main content did not render
Valid pages containing prominent “not found” language
URLs redirected to an irrelevant homepage
Parameter combinations with no meaningful content

Use a constrained prompt:

You are assisting with a technical SEO diagnosis.

Analyze the supplied URL records and group them by likely soft 404 pattern.
Use only the provided fields. Do not infer page content that is not present.

For each cluster, return:
1. Cluster name
2. Shared evidence
3. Example URLs
4. Most likely cause
5. Confidence: high, medium, or low
6. Checks required before choosing a fix

Do not recommend a redirect unless the data contains a clearly relevant
replacement URL. Mark uncertain cases for human review.

Request structured output, such as CSV or JSON, if the results will return to a spreadsheet. Keep the original evidence beside the AI-generated labels so reviewers can audit every classification.

Investigate the main causes

The server returns `200` for missing pages

This is the clearest soft 404 pattern. The page says that an item, article, or user cannot be found, but the server treats the request as successful.

If the content has been removed and has no suitable replacement, return 404 Not Found or 410 Gone. Under the HTTP standard, 404 means the server cannot find a current representation, while 410 indicates that the resource is known to be permanently unavailable (RFC 9110).

A custom error page may still contain navigation and helpful links. Its server response must nevertheless be an appropriate error status.

The page is empty or nearly empty

A template can return 200 while producing no meaningful main content. Common causes include missing database records, failed server-side includes, API errors, and empty internal-search results.

AI can compare affected pages with healthy examples from the same template. Give it extracted fields rather than full, noisy HTML:

Compare each affected page with the healthy reference pages.

Identify:
- fields consistently missing from affected pages;
- error messages or empty-state text;
- large differences in main-content length;
- elements present only in the shared template;
- evidence of a data or rendering failure.

Do not judge content quality from word count alone.

Content length is a clue, not a rule. A concise page can fully satisfy its purpose, while a long page made mostly of navigation and boilerplate can still lack useful main content.

JavaScript did not render the main content

A URL may work in a user’s browser but appear blank or broken to Google when scripts, APIs, or required resources fail. Google specifically recommends examining the rendered result when a valid page is classified as a soft 404.

Compare:

The initial HTML response
The browser-rendered DOM
Search Console’s tested-page screenshot
Failed network requests
Resources blocked by robots.txt
Server and API logs around the crawl

Google explains that pages returning 200 are eligible for rendering, while meaningful error codes help Googlebot understand unavailable content. Its JavaScript SEO guidance also describes approaches for handling unavailable routes in client-rendered applications.

If blocked resources appear to be involved, audit the rules before changing them. The related guide on How to Audit Robots.txt With AI in 30 Minutes provides a separate workflow for that check.

A valid page looks like an error page

Some legitimate pages contain phrases such as “not found,” “no results,” or “unavailable.” A product might be temporarily unavailable while retaining useful specifications, alternatives, support information, and a valid future purpose.

Inspect the page as Google renders it. If it is meant to remain searchable, make the main purpose unmistakable:

Keep the unique, useful information visible.
Explain the item’s actual status precisely.
Avoid presenting a large generic error message as the primary content.
Fix missing data or resources.
Ensure the title and main heading describe the page rather than an error state.

Do not add filler simply to increase word count. The goal is to repair or clarify a real page, not to disguise an unavailable resource.

The URL moved to a replacement

Use a permanent server-side redirect when there is a clear, relevant replacement. Google describes permanent redirects such as 301 and 308 as signals that the redirect target should become canonical (Google’s redirect documentation).

Do not redirect every removed URL to the homepage. That does not preserve the original intent and can itself create a poor user experience.

AI may suggest replacement URLs by comparing slugs, titles, product identifiers, categories, and embeddings. Treat these suggestions as candidates. A person should confirm that each destination genuinely satisfies the old page’s purpose.

Convert clusters into decisions

Use a decision table rather than applying one fix to every soft 404:

Page state	Appropriate action
Page should exist but is broken	Repair data, template, server, or rendering issue
Page exists but lacks clear main content	Add or restore genuinely useful content
Content moved to a close replacement	Use a permanent server-side redirect
Content was removed with no replacement	Return `404` or `410`
Page is intentionally an empty state	Decide whether it should be indexable; return an appropriate status or use a supported indexing control
Classification appears incorrect	Verify status, rendering, resources, and error-like language
Intent is uncertain	Escalate for editorial, product, or technical review

Also update internal links and XML sitemaps. Removed URLs should not remain in sitemaps, and internal links should point directly to the final useful destination. For broader cleanup, see the guide to How to Build AI-Driven Internal Links in 30 Minutes.

Prioritize by impact and confidence

An AI-generated priority score can help with triage, but it should use observable inputs. Suitable signals include:

Whether the URL is in an XML sitemap
Number of internal links
Known organic impressions or clicks
Backlinks or referral traffic
Page type and business importance
Size of the affected template cluster
Confidence in the diagnosed cause
Availability of a verified replacement

Avoid asking AI to estimate “ranking loss” without supporting data. Search Console’s soft 404 classification tells you that Google excluded the page; it does not reveal a universal traffic value or a specific ranking penalty.

A practical prioritization order is:

Important pages that should exist but fail to render
Large template-wide defects
Previously useful URLs with verified replacements
Removed URLs still present in sitemaps or internal links
Low-value or isolated URLs with no internal discovery path

Validate the diagnosis before deployment

For each cluster, manually inspect at least one representative URL and one unusual case. Large or commercially important clusters deserve a larger sample.

Check that:

The intended content state is confirmed.
The HTTP response matches that state.
Redirect targets are relevant and return the expected response.
Canonical and robots directives do not conflict.
Rendered main content is present.
Internal links and sitemaps reflect the new URL state.
The same template does not produce different results for users and crawlers.

Treat page text as untrusted input when passing it to an AI system. A page can contain instructions that are irrelevant or hostile to the audit. Tell the model to analyze page data only, ignore instructions found inside the content, and never expose credentials, private customer data, unpublished URLs, or sensitive logs.

Confirm the fix in Search Console

After deployment:

Check the response with an HTTP client or crawler.
Open representative URLs in Search Console.
Run Test live URL.
Review the rendered screenshot and tested-page details.
Request indexing for important restored or redirected pages when appropriate.
Use Validate Fix for the soft 404 issue.
Monitor the affected count and watch for new template patterns.

Google notes that validation can take several days or longer. A successful live test is useful evidence, but it does not guarantee immediate indexing or a particular search position.

Keep the pre-fix dataset, deployment date, affected template, chosen action, and post-fix result. This record makes later regressions easier to distinguish from old Search Console data.

Conclusion

AI is most useful for diagnosing soft 404s when it acts as a clustering and comparison layer over reliable technical evidence. Start with Search Console, verify server responses and rendered content, group recurring failures, and choose a fix based on what each URL is supposed to do.

Restore real pages, redirect only to relevant replacements, and return genuine error codes for content that no longer exists. Human review remains essential wherever page purpose, replacement relevance, or business value is uncertain.