Crawling and indexing: the two-step process
Understanding the difference between crawling and indexing is the first step to diagnosing why specific pages do not appear in Google search results.
What crawling is
Crawling is Google's discovery process. Googlebot a software program that visits URLs and reads their HTML follows links from page to page across the web. It finds new pages by following links from pages it has already discovered, or from URLs submitted in XML sitemaps.
When Googlebot visits a page, it reads the HTML, identifies the links, and adds those linked URLs to its crawl queue. The depth of your crawl coverage depends on how many internal links your site has (more links = more pages discovered), how fast your server responds (faster server = more pages crawled per session), and how authoritative Google considers your domain (more authority = higher crawl budget).
What can prevent crawling:
Disallow: /in your robots.txt file blocks Googlebot from visiting URLs- Firewall or bot-blocking rules that block Googlebot's IP ranges
- Slow server response times that exhaust crawl budget before pages are reached
- No internal links to a page (orphaned pages may never be discovered)
Ready to take the next step?
Let Perceptra scope the right approach for your business.
Book a Free Strategy Session ?What indexing is
Indexing is Google's decision process. After crawling a page, Google evaluates whether to add it to its search index the database of pages that can appear in search results. Not every crawled page gets indexed.
Google will not index a page if:
- It has a
tag - Its content is substantially similar to another page (duplicate content)
- It has very little useful content (thin content)
- Its quality signals are too low to justify inclusion
- The canonical tag points to a different URL (Google indexes the canonical, not the duplicate)
The most common indexing problem: A page is crawled but Google considers it "Discovered currently not indexed" meaning Google found it but has not decided to include it. This typically means the page needs better content, more internal links, or better authority signals to justify indexation.
How to check your crawl and index status
Google Search Console ? Coverage report (Indexed tab): Shows how many pages are indexed and which specific URLs are in Google's index.
Google Search Console ? Coverage report (Not Indexed tab): Shows pages that are crawled but not indexed, and the reason. "Excluded by 'noindex' tag," "Duplicate without user-selected canonical," "Not found (404)," "Crawled currently not indexed."
URL Inspection tool: Enter a specific URL to see its crawl and index status, when it was last crawled, and what Google's rendered version looks like.
Site: search operator: Type site:yourdomain.com in Google search to see an approximate count of indexed pages. Compare to your expected page count.
The most common crawl and index problems
Pages blocked by robots.txt that should rank: Fix by updating robots.txt to allow Googlebot.
Pages with noindex tags that should rank: Fix by removing the noindex meta tag.
Thin pages not being indexed: Fix by adding substantive, unique content that provides genuine value to searchers.
Orphaned pages with no internal links: Fix by adding internal links from relevant pages.
Frequently asked questions
How long does it take for a new page to be crawled and indexed? For a new page on an established, frequently crawled site: typically 1 7 days. For a page on a new or infrequently crawled site: 2 6 weeks. Submit the URL via Search Console URL Inspection to speed up discovery.
Can a page rank without being in Google's index? No. A page must be indexed to appear in search results.
If Google has crawled my page but not indexed it, does that mean it never will be? Not necessarily. "Crawled currently not indexed" can mean Google found the page but has not yet decided to include it. Improving the page's content, adding internal links, and building authority can result in indexation weeks later.