Website indexing is one of the first steps (post-crawl) in the complex process of understanding what web pages are ranked and served as search results by search engines.
Search engines are constantly improving the way they crawl and index websites.
Understanding how Google and Bing approach website crawling and indexing is essential for technical SEO and can help you develop strategies for improving your search visibility. increase.
Indexing: How search engines work today
Let’s take a look at the basics of how search engines work.
This article focuses on indexing. So let’s dive in.
Web indexing
Indexing is where the ranking process begins after your website has been crawled.
Indexing is basically adding the content of a web page to Google for ranking purposes.
When you create a new page on your site, you can index it in several ways.
The easiest way to get your page indexed is to do nothing.
Google has crawlers that follow links, so if your site is already indexed and new content is linked from within your site, Google will eventually find it and add it to the index. More on this later.
How to speed up page indexing
But what can you do to help Googlebot access your page quickly?
This is important if you have timely content or need to let Google know when you make important changes to your page.
Use faster methods when optimizing important pages or adjusting titles and descriptions to improve click-throughs.
I’d like to know specifically when they were picked up and displayed in the SERPs so that I know where to start measuring improvements.
In such cases, there are some additional methods that can be used.
1. XML Sitemap
XML sitemaps are the oldest and most commonly trusted way of getting search engines to pay attention to your content.
An XML sitemap provides search engines with a list of all the pages on your site, along with additional details such as when they were last modified.
Sitemaps can be submitted to Bing via Bing Webmaster Tools. It can also be submitted to Google via Search Console.
Definitely recommended!
But it’s not particularly reliable if the page needs to be indexed immediately.
2. Request indexing in Google Search Console
In Search Console, you can “Request Indexing”.
Start by clicking the search field at the top. By default, it says Inspect and URL for domain.com.
Enter the URL you want to index and press Enter.
If the page is already known to Google, you’ll see a set of information. I won’t go into detail here, but if you’re not logged in yet, I recommend logging in and seeing what you see.
For our purposes here, the important buttons are displayed regardless of whether the page is indexed or not. This means it can help you discover content or ask Google to understand recent changes.
The following buttons are displayed.
Google Search Console screenshot, December 2022Within seconds to minutes, you can search Google for new content or URLs and find changes or new content picked up.
3. Join Bing’s IndexNow
Bing has an open protocol based on the push method to alert search engines of new or updated content.
The indexing protocol for this new search engine is called IndexNow.
It’s called a push protocol because IndexNow is used to alert search engines about new or updated content so they can come and index it.
An example of a pull protocol is the old XML sitemap method. It relies on search engine crawlers deciding to access and index it (or fetch it by Search Console).
The advantage of IndexNow is less waste of web hosting and data center resources. This is not only environmentally friendly, but also saves bandwidth resources.
The biggest advantage, however, is fast content indexing.
IndexNow is currently only used by Bing and Yandex.
Implementing IndexNow is straightforward.
4. Bing Webmaster Tools
In addition to participating in IndexNow, consider a Bing Webmaster Tools account.
Not recommended unless you have a Bing Webmaster Tools account.
The information provided is substantial and will help you better assess problem areas, improve your rankings on Bing, Google, and elsewhere, and perhaps provide a better user experience.
However, to index content, simply click: [個人用サイト]>[URL を送信]Configure
Enter the URL you want to index,[送信]Click.
Screenshot of Bing Webmaster Tools, January 2023That’s pretty much all you need to know about indexing and how search engines index it (notice where things are going).
For more information, see the Bing Webmaster Tools URL Submission Tool help page.
There’s also the Bing Webmaster Tools Indexing API, which helps reduce the time it takes for your content to appear in Bing search results within hours.
More information on the Bing Indexing API can be found here.
crawl budget
You can’t talk about indexing without talking about crawl budget.
Basically, crawl budget is a term used to describe the number of resources Google spends crawling a website.
The allocated budget is based on a combination of two factors:
- Server speed (how far Google can crawl without degrading the user experience).
- How important is your site?
If you run a major news site that is constantly updating content that search engine users want to know about, your site will be crawled frequently (dare I say it all the time).
If you run a small barber shop and have dozens of links, which are of course not considered important in this context (you may be an important barber in your area, but It doesn’t matter in terms of crawl budget), the budget will be less.
For more information about crawl budgets and how they are determined, see Google’s instructions.
Google has two types of crawls
Indexing by Google starts with crawling. Crawl has two types of him:
- The first type of crawl is discovery, which finds new web pages for Google to add to its index.
- The second kind of crawl is an update. Google detects changes to web pages that are already indexed.
Understand how search engines work
Optimizing your website for search engines starts with great content and ends with submitting it to be indexed.
Whether you use an XML Sitemap, Google Search Console URL Submission Tool, Bing Webmaster Tools, or IndexNow, index that content and your web pages will start to move higher in search results (if all goes well). !).
That’s why it’s important to understand how search indexes work.
For more information on the main factors that affect search engine results pages, see How search engines work.
Featured Image: Over-Earth/Shutterstock
