SEO and search engines

What is a search engine?

A search engine is a computer program used to find information on the internet, usually in response to one or more keywords being typed in but also, increasingly, to voice commands.

Google is by far the best known and most widely used search engine, with over 90% of market share (see this search engine data from StatCounter).

The aim of Google and other search engines, such as Bing, Yahoo and Duck Duck Go, is to present users with the most relevant results for their search, in the form of web pages, images, videos and other materials.

For more information, see Techopedia’s definition of a Search Engine.

How do search engines work?

As explained below, search engines work by performing three distinct actions: crawling, indexing and ranking:

Stage 1 – Crawling

Crawlers (aka ‘spiders’, ‘robots’, ‘bots’) find and analyse information on websites.

Starting from a page it already knows about, a crawler follows links not only to pages on the same site (internal links) but also to pages on other sites (external links).

Following internal links enables the spider to discover both new content (e.g. a new blog post) and changes to existing content (e.g. updated contact details) on the original site, while tracing external links helps assess its relative importance (such as how many sites are linked to and link back to it, and how important they are).

By following links, analysing content and seeing how various pages and sites are connected, the crawler builds up a picture of what websites and their individual pages are about – and how relevant they might be to a search query.

To assess that relevance, search engines look at keywords, including how frequently they appear, whether they appear in headings and, if they do, how well those headings match the content of the page and site.

Note that it’s not all about keywords: when crawling a site, search engines also take other factors into consideration, such as how fast a page loads and how easy it is for users to navigate a site.

Stage 2 – Indexing

Once a website has been crawled, the search engine decides if the site’s content is worth indexing. If it is, the content is added to the engine’s database and becomes available for display in response to keyword searches.

With the exception of ‘stop’ words every word on a page that Google indexes is added to the database and Google’s index now contains hundreds of billions of web pages.

(See How Google Search organises information; for information about stop words, see this CSEO article: SEO Stop Words: The Definitive Guide.)

Because Google and other search engines are in business to present their users with the most relevant information, not all content is indexed.

To guarantee being indexed, a site should offer content that is both unique and valuable. Duplicating material available elsewhere or offering poor quality, spam-laden content will hurt a site’s chances of being indexed – as will anything that stops search engines crawling a site.

Stage 3 – Ranking

The final stage is for a website to be ranked.

Just as some websites are of such poor quality that they are not indexed, some web pages are considered to be more appropriate to a particular query and are therefore displayed nearer the top of the search results.

Deciding which of the pages that it’s indexed should be displayed in response to a search is a task for a search engine’s algorithms.

It’s generally accepted that Google uses over 200 factors to assess how well a page ranks. They include whether and where keywords are found, whether a page loads quickly and is mobile friendly, and whether a web page and the site of which it is part can be considered a reputable source of information for a particular subject.

In response to a search, the most relevant organic results (not paid advertisements) are ranked and then displayed on a series of search engine results pages (SERPs).

For more information, see the Ahrefs page How Search Engines Work.

To discuss how I can help improve your SEO and get your pages ranking higher on Google, please get in touch.