How Do Search Engines Work?

How Do Search Engines Work?

In the past 2 years, SEO has undergone some of the biggest changes in its relatively short history, thanks to Google’s  infamous zoo animals Panda and Penguin. Tactics have dramatically shifted from the days of on-page optimisation, to link building, to content marketing, to relationship building – the list goes on and on. With so many buzzwords and new concepts being thrown about, is it any wonder why many are confused about how to approach SEO in the first place?

Whether you are looking for an SEO company or wanting to implement your own in-house campaign, it’s essential to have a fundamental understanding of how search engines work. That way, when you are approached by a company promising the “latest proven strategy” or reading up on breaking search news, you’ll be in the position to analyse the logic behind these claims for yourself.

Read on for an introduction to the various whats, whys and hows surrounding search engines and SEO.

What is a Search Engine?

In short, a search engine is a system run by programs which “crawl” the web for information including web pages, images, files, videos and other detectable documents. When a new document is detected (or the crawler detects an update on that document), its data is stored on the search engine server in a process referred to as caching. The documents in the search engine’s cache are then ranked in order of importance on its Search Engine Result Page (or SERP, in its abbreviated form).







What is crawling?

Crawling refers to the ability of a search engine to traverse the billions of interlinked pages on the world wide web. With massive amounts of pages being generated on an hourly basis, it is impossible for us humans alone to visit, record and organise them on our own. Instead, automated search crawlers or “bots” conduct regular searches to save us the agony of finding relevant content ourselves.

Search bots wait upon signals from previously indexed pages, such as links, to be notified of new content. So if you have created a new page on your website and linked to it from an existing page or the main menu, this would be a signal to search bots that they should come visit and index it. New pages can also be introduced to bots through measures such as Sitemaps and robots.txt files. Platforms such as WordPress will automatically alert search engines that you have created a new page. Detection can be accelerated by verifying your website with search engines using Google Webmaster Tools or Bing Webmaster Tools.

What is indexing?

After a web page or document has been detected by crawlers, all its accessible data is stored (cached) on search engine servers so it can be retrieved when a user performs a search query. Indexing serves two purposes:

  • to return results related to a search engine user’s query
  • to rank those results in order of importance and relevancy

The order of ranking is dependent with each search engine’s ranking algorithm. These algorithms are highly complex formulas, made even more advanced by the relationship your website has with external sites and its on-page SEO factors.

To sum up, indexing exists to ensure that users questions are promptly answered as quickly as possible.

What are the most popular search engines?

In Australia, Google takes holds the largest piece of the pie as illustrated in the image below:

Google is by far Australia’s most used search engine. Source: Margin Media

The next largest search engine (though it holds only sliver of the Australian market) is Bing. While Google and Bing are great all purpose search engines, how you optimise your website may also depend on the type of documents your website specialises in. There are hundreds of specialised search engines providing results for users different needs: accounting, bit torrents, blogs, jobs, multimedia, news, Q&A, shopping and more. You may also want to optimise your website for the search engine that your audience uses the most. Outside of Australia and other English speaking countries, your audience may be using the leading search engine in their own country.

 Check out the International list of search engines for more information.


 Download WebCertain Search and Social Report (2012) for International search insight


Why should I care about how a search engine works?

When I first got into SEO, I was given this analogy to explain its importance. Imagine you built or moved to a house that was in the middle of nowhere. How would people that you want visiting you know that you were living there? You would need to give them your address and invite them over, have people who already know where you live tell others, as well as build access routes from other known routes so visitors have a way to get to you.

The very same could be said for search engines. They serve the purpose of telling others who may never have heard about your website that they should visit you if they want the answer to their search query.

If your business relies on online traffic, simply being crawled and indexed by the search engine is not enough to have your website found in search. With millions of pages being created, crawled and indexed daily – your website now has to employ Search Engine Optimisation strategies to ensure that it ranks highly enough above your competitors for it to be found.

Essentially this means your website must appear in the SERP for search queries that relate to your products or services. It must also answer the user questions inherent to these search queries, as this is key to converting after being found, but this is another post altogether.

Why does ranking matter?

In this day and age of instant information gratification, everyone is time poor with attention spans rivalling those of goldfish. The Click Through Rate (CTR) for SERPs clearly indicate this, with the majority of users clicking on the first-ranked website to answer their queries. This figure diminishes as a user scrolls down the page. Having your website ranked highly for a query will drive significant portions of your audience to your website. See the following search data below to see a typical CTR on a SERP page:

Why is SEO an ongoing investment?

How we rank has dramatically changed in the past 2 years. For webmasters optimising for Google, you may be aware that there are at least two changes made to the algorithm on a daily basis. Reputable SEO providers are continually researching and preempting search trends. Apart from getting their clients high rankings in the first place, SEO practitioners have the ongoing responsibilities of minimising fluctuations in the SERPs and taking advantage of emerging ranking opportunities.

If you are looking into SEO for your online business and rely on search traffic, employing the services of a SEO company or hiring an employee dedicated to SEO is a basic requirement for online success.

At the time of writing, the algorithm Google Panda  is up to its 21st update, and Google Penguin to its third. For a full list of changes, make sure you check our SEOMoz’ awesome Google Algorithm Change History timeline.

Google continues to modify search signals affecting your websites ranking on a daily basis. You may want to have a read of my previous Search Highlight posts for 2012:

(N.B.: These highlights are now published every second month, so make sure you are subscribed to our blog for October and November changes.)

How Do Search Engines Work?

The TL:DR version!

Search engines have programs that crawl the world wide web though links and other means to find new and updated web content. Once found, each web page is processed using an algorithm and indexed. When a search user performs a query, the indexed result is retrieved and arranged (ranked) in order of relevancy to the users query.

This order of relevancy can be optimised through a process known as search engine optimisation.

Now that you understand how a search engine works, take the next step and perform a Free Website Health Check on your website. You will find:

  • What your competitors are doing
  • What it will take to beat them in the rankings
  • How to get more visitors
  • How to convert those visitors
  • … and more!

Click here to begin your FREE Website Health Check and find out where your business stands in under 5 minutes.

E-Web Marketing
Follow us
E-Web Marketing

We’ve been in the digital marketing field for over 18 years and worked with hundreds of Australian (and international) businesses to grow their web presence. Specialising in SEO, search ads (PPC), social media, content marketing, email marketing and conversion rate optimisation.

Get your 100% free SEO Analysis


    100% Free. No obligation. Done by a human.