Lesson (1): Classification of Search Engines

The term "search engine" (SE) is often misused to describe both directories and pure search engines. In fact, they are not the same; the difference lies in how result listings are generated.

There are four major search engine types you should know about. They are:

• crawler-based (traditional, common) search engines;
• directories (mostly human-edited catalogs);
• hybrid engines (META engines and those using other engines' results);
• pay-per-performance and paid inclusion engines.

Crawler-based SEs, also referred to as spiders or Web crawlers, use special software to automatically and regularly visit websites to create and supplement their giant Web page repositories.

This software is referred to as a "bot", "robot", "spider", or "crawler". All these terms denote the same concept. These programs run on the search engines. They browse pages that already exist in their repositories, and find your site by following links from those pages. Alternatively, after you have submitted pages to a search engine, these pages are queued for scanning by a spider; it finds your page by looking through the lists of pages pending review in this queue.

After a spider has found a page to scan, it retrieves this page via HTTP (like any ordinary Web surfer who types an URL into a browser's address field and presses "enter"). Just like any human visitor, the crawling software leaves a record on your server about its visit. Therefore, it?s possible to know from your server log when a search engine has dropped in on your online estate.

Your Web server returns the HTML source code of your page to the spider. The spider then reads it (this process is referred to as "crawling" or "spidering") ? and this is where the difference begins between a human visitor and crawling software.

While a human visitor can appreciate the quality graphics and impressive Flash animation you've loaded onto your page, a spider won't. A human visitor does not normally read the META tags, a spider can. Only seasoned users might be curious enough to read the code of the page when seeking additional information about the Web page. A human visitor will first notice the largest and most attractive text on the page. A spider, on the other hand, will give more value to text that's closest to the beginning and end of the page, and the text wrapped in links.

Perhaps you've spent a fortune creating a killer website designed to immediately captivate your visitors and gain their admiration. You've even embedded lots of quality Flash animation and JavaScript tricks. Yet, a search engine spider is a robot which only sees that there are some images on the page and some code embedded into the "script" tag that it is instructed to skip. These design elements are additional obstacles on its way to your content. What's the result? The spider ranks your page low, no one finds it on the search engine, and no one is able to appreciate the design.

SEO (search engine optimization) is the solution for making your page more search-engine friendly. The optimization is mostly oriented towards crawler-based engines, which are the most-popular on the Internet. We're not telling you to avoid design innovations; instead, we will teach you how to properly combine them with your optimization needs.

Let's return to the way a spider works. After it reads your pages, it will compress them in a way that is convenient to store in a giant repository of Web pages called a search engine index. The data are stored in the search engine index the way that makes it possible to quickly determine whether this page is relevant to a particular query and to pull it out for inclusion in the result page shown in response to the query. The process of placing your page in the index is referred to as "indexing". After your page has been indexed, it will appear on search engine results pages for the words and phrases most common on the indexed Web page. Its position in the list, however, may vary.

Later, when someone searches the engine for particular terms, your page will be pulled out of the index and included in the search results. The search engine now applies a sophisticated technique to determine how relevant your page is to these terms. It considers many on-page and off-page factors and the page is given a certain position, or rank, within other results found for the surfer's query. This process is called "ranking".

Google (www.google.com) is a perfect example of a crawler-based SE.

Human-edited directories are different. The pages that are stored in their repository are added solely through manual submission. The directories, for the most part, require manual submission and use certain mechanisms (particularly, CAPTCHA images) to prevent pages from being submitted automatically. After completing the submission procedure, your URL will be queued for review by an editor, who is, luckily, a human.

When directory editors visit and read your site, the only decision they make is to accept or reject the page. Most directories do not have their own ranking mechanism ? they use various obvious factors to sort URLs, such as alphabetic sequence or Google PageRank^TM (explained later in this course). It is very important to submit a relevant and precise description to the directory editor, as well as take other parts of this manual submission seriously.

Spider-based engines often use directories as a source of new pages to crawl. As a result, it's self-evident in SEO that you should treat directory submission and directory listings as seriously and responsibly as possible.

While a crawler-based engine would visit your site regularly after it has first indexed it, and detect any change you make to your pages, it's not the same with directories. In a directory, result listings are influenced by humans. Either you enter a short description of your website, or the editors will. When searching, only these descriptions are scanned for matches, so website changes do not affect the result listing at all.

As directories are usually created by experienced editors, they generally produce better (at least better filtered) results. The best-known and most important directories are Yahoo (www.yahoo.com) and DMOZ (www.dmoz.org).

Hybrid engines. Some engines also have an integrated directory linking to them. They contain websites which have already been discussed or evaluated. When sending a search query to a hybrid engine, the sites already evaluated are usually not scanned for matches; the user has to explicitly select them. Whether a site is added to an engine's directory generally depends on a mixture of luck and content quality. Sometimes you may "apply" for a discussion of your website, but there?s no guarantee that it will be done.

Yahoo (www.yahoo.com) and Google (www.google.com), although mentioned here as examples of a directory and crawler respectively, are in fact hybrid engines, as are nowadays most major search machines. As a rule, a hybrid search engine will favor one type of listing over another. For example, Yahoo is more likely to present human-powered listings, while Google prefers its crawled listings.

Meta Search Engines. Another approach to searching the vast Internet is the use of a multi-engine search, or meta-search engine that combines results from a number of search engines at the same time and lays them out in a formatted result page. A common or natural language request is translated to multiple search engines, each directed to find the information the searcher requested. The search engine's responses thus obtained are gathered into a single result list. This search type allows the user to cover a great deal of material in a very efficient way, retaining some tolerance for imprecise search questions or keywords.

Examples of multi-engines are MetaCrawler (http://www.metacrawler.com) and DogPile (http://www.dogpile.com). MetaCrawler refers your search to seven of the most popular search engines (including AltaVista and Lycos), then compiles and ranks the results for you.

Pay-for-performance and paid inclusion engines. As is clear from the title, with these engines you have no way other than to pay a recurring or one-time fee to keep your site either listed, re-spidered, or top-ranked for keywords of your choice. There are very few search engines that solely focus on paid listings. However, most major search engines offer a paid listing option as a part of their indexing and ranking system.

Unlike paid inclusion where you just pay to be included in search results, in an advertising program listings are guaranteed to appear in response to particular search terms, and the higher your bid, the higher your position will be for these terms. Paid placement listings can be purchased from a portal or a search network. Search networks are often set up in an auction environment where keywords and phrases are associated with a cost-per-click (CPC) fee. Such a scheme is referred to as Pay-Per-Click (PPC). Yahoo and Google are the largest paid listing providers, and Windows Live Search (formerly MSN) also sells paid placement listings.

So here's what you should remember from this lesson:

1. Search engines (SEs) are classified into crawlers, directories, META engines and paid-inclusion engines.
2. Crawler-based SEs use software called robots, spiders, or crawlers to add new pages to its database which is called an index. Directories use humans to manually fill their databases.
3. After your site has been included in an index of a crawler-based search engine, you will appear in its results, and your position for a certain search query depends on how relevant the spider finds your page for this query.
4. Your directory listings are quite influential to your positions in crawling search engines.

Part 1: Understanding Search Engines

This step is an introduction into search engines and the relationships between them. This information is valuable in helping SEOs understand how site rankings in one search engine / directory can influence rankings in other engines.

Throughout this course, (and commonly used on the Internet as well) the term "SE" is used as an abbreviation for "search engine(s)".

Search engines are the most popular method for target customers to find you. As such, SEs are the most vital avenue for letting customers find you.

Currently, search engines around the world together receive around 400,000,000 searches per day. The searches are done with the help of keywords: as a rule, people type a short phrase consisting of two to five keywords to find what they are looking for. It may be information, products, or services.

(For more information on the share of world searches received by major search engines per day, see http://www.freeseosemtraining.com/blog/search-engine-market-share-july-2007.htm).

In response to this query, a search engine will pick from its huge database of Web pages those results it considers relevant for the Web surfer's terms, and display the list of these results to the surfer. The list may be very long and include several million results (remember that nowadays the number of pages on the Web reaches 2.1 trillion, i.e. 2,100,000,000,000); so the results are displayed in order of their relevancy and broken into many pages (most commonly 10 results per page). Most Web surfers rarely go further than the third page of results, unless they are considerably interested in a wide range of materials (e.g. for a scientific research). One reason for this is that they commonly find what they look for on those first pages without the needing to dive in any deeper.

That's why a position among the first 30 results (or "top-30 listing") is a coveted goal.

There used to be a great variety of search engines, but now after major reshuffles and partnerships there are just several giant search monopolies that are most popular among Web surfers and which need to be targeted by optimizers.

There are – and the search engines are aware of this – more popular searches and less popular searches. For instance, a search on the word "myrmecology" is conducted on the Web much more rarely than a search for "Web hosting". Search engines make money by offering special high positions (most often called "sponsored results") for popular terms, ensuring that a site will appear to Web surfers when they search for this term, and that it will have the best visibility. The more popular the term, the more you will have to pay for such a listing.

In this step, we provide insight into how search engines work, how they relate to each other and how it can be of use for you and your Web business.

Stage 1: Search Engine Marketing

Search Marketing has become a buzzword that is now heard all over the place many times a day. Here we provide an exact definition of what it refers to, and how it relates to both Web Search and Web Marketing.

Search Marketing is also known as Search Engine Marketing (SEM), and as such we will refer to is as SEM throughout this course. The definitions that follow are the basics; if you are an expert / advanced Search Marketer, you can skip these terms; otherwise we recommend that you read and understand them.

Search Marketing is a part of business marketing efforts that is aimed at increasing traffic (the number of visitors) to the website from the search engines. Additionally, it addresses conversion (the percent of visitors who become buyers). The first is achieved by increasing search engine visibility, i.e. the position of your site in search engine results for certain keywords that people type in the search box to obtain these results.

For instance, if someone wants to find and buy a digital camera, they will go to a search engine such as Google and type "digital camera" in the search box. Google will list, in this case, 138 million results (these are the real figures extracted while creating this course). If you sell digital cameras or offer any related services, your site may be listed among these 138,000,000 results. Here, everything depends on how deep you are. If you are on the first or second page of the search results, it's more likely that such visibility will bring many visitors and customers from Google. If you are the 300th result, it's unlikely that anyone at all will come to you from Google.

Together with the power and size of your banner / ad network, your affiliations and partnerships, SE visibility comprises a broader concept - Web visibility (aka online visibility).

Generally, there are two main methods of carrying out SEM: a) Search Engine Optimization (SEO) b) using pay-per-click and paid inclusion listing models. They are briefly depicted in the following lessons.

Although paid inclusion and pay-per-click advertising methods seem like the fastest methods to search engine marketing, website owners prefer to adopt a more time consuming search engine optimization method to obtain better marketing of their website on search engines.

Organic rankings are results that you get for free. That is, you create Web copy and publish it, then after a certain period of time a search engine robot finds it (either by itself or as a result of your submission). Finally, the robot reads your content and puts your site into its index. Now your site will be found by this search engine when people query for some words contained within your pages. Obtained this way, your positions in the result list are called your "organic search engine rankings".

Paid listings are different: pay a search engine and it guarantees the inclusion of your site in the index. Moreover, many search engines offer advanced pay-for-performance programs, such as showing your site / ad in the search results for keywords of your choice. These are the so-called "sponsored" results. Most commonly, you will have to pay a specified rate for each visitor that comes to your site from this search engine that clicks on these ads.

Mastering both methods and their proper combination can provide maximum search engine visibility. Because things keep changing, search engine marketers need to devote a good deal of time staying on top of the SEO industry and its trends.

The aim of SEM is not only to find a proper balance between organic and paid listings, but also to achieve maximum conversion of visitors into loyal customers. Nowadays SEM relies on the statement that it's not the traffic itself that matters, but how targeted and convertible it is. The way your traffic converts also matters a lot – even more than your site rank on a search engine. You can rank worse than a competitor and yet the percentage of your visitors that turn into buyers can be high enough to actually outperform a competitor several times over.

The following are the main goals of Search Engine Marketing:

1. Improve Web visibility and get as much traffic as possible.
2. Improve traffic quality: get high rankings for exactly those keywords that bring visitors with the best conversion rate.
3. Decrease expenditures by switching off advertising for underperforming keywords.

Methods used by Search Marketing

The main methods used for achieving the goals of Search marketing are Search Engine Optimization (for organic listings), Bid Management (for paid listings) and Web Analytics (for both types of listings).

Search Engine Optimization (further referred to in this course as SEO) is about changing the HTML code of your pages and the structure of your site in such a way that when an SE robot reads the site, it can understand that the pages have valuable content related to your keywords, and then rank them high. SEO also tells about ways to increase your link popularity - the number of links from other high-ranked pages to your site. This is important because most search engines consider your link popularity a vital ranking factor.

Bid Management is about controlling bids, i.e. the amount of money you spend maintaining your visibility in the sponsored listings. Usually you try to detect the best converting keywords and keyword groups, in order to increase bids on them; as well as decrease or take off bids on keywords that don't break even. Attention also should be paid to leveraging your paid and organic listings, so to spend less on paid advertising campaigns when you get enough traffic from natural results, and invest in paid advertising when an algorithm changes or strong competitors force you out from the top positions in the organic listings.

Web Analytics (further referred to as WA) is about getting, analyzing and using the information about your visitors, their details, their behavior on your site, the ways they have found your site, the efficiency of referrers and advertising, conversion rates, and, together with all that, eCommerce information.

So here's what you should remember from this lesson:

1. Search Marketing, or Search Engine Marketing, or SEM, is the aggregate of efforts aiming at increasing your search engine visibility.
2. SEM deals with your organic and paid listings on the search engines.
3. SEM includes and uses the techniques of Search Engine Optimization, Bid Management and Web Analytics.

Get Acquainted with Search Engine Optimization Tools

If you're promoting a website / business online, you can facilitate your work using special software. As with website maintenance, where you normally use some software to build web pages and to upload them to the hosting server, search engine optimization has its industry tools that take many routines off your shoulders.

There are several typical tasks every search marketer has to deal with on the daily basis. The most popular of them are: researching keywords, optimizing keyword positioning across Web pages, submission to search engines, checking rankings, searching for link partners, launching and controlling PPC campaigns, and many others.

What You Should Know before Getting Started with Online Marketing

The good news is that Internet marketing has rapidly grown and offers a broad array of opportunities for small and large businesses. From the previous introduction we became familiar with the Internet marketing science structure and the particular steps to follow do develop an online business.

At this point, before delving more deeply into the subject, let's define some main terms. Online marketing deals with websites and Web pages, search engines, email and the Internet as the base of the World Wide Web. All of these areas are used to advertise and sell goods and services.

Referencing the original Wikipedia encyclopedia project definitions of our basic terms, they are as follows:

The World Wide Web ("WWW" or simply the "Web") is a system of interlinked, hypertext documents that runs over the Internet.

The hypertext documents, or Web pages, reside on Web servers - special computers that receive requests for Web pages and can "serve" them to the requesting side.

Each Web server, or host, has its unique global address used to find it over the Internet. This address is called an "IP address". A typical IP address looks like four numbers separated by dots. For example, 63.146.123.0 is the address of the server where google.com is situated.

Each server can hold one or many websites. A website (or Web site) is a collection of Web pages, typically bound to a particular domain name or subdomain on the World Wide Web on the Internet. A website is identified uniquely by its domain name, e.g. www.asiawebpro.com.

Domain names are translated into IP addresses by the global DNS - domain name system. That is, when you type www.asiawebpro.com in your browser, the latter first sends a DNS request, and receives the IP address of the server where www.asiawebpro.com is hosted. The browser then connects to that server directly and asks for Web CEO's home page.

Each website is composed of many Web pages. A Web page is a document, typically written in HTML, that is always accessible via HTTP, a protocol that transfers information from the website's server to display in the user's Web browser.

So when a new business is born on the Internet, or when an established brick and mortar enterprise goes online, it starts by creating a website. This is done by purchasing a domain name from an organization accredited to sell domain names, e.g. godaddy.com or register.com. Also, some hosting space must be purchased, so that there's some server ready to store the website. Then, several Web pages are created and linked to each other to make up this website. Finally, the website is linked to the domain name so that everyone can type this domain name in a browser and view these pages.

With a Web browser, a user views Web pages that may contain text, images, and other multimedia and navigates between them using hyperlinks. A client program called a "user agent" retrieves information resources, such as Web pages and other computer files, from Web servers using their URLs.

Most commonly, the user agent is a kind of Web browser: Internet Explorer, Mozilla FireFox, Opera, Netscape, or the program that you use to view this lesson. It retrieves content from remote Web servers and displays it on your computer. You can then follow hyperlinks in each Web page to other World Wide Web resources, whose location (including their domain name) is embedded in the hyperlinks. The act of following hyperlinks from one website to another is referred to as "browsing" or sometimes as "surfing" the Web.

To find a Web page, you can always type its address in the address field of your browser. But what if you don't know the exact address, or want to find all Web pages from different websites on a particular topic?

This is when you use a search engine or directory for the search. There are 3 top search engines: Google, Yahoo! and MSN (Windows Live Search). These search engines are most preferred by Web surfers and every site owner strives to get included in their databases. If people can find your website through search engines, this search engine creates an invaluable source of traffic for you, which translates into income if you sell goods or services.

Actually, search engines are very numerous and differ according to how they work. We have a specific section for "Understanding Search Engines" which includes seven lessons to guide and familiarize you with their core principles.

There is one final point we'll look at, and that is the use of email for your online marketing activity. The topic is email marketing and using this form of communication to send goods and service offers to clients.

Electronic mail (abbreviated "email") is a store and forward method of composing, sending, storing, and receiving messages over electronic communication systems.

Email predates the Internet; existing email systems were a crucial tool in creating the Internet. Email was quickly extended and became additional and highly useful tool for the online marketers.

Email Marketing and other forms of Internet Marketing are depicted in this course to select and master your niche for a future online business.

Introduction into Internet Marketing

Starting from the early 1990s Internet marketing made an amazing development from simple text-based websites that offered product information into highly evolved and complete online businesses promoting and selling their services on the Internet.

Nowadays, the Internet marketing industry has become a complicated and branchy science involving a great deal of theoretical knowledge in combination with applied techniques. As a science, it ranges from browser-side and server-side programming and coding on one end to marketing and economics on the other.

Internet marketing means the use of the Internet to advertise and sell goods and services. It includes Banner and Text Advertising, Email Marketing, Interactive Advertising, Affiliate Marketing and Search Engine Marketing (including Search Engine Optimization and Pay-Per-Click Advertising).

Our first stage of the Internet Marketing course will start with Search Engine Marketing (SEM) as a specific area of an online marketers' business. Mainly, its purpose is to increase targeted traffic from search engines via organic search engine ranking, paid listing and advertising. Here you'll be shown the main principles of Search Engine Optimization (SEO), link building, and paid advertising campaigns.

Actually, every successful search engine optimizer should be aware of top search engine demands and consider them while creating website and improving on-page and off-page factors for Web pages. There are numerous important factors influencing search engine ranking of a Web page. The SEO division of the course provides profound and consecutive lessons depicting each step of your optimization work. Search Engine Advertising is the last topic of the SEM Stage.

There are certain methods that go beyond SEM that can help improve your site online visibility. These include, for instance, creating and spreading a banner / ad network and / or paid link partnerships, as well as email marketing and building affiliate relationships with other websites.

Email marketing is an independent branch which has to be dealt with separately and does not have much in common with SEM. Email marketing is a subject of our next stage and there we will provide insight into the main direct mail campaign steps and guidelines.

Banner networks relate to SEM as long as they touch upon your link popularity (which is a component of SEM).

In the following stage you'll study the Affiliate Marketing division of Internet Marketing. It is a popular method for promoting web businesses when with few marketing dollars marketers can establish a presence and earn a profit recruiting affiliates. Such partner networks can grow with your company business projects and add its profit to your marketing budget.

The most vital stage of the whole course is Web Analytics. Its role can be hardly underestimated as Web Analytics is an essential measure for continually improving web business performance, advertising campaigns, organic search engine results, ranking positions and others. Generally, Web Analytics deals with the traffic already generated at the previous stages. Its primary goal is to improve traffic quality and enhance conversion.

Although it is possible (and advisable) to understand every theoretical aspect of Internet marketing, in practice you may do much better by specialization in a specific area or technique and simply start your Internet Marketing business. Our last stage provides a proper and clear scheme about how to estimate your potential, find a niche, manage projects and promote your services as online marketer, SEO consultant, etc.