Directories and Search Engines

First and foremost, it’s important to know the difference between a search engine and a directory. For some folks this is pretty obvious, but to those new to search marketing the differences might not be so obvious, though they are vitally important.

Directories

You can think of a directory as basically being like a phone book’s yellow pages. It is a very large list of websites grouped into categories. Most directories are VERY deep, meaning that theirs is most likely a proper place for everything. This depth also means you will have to spend time looking for what you want, as different directories list topics in different ways. In one directory, “Automotive” might appear under their “Business” category, while in another directory “Automotive” appears in the “Sports” category.

How a directory groups topics is only of concern to you in as much as you’ll need to know where your website will properly fit in. Some directories have organizational groups that make perfect sense, others, not so much.

The key with directories is to NOT over-think things. Whatever might be the most obvious path to what you’re looking for probably is the actual path.

Search Engines

As the name implies, these guys search for data. They are the private investigators of the Internet. Engines find links and follow them. More precisely, search engines send out spiders, or robots, to catalogue “every” page on the Internet. The job these spiders have is simple – if they see a link, they follow it and report back on what they see.

Being robots, they respond to commands and are not biased in their reporting – meaning, if you have an error on the page when a spider visits, it sees the error and sends the info, along with everything else it sees, back to the main search engine.

The word “cache” comes up when speaking about engines. The “cache” is the collection of catalogued web pages an engine has. Most of these caches reach well into the BILLIONS of pages. “Caching” may also refer to the action the spider takes when visiting your site, as in “The spider cached my page”. This simply refers to the spider taking a snapshot of what it saw. The spiders act of ‘caching” is what leads to your pages being included in the engines “cache”.

When a user performs a search, the engine looks through it’s “cache” of pages for matches to the users query. If your pages are in the “cache” and properly optimized, you show up well in the results the user sees.

Differences

The main difference between directories and search engines is simple – with a directory, the user looks through a list of links to find what they’re after. With a search engine, the engine does the looking for you based on your request. You CAN ‘search” a directory list, but who’s actually included in directory lists is very different from who actually exists online.

To be included in a directory, YOU must fill in a form making the request, and in many cases, you must pay a fee for the inclusion. To be found by a spider and included in the engine’s “cache”, you need simply place a link on a web page which the spider already knows about – this is why exchanging links with other sites is often the first step for those with a new website.

Getting listing in directories also provides this vital first link as the search engines actively crawl most of the directories online each day – sometimes several times per day. The spider’s goal is to find the latest additions to the Internet and get them catalogued.

Submitting

We’ll deal with search engines first, as it’s very simple – you do not submit to search engines – bottom line. I know Google has a “Submit a URL” feature, but while using this may alert them that your URL exists, it’s not the first place they look for new additions to the Internet – they crawl links to see what’s new. Links are so important to how you rank in the search results pages (SERPs) that Google pays more attention to WHO is linking to you than your obvious request for attention.

Search Engines – who is who

We’ll keep this list simple. There are basically two tiers of engines.

Tier one consists of:

1 – Google

2 – Yahoo

3 – MSN

4 – Ask.com (used to be askjeeves.com)

Tier two consists of everyone else basically:

• Brainfox
• Enhance
• ePilot
• Miva
• Goclick
• Kanoodle
• Search123
• Searchfeed
• Findology

…and more.

The tier one engines handle about 97% of all search traffic daily, so it’s pretty obvious why these are the ones to get ranked well with. In this select group, I’m including ask.com because they’re going to break into the group fully in the near future. Within the Tier one group, the engines break out as follows regarding who handles how much of the search traffic:

Google – 50%

Yahoo – 28%

MSN – 13%

Ask – 6%

Other – 3%

Directories are a different story.

Every directory will follow a version of the following explanation for submissions.

You will find a simply page in most directory submission areas which will first request that you confirm whether the website exists in the directory currently – most directories allow only ONE listing, so make absolutely certain your URL does not currently exist before proceeding. This step may also uncover surprising information about your domain.

Many domains now have been bought and sold, so it’s plausible that you may find your URL listed already in a directory, though it’s brand new to you.

Next up will be a section for the URL, a title and a description of the website’s main content or theme. Be accurate and factual here – if you try to fudge things, you’re application will likely be rejected and rarely are refunds given. At this point you’ll also be asked to select the category the website should be listed under. Again, be real here – stick to the spot that best fits your site – not where YOU think YOU might like it to be found. Most directories will require you to submit to the lowest point in the directory, too. This means that a website about “Apples” would not be listed at the level called “Fruits”. Your site would be listed in the category which best describes it. If the site is about red delicious apples, you’ll need to follow down the levels until you find the section for sites about “Red Delicious Apples”. If no such level exists, you’ll stop and submit at the deepest point in the list. If you try to submit somewhere inappropriate, you’ll be rejected and any funds paid will be non-refundable. If you submit too high in the list, the directory may simply place you where THEY think you fit. They may also reject the application and you’ll have to start over.

Directories will most likely require you to submit only the main page of your website. It’s best to simply supply the most direct URL – your root: http://www.yourdomain.com Most directories will reject an application with a page from inside the website. Some will accept them, but you’ll still fall under any rules they have regarding single URLs being included in the directory list.

Given the value of links from places like the Yahoo Directory and the Open Directory Project (DMOZ.org), it’s best to carefully read the inclusion instructions and follow them to the letter.

The final portion of the application process will be the review everything one final time before committing the submission. Spend some time here refining the description you entered, etc.

If there are fees for submitting your application, you’ll either be asked to confirm payment immediately up front, before beginning the application process, or they’ll tag you near the end – think you’ve done all the work, so you’ll pay at this point. Don’t feel like you’re hostage to this. If you have no budget to spend on directory submissions, then don’t spend the money. Just submit to the free directories to start and after the website has generated some income, use that to pay for other directory submissions. Remember, the goal here is to get some valuable one-way links that the spiders crawl to find you.

Directories - Who is who

The two most obvious and well-known directories also embody the two opposite ends of the “pay for inclusion” spectrum.

Some time ago, Yahoo woke up and realized Google was using its directory as a sort of “priority list” of who was new and worth looking at. Yahoo then began charging users a fee for being CONSIDERED for inclusion. Note, this payment did not mean your site WOULD be included in the Yahoo directory, it meant they’d look you over, CONSIDER you, for inclusion. Any errors or attempt to list internal pages, etc. were met with rejections, though they still kept the money you paid. The rules have relaxes slightly over time, but the fee now stands at a hefty $299 USD – to be reviewed for inclusion. If you don’t pass muster, or make a mistake on the application, you’re still rejected, the money is kept and any further submissions must be accompanied by ANOTHER $299 USD payment. It sucks, but that’s the cost of a link from Yahoo, apparently.

There used to be a back door though Yahoo.co.uk as they were still actively culling their free submissions listings a few years ago and including folks. The benefit was that when Yahoo.com updated their directory, it would include the listing from all the other Yahoo directories, thus popping new sites into the main directory without them paying the $299 USD fee. Today, though, that’s not really an option – they’ve clamped down on sites getting into the main directory list using this back door.

The opposite end of the spectrum is The Open Directory Project (DMOZ.org). Started back when the Internet way young, DMOZ (or OPD, they’re used interchangeably it seems) is THE place you want a free listing. That’s’ right – Google values listing here and it’s free.

Here is a the best place to find the most current directories and some relevant data about them:

http://www.strongestlinks.com/directories.php

There is a complete directory list available there. These guys do an excellent job keeping the list up-to-date, too, so it’s a trusted source for me.