The basics

First, you need to understand how the search engines - or, rather, the 'spider' programs the engines use - actually work.

Put most simply, a search engine spider is a program designed to follow URL links and index what it finds. They extract the text from the various parts of the HTML code, comparing different elements such as the <title> tag and other components of the header section of the code, regular page text content (often giving weight to items according to their formatting), and extra items such as alt text descriptions. Any links to other pages that are found in the process are then followed, taking the spider through all the linked pages in your site as well as onto any other sites to which you've linked.

The formatting of text in your pages will have a direct effect on how important the text itself is regarded by the spiders. This doesn't have to mean designing your pages to suit spiders at the expense of your readers; in virtually every case a page that follows the basic rules of Web design (easy to read, easy to use and loads quickly) will be fine for both. However, there are ways of ensuring that things work more efficiently for the spiders without impacting the human viewer's experience.

Using the traditional h tags - <h1>, <h2> and the rest - to format content will go down well with most spiders, as these are well-established methods of defining a hierarchy of importance for formatted information. Of course, simply applying these standard header tags to text gives results which are boring and inflexible - do you really want all your headlines, subheads and other text elements shown in varying sizes of Times Bold?

The solution is to redefine the formatting a browser uses for these tags by creating custom CSS style definitions which use the regular h names; h1, h2 and so on. The method you use depends on your preferred site authoring tool - for example, Dreamweaver users will prefer to build their own custom h tag CSS definitions, while Softpress Freeway 4 users can edit the H tag styles in the Styles palette. (For Freeway 3 users, the CSS Suite from will remap existing styles in Freeway to the appropriate items.)

The title tag for a Web page, the element that defines the name shown in the browser window's titlebar, is an extremely important bit of text. Not only is this taken into consideration by a spider when it evaluates a page, this is also what many search engines show first in the results listings. Because of this, it's important to make sure your title tag for each page is descriptive for those looking at search results, as well as containing the appropriate keywords for indexing. Don't pack these keywords into the title: make sure the title text is informative as well, otherwise people will pass the pages by even if they rank highly in search engine indexes.

Next: Setting the right Keywords


The Basics

Setting the right Keywords

Tricks to avoid

Commercial options

The final word


Index or Directory?

There are two distinct kinds of search engine, and you need to know the difference between them so you can optimise your sites to take advantage of both. The first is the text index variety, built using spider programs that read, sort and index the text content of every page they find. (Although to be precise other programs do the sorting and indexing work.) Google and Alta Vista are good examples of this approach; the indexes used by these sites are principally built by automated tools that read pages and work out ranking methods. The process of determining the ranking and relevance for different search terms is complex to say the least, but it is automatic.


The other is the directory variety, a categorised listing that sorts sites into different areas according to their content. Directories are maintained by humans rather than spiders and are organised according to large hierarchical directory structures. Two good examples of Web directories are the Open Directory Project at and Yahoo. These rely more on site submissions than the automated search engine sites, but, of course, content is found for these places using any means they deem suitable. Getting into a directory can take a very long time, as each item is checked and placed - at least in theory - by a real 'editor' rather than by a computer algorithm.

Yahoo showing Google


The two forms of search engine do mix. Google and other index databases refer to directories for finding content, and directory searches will often give results from indexes (sometimes flagged as 'web results') as well as from their own site directory databases.

Be Found: Designing Findable Sites (2)