A very common question that I get asked is, “How do I get my Web site indexed” by Yahoo! or Google? It’s a simple answer really, and most people unwittingly do it incorrectly. You want to make sure that you get indexed properly so Google and Yahoo! fully trust you and your site. If you’re a Web developer or working on search engine optimization for your clients, you need to make sure you don’t set them up incorrectly.

What’s the WRONG Way?

Many people know that Google and Yahoo! have a place in their system to “Submit a URL”. These places hark back to the “olden” days of the search engines, and guess what? Spammers like to use these pages to submit their sites, with the idea that it’ll help them get onto Google more quickly. So if you want to look like a spammer in Google’s eyes, please go ahead and use the “Submit a URL” page.

However, note that I don’t have a link to them here. Most of us are legitimate business owners who want to get listed properly, and want Google and Yahoo! to trust us and our sites. So my advice is to NEVER use the “Submit a URL” link – ever.

How Do I Get Indexed Properly?

There are many ways to get Google or Yahoo! to index your site. None of them are going to be fast. It just takes time for them to come around and check you out, especially if you’re a new Web site. One of the best and most common ways is to get links to point to your new site from somewhere else. The search engines will discover these links and someday, get around to checking you out.

Unfortunately, it’s slow, and can take weeks. I’m a little more impatient than that, so I try to speed things up as much as I can.

Getting Indexed By Google

If you read yesterday’s blog about XML sitemaps, then hopefully you’ve already created a free account with Google’s Webmasters Tools and the Yahoo! Site Explorer. Assuming you’ve done so, then when you add a new site into their system, they will tell you that you haven’t validated the site yet.

Google will tell you that you have two choices to validate your site:

  • Create a validation file and put it in the root of your Web site
  • Create a special META tag that you put on the Home page of your site

Either one of these will work just fine, but because I tend to work with DotNetNuke Web sites, I don’t generally have the control to pick #2. So it’s simple to create an empty file (0 bytes) with a text editor, and give it a name that Google tells you. It will be a long hexadecimal file name like:

googlef9461b9e918b4846.html

This file name is unique to every Web site, so you can’t just use one for all your sites. Copy the file up to your root directory on the Web site, and push the button to tell Google that it’s there. You only have to do this once, and then you’re validated with them.

Getting Indexed By Yahoo!

Yahoo! has a very similar process in their Site Explorer tool. Once you add the site, you’ll be given the opportunity to validate it. They make it a little easier for you by just giving you a one-click button to create their special file. It too is unique to each Web site, so you need to create a new file each time. These files are not empty either, so just let Yahoo! create the file for you. It will have a file name something like:

y_key_86dfbe044123f5da.html

Again, copy it to the root directory of your Web site, and tell Yahoo! it’s there.

Final Indexing Thoughts

Once you’ve done this, make absolutely sure that you’ve created an XML sitemap for the site. The XML sitemap can be the same for Yahoo! and Google, but you need to let them know that it’s there.

Following this fairly simple one-two process will get your site indexed pretty quickly by both search engines. Google tends to pick up the new pages more quickly. I’ve seen them show up in as little as a week (but don’t count on that!).

Sometimes they’ll show up, then disappear again. Don’t panic! Just leave it alone, and within another week or two, it’ll start showing up again. Don’t keep pushing the “button” like everyone does at the crosswalk. Just let it ride, and you’ll drop into place with no more effort. Tell your clients to just sit, and it’ll take 3-4 weeks, then they’ll be all good with the universe.