Internet Entrepreneurs Blog

We’ve spent a lot of time on this blog talking about how to get your content into search engines but sometimes there are things you don’t want to get indexed. Sure you can use robots.txt and no-follow links to block search engines from indexing entire pages but we were curious about one thing: HTML comments. Do search engines read and index them?

For many of us HTML comments can contain embarassing or even proprietary information we’d prefer to keep hidden from the general page-viewing public. How many times have you written something like ‘messy code, I’ll clean this up later’ in the comments or even left database connection strings or development notes on a page by mistake? We certainly don’t condone the latter because while HTML comments are hidden from the casual browser a simple ‘view source’ will reveal your notes to anyone and everyone who pokes around.

But back to the question at hand: Do search engine spiders index HTML comments? Based on our tests the major search engines (Google, Yahoo, and MSN) do not index HTML comments - fortunate for many of us. It appears there is no need to go through and scrub your pages with comments right away but it’s still a good idea to write HTML comments AS IF they were potentially being indexed because you never know who is viewing your source.

If you were considering keyword stuffing your page through the use of HTML comments clearly this won’t work but you might be interested in our hidden DIV and hidden text posts.

Once again we’re out of SEO test results so let us know what SEO myths you’d like to see put to an objective test.

I suppose every Web 2.0 designer should know the answer to this question but admittedly I did not. We’re working on a redesign for one of our outdoor sites and the design calls for some of the content to be placed in a show/hide div to keep the page neat. I wasn’t sure if this would be a good idea from an SEO perspective so I decided to run a test.

I created a simple page with a medium amount of text visible and outside any hidden divs. I also placed more text within a div containing my target keyword that was set to display: none. This particular page did not contain the requisite javascript link to show/hide the div but I didn’t care - this page wasn’t for users, it was for search engines.

The result: within a week the page was indexed by Google and the hidden div text was picked up along with our target keyword. This is a useful finding for two reasons. First, it means unscrupulous publishers can place unrelated text on a page and keep it completely hidden from users while search engines slurp up the illegitimate content. But it also means that if, for design reasons, you need to place some of your content in a show/hide div arrangement the content is still visible to Google. Our test page hasn’t been picked up by Yahoo! yet but we’ll keep you posted if/when it does.

As with all search engine optimization knowledge, we hope you use this tip for good and not evil

If you’ve been following web development over the past couple years you’ve undoubtedly heard the arguments for separating markup from style information. You know, instead of using tags like <font> and <b> you should be using <span> and applying formatting via CSS. Great advice for sure - but what do the search engines think?

For this test we decided to see whether markup or style could communicate the importance of a single keyword to the search engine crawlers. In our test we created 3 pages: the first contained our keyword with no markup or styling, the second contained the same keyword but placed within bold tags (<b></b>), and the third contained the keyword within <strong> tags. Common wisdom is that <strong> is the better choice because it is markup (rather than a specific style) and thus communicates something (importance) about the words other than how they should look.

Anyway, the result: the search engines still prefer bold to strong. In fact, this is the first time in all our tests that both Google and Yahoo! agree - bold is more important than strong. Interestingly neither search engine picked up our page without any emphasis on the test keyword.

What does this mean? For me it means that I’m not quite ready to start replacing all my bold tags with strong tags. Honestly I understand and agree with the need for separating markup from style information. But it’s just too convenient to simply slap a <b> around a couple words than to go through the trouble of defining a style and calling it via <span class=”important”> or even <strong> which is 3 times as long as good old <b>. Funny thing is, it looks like for now Google and Yahoo agree with me

You may have heard from your search engine optimization consultant that duplicate content is a no-no in the search engine ranking game - but what does this really mean? How strict are search engines in terms of duplicate content?

Let’s first consider the types of duplicate content that a webmaster might post on his or her website. Direct quotes, from articles or speakers, are a pretty common form of duplicate content but we wouldn’t expect to be penalized for this type of content, especially if it is properly credited. Online yellow pages and directories contain mostly business names, addresses, and phone numbers - not original content by any means - and yet somehow these sites get indexed and ranked by search engines as well. And how about online news sites that reproduce stories from the AP wire - these posts are IDENTICAL to content posted on hundreds of other sites yet there doesn’t seem to be any “penalty” applied to the local news sites.

So perhaps the search engines have written special rules to deal with this type of content - but I really doubt it. Instead I think search engines don’t expect web pages to be completely unique but they prefer pages that are MOSTLY unique. What constitutes a MOSTLY unique page versus a not-unique-enough page is certainly a mystery but if I were writing a search alogorithm I’d want to see the ratio of unique to duplicate content at about 1 to 1.

Applying a penalty to non-unique sites can also pose a challenge because search engines don’t always know which site posted the content first (i.e. who the owner is). If a plaigerizer copies your content within a day of posting it on your website, the search crawlers have a good chance of finding your content on his site first and thus applying the “penalty” to your site even though you did nothing wrong. Again, this leads me to believe the duplicate content penalty is a myth because it’s just too difficult to enforce fairly and accurately.

In our own search engine tests of duplicate content we found that two pages, each with identical content (title tags, content, markup, etc.), posted on the same domain, were both indexed by the major search engines. The only difference between the files was the filenames and the resulting search engine listing randomly placed one file as a part of “supplemental results.” Since both files were created at the same time the choice of “supplemental” versus “original” was complete chance.

The moral of the story is that you shouldn’t stress about original content on your website. Sure, you should never copy content from other sites but that’s because you’ll eventually receive nasty-grams accusing you of plaigerism - not because the search engines will start to hate you. Just use duplicate content judiciously and keep increasing your (mostly) unique pages to watch your search traffic soar!

In the old days it was understood that search engines could not crawl web content (particulary dynamic content) that had to be accessed from drop down “select” menus. For example, if a site was about restaurant reviews in the US and users were asked to choose a state from a form menu (like the one below), the state listing pages would be all but invisible to search engines. But is this still the case?

In our test we “hid” pages with our keyword behind a select menu and found (to our surprise) that Google was able to quickly navigate the form with no problems. In fact the pages behind the select menu were picked up and indexed just as quickly (within a few days) as other, normal pages on our site. Yahoo!, on the other hand, still has not indexed the pages behind our select menu and it’s doubtful they ever will (we’ll keep you posted here).

This new finding begs the question: is it still necessary to have alternate entry points for search engine crawlers to access dynamic content? I say yes for two reasons: 1. because less sophisticated crawlers aren’t able to get past menus and 2. because it’s better to be safe than sorry for those that can access your content. It’s also important to consider the other ways folks may be accessing your content where drop-down menus aren’t ideal (mobile phones, text based browsers, screen readers, etc.). Sure it means more pages and more development but it also means even more pages indexed by those sophisticated search engines

We’re back with a fresh set of SEO tests results to share and this first one is pretty interesting: Do search engines count anchor text as page content?

One SEO specialist we spoke to claimed text within anchor (link) tags did not count toward the content of the page itself but rather assigned that text to the linked page. This makes alot of sense because the text within an anchor is describing the page you’re linking to and not necessarily the page the link is on. But it’s sorta wrong.

In our test, we built two pages - one with our keyword placed 6 times within the page content, the other with the keyword placed on the page in 6 anchors pointing to a third page on a different domain. After about a week or so we had 3 pages indexed for our keyword - the non-anchor text page, the anchor text page, and the linked page itself (which, by the way, does not contain our keyword at all). The non-anchor text page gets top billing which does lend some credence to the theory that linked keywords hold less power for a page than non-linked words.

We’re still waiting for Yahoo! to pick up our test page but the way things have been going over there it might be a while. We’ll keep you posted once the pages are indexed…

So what does this mean? For starters it means you shouldn’t “send” your most important keywords off your page through links, though doing so won’t completely tank your ranking. It also means that pages can be associated with keywords even without having the keyword itself anywhere on the page - so long as the keyword is in referring link text. Stay tuned next week for more search engine test results!

Here’s an SEO trick you may have heard: placing Google Adsense advertisements on your new website will get your pages indexed sooner since Adsense relies on knowing what your site is about. Or perhaps you’ve heard the opposite (from an SEO friend of a friend): Placing Google Adsense on your new site will actually undermine your ranking because Google will think you are a spammer/domainer. So which one is correct?

We built two similar pages with similar keyword densities, page rank, etc. and placed a Google Adsense ad on one page, no ad on the other. After just a few days, the page without our Google Adsense code was indexed and a full week later the page WITH the Google ad was finally indexed. Even today the non-Adsense page is listed first in Google search results.

So what does Yahoo! think about all this? Yahoo! prefers the page without the ad which shouldn’t come as a suprise given that these two are bitter rivals. Well, maybe not that bitter since Yahoo! is itself a Google Adsense publisher these days but you get the point. Perhaps Yahoo! is jealous of us

So what does this mean? Should we take down the Google Adsense ads from our web pages to improve search engine rankings? The answer for me is no, especially given the revenue these ads generate for my sites. However, if you’re building a new site you may want to hold off on the Google Adsense ads until your site has been indexed and you’re generating at least a few “real” visits per day that can be monetized.

The cool thing about Google’s whole pagerank set-up is that, for the most part, it’s very difficult to game. Roughly speaking, pagerank assigns a value to a website based on the number and quality of outside sites that point to said website. The more links to your website, the higher your pagerank and the higher your site shows up in online search results.

While virtually every other portion of search engine optimization can be gamed to some degree, pagerank is a difficult thing to fake. Of course people have tried all sorts of schemes to increase the number of incoming links to their sites - link farms, paid links, spam links (posted on blogs / forums / etc.) - but in the end, genuine pagerank-enhancing links are made the old fashioned way.

In the early days we used to email folks and ask them to trade links with our sites to build reciprocal traffic (even though there was no such thing as pagerank back then). Today people still try to build links this way but it’s tough work and can come across as spammy to the recipients of link request emails. Even if your emails are genuine and get a good response from webmasters, we’re still talking a 5% hit rate or less - meaning you’ll need to identify and send 100 emails to get 5 new incoming links. Not an effective strategy if you ask me.

Our approach to collecting backlinks is actually pretty simple: we try to create the best content possible. If your site has useful, interesting, and relevant content you won’t need to ask people to link to you - they’ll do it automatically! Focus on making your site useful and not only will you collect backlinks but your users will actually stay on your site longer, generating higher pageviews and perhaps even a little word of mouth marketing for your site.

Now I know, creating great site content is tough but it’s really the only foolproof way I know of building a lasting online brand. A wise man once said “if you build it, they will come” - great advice if you’re considering a creating a baseball field in Iowa. But if you’re launching a new website I like to say “If you create great content, they will visit and link.”

How does the text within an anchor tag linking to your website affect your search engine ranking for a particular keyword? Anecdotally I can tell you it’s hugely important - we saw one of our sites rise to the top of Google results for ‘mountain bike blog’ simply by including that text in our trackback links. We decided to test the theory to see if this is always the case.

By now you know our setup - two pages, same keyword densities, pagerank, etc. but with one difference: one page has our target keyword in an incoming text link, the other does not. Our page without the keyword in the incoming link was actually indexed first by Google but once the keyword-link page was indexed it quickly took over the top spot in the search results. Today the keyword-link page is in second place but I think that is a function of Google getting zero feedback on the search results for this particular word (i.e. no clicks on either link since the keyword sees virtually zero search traffic, more on this another time).

Yahoo! chose to place the LINKING page at the top of results which shows us something else entirely. The linking page is a third page we didn’t intend to test here - it’s the page where the keyword link itself was placed. This shows us that Yahoo! indexes anchor text as page content rather than assign that text to the LINKED page, something that most of us assume Google does. The second result on Yahoo! is the non-keyword linked page while the keyword-linked page wasn’t indexed by Yahoo! at all.

These are certainly interesting results and the anchor-text as content finding definitely warrants further testing. Drop a comment and let us know what other SEO tests you’d like to see tackled…

Here’s a relatively new search engine optimization myth: The simpler a page markup, the more likely the page will be indexed and the higher its rank. This myth seems to have grown out of the movement toward valid XHTML and the separation of web content from the actual page layout itself. Instead of laying out pages using complicated table structures, designers are encouraged to use separate style sheets for a cleaner code and (conceivably) better search engine placement.

We tested two page layouts, each with the same target keyword with similar keyword densities, page titles, etc. to see if there was a difference in 1. WHEN the page was indexed and 2. Which page ranked HIGHER once both pages were listed in the major search engines. The first page used a simple, valid XHTML layout while the content on the second page was nested in nasty, redundant, overly complex HTML tables.

On the first point, WHEN a page was indexed, the complex page came out on top, showing up in Google almost a week before our simple page. Over the following weeks there was some jockeying of position but ultimately the complex page has settled in the top spot over the simple page. So for our Google test, it appears the venerable search engine doesn’t have a problem with complex markup and in fact may prefer it on some level.

Yahoo! had the opposite take on our test and placed the simple page layout on top of the complex page in their search results. If you’ve been keeping up with our test results on this blog you’ll know this isn’t the first or even second time that Google and Yahoo! have disagreed on such a seemingly fundamental point. Based on our limited testing it appears Yahoo! tends to reward many of the “traditional” SEO tricks while Google often yields counter-conventional results.

We have one more search engine ranking test result to share with you, then we’re out. If you have an SEO myth you’d like us to test, leave a comment here and we’ll put one in the hopper for ya.