Software Voices - Opinion

Search Engine Ethics: Even Google has a Learning Curve (Google Music Search)

nospam@example.com (Craig Ogg) — Wed, 21 Dec 2005 00:05:00 -0600

UPDATED 02-19-2006 I incorrectly stated that Google was no longer showing Epinions reviews for newer music. My recent searches say I was either mistaken before, or Google changed its policy again.

Eric Hammond pointed out Google Music Search to me a few days ago. I eagerly tried it out and was thunderstruck -- Google stepped over a line I thought they would never cross. As I looked up bands, I found most of the reviews were from Epinions and were offered alongside comparison shopping. Which; of course, is Epinions' business model. One of the bands I looked up had 7 pages of reviews on Google Music Search, all provided by Epinions.

At this point, the alert reader will notice that I said "had." More on that in a minute.

First, why is this an ethical issue? Search results are search results, right? In my opinion, there is a big difference between searching and aggregating. By aggregating Epinions' reviews and ratings, Google was trying to replace Epinion as a destination for this type of information. This undercuts Epinions business model by destroying the community and makes it less likely that this type of information will be created in the future. Had this continued, this would have left Epinions in the position of having to choose between giving up all traffic from Google (by excluding them in robots.txt) or providing them content for free to compete with them. Truly a Hobson's Choice for an Internet company.

Based on the searches I did today as I was starting to write this blog entry, I have to say that it appears that Google has modified Google Music Search to be much more reasonable. While the Pixies search above shows Epinions reviews, all of the reviews are years old. Searches on more recent albums (like anything by Norah Jones) shows a large number of reviews on Epinions, none of which show up on Google Music Search. This seems to me to be a great compromise: it preserves Epinions incentive to grow its community while extending the life of the content created there (by exposing it to more people after it ceases to be a draw at Epinions). Bravo Google.The reviews are back for new music. Google is taking a particularly interesting approach with the reviews from the BBC web site. More on that in a future blog post.

I should emphasize that this is all speculation based on a few searches several days apart -- there is a good chance that it is entirely off-base. That said, I think the ethical issue is real and will play itself out again and again as the importance of community continues to rise on the Internet. For more on this, see John Clippinger's and David Bollier's thoughts on the conflict between the "market" and the "commons."

For another example of this type of conflict, look to the job listings search market. Indeed is a search engine, and Oodle is an aggregator similar to Google Music Search (I think they might disagree with this label, to be fair). Craigslist, seeing this distinction, allows Indeed to crawl them but not Oodle. Indeed's model doesn't threaten Craiglist's community, where Oodle's does because Oodle is seeking to replace Craigslist as a destination. A quick glance at the homepage makes this clear. I offered my opinion on this on Oodle's blog.

Opportunities for Innovators in Search - An Entrepreneur's View

nospam@example.com (Craig Ogg) — Fri, 09 Dec 2005 10:51:40 -0600

On Saturday, December 10th I have my first speaking engagement in a long time. I am on the panel as an entrepreneur at the Caltech/MIT Enterprise Forum panel on Opportunities for Innovators: Venturing in Online Search, Advertising & Sales.

I am using this blog entry to get my thoughts together for my (short) presentation.

I am co-founder of ThisNext, a "Web 2.0" social commerce start-up (vague enough? ) that is in the process of closing its Series A. Prior to ThisNext, I was VP of R&D of Stamps.com. I was employee #12 at Stamps.com and left at the end of January.

First let met me give you a quick intro to web advertising terms. CPM is traditional web advertising where the publisher is paid based on how often an ad is shown. CPC is a newer form of advertising where the publisher is only paid for readers who actually click on the ad. CPA advertising only pays the publisher when some action is accomplished. This can be as simple as the user providing an email address or as complicated as the user signing up for a service and being successfully billed for a certain number of months. CPA can also be referred to as lead generation or affiliate marketing

Each type of advertising splits the risk between the advertiser and the publisher in different ways. In CPM, the advertiser is taking all of the risk for the success of the campaign. In CPA, the publisher is taking all of the risk. This difference in risk usually represents itself as a spread between the advertising rates that is subject to arbitrage. Affiliate marketers often take advantage of this spread by using CPC advertising to drive CPA sales.

Most of the business models for search-related companies are fueled by CPC and/or CPA advertising.

What follows is a whirlwind tour of the current search space and where I think the opportunities are. Let me note that it is often hard to tell the difference from the outside between a search engine (Froogle) and an aggregator (Shopzilla). For purposes of this discussion I am just going to gloss over the difference.

There is a lot of action right now in what is known broadly as "vertical search." Vertical search is search that focuses on one narrow area and attempts to offer more functionality than could be offered by traditional search. Vertical search engines exist for shopping (Froogle, Yahoo! Shopping, Shopzilla), jobs (SimplyHired, Indeed, Oodle), tickets to events (FatLens), travel (Kayak), dating (Lycos' Dating Search), apartments (Rent.com) and more. There is still plenty of opportunity in this space, though Google may end up gutting this market some with Google Base. I think successful ventures in this area will bring technology or domain knowledge to bear in a way that transparently enhances the user experience. Businesses that still depend on knowledge inequities for their profit are particularly disruptable.

Another area that has received a lot of attention is "local search" -- finding businesses and/or products that are near you geographically. Given that all of the major players and several minor players are pursuing this space, I think this area is one to avoid if you are starting today unless you really have a unique and defendable idea. One challenge in this space is finding a business model that incents local businesses to do the extra work to be on-line. Yelp is one new player in this space.

"Media search" is just starting to get off the ground. Podoscope and Podzinger allows you to search audio that has been transcribed, Riya allows you to identify people in your digital photos so you can locate them easily later.

The current explosion in search revenues was really launched not by a new way of searching, but a new financial model for advertising (CPC, thanks Bill Gross!). Several companies are exploring this theme. Snap is exploring CPA where higher conversions actually affect your organic search ranking. /Root is exploring the securitization of attention along with Attention Trust (it is as complicated as it sounds), similar to the way that mortgages were securitized creating REITs. While it is hard to get advertisers and publishers to accept new financial models, the rewards can be huge. This area is ripe for innovation.

A new type of technology is starting to appear that I call "augmented reality search." This type of search is trying to bring the same type of efficiencies we have on-line to the physical world. Searching physical books, identifying products by entering the UPC or taking a picture of it with your phonecam, and even identifying animals and plants from their mitochondria. These are deep areas that are expensive to get into and therefore expensive to replicate as well.

One movement that is threatening to shake things up is "open search." Both Amazon and Google are attempting to change the way that search operates to allow the data that is in sites to be as searchable as the content is today. Google Base allows publishers to attach arbitrary attributes to an URL to allow it to be searched more intelligently: price, location, size, etc. Most see this as a threat to eBay and Craigslist, I see it more as an entirely new model that (if it works) does more to allow the creation of new businesses than the destruction of existing ones.

The area where my company, ThisNext, plays is in what I call "discovery." Related to search and sharing most of the business model characteristics, companies focused on discovery help you find things you didn't know that you were looking for. Del.icio.us, Wists, Pandora, Last.fm and in some cases, blogs are all in this space. Whether the recommendations are social or from experts, I think this area still has huge potential for growth.

Update 12/14/2005 Attempted to add the rest of the companies I mentioned during my talk to this post. The talk I gave ended up differing significantly from this, this was just to organize my thoughts.

Did Adam Bosworth reveal the real Google Base at the MySQL Users Conference?

nospam@example.com (Craig Ogg) — Mon, 07 Nov 2005 11:58:49 -0600

Back in April of this year, Adam Bosworth laid out a compelling vision of how to create "internet scale" databases at the MySQL User Conference that sounds suspiciously like something Google Base might be designed to support (listen to the podcast).

Paraphrase: "Imagine being able to query all of the world's websites for their information -- not their content, but their information. You could find out what is for sale and at what price or where they are located accurately enough to place them on a map."

The examples he gave were all things that were in the (briefly exposed) Google Base interface: searching for real estate listings, finding local businesses, and findings products for sale.

To enable this vision Adam calls for the creation of a universal data-verse (my word, not his) that mirrors HTTP/HTML: standardized, simple, sloppy, stateless, and thus scaleable. This data-verse would have queries in the form of URLs and results bassed on RSS/Atom, similar to what Amazon is doing with OpenSearch at A9. It would be inherently distributed and any "P" programmer (Perl/Python/PHP) can easily participate because the RSS and Atom are so straightforward to implement (as evidenced by their large adoption).

Most interestingly, Adam suggested that Google would participate as a peer in this new global database -- encouraging the MySQL development community to create one of the first query engines that supports it. (Paraphrased) "In a few years we will look back and view the closed, centralized database servers of today as 'quaint'."

As the Oodle controversy with Craigslist points out, liberating information from content has real world implications that need to be worked out. That said, if this idea gets traction I am sure it will spur a new round of innovation that will make the current RSS-based innovation pale in comparison.

NB: I was listening to this podcast while driving so I am sure I got some of the details wrong.

Update 11/16/2005 While I still believe the Bosworth talk refers to the ideas behind Google Base, it appears I was wrong about Google intending to participate as a peer. At least currently there is no way to access the results of a Google Base query programmatically. No RSS output, nada.