Software Voices

I have written previously on Adam Bosworth's talk at the MySQL Users Conference. At that time Google Base had not yet been revealed to the public. I now have had a chance to play with Google Base, have listened to the Bosworth's talk again and I have some opinions on how his vision of Google Base as the "http and html" of information access is playing out.

Adam believes that the success of HTML is attributable to its having one schema and the fact that browsers handle badly formatted HTML without barfing -- it is inherently a "sloppy" format and user-friendly. He says RSS 2.0 and Atom are similarly sloppy formats that can express information in a reusable way.

In other words, the formats that Google defines for bulk uploading (RSS 2.0, Atom) are intended to be nothing less than a new universal schema for all data. While these are uploaded to Google today, Adam imagines a world where these files are produced as the result of a query ala Open Search. Adam goes so far as to say that Google will publish their data in this manner.

What I wonder is whether Dave Winer will beat them to the punch in making this a reality. A robots.txt or <link> tag seems to be all that is necessary for publishers to make this data available to all comers. Alternatively, the developers of A9's Open Search could adopt Google's upload format for results. Either step would bring us one step closer to Adam's grand vision of the democratization of information.

Back in April of this year, Adam Bosworth laid out a compelling vision of how to create "internet scale" databases at the MySQL User Conference that sounds suspiciously like something Google Base might be designed to support (listen to the podcast).

Paraphrase: "Imagine being able to query all of the world's websites for their information -- not their content, but their information. You could find out what is for sale and at what price or where they are located accurately enough to place them on a map."

The examples he gave were all things that were in the (briefly exposed) Google Base interface: searching for real estate listings, finding local businesses, and findings products for sale.

To enable this vision Adam calls for the creation of a universal data-verse (my word, not his) that mirrors HTTP/HTML: standardized, simple, sloppy, stateless, and thus scaleable. This data-verse would have queries in the form of URLs and results bassed on RSS/Atom, similar to what Amazon is doing with OpenSearch at A9. It would be inherently distributed and any "P" programmer (Perl/Python/PHP) can easily participate because the RSS and Atom are so straightforward to implement (as evidenced by their large adoption).

Most interestingly, Adam suggested that Google would participate as a peer in this new global database -- encouraging the MySQL development community to create one of the first query engines that supports it. (Paraphrased) "In a few years we will look back and view the closed, centralized database servers of today as 'quaint'."

As the Oodle controversy with Craigslist points out, liberating information from content has real world implications that need to be worked out. That said, if this idea gets traction I am sure it will spur a new round of innovation that will make the current RSS-based innovation pale in comparison.

NB: I was listening to this podcast while driving so I am sure I got some of the details wrong.

Update 11/16/2005 While I still believe the Bosworth talk refers to the ideas behind Google Base, it appears I was wrong about Google intending to participate as a peer. At least currently there is no way to access the results of a Google Base query programmatically. No RSS output, nada.