Mary Hodder points out that charging $240/year per group is the least of the problems with MeetUp.com's new business model. The real long-term threat to MeetUp and other companies who rely on similar business models is Structured Blogging. The combination of search engines (both retrospective and prospective) with Structured Blogging will force many of today's web businesses to either adopt new business models or cease operations. At the same time, a tremendous number of opportunities for entirely new and innovative businesses will be created.
Structured Blogging is an almost trivially simple idea which delivers on the many promises made for the semantic web. But, unlike previous semantic web proposals which have been crushed under the weight of new and exotic technologies, Structured Blogging as proposed by PubSub is practical and relies only on existing, proven, and widely available methods. As a practical approach, not merely a vision, Structured Blogging has the capacity to be one of the most disruptive Internet technologies of the next few years.
Sites like MeetUp, eBay, Monster, EVDB, and hundreds of others have business and usage models that work only because they exploit a small number of temporary limitations of today's most commonly used web publishing and searching tools. When those limitations are eliminated through deployment of Structured Blogging capabilities in millions of syndication feeds, by enhancing search engines to handle structured data, and by broad deployment of prospective search, the very foundation upon which these businesses are constructed will disappear. Their market window will have closed...
The Gray Web -- Hidden in Plain Sight
There are many who have commented on the differences between the "visible web" and the "hidden web." The hidden web is commonly defined as "the publicly accessible pages on the World Wide Web that are not indexed by search engines." To these two webs (visible and hidden) we should add a third web -- the Gray Web. By this I mean publicly accessible pages on the web that are not usefully indexed by search engines today. The Gray Web is both dark and visible at the same time. The Gray Web is composed of the structured data and the timely or rapidly changing data not well supported by most of today's search engines.
Structured Data -- not just text
Virtually all search engines used with the visible-web today are limited to only indexing simple text. They have no ability to deal with the kind of structured data that must be used to describe events, job openings, offers-to-sell, offers-to-buy, etc.
On a site like Google, you can find all the pages that mention "ballet", "performance" and "New York" but can't give you much direct help if what you're searching for is "Ballet Performances in New York this weekend." For an answer to such a query, you need to use a hidden-web site like CitySearch. Google can find pages which contain keywords but it can't decipher their meaning or semantics within the context of an announcement and Google has no idea what "this weekend" means. For Google to do what CitySearch does, it would have to recognize that something was an Event-Announcement and it would have to index the individual "fields" of the many event announcements that it finds while crawling the web. Even though Google, Yahoo!, Altavista, etc. actually index millions of pages that announce events, job openings, offers-to-sell, meet-ups, etc. today, these search engines index them simply as blobs of text. Because they don't recognize what they are reading they can't index these things in a usefully searchable manner.
Structured Blogging provides a means to include tags in web pages and blog postings that clearly identify structured data for what it is -- rich, fielded data -- not just a blob of text. Using Structured Blogging, you can post an item which is an "Event Announcement" and, because of tags in the item, that item can be identified by any one of potentially hundreds of search engines as being an "Event Announcement" which has fields that conform to a known or discoverable schema and can thus be indexed, stored and searched for with all the power, precision and flexibility of the normal structured data searches provided by databases.
For a taste of the utility of publishing structured data on the visible web, take a look at the Video Search system that Yahoo! recently released. In part, this service relies on Yahoo!'s Media-RSS extensions which empower bloggers to clearly identify and describe, using structured data, the multimedia items that their blogs point to or discuss. Now that it is exposed, Yahoo! can and does index the structured data -- not just the text in the blog posts! (Unfortunately, the Media-RSS stuff only works in RSS files. Hopefully, an adaption of Media-RSS to Structured Blogging norms will enable Media-RSS to work in (X)HTML pages as well as Atom feeds.) An older example of search engine support for structured data can be found in Google's Froogle service. Froogle relies on vendors sending to Google structured data which describes their product offerings, prices, etc. Imagine if those vendors were to make their Froogle data available for use by all search engines using openly syndicated feeds of structured data... We are moving toward a world of ubiquitous syndication of data by businesses as envisioned by Paul Kedrosky in his June-2004 Harvard Business Review article entitled "Feeding Time."
With Structured Blogging, we'll be able to post structured items in any of millions of blogs or web sites and have those items recognized, indexed, and searched on any number of search sites -- just like HTML pages are today. No longer will we need to rely on going to a small number of centralized, walled-garden, closed sites like MeetUp, eBay, Monster, or EVDB to publish or search for the kind of information that requires structure. Common search engines like Google, Yahoo! and PubSub will be able to usefully index this data. In the future, as search engines come to better support structured data, we will all benefit as the Gray Web grows smaller and the visible web grows larger.
Timely and rapidly changing data -- not static texts
The retrospective search engines that we rely on today (Google, Yahoo!, etc.) don't provide good support for data whose timeliness is important. While they scan a small number of sites fairly frequently, they only visit average web sites once every few weeks at best and often only update their primary indexes monthly. Thus, while today's visible-web search engines index billions of pages, they simply aren't very useful for discovering recently posted information or tracking information that changes frequently. Vast amounts of the data on the web, even if theoretically visible to search engines, either won't ever be found by them or won't be indexed in a usefully timely manner. A great deal of Gray Web data becomes out-of-date, is replaced, or is removed from the web long before the search engines even try to index it. This weakness in the timeliness of visible-web search engines has opened a vibrant but hopefully temporary niche for exploitation by hidden-web service providers.
The walled-garden hidden-web providers extract a terrible price for servicing data whose structure or timeliness makes it unsuitable for today's visible web search engines. Typically, the price of service is exclusivity. Hidden web sites usually require that if data is to be included in their databases, it must be input or published directly into their systems using custom input forms or methods. While they may not formally require that information is exclusively published at their sites, the reality is that by imposing this "cost" of explicit publishing, they make it unlikely that many people will publish the same data on many different sites. It's just too hard... In essence, the hidden web sites are exploiting the weaknesses of visible-web engines to pull data out of the Gray Web and into the hidden web. These sites then monetize the "network effects" that result from their drawing large quantities of data into their hidden, walled-gardens. The network effects which should benefit the visible web as a whole are thus channeled to the benefit of a small number of hidden web sites. Often, these sites will start "open" and free, but once they have established their dominance in a particular category, they will begin to charge their now captive markets for what might otherwise might have been a free service. Such is the privilege of monopoly... MeetUp, eBay, and many others have followed, or are tying to follow, this path to profit.
The Blogosphere has taught us how to build systems that can usefully address the need for timely access to rapidly changing information. Today, rather than relying on slowly crawling the entire web to find updated blogs, blog search engines like Feedster, Blogdigger and PubSub rely largely on "ping" messages that are sent by millions of blogs whenever they update their content. The result is that updates are almost instantly announced to those who seek them. There is even a FeedMesh project which has shown excellent results in distributing notices of blog updates between the various blog search and monitoring services who have decided to share data while competing based on the quality of the services they provide rather than by trying to capture and horde information about blog updates. FeedMesh helps push the fast-changing Blogosphere from the Gray Web into the visible-web.
This new pattern of publishers announcing their updates to search engines has enabled us to implement prospective search services such as PubSub's. Prospective search services compliment traditional retrospective search engines by allowing users to "subscribe" to be notified instantly whenever new information in which they have a persistent interest is published. The result is a level of service for timely information that is at least equal to and normally far superior to that provided by most of today's hidden web providers.
The Visible-Web will grow while the Gray and Hidden Webs shrink
Structured Blogging combined with timely updates and prospective search will challenge the ability of many of today's hidden web providers to rely on hording otherwise public data in their walled gardens. The result will be significant impact on the business models of what are today some of the Internet's most venerable brands. But this is only natural. Just as Internet innovation in the 90's allowed these hidden web sites to be conceived and created, further innovation in this new decade will provide opportunities for others to establish themselves. The innovators of the 90's have become today's "old guard" and they will be challenged when further innovation eliminates or reduces the weaknesses that they have come to rely on exploiting.
A growing focus on competing through quality of service -- not database size
It should be understood that companies like eBay, Monster, etc. are not doomed... Rather, they will be pressured into changing their business models to adjust to the new environment. Today, the hidden web providers can rely on monetizing their exclusive access to data. Tomorrow, these same providers will find that their real business value lies in the services they provide based on visible web data. A company like eBay is likely to find that superior auction management, reputation management and payment processing will be enduring sources of revenue and distinction. They will undoubtedly find ways to provide these services even if they don't have exclusive access to large collections of hidden-web data. Given superior services offered by eBay, people or companies who publish offers-to-sell on their websites using Structured Blogging are likely to specify in those offers that eBay is their reputation management provider or their preferred payment processor. Thus, even if the offer-to-sell is discovered on a site other than eBay's, it will still result in business for eBay. (Of course, if eBay doesn't provide good service, publishers of such offers are likely to specify some other service provider -- therein lies an opportunity for many new businesses...) A company like Monster may find that their real business strength is in evaluating or screening applications and resumes rather than in simply collecting and selling access to the large number of offers-to-hire and resumes in their hidden-web database. Potential employers are likely to simply post many of their job openings on company "hiring blogs" using Structured Blogging since they will know that many job search engines will find and process their data. But, those employers might still specify that Monster is their preferred "application processor" -- no matter which search engine is used by a candidate to discover the opening. Of course, Monster could still continue to provide the "job search" function -- they would simply expand it to include content discovered on the visible web.
The same sort of shift in the source of competitive advantage will happen for other sites as well. Today's players probably won't go away... They will simply change the focus of their efforts while learning to deal with and service the Gray Web data turned Visible.
This overly-long note can be summarized by simply saying, once again, that I agree whole-heartily with Mary Hodder's observation that the real long-term challenge to MeetUp and companies like them is further innovation in the Internet -- particularly in the areas of Structured Blogging and Prospective Search. As the inevitable innovation eliminates the technical weaknesses that have allowed the establishment of business model that rely on driving Gray Web data into the hidden web, those vendors who rely on building walled gardens around Gray Web data will find that their current business models simply won't be sustainable.
At PubSub.com, we believe that our proven capability in prospective search as well as our structured data matching and handling capabilities uniquely position us to take a leadership position in servicing customer needs in this new post-90's environment. Mary, soon you will be able to publicize your MeetUps through PubSub by simply posting event details on your blog with Structured Blogging -- and we won't charge you a penny! (But, we might slip one or two discrete advertisements onto the page... )
bob wyman
This is an absolutely wonderful post. I agree 100% with this (and I don't always agree with you, Bob :-) )
Posted by: Michael Arrington | May 08, 2005 at 16:38
Lots of stuff to digest in this one. Very good.
Posted by: Phil | May 09, 2005 at 17:25
An extremely interesting and convincing piece. Somebody could make a lot of money if they really thought through the implications.
Posted by: Peter Dean | May 17, 2005 at 11:54
Supplanting eBay, Monster, etc. would be very difficult. Don't forget that eBay has successfully sued to keep its site from being crawled. If you can't crawl eBay to aggregate the info your auction service would be sparsely populated. This (legal) is probably the only thing stopping Google from aggregating auctions.
Posted by: Johannes Kepler | May 18, 2005 at 21:56
I think structured blogging is a great start toward supplanting eBay, but you also need business transactions. eBay registers auction wins, PayPal pays for the goods and can track shipments, and they even have a relationship with Escrow.com.
It would be possible to do a better job of all of the above features, but at least eBay can complete deals in some fashion.
Are you interested in completing the circle?
Posted by: Bob Haugen | May 24, 2005 at 09:23