In his recent article "If RSS ain't broke...", Steve Gillmor of ZD News continues to reinforce CNET/ZDNet's leadership in reporting on the "blog clog" issue. Gillmor responds negatively to the often made suggestion that news aggregators should poll less frequently in order to reduce server and network load. After pointing out that POP3 email clients typically poll for new email every few seconds he says: “Then there are the real fanatics like me, who don't want no stinkin' hourly intervals, thank you. I want my POP3 intervals, or even better, event-driven intervals powered by peer-to-peer.” Unfortunately, of course, if RSS Aggregators were to poll as frequently as POP3 clients do, the blogosphere would melt down instantly. Certainly, there is a real problem here that must be addressed, however, there must be a better solution than forcing everyone to accept lower quality of service.
Fortunately, PubSub.com offers the update frequency that Gillmor is looking for: i.e. "No Interval -- near zero-latency". Since we're a "push" based system (although we'll let you poll if you want...) as soon as we find new data we immediately send it to our subscribers. There is no polling interval and since millions of blogs currently push pings to us whenever they've been updated, we're able to fetch most new entries almost the instant they are published. The result is that with PubSub, you get:
- More rapid distribution of data: New Messages are delivered more quickly since you don't have to wait until the next polling interval expires to ask for new data.
- More efficient use of network bandwidth: Fewer bytes are used since the network doesn't have to deal with millions of "if-modified-since" requests and their responses. Bytes are only sent over the network when new data is available.
PubSub's push technology is already integrated into the Gush news aggregator and we offer extensions for IE and Firefox that use "Atom over XMPP" to allow near "zero-latency" delivery of updated items to subscribers.
The eventual solution to the "blog clog" issue will come when we transition from "pull" technology to "push" technology; however, there is much that can be done to improve on the efficiency of the current polling based systems while the blogosphere is still in its early stages and some inefficiency can be tolerated.
Some improvements in efficiency can be had by simply educating webmasters in how to properly manage nodes that publish RSS and Atom files. For instance, it has frequently been pointed out that many sites still don't support if-modified-since, Etags, gzip compression and other well-established methods for reducing the bandwidth requirements of serving RSS/Atom files. Further gains will be had once more people implement the RFC3229 + "feed" method that I specified recently and which has already gained quite a number of implementations. Broadened use of these mechanisms will lengthen the useful life of polling on the web -- but even more efficient polling won't ever get us to the levels of efficiency that we'll need if the blogosphere is to continue to grow in size and usage at its current rate.
The blog-clog issue is a real one and one that must be addressed seriously if we're to reach our goal of making the "writeable web" -- enabled by blogging technology -- available to the widest number of people. But solving the blog-clog issue is going to take us far from the "prototyping" technology of polling that we rely on today. Over the next few years, we're all going to be dealing with issues of push technology, peer-to-peer networks, overlay networks, meshes, grids, distributed matching -- all sorts of things that are much more complex then the simplistic systems used by today's blogosphere. We're fortunate, however, in that while we're exploring and implementing these more advanced, more efficient, and more timely techniques, we have available to us some very clear use cases and applications that have been and will be worked out on the simpler, but less scalable technology of polling. A key challenge will be to deliver systems that offer higher efficiency and higher quality of service without compromising on what users perceive as the simplicity of the existing polling based systems...
Watching the blogsphere evolve over the next few years is going to be great fun...
bob wyman
Comments