As I May Think...

What's up (or down) with Zeldman?

[Updated 19-Nov-2004]

Robert Scoble wrote on September 30:

I get to see trends that happen over time. For instance, when I started blogging, tons of people would link to Jeffrey Zeldman. He's a design guru who's written great books on designing with Web standards and such. But lately no one has even mentioned him in the blogs I'm reading. Why?

Well, this got me curious so I looked up the PubSub LinkRank for zeldman.com and found that it was 648 on September 30, down from 629 the day before. Being the 648th most popular link destination isn't bad! But, what was it awhile ago? So, I dug into our databases and created a chart of the Linkrank for zeldman.com over time which I published in this blog. Time has passed, so I've dug into the database again and created a new chart which shows Zeldman.com's LinkRank going back to August: (Click on the image to see a larger version.)

Clearly, Zeldman was losing popularity rapidly back when Scoble first made his comments and he continued to lose popularity for some time afterwards. However, there has been a pickup lately -- which presumably indicates that while Zeldman may have gone through a period of publishing less-then-enthralling content, he has somewhat returned to being able to produce what readers find interesting -- or, at least bloggers want to link to (there is a difference there...). But, of course, one can't really be sure what is going on unless you study the day-to-day publishing and linking history -- not just the rough evidence of results as shown by LinkRanks.

Blog Clog Continued...

In his recent article "If RSS ain't broke...", Steve Gillmor of ZD News continues to reinforce CNET/ZDNet's leadership in reporting on the "blog clog" issue. Gillmor responds negatively to the often made suggestion that news aggregators should poll less frequently in order to reduce server and network load. After pointing out that POP3 email clients typically poll for new email every few seconds he says: “Then there are the real fanatics like me, who don't want no stinkin' hourly intervals, thank you. I want my POP3 intervals, or even better, event-driven intervals powered by peer-to-peer.” Unfortunately, of course, if RSS Aggregators were to poll as frequently as POP3 clients do, the blogosphere would melt down instantly. Certainly, there is a real problem here that must be addressed, however, there must be a better solution than forcing everyone to accept lower quality of service.

Fortunately, PubSub.com offers the update frequency that Gillmor is looking for: i.e. "No Interval -- near zero-latency". Since we're a "push" based system (although we'll let you poll if you want...) as soon as we find new data we immediately send it to our subscribers. There is no polling interval and since millions of blogs currently push pings to us whenever they've been updated, we're able to fetch most new entries almost the instant they are published. The result is that with PubSub, you get:

Implementations of RFC3229 with "feed"

[Updated: 18-June-2006 to include support by Rawdog 2.9 aggregator]

This blog entry attempts to provide a catalog of systems that implement RFC3229 as described in my post on RFC3229+feed. I'm hoping that this will grow to be a very long list.

If we can get RFC3229 well and fully deployed in the blogging community, it is certain that we'll see quite a dramatic drop in the bandwidth requirements that result from polling for RSS and Atom files. While I'm convinced that eventually we'll have to leave the inefficiencies of polling behind and move syndication to a push model, in the meantime, we should be doing polling as well as we can.

Below, you'll find a list of servers and clients that support RFC3229.

Using RFC3229 with Feeds.

[Updated: 22-Sep-2004 14:33 with link to list of implementations]

The other day I wrote that we really should be adopting RFC3229 "Delta Encoding in HTTP" in order to reduce the amount of bandwidth, etc. that is wasted in serving RSS and Atom files. I'm fairly convinced that if the folk at Microsoft had been using what I propose here, they would not have been forced to take the drastic measures that they did when they did.

Of course, use of RFC3229 would have only delayed, not eliminated the day when the current practice of polling for updates to RSS files would have become excessively expensive for Microsoft. The real solution to the bandwidth problem is to move from polling to a push-based solution. But, at least by implementing RFC3229, we can take the polling solution just about as far as it can be taken -- in terms of efficiency... This is a good intermediate step on the way to the push-based solutions that we won't have much choice but to implement as the audience for RSS and Atom data grows.

This post is intended to provide additional detail on what I'm proposing. It is my intention to create a Internet Draft describing the ideas here once I've had reasonable time to receive comments from folk and work out the inevitable bugs. Please feel free to comment on what is below:

If you must poll, at least poll well...

Microsoft's recent troubles with RSS files have focused the RSS/Atom communities squarely on fundamental problems with feed syndication as practiced today. While some of us have been warning about this problem for quite a while, it has taken Microsoft's recent embarrasment to convince folk that the problems are real -- not just theoretical. The fundamental problem is that polling, the mechanism used by today's news aggregators, is simply not scalable enough to take us where we need to go -- to world where reading feeds is just as common as reading HTML pages. Eventually, we're going to have to move to a push-based architecture -- such as what is implemented by PubSub.com. However, in the mean time, there are things that we can do to give the old polling model a bit more life and save providers of popular feeds, like Microsoft, quite a bit of bandwidth expense. In the process, we'll be facilitating the continued growth of feed syndication rather then seeing it die off or crippled just as it is finally gaining popularity.

Arthur C. Clarke and Prospective Search

Arthur C. Clarke is well known for having written about many technologies long before they are invented -- or at least before they become popular... In his 1979 book The Fountains of Paradise, about Space Elevators, Clarke anticipates the kind of prospective search system that we're building at PubSub.com. Of course, given his ability to see far into the future, Clarke describes prospective search as something which is as normal as looking up an address or getting a birthday reminder...