As I May Think...

Pinging Power Laws: 1% generate 33%

Questions have recently been raised about the state of the blogosphere's pinging infrastructure. I find the recent discussions to be a bit light on data...  So, I did a quick analysis of the pings that we receive daily at PubSub.com to provide some context and data for further discussions.

As would be expected, when one plots the number of pingers against a ranked ordering of the number of pings received per pinger per day, what results is a power-law curve. (click on the top chart at the right for more detail.) This particular chart is based on an analysis of 1,171,122 pings received from 468,795 pingers. In this sample, 313,340 pingers only pinged once while the most frequent pinger pinged 7,340 times.

The average number of pings per pinger was about 2.5. However, as the chart at the right demonstrates clearly, that average is heavily influenced by a large number of sites that pinged very frequently. The majority of sites (67%) only pinged once during this sample. These "single pingers" generated about 27% of all pings received. 99% of all pingers pinged 18 or fewer times while generating only 67% of all pings. The remaining 33% of pings were generated by only 1% of all pingers -- many of them "spingers"...

A comparision of the number of actual feed "updates" to the number of pings received from the top 1% of pingers indicates that very, very few of those in the top 1% actually generate anywhere near the number of new or updated entries that their pinging would indicate. Some exceptions include pinging newspaper sites that do generate an unusual number of updates during the day. On inspection, the vast majority of "fast pingers" turn out to be sploggers or spammers, "legitimate" sites that appear to ping from timer-driven scripts and sites that appear to incorrectly generate a ping whenever a comment is entered on an entry in their blogs.

I believe that we'll see the proportion of "fast pingers" and "fake pingers" reduced in the future as spammers begin to realize that fast-pinging is not in their interests. Spammers and Sploggers are beginning to learn that pinging too frequently makes them stick out in the logs of search services. We can expect that in the future they will seek to make themselves less visible and thus more difficult to filter out. Also, we'll probably see fewer timer-driving pingers in the future as the exisiting offenders are contacted one-by-one by various search services and as people tend to use less home-grown technology to do blogging and feed generation (i.e. more folk are moving to the common platforms like WordPress, TypePad, etc. that ping properly.)

In summary: The existing "ping infrastructure" is not broken. The frequency of pings per pinger tends to follow the power-law that we would expect in such an environment (R-squared=.92) Yes, there are some fast pingers out there, however, since we've got a power-law curve they are expected. Additionally, it turns out that fast-pinging helps us to identify some spammers and sploggers. Thus, the high end of the scale is likely to be pulled back somewhat over time while not changing the actual curve or slope of the curve too much.

bob wyman

Dave Winer: Show me that mathematical proof!

Dave, even endless repetition of a false statement won't make it true...

Pito Salas of BlogBridge recently wrote that they have implemented "Reading List Pings." This brings to OPML the same push-based bandwidth saving technology pioneered and proven in the realm of RSS/Atom syndication. As Pito says: "As more and more people both publish and subscribe to OPML Reading Lists, the polling that we are all doing isn't going to scale."

Almost on queue, Dave Winer slams the idea of Reading List Pings while inferring that Pito and others are "doomsayers." Winer claims that he can produce a "mathematical proof" that polling does, in fact, scale. In the past, Dave has made similar statements repeatedly while arguing against push-based notification, but he is no more correct today than he's ever been. No valid proof can be concocted that shows that polling is more efficient than push-based "notifications." It simply can't be done. No amount of name-calling or accusing folk of being "doomsayers" can change the simple fact that polling has serious and unsolvable scaling problems.

Web 2.0: TANSTAAFL!

There has been all sorts of commentary recently about the new world of Web 2.0 applications and mashups built on "free" API's. Somewhere in all the excitement, people seem to have forgotten Robert Heinlein's famous warning:TANSTAAFL ("There Ain't No Such Thing As A Free Lunch..."). While mashups based on free API use are often innovative and sometimes delightful, the providers of the APIs being exploited in the mashups will eventually need to find some way to monetize the services they are providing or to at least limit the cost of providing these valuable services.

Google demonstrates what we can expect to soon see from other "Web 2.0" free API service providers. On April 11, Google announced on their AdWords blog that they will soon start charging a nominal fee for user of their AdWords API. They explain that they wish to "level playing field that encourages efficient coding and application design". Certainly, one could get crass and suggest that they are doing this "just for the money," however, the fees look to be so low that it is hard to believe that they will generate what Google would consider to be a significant revenue stream. It is easier to believe that Google has discovered something very similar to the "Tragedy of the Commons" in free API usage patterns. If a service is provided free, users of that service are simply not motivated to conserve or be efficient in their use of the service. Low efficiencies or excessive use of a free service impacts only the provider of the free service -- not the consumers of that service. Thus, Google is charging in order to move to its consumers some of the burden of use and thus some motivation to conserve. By aligning the interests of consumers more with those of the producer, it is hoped that all will benefit. The theory is good. Time will tell if it works.

PeopleAggregator: Numerical vs. Qualitative Identity

Marc Canter posted on the progress of his PeopleAggregator. He mentions that PeopleAggregator will support a wide range of "identity" systems including: Sxip, OpenID, XDI I-Names, YADIS, Microsoft Infocard, Yahoo, AOL, and Google. Basically, if a system exists and has anything to do with identity, Marc's folk appear to intend to support it.

This is a rather drastic but very rational departure from the customary practice of only supporting a single identity system per site. In essence, what Marc is telling us is that PeopleAggregator won't care *how* you identify yourself -- as long as you can present a numerically unique identity that will serve as your proxy in future interactions. In fact, we're even being told that PeopleAggregator doesn't even care *who* you are. Since they will accept numerical identities managed by different systems, it is entirely possible that you could have multiple PeopleAggregator "accounts" -- each with an identity managed by a different service. It is ironic that even though the subject of PeopleAggregator is people, it isn't "person-ness" that will distinguish one user from another. Rather, what is identified is the ability to demonstrate ability to control or exercise use of an identity token on some supported identity system. This is as it should be.

Microsoft to support RFC3229+feed

While there have been hints of it before, Microsoft's commitment to support RFC3229+feed in the Windows RSS Platform Engine seems to have been restated more strongly today in a post on the Microsoft Team RSS Blog. The post is quite an interesting read as it gives some good hints as to what the folk at Microsoft are implementing. Anyone interested in the technical aspects of the blogging and syndication space should read this post in full. On the subject of RFC3229+feed they say:

Delta Encoding In addition to standard conditional GETs the RSS Platform download engine supports Delta Encoding (for details, see Bob Wyman's post "Using RFC 3229 with Feeds") which allows the server to respond with only the feed items that are new or have been updated, thereby possibly reducing the response significantly.

I think there will be a great deal of support in the community for Microsoft's helping us all reduce the bandwidth cost of growing the syndication web. RFC3229+feed will save many, many terabytes of otherwise wasted bandwidth. Unsurprisingly, some folk have already begun to blog in support of Microsoft's latest announcement.

This reminds me that it really has come time to prepare a formal IETF Internet Draft describing RFC3229+feed. It is time that we had something a bit more formal than a post on my blog as the documentation for this method. If anyone has any issues that should be addressed in such an ID, please let me know.

bob wyman

Edgio = Open, Craigslist = Walled Garden

Mike Masnick of Techdirt asked in a comment on Rob Huf's blog at BusinessWeek: "What's the benefit [of Edgio] over Craigslist or eBay?" Keith Teare of Edgio has already responded, but I think an essential and profoundly important point is being missed... The key difference between these three services can be found in how they get their data. Edgio is an "open" service that aggregates data which is openly published anywhere on the web while CraigsList and eBay are closed, wall-garden sites that only provide service to data that has been published within the confines and control of their "walled-gardens."