Google Lock Secrets Unlocked: Chubby unveiled...

Seekers after the inner secrets of Google have  undoubtedly been frustrated by the fact that several of the papers describing Google's internals refer to a distributed lock management service (Chubby)  but provide little information on the lock service's design or implementation. For example, the recently released paper on BigTable refers frequently to Chubby but points to a paper by Mike Burrows to be presented at OSDI'06 in November -- still two months away! Well, sometime in the last few days, Google published a pre-print of Burrows' paper. For the inside story on Chubby, and thus more to the story of BigTable, the Google File System (GFS), etc. see the paper: The Chubby Lock Service for Loosely-Coupled Distributed Systems .

The paper's abstract reads in part:

We describe our experiences with the Chubby lock service, which is intended to provide coarse-grained locking as well as reliable (though low-volume) storage for a loosely-coupled distributed system. Chubby provides an interface much like a distributed file system with advisory locks, but the design emphasis is on availability and reliability, as opposed to high performance. Many instances of the service have been used for over a year, with several of them each handling a few tens of thousands of clients concurrently. The paper describes the initial design and expected use, compares it with actual use, and explains how the design had to be modified to accommodate the differences.

The author (ex-DEC) is refreshingly modest about what has been accomplished and clarifies that:

Building Chubby was an engineering effort ...; it was not research. We claim no new algorithms or techniques. The purpose of this paper is to describe what we did and why, rather than to advocate it.

As is always seems to be the case with code that actually works, the implementers were surprised to discover that their system was being used in unanticipated ways --- some reasonable, some not. What they had intended to be a lock manager is apparently also getting heavy use within Google as a DNS-like name service and as a repository for configuration information and small files. Given my interest in publish/subscribe technologies, I was also intrigued to see that they have had to dissuade a number of groups from using Chubby as a general Publish/Subscribe system. (Does this indicate that Google needs a general PubSub service for internal use? Will we one day see a paper published by Google describing a Pub/Sub system?)

The paper includes an interesting discussion of the use of lock management mechanisms as an alternative to an asynchronous consensus algorithm such as Leslie Lamport's (another ex-DECie) Paxos protocol.

This paper on Chubby is an interesting read. I recommend it highly.

bob wyman

Comments

[Phil Aaronson]

I'm sure this quote from the paper brought a smile to your face. This is with respect to Chubby being used as a name server:

The ability to provide swift name updates without polling each name individually is so appealing that Chubby now provides name service for most of the company's systems.