Wide Awake Developers

Heads Down

| Comments

I’ve been quiet lately for a couple of reasons.

First, I’m thrilled to say that I’m joining the No Fluff, Just Stuff stable of speakers.  It’s an honor and a pleasure to be invited to keep such company.  The flip side is, I’m spending a lot of my free time polishing up my inventory of presentations.  More frankly, I’m rebuilding them all with Keynote.  (Brief aside, I’m coming to love Keynote.  It has some flaws and annoyances, but the result is worth it!)

I’ll debut the first of these new presentations at OTUG on May 15th.  I’ll be speaking about "Design for Operations".  The talk will be about 70% from the last part of Release It, and about 30% original content.  OTUG will be giving away a couple of copies of my book, but you have to be there to win!

Finally, I’m working on an article about performance and capacity management.  Most capacity planning work is done entirely within Operations, without much involvement from Development.  At the same time, most developers don’t have a visceral appreciation for how dramatically the application’s efficiency can affect the system’s overall profitability.

This article will show the relationship between application response time, system capacity, and financial success.  I’m hoping to include a simulator app for download that you can use to play with different scenarios to see what a dramatic difference 100ms can make. 

Coach and Team From Same Firm

| Comments

Is it an antipattern to have a consulting firm provide both the coach and developers?  By providing the developers, the firm is motivated to deliver on the project, with coaching as an adjunct.  If, instead, the firm provides just the coach, it will be judged by how well the client adopts the process.  These two motives can easily conflict.

Case in point: at a previous client of mine, my employer was charged with completing the project, using a 50-50 mix of contractors and client developers.  My employer, a consulting firm, provided several developers experienced with XP and Scrum, as well as an agile coach.  The firm was thus charged with two imperatives: first, deliver the project; second, introduce agile methods within the client. 

With project success as a requirement, the firm decided to intereview the developers at the outset of the project. The client’s developers (rightly) perceived that they were interviewing for their own jobs.  This started a negative dynamic that ultimately resulted in 80% attrition among the client’s developers.

On a pure coaching engagement, the coach would probably have "made do" with whomever the client provided. 

We delivered all the features, basically on time, with very high quality. Financially speaking, it was a success, generating more orders and more revenue per order than its predecessor.  It is harder to say that the engagement as a whole was a success, though.  Almost all of the developers were contractors, so the client got their product, but very little adoption of agile methods.

Perhaps if the coach and the contract developers had come from different firms, the motivations would not have been as tangled, and more of the client’s valuable people would have stayed.  The team might not have suffered from the strained, unhealthy environment from the early days of the project.

Then again, perhaps not.  The client may have been expecting that level of attrition. Maybe that’s just to be expected when you trying to bring a random selection of corporate developers over to agile methods, especially if the methods are decreed from above instead of brought upward by grass-roots. Maybe the dynamic would have existed even with a coach that was totally disinterested in the project outcome.

Moving Your Home Directory on Leopard

| Comments

Since NetInfo Manager is going away under Leopard, we’ve got a gap in capability. How do you relocate your home directory without the GUI?

There are a few reasons you might want to move your home directory to another volume. For example, you might reinstall your OS frequently. Or, perhaps you just want to keep your data on a bigger disk than the one that came in the machine. In my case, both.

The venerable NetInfo is being replaced entirely with Directory Services. (Try "man 8 DirectoryServices" for more information.) There’s a handy command-line tool you can use to interact with the DirectoryServices.

Let’s start by opening up a Terminal window. (Applications > Utilities > Terminal) At first, you’ll be logged in as yourself, not as root.

Last login: Wed Dec 31 18:00:00 on ttyp0
donk:~ mtnygard$ 

The first thing is to get out of your home directory, because we’re going to delete it in about a minute and a half. Change to the root directory and make yourself into the root user with "sudo".

Last login: Wed Dec 31 18:00:00 on ttyp0
donk:~ mtnygard$ sudo su -
donk:~ root#

Next, fire up "dscl", the directory services command line. Without arguments, this gives you an interactive, shell-like environment to explore the directory. It also spews a bunch of help messages. If you give it "localhost", then it quietly assumes you wanted to interact with the directory.

You can list entries, cd around the directory hierarchy, and even create entries or change attributes.

User information is stored under /Local/Users, so we’ll cd to that now.

donk:~ root# dscl localhost
 > cd /Local/Users
/Local/Users >

Now, running "ls" will show you all the users that your machine knows.  Try it now.

donk:~ root# dscl localhost
 > cd /Local/Users
/Local/Users > ls
/Local/Users >

Holy crap!  Who the hell are all these people?

Well, of course, they aren’t people.  All the usernames starting with an underscore are application IDs.  Root, nobody, and daemon are all part of the OS.  Once you eliminate them, there should just be the people you’ve actually created accounts for.  If you see any names you don’t recognize at this point, this would be a good time to shut off your network connection.

At this point, you could "cd" directly into the entry for your user.  It won’t show you anything special; users do not have subnodes in the directory.  It would set up your context for future commands, limiting them to just that user.  In this case, however, we’ll stay at /Local/Users and run "cat" on my username.

/Local/Users > cat mtnygard
dsAttrTypeNative:_writers_hint: mtnygard
dsAttrTypeNative:_writers_jpegphoto: mtnygard
dsAttrTypeNative:_writers_passwd: mtnygard
dsAttrTypeNative:_writers_picture: mtnygard
dsAttrTypeNative:_writers_realname: mtnygard
dsAttrTypeNative:authentication_authority: ;ShadowHash;
dsAttrTypeNative:generateduid: 7F6A8EDE-63EC-4A34-9391-031A9C77806D
dsAttrTypeNative:gid: 501
dsAttrTypeNative:home: /Users/mtnygard
 ffd8ffe0 00104a46 49460001 01000001 00010000 ffdb0043 00020202 ... 7fffd9
dsAttrTypeNative:name: mtnygard
dsAttrTypeNative:passwd: ********
 /Library/User Pictures/Sports/Tennis.tif
 Michael Nygard
dsAttrTypeNative:shell: /bin/bash
dsAttrTypeNative:uid: 501
AppleMetaNodeLocation: /Local/Default
AuthenticationAuthority: ;ShadowHash;
GeneratedUID: 7F6A8EDE-63EC-4A34-9391-031A9C77806D
 ffd8ffe0 00104a46 49460001 01000001 00010000 ffdb0043 00020202 ... 7fffd9
NFSHomeDirectory: /Users/mtnygard
Password: ********
 /Library/User Pictures/Sports/Tennis.tif
PrimaryGroupID: 501
 Michael Nygard
RecordName: mtnygard
RecordType: dsRecTypeStandard:Users
UniqueID: 501
UserShell: /bin/bash
/Local/Users >

Hmm. Seems like it must mean something. This is listing the values of all the attributes of my user profile. It’s what I want, but there’s a big pile of noise in the middle. That noise is a textual representation of my profile’s JPEG. (I’ve edited it out of this transcript.) If you scroll up past that, you’ll see the attribute of real interest.

The property dsAttrTypeNative:home tells the OS where to find my home directory.

I can change it with dscl’s "change" command. The format of change is a little strange because it has to deal with multi-valued properties (as do all of the directory services commands.)

/Local/Users > change mtnygard dsAttrTypeNative:home /Users/mtnygard /Volumes/Data/mtnygard
/Local/Users >

The first parameter is the object to change, the second parameter is the attribute to change. The third parameter is the old value that you want to replace (multi-valued list for each attribute, remember.) Finally, the fourth parameter is the new value you want to set.


Not quite done yet, though. I’ve given the OS a bogus home directory. There’s no such directory as /Volumes/Data/mtnygard yet.

To get there, I have to move my directory from under /Users to the new location. I have to do this as root, but I don’t want root to end up owning all my personal stuff. Fortunately, there’s a "cp" option for that.

donk:~ # cp -Rp /Users/mtnygard /Volumes/Data/

Now, we’re almost, almost done. Log off and log back on into your roomy new home directory.


  1. I don’t know how to do this if you’ve got a shared directory tree set up.  You might have that if you’re on a Mac network at work, for example.  You should definitely try this at home.
  2. The "cp" command I use will do really funky things if you’ve got hard links, symlinks, or especially circular symlinks in your home directory.  Then again, if you’ve done that to yourself, you probably know enough Unix to work out your own parameters for "cp", "tar", "mv" or "cpio".
  3. One more thing: I’m not sure if this is from running a developer seed of Leopard, or if it’s due to this home directory move technique, but I keep running into permissions problems. I couldn’t automatically install dashboard widgets, for example. Adium complained that it couldn’t create its "sounds" directory.

What Makes a POJO So Great, Anyway?

| Comments

My friend David Hussman once said to me, "The next person that says the word ‘POJO’ to me is going to get stabbed in the eye with a pen."  At the time, I just commiserated about people who follow crowds rather than making their own decisions.

David’s not a violent person.  He’s not prone to fits of violence or even hyperbole.  What made this otherwise level-headed coach and guru resort to non-approved uses of a Bic?

This weekend in No Fluff, Just Stuff, I had occasion to contemplate POJOs again.  There were many presentations about "me too" web frameworks.  These are the latest crop of Java web frameworks that are furiously copying Ruby on Rails features as fast as they can.  These invariably make a big deal out of using POJOs for data-mapped entities or for the beans accessed by whatever flavor of page template they use. (See JSF, Seam, WebFlow, Grails, and Tapestry 5 for examples.)

Mainly, I think the infuriating bit is the use of the word "POJO" as if it’s a synonym for "good".  There’s nothing inherently virtuous about plain old Java objects.  It’s a retronym; a name made up for an old thing to distinguish it from the inferior new replacement.

People only care about POJOs because EJB2 was so unbelievably bad.

Nobody gives a crap about "POROs" (Plain old Ruby objects) because ActiveRecord doesn’t suck.

Flash Mobs and TCP/IP Connections

| Comments

In Release It, I talk about users and the harm they do to our systems.  One of the toughest types of user to deal with is the flash mob.  A flash mob often results from Attacks of Self-Denial, like when you suddenly offer a $3000 laptop for $300 by mistake.

When a flash mob starts to arrive, you will suddenly see a surge of TCP/IP connection requests at your load-distribution layer.  If the mob arrives slowly enough (less than 1,000 connections per second) then the app servers will be hurt the most.  For a really fast mob, like when your site hits the top spot on digg.com, you can get way more than 1,000 connections per second.  This puts the hurt on your web servers.

As the TCP/IP connection requests arrive, the OS queues them for servicing by the application.  As the application gets around to calling "accept" on the server socket, the server’s TCP/IP stack sends back the SYN/ACK packet and the connection is established.  (There’s a third step, but we can skip it for the moment.)  At that point, the server hands the established connection off to a worker thread to process the request.  Meanwhile, the thread that accepted the connection goes back to accept the next one.

Well, when a flash mob arrives, the connection requests arrive faster than the application can accept and dispatch them.   The TCP/IP stack protects itself by limiting the number of pending connection requests, so if the requests arrive faster than the application can accept them, the queue will grow until the stack has to start refusing connection requests.  At that point, your server will be returning intermittent errors and you’re already failing.

The solution is much easier said than done: accept and dispatch connections faster than they arrive.

Filip Hanik compares some popular open-source servlet containers to see how well they stand up to floods of connection requests.  In particular, he demonstrates the value of Tomcat 6’s new NIO connector.  Thanks to some very careful coding, this connector can accept 4,000 connections in 4 seconds on one server.  Ultimately, he gets it to accept 16,000 concurrent connections on a single server.  (Not surprisingly, RAM becomes the limiting factor.)

It’s not clear that these connections can actually be serviced at that point, but that’s a story for another day.

Release It! Is Released!

| Comments

"Release It!" has been officially announced in this press release.  Andy Hunt, my editor, also posted announcements to several mailing lists.

It’s been a long road, so I’m thrilled to see this release.

When you release a new software system, that’s not the end of the process, but just the beginning of the system’s life.  It is the same thing here.  Though it’s taken me two years to get this book done and on the market, this is not the end of the book’s creation, but the beginning of it’s life.


Self-Inflicted Wounds

| Comments

My friend and colleague Paul Lord said, "Good marketing can kill you at any time."

He was describing a failure mode that I discuss in Release It!: Design and Deploy Production-Ready Software as "Attacks of Self-Denial".  These have all the characteristics of a distributed denial-of-service attack (DDoS), except that a company asks for it.  No, I’m not blaming the victim for electronic vandalism… I mean, they actually ask for the attack.

The anti-pattern goes something like this: marketing conceives of a brilliant promotion, which they send to 10,000 customers.  Some of those 10,000 pass the offer along to their friends.  Some of them post it to sites like FatWallet or TechBargains.  On the appointed day, hour, and minute, the site has a date with destiny as a million or more potential customers hit the deep link that marketing sent around in the email.  You know, the one that bypasses the content distribution network, embeds a session ID in the URL, and uses SSL?

Nearly every retailer I know has done this to themselves at one point.  Two holidays ago, one of my clients did it to themselves, when they announced that XBox 360 preorders would begin at a certain day and time.  Between actual customers and the amateur shop-bots that the tech-savvy segment cobbled together, the site got crushed.  (Yes, this was one where marketing sent the deep link that bypassed all the caching and bot-traps.)

Last holiday, Amazon did it to themselves when they discounted the XBox 360 by $300.  (What is it about the XBox 360?)  They offered a thousand units at the discounted price and got ten million shoppers.  All of Amazon was inaccessible for at least 20 minutes.  (It may not sound like much, but some estimates say Amazon generates $1,000,000 per hour during the holiday season, so that 20 minute outage probably cost them around $200,000!)

In Release It!, I discuss some non-technical ways to mitigate this behavior, as well as some design and architecture patterns you can apply to minimize damage when one of these Attacks of Self-Denial occur.

Design Patterns in Real Life

| Comments

I’ve seen walking cliches before.  There was this one time in the Skyway that I actually saw a guy with a white cane being led by a woman with huge dark sunglasses and a guide dog.  Today, though, I realized I was watching a design pattern played out with people instead of objects.

I’ve used the Reactor pattern in my software before.  It’s particularly helpful when you combine it with non-blocking multiplexed I/O, such as Java’s NIO package.

Consider a server application such as a web server or mail transfer agent.  A client connects to a socket on the server to send a request. The server and client talk back and forth a little bit, then the server either processes or denies the client’s request.

If the server just used one thread, then it could only handle a single client at a time.  That’s not likely to make a winning product. Instead, the server uses multiple threads to handle many client connections.

The obvious approach is to have one thread handle each connection.  In other words, the server keeps a pool of threads that are ready and waiting for a request.  Each time through its main loop, the server gets a thread from the pool and, on that thread, calls the socket "accept" method.  If there’s already a client connection request waiting, then "accept" returns right away.  If not, the thread blocks until a client connects.  Either way, once "accept" returns, the server’s thread has an open connection to a client.

At that point, the thread goes on to read from the socket (which blocks again) and, depending on the protocol, may write a response or exchange more protocol handshaking.  Eventually, the demands of protocol satisfied, the client and server say goodbye and each end closes the socket.  The worker thread pulls a Gordon Freeman and disappears into the pool until it gets called up for duty again.

It’s a simple, obvious model.  It’s also really inefficient.  Any given thread spends most of its life doing nothing.  It’s either blocked in the pool, waiting for work, or it’s blocked on a socket "accept", "read", or "write" call.

If you think about it, you’ll also see that the naive server can handle only as many connections as it has threads.  To handle more connections, it must fork more threads.  Forking threads is expensive in two ways.  First, starting the thread itself is slow.  Second, each thread requires a certain amount of scheduling overhead.  Modern JVMs scale well to large numbers of threads, but sooner or later, you’ll still hit the ceiling.

I won’t go into all the details of non-blocking I/O here.  (I can point you to a decent article on the subject, though.)  Its greatest benefit is you do not need to dedicate a thread to each connection.  Instead, a much smaller pool of threads can be allocated, as needed, to handle individual steps of the protocol.  In other words, thread 13 doesn’t necessarily handle the whole conversation. Instead, thread 4 might accept the connection, thread 29 reads the initial request, thread 17 starts writing the response and thread 99 finishes sending the response.

This model employs threads much more efficiently.  It also scales to many more concurrent requests.  Bookkeeping becomes a hassle, though. Keeping track of the state of the protocol when each thread only does a little bit with the conversation becomes a challenge.  Finally, the (hideously broken) multithreading restrictions in Java’s "selector" API make fully multiplexed threads impossible.

The Reactor pattern predates Java’s NIO, but works very well here.  It uses a single thread, called the Acceptor, to await incoming "events". This one thread sleeps until any of the connections needs service: either due to an incoming connection request, a socket ready to read, or a socket ready for write.  As soon as one of these events occurs, the Acceptor hands the event off to a dispatcher (worker) thread that then processes the event.

You can visualize this by sitting in a TGI Friday’s or Chili’s restaurant.  (I’m fond of the crowded little ones inside airports. You know, the ones with a third of the regular menu and a line stretching out the door.  Like a home away from home for me lately.) The "greeter" accepts incoming connections (people) and hands them off to a "worker" (server).  The greeter is then ready for the next incoming request.  (The line out the door is the listen queue, in case you’re keeping score.)  When the kitchen delivers the food, it doesn’t wait for the original worker thread.  Instead, a different worker thread (a runner) brings the food out to the table.

I’ll keep my eyes open for other examples of object-oriented design patterns in real life–though I don’t expect to see many based on polymorphism.

Another Path to a Killer Product

| Comments

Give individuals powers once reserved for masses

Here’s a common trajectory:

1. Something is so expensive that groups (or even an entire government) have to share them.  Think about mainframe computers in the Sixties.

2. The price comes down until a committed individual can own one.  Think homebrew computers in the Seventies.  The "average" person  wouldn’t own one, but the dedicated geek-hobbyist would.

3. The price comes down until the average individual can own one.  Think PCs in the Eighties.

4. The price comes down until the average person owns dozens.  PCs, game consoles, MP3 players, GPS navigators, laptops, embedded processors in toasters and cars.  An average person may have half a dozen devices that once were considered computers.

Along the way, the product first gains broader and broader functionality, then becomes more specific and dedicated.

Telephones, radios and televisions all followed the same trajectory.  You would probably call these moderately successful products.

So: find something so expensive that groups have to purchase and share it.  Make it cheap enough for a private individual.