Wide Awake Developers

"Us" and "Them"

| Comments

As a consultant, I’ve joined a lot of projects, usually not right when the team is forming. Over the years, I’ve developed a few heuristics that tell me a lot about the psychological health of the team. Who lunches together? When someone says "whole team meeting," who is invited? Listen for the "us and them" language. How inclusive is the "us" and who is relegated to "them?" These simple observations speak volumes about the perception of the development team. You can see who they consider their stakeholders, their allies, and their opponents.

Ten years ago, for example, the users were always "them." Testing and QA was always "them." Today, particularly on agile teams, testers and users often get "us" status (As an aside, this may be why startups show such great productivity in the early days. The company isn’t big enough to allow "us" and "them" thinking to set in. Of course, the converse is true as well: us and them thinking in a startup might be a failure indicator to watch out for!). Watch out if an "us" suddenly becomes "them." Trouble is brewing!

Any conversation can create a "happy accident;" some understanding that obviates a requirement, avoids a potential bug, reduces cost, or improves the outcome in some other way. Conversations prevented thanks to an armed-camp mentality are opportunities lost.

One of the most persistent and perplexing "us" and "them" divisions I see is between development and operations. Maybe it’s due to the high org-chart distance (OCD) between development groups and operations groups. Maybe it’s because development doesn’t tend to plan as far ahead as operations does. Maybe it’s just due to a long-term dynamic of requests and refusals that sets each new conversation up for conflict. Whatever the cause, two groups that should absolutely be working as partners often end up in conflict, or worse, barely speaking at all.

This has serious consequences. People in the "us" tent get their requests built very quickly and accurately. People in the "them" tent get told to write specifications. Specifications have their place. Specifications are great for the fourth or fifth iteration of a well-defined process. During development, though, ideas need to be explored, not specified. If a developer has a vague idea about using the storage area network to rapidly move large volumes of data from the content management system into production, but he doesn’t know how to write the request, the idea will wither on the vine.

The development-operations divide virtually ensures that applications will not be transitioned to operations as effectively as possible. Some vital bits of knowledge just don’t fit into a document template. For example, developers have knowledge about the internals of the application that can help diagnose and recover from system failures. (Developer: "Oh, when you see all the request handling threads blocked inside the K2 client library, just bounce the search servers. The app will come right back." Operations: "Roger that. What’s a thread?") These gaps in knowledge degrade uptime, either by extending outages or preventing operations from intervening. If the company culture is at all political, one or two incidents of downtime will be enough to start the finger-pointing between development and operations. Once that corrosive dynamic gets started, nothing short of changing the personnel or the leadership will stop it.

Inviting Domestic Disaster

| Comments

We had a minor domestic disaster this morning. It’s not unusual. With four children, there’s always some kind of crisis. Today, I followed a trail of water along the floor to my youngest daughter. She was shaking her "sippy cup" upside down, depositing a full cup of water on the carpet… and on my new digital grand piano. 

Since the entire purpose of the "sippy cup" is to contain the water, not to spread it around this house, this was perplexing.

On investigation, I found that this failure in function actually mimicked common dynamics of major disasters. In Inviting Disaster, James R. Chiles describes numerous mechanical and industrial disasters, each with a terrible cost in lives. In Release It, I discuss software failures that cost millions of dollars—though, thankfully, no lives. None of these failures come as a bolt from the blue. Rather, each one has precursor incidents: small issues whose significance are only obvious in retrospect. Most of these chains of events also involve humans and human interaction with the technological environment.

The proximate cause of this morning’s problem was inside the sippy cup itself. The removable valve was inserted into the lid backwards, completely negating its purpose. A few weeks earlier, I had pulled a sippy cup from the cupboard with a similarly backward valve. I knew it had been assembled by my oldest, who has the job of emptying the dishwasher, so I made a mental note to provide some additional instruction. Of course, mental notes are only worth the paper they’re written on. I never did get around to speaking with her about it.

Today, my wonderful mother-in-law, who is visiting for the holidays, filled the cup and gave it to my youngest child. My mother-in-law, not having dealt with thousands of sippy cup fillings, as I have, did not notice the reversed valve, or did not catch its significance.

My small-scale mess was much easier to clean up than the disasters in "Release It!" or "Inviting Disaster". It shared some similar features, though. The individual with experience and knowledge to avert the problem–me–was not present at the crucial moment. The preconditions were created by someone who did not recognize the potential significance of her actions. The last person who could have stopped the chain of events did not have the experience to catch and stop the problem. Change any one of those factors and the crisis would not have occurred.

Book Completed

| Comments

I’m thrilled to report that my book is now out of my hands and into the hands of copy editors and layout artists.

It’s been a long trip. At the beginning, I had no idea just how much work was needed to write an entire book. I started this project 18 months ago, with a sample chapter, a table of contents, and a proposal. That was a few hundred pages, three titles, and a thousand hours ago.

Now "Release It! Design and Deploy Production-Ready Software" is close to print. Even in these days of the permanent ephemerance of electronic speech, there’s still something incomparably electric about seeing your name in print.

Along with publication of the book, I will be making some changes to this blog. First, it’s time to find a real home. That means a new host, but it should be transparent to everyone but me. Second, I will be adding non-blog content: excerpts from the book, articles, and related content. (I have some thoughts about capacity management that need a home.) Third, if there is interest, I will start a discussion group or mailing list for conversation about survivable software.


Reflexivity and Introspection

| Comments

A fascinating niche of programming languages consists of those languages which are constructed in themselves. For instance, Squeak is a Smalltalk whose interpreter is written in Squeak. Likewise, the best language for writing a LISP interpreter turns out to be LISP itself. (That one is more like nesting than bootstrapping, but it’s closely related.)

I think Ruby has enough introspection to be built the same way. Recently, a friend clued me in to PyPy, a Python interpreter written in Python.

I’m sure there are many others. In fact the venerable GCC is written in its own flavor of C. Compiling GCC from scratch requires a bootstrapping phase, by compiling a small version of GCC, written in a more portable form of C, with some other C compiler. Then, the phase I micro-GCC compiles the whole GCC for the target platform.

Reflexivity arises when the language has sufficient introspective capabilities to describe itself. I cannot help but be reminded of Godel, Escher, Bach and the difficulties that reflexivity cause. Godel’s Theorem doesn’t really kick in until a formal system is complex enough to describe itself. At that point, Godel’s Theorem proves that there will be true statements, expressed in the language of the formal system, that cannot be proven true. These are inevitably statements about themselves—the symbolic logic form of, "This sentence is false."

Long-time LISP programmers create works with such economy of expression that we can only use artistic metaphors to describe them. Minimalist. Elegant. Spare. Rococo.

Forth was my first introduction to self-creating languages. FORTH starts with a tiny kernel (small enough that it fit into a 3KB cartridge for my VIC-20) that gets extended one "word" at a time. Each word adds to the vocabulary, essentially customizing the language to solve a particular problem. It’s really true that in FORTH, you don’t write programs to solve problems. Instead, you invent a language in which solving the problem is trivial, then you spend your time implementing that language.

Another common aspect of these self-describing languages seems to be that they never become widely popular. I’ve heard several theories that attempted to explain this. One says that individual LISP programmers are so productive that they never need large teams. Hence, cross-pollination is limited and it is hard to demonstrate enough commercial demand to seem convincing. Put another way: if your started with equal populations of Java and LISP programmers, demand for Java programmers would quickly outstrip demand for LISP programmers… not because it’s a superior language, but just because you need more Java programmers for any given task. This demand becomes self-reinforcing, as commercial programmers go where the demand is, and companies demand what they see is available.

I also think there’s a particular mindset that admires and relates to the dynamic of the self-creating language. I suspect that programmers possessing that mindset are also the ones who get excited by metaprogramming.

Education as Mental Immune System

| Comments

Education and intelligence act like a memetic immune system. For instance, anyone with knowledge of chemistry understands that "binary liquid explosives" are a movie plot, not a security threat. On the other hand, lacking education, TSA officials told a woman in front of me to throw away her Dairy Queen ice cream cones before she could board the plane. Ice cream.

How in the hell is anyone supposed to blow up a plane with ice cream? It defies imagination.

She was firmly and seriously told, "Once it melts, it will be a liquid and all liquids and gels are banned from the aircraft."

I wanted to ask him what the TSA’s official position was on collodal solids. They aren’t gels or liquids, but amorphous liquids trapped in a suspension of solid crystals. Like a creamy mixture of dairy fats, egg yolks, and flavoring trapped in a suspension of water ice crystals.

I didn’t of course. I’ve heard the chilling warnings, "Jokes or inappropriate remarks to security officials will result in your detention and arrest." (Real announcement. I heard it in Houston.) In other words, mouth off about the idiocy of the system and you’ll be grooving to Brittney Spears in Gitmo.

On the other hand, there are other ideas that only make sense if you’re overly educated. Dennis Prager is fond of saying that you have to go to graduate school to believe things like, "The Republican party is more dangerous than Hizbollah."

Of course, I don’t think he’s really talking about post-docs in Chemical Engineering.

Expressiveness, Revisited

| Comments

I previously mused about the expressiveness of Ruby compared to Java. Dion Stewart pointed me toward F-Script, an interpreted, Smalltalk-like scripting language for Mac OS X and Cocoa. In F-Script, invoking a method on every object in an array is built-in syntax. Assuming that updates is an array containing objects that understand the preProcess and postProcess messages.

updates preProcess
updates postProcess

That’s it. Iterating over the elements of the collection is automatic.

F-Script admits much more sophisticated array processing; multilevel iteration, row-major processing, column-major processing, inner products, outer products, "compression" and "reduction" operations. The most amazing thing is how natural the idioms look, thanks to their clean syntax and the dynamic nature of the language.

It reminds me of a remark about General Relativity, that economy of expression allowed vast truths to be stated in one simple, compact equation. It would, however, require fourteen years of study to understand the notation used to write the equation, and that one could spend a lifetime understanding the implications.

Technorati Tags: java, beyondjava, ruby, fscript

Inviting Disaster

| Comments

I’m reading a fabulous book called "Inviting Disaster", by James R. Chiles. He discusses hundreds of engineering and mechanical disasters. Most of them caused serious loss of life.

There are several common themes:

1. Enormously complex systems that react in sometimes unpredictable ways

2. Inadequate testing, training, or preparedness for failures – particularly for multiple concurrent failures

3. A chain of events leading to the "system fracture". Usually exacerbated by human error

4. Politics or budget pressure causing otherwise responsible people to rush things out. This often involves whitewashing or pooh-poohing legitimate criticism and concern from experts involved.

The parallels to some projects I’ve worked on are kind of eerie. Particularly when he’s talking about things like the DC-10 and the Hubble Space Telescope. In both of those cases, warning signs were visible during the construction and early testing, but because each of the people involved had tunnel vision limited to that person’s silo, the clues got missed.

The scary part is that there is no solution here. Sometimes, you can’t even place the blame very squarely. When half-a-dozen people were involved with unloading and handling of oxygen-generating cylinders on a ValuJet flight, no single individual really did something wrong (or contrary to procedure, anyway). Still, the net effect of their actions cost the lives of every single person on that flight.

It’s grim stuff, but it ought to be required reading. If you ever leave your house again, you’ll be much better prepared for building and operating complex systems.

Technorati Tags: operations, systems


New Interview Question

| Comments

So many frameworks… so much alphabet soup on the resumes.

Anyone that ever reads The Server Side or Monster.com knows exactly which boxes to hit when they’re writing a resume. The recruiters telegraph their needs a mile away. (Usually because they couldn’t care less about the differences or similarities between Struts, JSF, WebWork, etc.) As long as the candidate knows how to spell Spring and Hibernate, they’ll get submitted to the "preferred vendor" system.

Being one of those candidates is tough, but that’s not the part I’m concerned about now. I’m interested in weeding out the know-nothings, the poseurs, and the fast talkers.

When I’m interviewing somebody, my main criterion is this: would I want to work on a two-person project with this candidate? My secondary criterion is "Would I feel comfortable leaving this person along at a client site? Will they deliver value to the client? Will they look like an idiot, and by extension, make me look like an idiot?"

My friend Dion Stewart had a great idea for a weed-out question. No matter what frameworks the candidate shows on the resume, ask them what they disliked the most about the framework. (I have my top three list for each framework I’ve worked in… except NeXT’s Enterprise Objects Framework. But that’s another story.)

If they can’t answer at all, then they haven’t actually worked with the framework. They’re just playing buzzword bingo.

If they answer, but it sounds like bullshit, then odds are they’re bullshitting you.

If they have never thought about it, haven’t formed an opinion, or say "it’s all good", then they lack passion about what they do.

A candidate that is driven, that cares about the quality-without-a-name should be able to go on a rant about something in each framework they’ve actually worked with. In fact, you’ve really hit the jackpot if your candidate <i>can</i> go on a rant, but does it in a professional, reasoned way. I love to see a candidate that can show some fire without seeming like a loon. That’s when I can see how they’ll react when the client makes a decision the candidate considers boneheaded. (I’ve seen some spectacular pyrotechnics from consultants that forgot whose money they’re spending. But that’s another story.)

Technorati Tags: resume, jobs

JAI 1.1.3 in Beta

| Comments

I’ve been using JAI 1.1.2 for the past year. It’s an incredibly powerful tool, though I will confess that the API is more than a bit quirky.

Early this year, Sun made JAI an open-source project available at java.net. That project has been working on the 1.1.3 release for most of the year. It’s now in beta, with a few enhancements and a lot of bug fixes.

The most significant enhancement is that JAI can now be used with Java WebStart. Previously it had to be installed as a JRE extension.

Also, one of the big bugs is fixed. Issue #13 is fixed in the beta. It could cause the JPEG codec to use excessive amounts of memory when decoding large untiled images. (Which we do in our app a lot!)

Technorati Tags: java

Ruby Expressiveness and Repeating Yourself

| Comments

Just this week, I was reminded again of how Java forces you to repeat yourself. I had an object that contains a sequence of "things to be processed". The sequence has to be traversed twice, once before an extended process runs and once afterwards.

The usual Java idiom looks like this:

public void preProcess(ActionContext context) {
  for (Iterator iter = updates.iterator(); iter.hasNext(); ) {
    TwoPhaseUpdate update = (TwoPhaseUpdate) iter.next();

public void postProcess(ActionContext context) {
  for (Iterator iter = updates.iterator(); iter.hasNext(); ) {
    TwoPhaseUpdate update = (TwoPhaseUpdate) iter.next();

Notice that there are only two symbols different between these two methods, out of 20 semantically significant symbols. According to the Pragmatic Programmers, even iterating over the collection counts as a kind of repetition (and therefore a violation of DRY - don’t repeat yourself.)

The Ruby equivalent would be something like:

def preProcess(context)
   updates.each { |u| u.preProcess(context) }

def postProcess(context)
   updates.each { |u| u.postProcess(context) }

Now, there are two differening symbols out of 10 (20% variance instead of 10%). There’s been no loss of expressiveness, in fact, the main intention of the code is clearer in the Ruby version than in the Java version.

Can we make the variance higher? Perhaps.

def preProcess(context)
   each_update(:preProcess, context)

def postProcess(context)
   each_update(:postProcess, context)

def each_update(method, context)
   updates.each { |u| u.send(method, context) }

Now the two primary methods have 2 symbols out of 7 different or nearly 28%. The expressiveness is damaged a little bit by the dynamic dispatch via "send". It would be unthinkable to use reflection in Java to make the code clearer. (Anyone who’s worked with reflection knows what I mean.) Here, it’s not unthinkable, but it might just not help clarity.

Technorati Tags: java, beyondjava, ruby