Wide Awake Developers

Main | April 2002 »

Designing for Emergent Behavior

Lately, I've been grooving on emergent behavior. This fuzzy term comes from the equally fuzzy field of complexity studies. Mix complex rules together with non-linear effects (like humans) and you are likely to observe emergent behavior.

Recent example: web browser security holes. Any program inherently constitutes a complex system. Add in some dynamic reprogramming, downloadable code, system-level scripting, and millions upon millions of users and you've got a perfect petri dish. Sit back and watch the show. Unpredictable behavior will surely result.

In fact, "emergent" sometimes gets used as a synonym for "unpredictable". By and large, I believe that's true. In traditional systems design, "unpredictable" definitely equals "sloppy". Command-and-control, baby. Emergent behavior is what happens when your program goes off the rails.

The thing is, emergent behavior is where all the really interesting things happen. Predictable programs are boring. Big batch runs are predictable.

But, you have to consider the complete system. In a big batch run, the system is linear: inputs, transformation, outputs. No feedback. No humans. When you include humans in your view of the system, all these messy feedback loops start to appear. It gets even worse when you have multiple humans connected via the programs. Feedback loops that stretch from one person, through at least two programs, out to another person and back.

Any system that involves humans will exhibit emergent behaviors -- and this is a very good thing.

Are "designed" behavior and "emergent" behavior inherently incompatible? I don't think so. I think it may be possible to design for emergent behavior. I mean that certain designs will encourage some kinds of emergent behavior, whereas other designs encourage other kinds of emergent behavior. We can study the behaviors produced by various systems and designs to build a compendium of factors that are likely to facilitate one class of behavior or another.

For example: In every corporation, I see large volumes of data stored and shared in two different formats. The nature of the two systems encourages very different behaviors.

First we have relational databases. These tend to be large, expensive systems. As a result, they are centralized to one degree or another. The nature of relational algebra is that of a static schema. Therefore, changes are rigidly controlled. Centralized, rigidly controlled assets require guardians (DBAs) and gatekeepers (data modelers). Because the schema is well-defined and changes slowly, the database gains a degree of transparency. Applications are integrated through their databases. Generic tools for backup, reporting, extraction, and modeling become possible. The data can be accessed from a variety of applications in a relatively generic fashion.

The other data storage tool I see used widely is the spreadsheet. I almost never see a spreadsheet used to calculate numbers. Instead, most are used as a schema-less data storage tool. Often created directly by the business analysts, these spreadsheets are very conducive to change. Sharing is as simple as sending the file through email. Of course, this leads to version conflicts and concurrent update issues that have to be settled by hand (usually by printing a timestamp on the hardcopies!) There is not a central definition of the data structure. Indeed, neither the data nor the structures from spreadsheets can be reused. A spreadsheet makes the 2-dimensional structure of a table obvious, but it makes relationships difficult, if not impossible, to represent. Ergo, spreadsheet users don't do relationships. Access to the spreadsheets is always mediated by a single application.

So, two different systems. Both store structured (or at least semi-structured) data. The nature of each produces very different emergent behaviors. In one case, we find the evolution of acolytes of the RDBMS. In the other case, we find that a numeric analysis tool is being used for widespread data storage and sharing.

Given enough examples, enough time, and enough study, can we not learn to extrapolate from the essential nature of our designs to the most probable emergent behaviors? Even perhaps, to select the emergent behaviors that we desire first, and, starting from those, decide what essential nature our designs must embody to most likely to encourage those behaviors?

Names have Power

Names have power. Shamanic primitives guard their true names -- give me your name and you give me power over you. In the ether, your name is your only identity. Give away your name and you give away yourself. No cause, issue, or crusade has a follower until it has a name. A good name evokes images, emotions. A well-named issue becomes uncontestable. (Who is really opposed to "family values", anyway?)

Naming things well may be one of the hardest jobs in design. Somebody once said that object-oriented design was about creating the language that you would use to solve the problem. Start with the language (a collection of names, and rules about how to assemble the names), then deal with the problem.

I'm struggling with naming something right now. I can sense what it is. There is a real thing there. I can feel it. I need to define it, give it boundaries. When I can name it, I will give it life.

Find the line, find the shape
Through the grain
Find the outline, things will
Tell you their name
--Susanne Vega

The best name I've come up with yet is fluid. There are fluid methods, fluid tools, fluid technologies, fluid designs, and so on. Things that are fluid welcome change. They adapt. They are pleasant to modify. If I have a fluid architecture, then integrating a new system into the mix does not cause massive headaches and heartburn. (Hmmm. So dropping a new system into a fluid architecture doesn't cause a ripple effect? Right. See how hard it is to name things?) Fluid "stuff" does not resist change. Being fluid means nothing is ever carved in stone. Things that are fluid encourage certian emergent properties that we value: fast, flexible, joyous.

Pah. That's damn close to gibberish.

Let's try analogy and contrast:

FluidNot fluid
Publish-subscribe messagingFlat file integration
Typeless languagesStrongly-typed languages
Tuple-spacesRelational databases
eXtreme ProgrammingSEI CMM Level 5
Cross-functional teamsSilos
Whiteboard task listsGANTT charts
WebClient-server
20-person startupThe same company at 150 people

Does that help? The items on the left share some essential, underlying attributes. The things on the right lack those attributes; they embody different values. (I don't like the semiotics of "fluid". Call that a working title, not a true name. Besides, the natural opposites of "fluid" would be "solid" or "concrete". These are both positively-connoted terms.)

So what can I name this quality? Is there really something essential there, or is this just reflecting nothing more than the way I like to work?

Lately, I have been struggling

Lately, I have been struggling to find the meaning in my work. I suppose that's not surprising. I am a human being--a mortal creature. My age will soon flip a decimal digit. (I decline to specify which.) These can certainly cause one to spend time reflecting on one's legacy. They can also cause one to buy a flaming red sports car. I may explore that option later.

 

I also work in a field of incredible transience. Two hundred years from now, no cathedral will bear my mark. No train depot of my design will grace the National Register of Historic Places. No literary critics will deconstruct the significance of my characters' middle initials. In truth, the shelf life of my work compares poorly to that of a gallon of milk.

I am a programmer.

I and my comrades can usually be found behind our glowing screens, working hour after hour to bring some other person's vision to life. We who grapple with chaos and ether and mud expend our spirit, energy, life, time, soul, and qi in the name of creation. We work long after the managers have left. We learn the janitors' names. I have often gazed out my window to the neon street below, full of the theater signs, restaurants, and wandering crowds seeking to be entertained. I have wondered what kind of life I should have led to be in that crowd instead of watching it. I've wondered how I could rejoin that human mass. I think I'd have to change careers.

I cannot deny, however, that my work brings me deep--if ephemeral--satisfaction. The harsh joy of self-sacrifice combines with the exultant delight of success when a project comes together. When I finally get my programs to work, it's a kind of magic, dense and layered. At one level, the thought that my work will be useful to someone--that it will make dozens, hundreds, maybe millions of people more individually powerful--it heady and exciting.

At another level, I have a fierce pride that my software works at all. Knowing that my creation is strong enough, powerful enough to survive the threat of millions of users doing their damndest to destroy it. Despite the teeming millions trying to prove that there is no such thing as "foolproof", my software keeps working. "Robust", we call it. "Resilient". "Come on", it says, "bring it on."

Deeper still, I take a craftman's pride in a job well done. Like a mason or a carpenter, I know what is under the surface. I know how well it is put together. I know what skill went into its construction. No one else may see this, but I know.