Wide Awake Developers

Uniting Reason and Passion

| Comments

Reason and Passion need not conflict. Reason without passion is dusty, dry, and dead. Reason without passion leads to moral relativity. If nothing moves the thinker to passion, then all subjects are equal and without distinction. As well to discuss the economic benefits of the euthanasia of infants as the artistic merits of urinals.

Passion without reason brings the indiscriminate energy of a summer’s thunderstorm. Too much energy unbound, without direction, it’s fury as constant as the winds of the air.

Passion provides energy, the drive to accomplish, change, improve, or destroy. Reason provides direction. Reason channels Passion and achieves goals by identifying targets, foci, leverage points. Passion powers Reason. It brings motive power. Passion knows that things must be done and that change is possible. Reason knows how change may be effected.

I was reminded of the fallacy of Passion without Reason recently. At lunch with a friend, she talked about working with a non-profit organization. Workers for non-profits epitomize those who are driven by Passion. Agree or disagree with their aims, you must admit that they earnestly mean to change the world. My friend, who comes from the profit-driven corporate world, was explaining some aspects of statistical process control and how it could be applied to improve fundraising results on their website. She was told that she needed to have more heart and feel for those unfortunates that this group helps.

Her critic obviously felt that her approach was too analytical. Too driven by Reason, not enough Passion. In fact, the opposite was true. She was applying the combination of Reason and Passion. Passion showed her that the cause was worthy and that she could help. Reason showed her where leverage could be gained and a small effort input could result in a large change in output.

In various disfunctional organizations which I have inhabited, I’ve seen many examples of the opposite. Reason reveals problems and solutions to those poor sapient cogs in the low levels of the machine. They lack the Passion to see that change is possible and so divest themselves of the power to improve their own lot in life. Problems or challenges will always overcome such people, because they give the problem power and remove it from themselves.

More Wiki

| Comments

My personal favorite is TWiki. It has some nice features like file attachments, a great search interface, high configurability, and a rich set of available plugins (including an XP tracker plugin.)

One cool thing about TWiki: configuration settings are accomplished through text on particular topics. For example, each "web" (set of interrelated topics) has a topic called "WebPreferences". The text on the WebPreferences topics actually controls the variables. Likewise, if you want to set personal preferences, you set them as variables–in text–on your personal topic. It’s a lot harder to describe than it is to use.

There are some other nice features like role-based access control (each topic can have a variable that says which users or groups can modify the topic), multiple "webs", and so on.

The search interface is available as variable interpolation on a topic, so something like the "recent changes" topic just ends up being a date-ordered search of changes, limited to ten topics. This means that you can build dynamic views based on content, metadata, attachments, or form values. I once put a search variable on my home topic that would show me any task I was assigned to work on or review.

I’ve also been looking at Oahu Wiki. It’s an open source Java wiki. It’s fairly short on features at this point, but it has by far the cleanest design I’ve seen yet. I look forward to seeing more from this project.

Wiki Proliferation

| Comments

Wikis have been thoroughly mainstreamed now. You know how I can tell? Spammers are targeting them.

Any wiki without access control is going to get steamrolled by a bunch of Russian computers that are editing wiki pages. They replace all the legitimate content with links to porn sites, warez, viagra, get rich now, and the usual panoply of digital plaque.

The purpose does not appear to be driving traffic directly to those sites from the wikis. Instead, they are trying to pollute Google’s page rankings by creating thousands upon thousands of additional inbound links.

If you run a wiki, be sure to enable access control and versioning (so you can recover after an attack). It is a shame that the open, freewheeling environment of the wiki has to end. It seems that the only way to preserve the value of the community is to weaken the core value of open participation that made the community worthwhile.

Moving On

| Comments

The latest in my not-exactly-daily news and commentary…

As of December 10th, I will be leaving Totality Corporation. It has been a challenge and an education. It has also been an interesting time, as we uncovered the hidden linkages from daily activities to ultimate profitability. The managed service provider space is still new enough that the business models are not all so well-defined and understood as in consulting. I earnestly hope that I am leaving Totality in a much better place than it was when I joined.

Still, a number of positive attractions to the new position and some negative forces away from my current position have overcome inertia.

I will be joining Advanced Technologies Integration as a consultant. I will be forming a team with Kyle Larson, Dale Schumacher, and Dion Stewart to do a development project for one of ATI’s clients. The project itself has some moderately interesting requirements… it’s not just another random commerce site. (I’m really, really bored with shopping carts!) The thing that really attracted me though, is that this is a hardcore agile methods project. We’ll be using a combination of Scrum and XP.

For a long time, I’ve advocated small teams of highly skilled developers. I have seen such teams produce many times the business value (and ROI) of the typical team. ATI and this client are willing to subscribe to the theory that a small, high-caliber team will outperform an army of cheap morons.

It’s going to be a blast proving them right!

Too Much Abstraction

| Comments

The more I deal with infrastructure architecture, the more I think that somewhere along the way, we have overspecialized. There are too many architects that have never lived with a system in production, or spent time on an operations team. Likewise, there are a lot of operations people that insulate themselves from the specification and development of systems for which they will ultimately take responsibility.

The net result is suboptimization in the hardware/software fit. As a result, overall availability of the application suffers.

Here’s a recent example.

First, we’re trying to address the general issue of flowing data from production back into pre-production systems – QA, production support, development, staging. The first attempt took 6 days to complete. Since the requirements of the QA environment stipulate that the data should be no more than one week out of date relative to production, that’s a big problem. On further investigation, it appears that the DBA who was executing this process spent most of the time doing scps from one host to another. It’s a lot of data, so in one respect 10 hour copies are reasonable.

But the DBA had never been told about the storage architecture. That’s the domain of a separate "enterprise service" group. They are fairly protective of their domain and do not often allow their architecture documents to be distributed. They want to reserve the right to change them at will. Now, they will be quite helpful if you approach them with a storage problem, but the trick is knowing when you have a storage problem on your hands.

You see, all of the servers that the DBA was copying files from and to are all on the same SAN. An scp from one host on the SAN to another host on the SAN is pretty redundant.

There’s an alternative solution that involves a few simple steps: Take a database snapshot onto a set of disks with mirrors, split the mirrors, and join them onto another set of mirrors, then do an RMAN "recovery" from that snapshot into the target database. Total execution time is about 4 hours.

From six days to four hours, just by restating the problem to the right people.

This is not intended to criticize any of the individuals involved. Far from it, they are all top-notch professionals. But the solution required merging the domains of knowledge from these two groups – and the organizational structure explicitly discouraged that merging.

Another recent example.

One of my favorite conferences is the Colorado Software Summit. It’s a very small, intensely technical crowd. I sometimes think half the participants are also speakers. There’s a year-round mailing list for people who are interested in, or have been to, the Summit. These are very skilled and talented people. This is easily the top 1% of the software development field.

Even there, I occasionally see questions about how to handle things like transparent database connection failover. I’ll admit that’s not exactly a journeyman topic. Bring it up at a party and you’ll have plenty of open space to move around in. What surprised me is that there are some fairly standard infrastructure patterns for enabling database connection failover that weren’t known to people with decades of experience in the field. (E.g., cluster software reassigns ownership of a virtual IP address to one node or the other, with all applications using the virtual IP address for connections).

This tells me that we’ve overspecialized, or at least, that the groups are not talking nearly enough. I don’t think it’s possible to be an expert in high availability, infrastructure architecture, enterprise data management, storage solutions, OOA/D, web design, and network architecture. Somehow, we need to find an effective way to create joint solutions, so we don’t have software being developed that’s completely ignorant of its deployment architecture, nor should we have infrastructure investments that are not capable of being used by the software. We need closer ties between operations, architecture, and development.

The Lights Are on, Is Anybody Home?

| Comments

We pay a lot of attention to stakeholders when we create systems. The end users get a say, as do the Gold Owners. Analysts put their imprimatur on the requirements. In better cases, operations and administration adds their own spin. It seems like the only group that doesn’t have any input during requirements gathering is the development team itself. That is truly unfortunate.

Not even the users will have to live with the system more than the developers will. Developers literally inhabit the system for most of their waking hours, just as much (or maybe more) than they inhabit their cubes or offices. When the code is messy, nobody suffers more than the developers. When living in the system becomes unpleasant, morale will suffer. Any time you hear a developer ask for a few weeks of "cleanup" after a release, what they are really saying is, "This room is a terrible mess. We need to remodel."

A code review is just like an episode of "Trading Spaces". Developers get to trade problems for a while, to see if somebody else can see possibilities in their dwelling. Rip out that clunky old design that doesn’t work any more! Hang some fabric on the walls and change the lighting.

Whether your virtual working environment becomes a cozy place, a model of efficiency, or a cold, drab prison, you create your own living space. It is worth taking some care to create a place you enjoy inhabiting. You will spend a lot of time there before the job is done.

Don’t Build Systems That Boink

| Comments

Note: This piece originally appeared in the "Marbles Monthly" newsletter in April 2003

I caught an incredibly entertaining special on The Learning Channel last week. A bunch of academics decided that they were going to build an authentic Roman-style catapult, based on some ancient descriptions. They had great plans, engineering expertise, and some really dedicated and creative builders. The plan was to hurl a 57 pound stone 400 yards, with a machine that weighed 30 tons. It was amazing to see the builders faces swing between hope and fear. The excitement mingled with apprehension.

At one point, the head carpenter said that it would be wonderful to see it work, but “I’m fairly certain it’s going to boink.” I immediately knew what he meant. “Boink” sums up all the myriad ways this massive device could go horribly wrong and wreak havoc upon them all. It could fall over on somebody. It could break, releasing all that kinetic energy in the wrong direction, or in every direction. The ball could fly off backwards. The rope might relax so much that it just did nothing. One of the throwing arms could break. They could both break. In other words, it could do anything other than what it was intended to do.

That sounds pretty familiar. I see the same expressions on my teammates’ faces every day. This enormous project we’re slaving on could fall over and crush us all into jelly. It could consume our hours, our minds, and our every waking hour. Worst case, it might cost us our families, our health, our passion. It could embarrass the company, or cost it tons of money. In fact, just about the most benign thing it could do is nothing.

So how do you make a system that don’t boink? It is hard enough just making the system do what it is supposed to. The good news is that some simple “do’s and don’ts” will take us a long way toward non-boinkage.

Automation is Your Friend #1: Runs lots of tests – and run them all the time

Automated unit tests and automated functional tests will guarantee that you don’t backslide. They provide concrete evidence of your functionality, and they force you to keep your code integrated.

Automation is Your Friend #2: Be fanatic about build and deployment processes

A reliable, fully automated build process will prevent headaches and heartbreaks. A bad process–or a manual process–will introduce errors and make it harder to deliver on an iterative cycle.

Start with a fully automated build script on day one. Start planning your first production-class deployment right away, and execute a deployment within the first three weeks. A build machine (it can be a workstation) should create a complete, installable numbered package. That same package should be delivered into each environment. That way, you can be absolutely certain that QA gets exactly the same build that went into integration testing.

Avoid the temptation to check out the source code to each environment. An unbelievable amount of downtime can be traced to a version label being changed between when the QA build and the production build got done.

Everything In Its Place

Keep things separated that either change at different speeds. Log files change very fast, so isolate them. Data changes a little less quickly but is still dynamic. “Content” changes slower yet, but is still faster than code. Configuration settings usually come somewhere between code and content. Each of these things should go in their own location, isolated and protected from each other.

Be transparent

Log everything interesting that happens. Log every exception or warning. Log the start and end of long-running tasks. Always make sure your logs include a timestamp!

Be sure to make the location of your log files configurable. It’s not usually a good idea to keep log files in the same filesystem as your code or data. Filling up a filesystem with logs should not bring your system down.

Keep your configuration out of your code

It is always a good idea to separate metadata from code. This includes settings like host names, port numbers, database URLs and passwords, and external integrations.

A good configuration plan will allow your system to exist in different environments – QA versus production, for example. It should also allow for clustered or replicated installations.

Keep your code and your data separated

The object-oriented approach is a good wasy to build software, but it’s a lousy way to deploy systems. Code changes at a different frequency than data. Keep them separated. For example, in a web system, it should be easy to deploy a new code drop without disrupting the content of the site. Likewise, new content should not affect the code.

Plugging the Marbles Newsletter

| Comments

Not too much going on here lately. Most of my waking hours have been billable for the past few months. That’s good and bad, in so many different ways.

Most of my recent writing has been for the Marbles, Inc. monthly newsletter.

Dec 2006 Edit: Marbles IT has not been a going concern for some time.  My articles for the Marbles Monthly newsletter are now available under the Marbles category of this blog.

Multiplier Effects

| Comments

Here’s one way to think about the ethics of software, in terms of multipliers. Think back to the last major email virus, or when the movie “The Two Towers” was released. No doubt, you heard or read a story about how much lost productivity this bane would cause. There is always some analyst willing to publish some outrageous estimate of damages due to these intrusions into the work life. I remember hearing about the millions of dollars supposedly lost to the economy when Star Wars Episode I was released.

(By the way, I have to take a minute to disassemble this kind of analysis. Stick with me, this won’t take long.

If you take 1.5 seconds to delete the virus, it costs nothing. It’s an absolutely immeasurable impact to your day. It won’t even affect your productivity. You will probably spend more time than that discussing sports scores, going to the bathroom, chatting with a client, or any of the hundreds of other things human beings do during a day. It’s literally lost in the noise. Nevertheless, some analyst who likes big numbers will take that 1.5 seconds and multiply it by the millions of other users and their 1.5 seconds, then multiply that by the “national average salary” or some such number.

So, even though it takes you longer to blow your nose than to delete the virus email, somehow it still ends up “costing the economy” 5x10^6 USD in “lost productivity”. The underlying assumptions here are so flawed that the result cannot be taken seriously. Nevertheless, this kind of analysis will be dragged out every time there’s a news story–or better yet, a trial–about an email worm.)

The real moral of this story isn’t about innumeracy in the press, or spotlight seekers exploiting said innumeracy. It’s about multipliers, and the very real effect they can have.

Suppose you have a decision to make about a particular feature. You can do it the easy way in about a day, or the hard way in about a week. (Hypothetical.) Which way should you do it? Suppose that the easy way makes four new fields required, whereas doing it the hard way makes the program smart enough to handle incomplete data. Which way should you do it?

Required fields seem innocuous, but they are always an imposition on the user. They require the user to gather more information before starting their jobs. This in turn often means they have to keep their data on Post-It notes until they are ready to enter it, resulting in lost data, delays, and general frustration.

Let’s consider an analogy. Suppose I’m putting a sign up on my building. Is it OK to mount the sign six feet up on the wall, so that pedestrians have to duck or go around it? It’s much easier for me to hang the sign if I don’t have to set up a ladder and scaffold. It’s only a minor annoyance to the pedestrians. It’s not like it would block the sidewalk or anything. All they have to do is duck. So, I get to save an hour installing the sign, at the expense of taking two seconds away from every pedestrian passing my store. Over the long run, all of those two second diversions are going to add up to many, many times more than the hour that I saved.

It’s not ethical to worsen the lives of others, even a small bit, just to make things easy for yourself. Successful software is measured in millions of people. Every requirements decision you make is an imposition of your will on your users’ lives, even if it is a tiny one. Always be mindful of the impact your decisions–even small ones–have on those people. You should be willing to bear large burdens to ease the burden on those people, even if your impact on any given individual is miniscule.

Keep Your Secrets

| Comments

Here’s a system I call "KeepYourSecrets.org". Recall a film noir detective telling the criminal mastermind that unless he drops a postcard in the mail in the next three days, all the details will go straight to the newspaper.

You can upload any kind of file – it’s all treated like binary. You can set some parameters like a distribution list and a checkin frequency. The system uses an IRC-like network to split your file in n parts, of which some k parts are needed to re-create the original. Up to n-k parts can be lost or compromised without losing or compromising the whole. (See "Applied Cryptography" for details.) With lots of hosts, you can split a document into multiple overlapping sets of pieces to provide another layer of resiliency against damage.

From then on, if you do not check in with the network on some periodic basis, the document goes out to the distribution list. NYTimes, Washington Post, CIA, whoever is on the distribution list for your file.

The network of server don’t ever have to know who you are. They just need to know that you hold the private key that matches the public key that was used to upload the package.

It’s possible to construct voting algorithms that the servers can use to decide if you have really checked in or not. This lets the network protect against a single compromised or hostile host. (You have to be resilient against hostile implementations.)

Because the hosts all communicate via some pub/sub or relay-chat protocol (Jabber, maybe?), the networks of hosts can be self-forming and self-identifying. If there is no central point of control, then the network as a whole cannot be stopped, subverted or forced to give up secrets by any single agency.

What you end up with is a secure, anonymous drop box that cannot be blocked, traced, or inflitrated. It is self-forming and highly resilient to the loss of constituent pieces.