Wide Awake Developers

I Forgot to Mention That I Will Be Speaking at OTUG

| Comments

I forgot to mention that I will be speaking at OTUG on April 19th! I will be speaking on "Living With Systems in Production: Avoiding Heartbreak in Long-Term Relationships With Your Code"

From the summary of the talk:

Everything changes after Release 1.0. One batch of consultants leave, key developers jockey to get themselves reassigned, and the free-wheeling development environment is replaced by the painful rigor of operations. Or, at least, it should be. Systems in production require a different kind of care and feeding. If you have to live with a system in production, your quality of life is largely determined by the things you put in place before Release 1.0. This talk covers the topics that will give you God-like powers over your production systems. If you are an architect or developer who has ever put a system in production–or expects to put a system into production–then this talk is for you.

Much of this will be derived from my experiences at Totality and Best Buy. Spending time in operations gave me a great education about building systems to run, instead of building them to pass QA.

Technorati Tags: operations, OTUG, speaking

Leaving AntHill for CruiseControl

| Comments

We’ve been using AntHill to do continuous builds. It has served us well, but we’re now moving away from it and towards CruiseControl.

There are a few main reasons for this. First and foremost, AntHill runs inside of Tomcat. This is billed as a feature, but for us, it was a big problem. There are two layers of Java containers between your OS and your build. Trying to get environment variables (like "DISPLAY=localhost:99.0") passed from an init script, to Tomcat, to AntHill, to ANT, was just becoming too burdensome.

We also experienced some serious classpath pollution. Some things just acted differently between ANT builds on our development boxes and ANT builds on the build box. That’s unacceptable, but we found that with AntHill it was impossible to eliminate the differences. Finally, through some jar file unpacking and decompilation, we found that our builds were picking up classes from AntHill’s ow n jars.

The ability to fix these things exists, but only in AntHill Pro. I downloaded CruiseControl today and spent an hour going through the quick start and FAQ. At the end of it, I had our build process replicated on CruiseControl.

I did run into a problem… the checkstyle task that we have been running as part of our build all along started failing. I assumed that it was something wrong with the build box, or with my project configuration for CruiseControl. After half an hour or so, I ran the same build on a dev box, but from the command line. It failed there, too. It turns out that checkstyle includes a version of the Jakarta commons-collections classes that is not compatible with the Jakarta digester version that we’ve added to our code base.

This problem existed all along. Running the build under CruiseControl was enough like running it from the command line that it uncovered a problem which had been present for over two weeks. For some reason, running under AntHill never revealed this problem.

Bottom line is, a CI server needs to be as close to running a command-line build as possible. If I have to spend time figuring out what environmental conditions the CI tool is imposing on my build, then it is defeating the purpose.

Now, I just have to figure out why in the hell checkstyle’s classpath is leaking into the classpath of the code it is checking.

Technorati Tags: agile, automation, CruiseControl, pragmatic

The Veteran and the Master

| Comments

The aged veteran said to the master, "See how many programs I have written in my labors. All of these works I have created needed no more than a text editor and a compiler." The master said, "I do have an editor; indeed, I have also a compiler."

Said the aged one, "Yet you shackle them within an ‘environment’. Why must your environment be integrated? My environment has never been integrated, yet I am a mighty programmer."

The master said, "You are truly a mighty programmer. I perceive that you, in your keen intellect, can hold entire class hierarchies in mind at once. Such abilities of apprehension are to be respected."

The veteran was well pleased and said, "It is true. Hence I am lead programmer."

The master nodded. "Sadly, I have not your powers of visualization. I cannot hold entire hierarchies in my minds eye at once. In my limited faculties, I must focus entirely on one class at a time. The tool remembers the rest, as I cannot."

Emboldened, the aged veteran boasted, "See the commands fly from my fingertips! I type faster than other programmers think!"

Again, the master nodded his agreement, "I am not so blessed with speed as you. It is a burden and a trial to move so slowly. Behold, this measure of the marvel of your fingers. Such is the flight of your keystrokes that in the time it takes you to execute a regex replace across thirty files; compile the project; note the errors; and edit the twelve files with failed replacements; I will have barely completed the ‘rename refactor’ which I started by typing shift-alt-r."

Brazen in his opponents weakness, the veteran cried, "While you sit meditating at the green bar, I pound out another four thousand lines of code!"

Again, the master nodded, "Yes. And worse, while you write the next thousand, I will surely erase a thousand more, leaving us barely past where we began. It is clear that I cannot long contend in this field against such as yourself."

The battle-scarred veteran, his opponent beaten, laughed aloud. Barely bothering to express his contempt, he sneered, "And what fine code it is, too! You write a fraction of the code a real programmer could produce. As a coward in the grain, you shrink from any real challenge. Fearing to tread where real programmers dwell, you trade in coin like a merchant, purchasing the work of others, or worse, living on the charity of those motley-clad coders who give away the fruits of their work."

"Again, your perspicacity has unmasked me," said the master. "Knowing myself to produce bugs in my code, I prefer to write little of it. I do rely upon the work of others who, if not being smarter than myself, are at least more numerous than I. Had I your fleet fingers, I might not need to download these gifts offered by others. Indeed, I am certain that your mighty editor would surely outpace my mere web browser, and you could then code a new SVG renderer long before I will finish downloading Batik to do the same work. Alas, lacking your skills, I must fend for myself as best I can by reusing that which I can. Since each line of code costs me so greatly, it behooves me to write little, and I must needs make use of what aids I can."

Shaking his head, the aged veteran stalked away, safely assured that he had gauged the so-called master truly. He returned to his labors, building a parser for the scripting language of his workflow engine. This would be placed inside of an application that would someday have users.

Shaking his head, the master returned his eye to the red bar of his users’ new acceptance tests. Reaching deliberately for the keyboard, he changed two methods and added one test case. In the serene green light of the test bar, he reflected a moment on the code he had added. Unruffled by the staccato typing in the direction of the veteran, he renamed four fields, extracted a method, and pulled it up into a new base class. Comforted by the tranquil green light, the master rested his hands a moment, then lifted them from the keyboard and walked away.

From the corner of his eye, the veteran observed the master leaving. "Charlatan," he snarled, as the regexes flew from his hands, long, long into the night.

Technorati Tags: agile, lean, pragmatic

On Relativism and Social Constructions

| Comments

The key operative precept of post-modernism is that all reality is a social construct. Since no institution or normative behavior stems from natural cause, and there is no objective, external reality, then all institutions and attitudes are just social constructs. They exist only through the agreement of the participants.

Nothing can be sacred, since sanctification comes from outside, by definition.

If nothing is sacred, and institutions have no more reality than a children’s amorphous game of ball, they deduce that any construct can be reconstructed through willful choice.

Even if you accept the precept that there is no objective, external (let alone universal) value system, you can still see the fundamental fallacy in this thinking.

Anyone who has ever tried to bring change into a hidebound organization knows that social constructs are far harder to change than any physical or legal structure. You can reorganize units, bring locations together, shuffle management, or get rid of half of the people. Still, underlying social organization will re-emerge as long as there is any vestige of continuity.

Much of the heat energy in the ongoing culture war arises from this inertia. Those who are so tiresomely labelled as "liberal", "progressive", the "Left", the "Cultural Elite", etc. represent a large force of people aimed at deliberately reconstructing every institution in Western life. They have decided, based on their own feelings, bereft of natural or religious law, that any institution observed by men for more than one hundred years cannot be endured. They are organized around the post-modern paradigm–armed with Hayakawa and Chomsky–and don’t accept that some hidebound Neanderthals will not welcome forceful re-education.

I suppose that I follow a third way. I can agree that our institutions are social cosntructs. That does not mean that they can, or should be, tampered with lightly. The concept of "natural law" teaches that certain modes of behavior, certain morals, generate a more successful society. Our social institutions–like marriage–have undergone the same forces of competitive pressures and differential reproduction that drive neo-Darwinian evolution. That means the institutions we observe today–such as preserving the integrity of personal property–are the ones that worked.

There is an argument to be made that I’m advocating cultural imperialism. It could perhaps be seen that way, though such is not my intent. Rather, just as we should justifiably be wary of changing our own genetic code, we should be wary of making large changes to our social institutions. We do not know what will result. There are many paths down the mountain, but only one upward. Most random mutations result in death. Even well-planned changes have unintended, sometimes catastrophic, effects.


An IKEA Weekend

| Comments

I’ve been building a new office in my downstairs space for quite a while now. It’s a "weekends" project for someone who doesn’t have very many weekends. In early December, I broke down and hired a contractor to install the laminate ("cardboard") flooring, which was the penultimate step in the master plan.

Last comes furniture, then moving in. (Which starts the chain of dominoes, as my eldest gets the bedroom which used to be my office, then my youngest takes her spot, which makes room for the new baby. The challenge is to finish with the hole migration before the new electron gets injected. No, that wasn’t a spelling error.)

So this weekend, I had thirty-six boxes of IKEA modular furniture from "Work IKEA" to assemble.

You have time to meditate on many lessons when you are assembling thirty-six boxes of IKEA modular furniture.

For example, I’ve never seen a company that makes it so difficult to purchase from them. I don’t really want to know that the six-shelf bookshelf I picked out from the design software actually comes as three separate SKUs. Just sell me the damn shelf.

I shouldn’t have to learn what a "CDO" is in order to pick out a bunch of stuff and have them deliver it on a specific day. I shouldn’t have to make three trips into the store because they cannot take my credit card number over the phone.

And can someone please explain why I have to remove items from my delivery order because the local store doesn’t have them in stock? In some fields of endeavor, timing is everything, but why should I have to call them every day to find out when the left-handed tabletop comes in, then rush to the store and place my order so the piece can be pulled from inventory?

It makes no sense to me. The whole process was implemented for the convenience of IKEA, not IKEA’s customers. They’ve made a business decision to optimize for cost control rather than customer satisfaction. IKEA is certainly free to make that choice, and they do seem to be making profits, but I’m not likely to choose them for future furniture purchases.

Exposing that much of your internal process to the customer–or end user–is never a good way to win the hearts and minds of your customers.

Most of the assembly went without incident, though I was often perplexed by trying to map the low-level components into the high-level items I designed with. IKEA offers zero-cost software for download to design a floorplan with their lines, but it works at a higher level of abstraction. I was often left wondering which item a particular component was supposed to construct.

The components were very well designed. Each piece can either fit together in only one way, or it is rotationally symmetric so either orientation works. In either case, I, the assembler, am not left with an ambiguous situation, where something might fit but does not work.

The toughest pieces were the desks. Desks can be configured in about eighty-nine different ways. The components are all modular and generally have the same interfaces. I have a lot of flexibility at my disposal, but at the expense of complexity. A significant number of sample configurations helped me understand the complexity of options and pick a reasonable structure, but I can’t help but wonder how the experience could be simplified.

The furniture is all assembled now, and the office sits expectantly waiting for its occupant, full of unrealized potential.

Uniting Reason and Passion

| Comments

Reason and Passion need not conflict. Reason without passion is dusty, dry, and dead. Reason without passion leads to moral relativity. If nothing moves the thinker to passion, then all subjects are equal and without distinction. As well to discuss the economic benefits of the euthanasia of infants as the artistic merits of urinals.

Passion without reason brings the indiscriminate energy of a summer’s thunderstorm. Too much energy unbound, without direction, it’s fury as constant as the winds of the air.

Passion provides energy, the drive to accomplish, change, improve, or destroy. Reason provides direction. Reason channels Passion and achieves goals by identifying targets, foci, leverage points. Passion powers Reason. It brings motive power. Passion knows that things must be done and that change is possible. Reason knows how change may be effected.

I was reminded of the fallacy of Passion without Reason recently. At lunch with a friend, she talked about working with a non-profit organization. Workers for non-profits epitomize those who are driven by Passion. Agree or disagree with their aims, you must admit that they earnestly mean to change the world. My friend, who comes from the profit-driven corporate world, was explaining some aspects of statistical process control and how it could be applied to improve fundraising results on their website. She was told that she needed to have more heart and feel for those unfortunates that this group helps.

Her critic obviously felt that her approach was too analytical. Too driven by Reason, not enough Passion. In fact, the opposite was true. She was applying the combination of Reason and Passion. Passion showed her that the cause was worthy and that she could help. Reason showed her where leverage could be gained and a small effort input could result in a large change in output.

In various disfunctional organizations which I have inhabited, I’ve seen many examples of the opposite. Reason reveals problems and solutions to those poor sapient cogs in the low levels of the machine. They lack the Passion to see that change is possible and so divest themselves of the power to improve their own lot in life. Problems or challenges will always overcome such people, because they give the problem power and remove it from themselves.

More Wiki

| Comments

My personal favorite is TWiki. It has some nice features like file attachments, a great search interface, high configurability, and a rich set of available plugins (including an XP tracker plugin.)

One cool thing about TWiki: configuration settings are accomplished through text on particular topics. For example, each "web" (set of interrelated topics) has a topic called "WebPreferences". The text on the WebPreferences topics actually controls the variables. Likewise, if you want to set personal preferences, you set them as variables–in text–on your personal topic. It’s a lot harder to describe than it is to use.

There are some other nice features like role-based access control (each topic can have a variable that says which users or groups can modify the topic), multiple "webs", and so on.

The search interface is available as variable interpolation on a topic, so something like the "recent changes" topic just ends up being a date-ordered search of changes, limited to ten topics. This means that you can build dynamic views based on content, metadata, attachments, or form values. I once put a search variable on my home topic that would show me any task I was assigned to work on or review.

I’ve also been looking at Oahu Wiki. It’s an open source Java wiki. It’s fairly short on features at this point, but it has by far the cleanest design I’ve seen yet. I look forward to seeing more from this project.

Wiki Proliferation

| Comments

Wikis have been thoroughly mainstreamed now. You know how I can tell? Spammers are targeting them.

Any wiki without access control is going to get steamrolled by a bunch of Russian computers that are editing wiki pages. They replace all the legitimate content with links to porn sites, warez, viagra, get rich now, and the usual panoply of digital plaque.

The purpose does not appear to be driving traffic directly to those sites from the wikis. Instead, they are trying to pollute Google’s page rankings by creating thousands upon thousands of additional inbound links.

If you run a wiki, be sure to enable access control and versioning (so you can recover after an attack). It is a shame that the open, freewheeling environment of the wiki has to end. It seems that the only way to preserve the value of the community is to weaken the core value of open participation that made the community worthwhile.

Moving On

| Comments

The latest in my not-exactly-daily news and commentary…

As of December 10th, I will be leaving Totality Corporation. It has been a challenge and an education. It has also been an interesting time, as we uncovered the hidden linkages from daily activities to ultimate profitability. The managed service provider space is still new enough that the business models are not all so well-defined and understood as in consulting. I earnestly hope that I am leaving Totality in a much better place than it was when I joined.

Still, a number of positive attractions to the new position and some negative forces away from my current position have overcome inertia.

I will be joining Advanced Technologies Integration as a consultant. I will be forming a team with Kyle Larson, Dale Schumacher, and Dion Stewart to do a development project for one of ATI’s clients. The project itself has some moderately interesting requirements… it’s not just another random commerce site. (I’m really, really bored with shopping carts!) The thing that really attracted me though, is that this is a hardcore agile methods project. We’ll be using a combination of Scrum and XP.

For a long time, I’ve advocated small teams of highly skilled developers. I have seen such teams produce many times the business value (and ROI) of the typical team. ATI and this client are willing to subscribe to the theory that a small, high-caliber team will outperform an army of cheap morons.

It’s going to be a blast proving them right!

Too Much Abstraction

| Comments

The more I deal with infrastructure architecture, the more I think that somewhere along the way, we have overspecialized. There are too many architects that have never lived with a system in production, or spent time on an operations team. Likewise, there are a lot of operations people that insulate themselves from the specification and development of systems for which they will ultimately take responsibility.

The net result is suboptimization in the hardware/software fit. As a result, overall availability of the application suffers.

Here’s a recent example.

First, we’re trying to address the general issue of flowing data from production back into pre-production systems – QA, production support, development, staging. The first attempt took 6 days to complete. Since the requirements of the QA environment stipulate that the data should be no more than one week out of date relative to production, that’s a big problem. On further investigation, it appears that the DBA who was executing this process spent most of the time doing scps from one host to another. It’s a lot of data, so in one respect 10 hour copies are reasonable.

But the DBA had never been told about the storage architecture. That’s the domain of a separate "enterprise service" group. They are fairly protective of their domain and do not often allow their architecture documents to be distributed. They want to reserve the right to change them at will. Now, they will be quite helpful if you approach them with a storage problem, but the trick is knowing when you have a storage problem on your hands.

You see, all of the servers that the DBA was copying files from and to are all on the same SAN. An scp from one host on the SAN to another host on the SAN is pretty redundant.

There’s an alternative solution that involves a few simple steps: Take a database snapshot onto a set of disks with mirrors, split the mirrors, and join them onto another set of mirrors, then do an RMAN "recovery" from that snapshot into the target database. Total execution time is about 4 hours.

From six days to four hours, just by restating the problem to the right people.

This is not intended to criticize any of the individuals involved. Far from it, they are all top-notch professionals. But the solution required merging the domains of knowledge from these two groups – and the organizational structure explicitly discouraged that merging.

Another recent example.

One of my favorite conferences is the Colorado Software Summit. It’s a very small, intensely technical crowd. I sometimes think half the participants are also speakers. There’s a year-round mailing list for people who are interested in, or have been to, the Summit. These are very skilled and talented people. This is easily the top 1% of the software development field.

Even there, I occasionally see questions about how to handle things like transparent database connection failover. I’ll admit that’s not exactly a journeyman topic. Bring it up at a party and you’ll have plenty of open space to move around in. What surprised me is that there are some fairly standard infrastructure patterns for enabling database connection failover that weren’t known to people with decades of experience in the field. (E.g., cluster software reassigns ownership of a virtual IP address to one node or the other, with all applications using the virtual IP address for connections).

This tells me that we’ve overspecialized, or at least, that the groups are not talking nearly enough. I don’t think it’s possible to be an expert in high availability, infrastructure architecture, enterprise data management, storage solutions, OOA/D, web design, and network architecture. Somehow, we need to find an effective way to create joint solutions, so we don’t have software being developed that’s completely ignorant of its deployment architecture, nor should we have infrastructure investments that are not capable of being used by the software. We need closer ties between operations, architecture, and development.