If software were like this, you would never care about its architecture. You would spend a few minutes to create the thing that was needed, it would be used for the job at hand, and then it would be thrown away. It really wouldn't matter how good the software was on the inside--how easy it was to change--because you'd never change it! It wouldn't matter how it adapted to changing business requirements, because you'd just create a new one when the new requirement came up. In this thought experiment we wouldn't worry about architecture.

The key difference between this thought experiment and actual software? Of course, actual software is not disposable. It has a lifespan over some amount of time. Really, it's the time dimension that makes architecture important.

Over time, we need for many different people to work effectively in the software. Over time, we need the throughput of features to stay constant, or hopefully not decrease too much. Maybe it even increases in particularly nice cases. Over time, the business needs change so we need to adapt the software.

It's really time that makes us care about architecture.

Isn't it interesting then, that we never include time as a dimension in our architecture descriptions?

Check out http://github.com/FaKod/Circuit-Breaker-for-Scala for the code.

The Circuit Breaker can be mixed in to any type. See http://wiki.github.com/FaKod/Circuit-Breaker-for-Scala/ for an example of usage.

• Two obvious trends are cloud computing and mobile access. They are complementary. As the number of people and devices on the net increases, our ability to shape traffic on the demand side gets worse. Spikes in demand will happen faster and reach higher levels over time. Mobile devices exacerbate the demand side problems by greatly increasing both the number of people on the net and the fraction of their time they are able to access it.

• Large traffic volumes both create and demand large data. Our tools for processing tera- and petabyte datasets will improve dramatically. Map/Reduce computing (a la Hadoop) has created attention and excitement in this space, but it is ultimately just one tool among many. We need better languages to help us think and express large data problems. In particular, we need a language that makes big data processing accessible to people with little background in statistics or algorithms.

• Speaking of languages, many of the problems we face today cannot be solved inside a single language or application. The behavior of a web site today cannot be adequately explained or reasoned about just by examining the application code. Instead, a site picks up attributes of behavior from a multitude of sources: application code, web server configuration, edge caching servers, data grid servers, offline or asynchronous processing, machine learning elements, active network devices (such as application firewalls), and data stores. "Programming" as we would describe it today--coding application behavior in a request handler--defines a diminishing portion of the behavior. We lack tools or languages to express and reason about these distributed, extended, fragmented systems. Consequently, it is difficult to predict the functionality, performance, capacity, scalability, and availability of these systems.

• Some of this will be mitigated naturally as application-specific functions disappear into tools and frameworks. Companies innovating at the leading edge of scalability today are doing things in application-specific behavior to compensate for deficiencies in tools and platforms. For example, caching servers could arguably disappear into storage engines and no-one would complain. In other words, don't count the database vendors out yet. You'll see key-value stores and in-memory data grid features popping up in relational databases any day now.

• In general, it appears that Objects will diminish as a programming paradigm. Object-oriented programming will still exist... I'm not claiming "the death of objects" or something silly like that. However, OO will become just one more paradigm among several, rather than the dominant paradigm it has been for the last 15 years. "Object oriented" will no longer be synonymous with "good".

• Some people have talked about "polyglot programming". I think this is a red herring. Polylgot is a reality, but it should not be a goal. That is, programmers should know many languages and paradigms, but deliberately mixing languages in a single application should be avoided. What I think we will find instead is mixing of paradigms, supported by a single primary language, with adjunct languages used only as needed for specialized functions. For example, an application written in Scala may mix OO, functional, and actor-based concepts, and it may have portions of behavior expressed in SQL and Javascript. Nevertheless, it will still primarily be a Scala application. The fact that Groovy, Scala, Clojure, and Java all run on Java Virtual Machine shouldn't mislead us into thinking that they are interchangeable... or even interoperable!

• Regarding Java. I fear that Java will have to be abandoned to the "Enterprise Development" world. It will be relegated to the hands of cut-rate business coders bashing out their gray business applications for $30 / hour. We've passed the tipping point on this one. We used to joke that Java would be the next COBOL, but that doesn't seem as funny now that it's true. Java will continue to exist. Millions of lines of it will be written each year. It won't be the driver of innovation, though. As individual programmers, I'd recommend that you learn another language immediately and differentiate yourself from the hordes of low-skill, low-rent outsource coders that will service the mainstream Java consumer.

• Where will innovation come from? Although some of the blush seems to be coming off Ruby, the reduction in hype has mainly allowed Ruby and Ruby on Rails developers to knuckle down and produce. That community continues to drive tremendous innovation. Many of the interesting developments here relate to process. Ruby developers have given us fantastic tools like Gems and Capistrano, that let small teams outperform and outproduce groups four times their size.

• To my great surprise, data storage has become a hotbed of innovation in the last few years. Some of this is driven by the high-scalability fetishists, which is probably the wrong reason for 98% of companies and teams. However, innovations around column stores, graph databases, and key-value stores offer developers new tools to reduce the impedance mismatch between their data storage and their programming language. We spent twenty years trying to squeeze objects into relational databases. Aside from the object databases, which were an early casualty of Oracle's ascension, we mostly focused on changing the application code through framework after framework and ORM after ORM. It's refreshing to see storage models that are easier to use and easier to modify.

• This will also cause another flurry of "reactive innovation" from the database vendors, just as we saw with "Universal Databases" in the mid-90s. The big players here--Microsoft and Oracle--won't let some schemaless little upstarts erode their market share. More significantly, they aren't about to let their flagship products--and the ones which give them beachheads inside every major corporation--get intermediated by some open-source frameworks banged up by the social network giants. Look for big moves by these vendors into high scalability, agile storage, and eventual consistency storage.

• Just add some servers!
• I want a report of every configuration setting that's different between production and QA!
• We're going to make sure this (outage) never happens again!

I've recently been reminded of this during some discussions about disaster recovery. This topic seems to breed misconceptions. Somewhere, I think most people carry around a mental model of failover that looks like this:

That is, failover is essentially automatic and magical.

Sadly, there are many intermediate states that aren't found in this mental model. For example, there can be quite some time between failure and it's detection. Depending on the detection and notification, there can be quite a delay before failover is initiated at all. (I once spoke with a retailer whose primary notification mechanism seemed to be the Marketing VP's wife.)

Once you account for delays, you also have to account for faulty mechanisms. Failover itself often fails, usually due to configuration drift. Regular drills and failover exercises are the only way to ensure that failover works when you need it. When the failover mechanisms themselves fail, your system gets thrown into one of these terminal states that require manual recovery.

Just off the cuff, I think the full model looks a lot more like this:

It's worth considering each of these states and asking yourself the following questions:

• Is the state transition triggered automatically or manually?
• Is the transition step executed by hand or through automation?
• How long will the state transition take?
• How can I tell whether it worked or not?
• How can I recover if it didn't work?

On my way to QCon Tokyo and QCon China, I had some time to kill so I headed over to Delta's Skyclub lounge. I've been a member for a few years now. And why not? I mean, who could pass up tepid coffee, stale party snacks, and a TV permanently locked to CNN? Wait... that actually doesn't sound like such a hot deal.

Oh! I remember, it's for the wifi access. (Well, that plus reliably clean bathrooms, but we need not discuss that.) Being able to count on wifi access without paying for yet another data plan has been pretty helpful for me. (As an aside, I might change my tune once I try a mifi box. Carrying my own hotspot sounds even better.)

Like most wifi providers, the Skyclub has a captive portal. Before you can get a TCP/IP connection to anything, you have to submit a form with a checkbox to agree to 89 pages of terms and conditions. I'm well aware that Delta's lawyers are trying to make sure the company isn't liable if I go downloading bootlegs of every Ally McBeal episode. But I really don't know if these agreements are enforceable. For all I know, page 83 has me agreeing to 7 years indentured servitude cleaning Delta's toilets.

Anyway, Delta has outsourced operations of their wifi network to Concourse Communications. And apparently, they've had an outage all morning that has blocked anyone from using wifi in the Minneapolis Skyclubs. When I submit the form with the checkbox, I get the following error page:

Including this bit of stacktrace:

There's a lot to dislike here.

1. Why is this yelling at me, the user? To anyone who isn't a web site developer, this makes it sound like the user did something wrong. There's a ton of scary language here: "instance-specific error", "allow remote connections", "Named Pipes Provider"... heck, this sounds like it's accusing the user of hacking servers. "Stack trace" sure sounds like the Feds are hot on somebody's trail, doesn't it?
2. Isn't it fabulous to know that Ken keeps his projects on his D: drive? If I had to lay bets, I'd say that Ken screwed up his configuration string. In fact, the whole problem smells like a failed deployment or poorly executed change. Ken probably pushed some code out late on a Friday afternoon, then boogied out of town. My prediction (totally unverifiable, of course) is that this problem will take less than 5 minutes to resolve, once Ken gets his ass back from the beach.
3. We mere users get to see quite a bit of internal information here. Nothing really damaging, unless of course Wilson ORMapper has some security defects or something like that.
4. Stepping back from this specific error message, we have the larger question: is it sensible to couple availability of the network to the availability of this check-the-box application? Accessing the network is the primary purpose of this whole system. It is the most critical feature. Is collecting a compulsory boolean "true" from every user really as important as the reason the whole damn thing was built in the first place? Of course not! (As an aside, this is an example of Le Chatelier's Principle: "Complex systems tend to oppose their own proper function.")

We see this kind of operational coupling all the time. Non-critical features are allowed to damage or destroy critical features. Maybe there's a single thread pool that services all kinds of requests, rather than reserving a separate pool for the important things. Maybe a process is overly linearized and doesn't allow for secondary, after-the-fact processing. Or, maybe a critical and a non-critical system both share an enterprise service---producing a common-mode dependency.

Whatever the proximate cause, the underlying problem is lack of diligence in operational decoupling.

Like all of my work about architecture, this covers technology, business, and strategic aspects, so there's an emphasis on creating high-velocity, competitive organizations.

In general, I'm aiming for a mark that's just behind the bleeding edge. So, I'm including several of the NoSQL persistence technologies, for example, but not including Erjang because it's too early. (Or is that "erl-y"? )

(What I'd really love to do is make a screencast series out of all of these. I'm daunted, though. There's a lot of ground to cover here!)

EDIT: Added function and OO styles of programming. (Thanks @deanwampler.) Added JRuby/Java under languages. (Thanks @glv.)

I'm interested in hearing your feedback. What would you add? Remove?

• Methods and Processes

  • Systems Thinking/Learning Organization
  • High Velocity Organizations
  • Safety Culture
  • Error-Inducing Systems ("Normal Accidents")
  • Points of Leverage
  • Fundamental Dynamics: Iteration, Variation, Selection, Feedback, Constraint
  • 5D architecture
  • Failures of Intuition
  • ToC
  • Critical Chain
  • Lean Software Development
  • Real Options
  • Strategic Navigation
  • OODA
  • Tempo, Adaptation
  • XP
  • Scrum
  • Lean
  • Kanban
  • TDD

• Architecture Styles

  • REST / ROA
  • SOA
  • Pipes & Filters
  • Actors
  • App-server centric
  • Event-Driven Architecture

• Web Foundations

  • The "architecture" of the web
  • HTTP 1.0 & 1.1
  • Browser fetch behaviors
  • HTTP Intermediaries

• The Nature of the Web

  • Crowdsourcing
  • Folksonomy
  • Mashups/APIs/Linked Open Data

• Testing

  • TDD
  • Unit testing
  • BDD/Spec testing
  • ScalaCheck
  • Selenium

• Persistence

  • Redis
  • CouchDB
  • Neo4J
  • eXist
  • "Web-shaped" persistence

• Technical architecture

  • 8 Fallacies of Distributed Computing
  • CAP Theorem
  • Scalability
  • Reliability
  • Performance
  • Latency
  • Capacity
  • Decoupling
  • Safety

• Languages and Frameworks

  • Spring
  • Groovy/Grails
  • Scala
  • Lift
  • Clojure
  • Compojure
  • JRuby
  • Rails
  • OSGi

• Design

  • Code Smells
  • Object Thinking
  • Object Design
  • Functional Thinking
  • API Design
  • Design for Operations
  • Information Hiding
  • Recognizing Coupling

• Deployment

  • Physical
  • Virtual
  • Multisite
  • Cloud (AWS)
  • Chef
  • Puppet
  • Capistrano

• Build and Version Control

  • Git
  • Ant
  • Maven
  • Leiningen
  • Private repos
  • Collaboration across projects

One of the very interesting things in Itasca State Park is a section of forest that is fenced off so that deer cannot enter it. It's part of a decades-long experiment to see how forests are affected by browsing herbivores. What's really interesting is that not only are the quantity of plants different inside the protected area, but the types of plants and trees are different, too. Because deer prefer to nibble on younger trees, fewer saplings survive in the main body of the forest than in the fenced-off portion. Outside the fence, the distribution of tree size and age is biased toward older trees. The population of trees is weighted more toward resinous species like pines, which deer prefer not to eat. Inside the fence, more saplings survive into young maturity, so you see a more even distribution of tree ages and a wider diversity of species represented in the mature trees. The changes in the canopy affect the ground cover which, in turn, change how deer could (if allowed) reach the trees and browse them.

So, here's a feedback loop that involves deer, trees, leaves and brush. The net result is a different ecosystem (albeit a slightly artificial one.)

Most physical and biological systems are like this in several ways, particularly relating to feedback. In our artificial systems (electrical, mechanical, symbolic, or semantic) we build in feedback mechanisms as a deliberate control. These are often one dimensional, proportional, and negative.

In natural systems, feedback arises everywhere. Sometimes, it proves to be helpful for the long-term stability of the system. In which case, the feedback itself gets reinforced by the existence and perpetuation of the system it exists within. In a sense, the system adapts to reinforce beneficial feedback. Conversely, feedback webs that cause too much instability will, like an overly aggressive virus, lead to destruction of their host system and disappear. So, we can see the constituents of a system co-evolving with each other and the system itself.

The old "microphone-amplifier-speaker-squealing" example of feedback really fails here. We lack both language and metaphor to really grasp this kind of interaction over time. In part, I think that's because we like to separate the world into isolated components and only talk about components at a single level of abstraction. The trouble is that abstractions like "level of abstraction" only exist in our minds.

Here's another example of coevolution, courtesy of Jared Diamond in "Guns, Germs, and Steel". I'll apologize in advance for oversimplifying; I'm devoting a paragraph to an argument he develops across entire chapters.

At some point, a group of nomads decided that the seeds of these particular grasses were tasty. In collecting the grasses, they spread it around. Some kinds of seeds survived the winter better and responded well to being sown by humans. Now, nobody sat down and systematically picked out which seeds grew better or worse. They didn't have to, because the seeds that grew better produced more seeds for the next generation. Over time, a tiny difference (fractions of a percent) in productivity would lead some strains to supplant the others. Meanwhile, inextricably linked, some humans figured out how to plants, harvest, and eat these early grains. These humans had an advantage over their neighbors, so they were able to feed more babies. That turns out to be a benefit, because farming is hard work and requires more offspring to help produce food. (Another feedback loop.) Oh, and this kind of labor makes it advantageous to keep livestock, too. Over time, these farmers would breed and feed more children than the nomads, so farmers would come to be a larger and larger percentage of the population. Just as an added wrinkle, keeping livestock and fertilizing fields both lead to diseases that simultaneously harm the individuals and occasionally decimate the population, but also provide some long-term benefits such as better disease resistance and inadvertent biological warfare when encountering other civilizations.

Try to diagram the feedback loops here: nomads, farmers, livestock, grains, birthrates, and so on. Everything is connected to everything else. It's really hard to avoid slipping into teleological language here. We've got feedback and feedforward at several different levels and timescales here, from the scale of microbes to livestock to civilizations, and across centuries. This dynamic altered the course of many species evolution: cattle, wheat, maize, and yes, good old H. Sapiens.

This complexity of interaction extends to planetary and stellar levels as well. At some sufficiently long time scale, the intergalactic medium is coupled to our planetary ecosystem.

The human intellectual penchant for decomposition, isolation, and leveled abstraction is purely an artifact of the size of our bodies and the duration of our lives.

...a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we're too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This perfectly describes the "Chain Reaction" stability antipattern from Release It!

I'm running on a laptop, using a pseudo-distributed cluster. That means all the different processes are running, but they're all running on one box. That makes it possible to test jobs with full network communication, but without deploying to a production cluster.

I'm also working remotely, connecting to the corporate network by VPN. As is commonly done, our VPN is configured to completely separate the client machine from its local network. (If it didn't, you could use the VPN machine to bridge the secure corporate network to your home ISP, coffeeshop, airport, etc.)

Here's the problem: when on the VPN, my machine can't talk to its own IP address. Right now, ifconfig reports the laptops IP address as 192.168.1.105. That's the address associated with the physical NIC on the machine.

The odd part is that Hadoop mostly works this way. I've configured the name node, job tracker, task tracker, datanodes, etc. to all use "localhost". I can use HDFS, I can submit jobs, and all the map tasks work fine. The only problem is that when the map tasks finish, the task tracker cannot send data from the map tasks to the reduce tasks. The job appears to hang.

In the task tracker's log file, I see reports every 20 seconds or so that say

2009-07-31 11:01:33,992 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200907310946_003_r_000000_0 0.0% reduce > copy >

The instant I disconnected from the VPN, the copy proceeded and the reduce job ran.

I'm sure there's a configuration property somewhere within Hadoop that I can change. When (if) I find it, I'll update this post.

Spiros' project page is here, and the code is (where else?) on GitHub. He appears to be quite actively developing the project.

The other one has actually been out for a couple of months now, but I forgot to blog about it. Scott Vlamnick created a Grails plugin that uses AOP to weave Circuit Breaker functionality as "around" advice. This one can also report its state via JMX. In a particularly nice feature, this plugin supports different configurations in different environments.

Well, can a chef make a souffle with a skillet?

Can a cabinetmaker round an edge with dull router bits?

I'm not going to rule it out. Perhaps there's a brilliant chef who---at this very moment---is preparing to introduce the world to the "skiffle." And, it's possible that one could coax a dull router into making a better quarter round through care, attention, and good speed control.

Going by the odds, though, I'd bet on scrambled eggs and splinters.

Like a lot of old sayings, this one doesn't make much sense in it's usual interpretation. Most people take this proverb to mean that you should be able to turn out top-notch work with whatever tools you're given. It's an excuse for bad tools, or lack of interest in improving them.

This homily dates back to a time when workers would bring their own tools to the job, leading to the popular origin story for the phrase "getting sacked". (No comments about møøse bites, please.) Some crafts have evaded the assembly line, and in those, craftsman still bring their own tools. Chefs bring their prized knives. Fine carpenters bring their own hand and bench tools.

There is a grain of truth in the common interpretation that good tools don't make a good workman. There's another level of truth under the surface, though. The 13th Century French version of this saying translates as, "A bad workman will never find a good tool." I like this version a lot better. Tools cannot make one good, but bad tools can hurt a good worker's performance. That sounds a lot less like "quit whining and use whatever's at hand," doesn't it?

On the other hand, if you supply your own tools, you're not as likely to tolerate bad ones, are you? I think this is the most important interpretation. Good workers---if given the choice---will select the best tools and keep them sharp.

Beginning Scala is a nice, gentle introduction to this language. It takes a gradual, example driven approach that emphasizes running code early. This makes it a good intro for people who want to use the language for applications first, then worry about creating frameworks later.

Don't let that fool you, though. Pollak gets to the sophisticated parts soon enough. I particularly like a example of creating a new "control structure" to execute stuff in the context of a JDBC connection. This puts some meat on the argument that Scala is a "scalable language." Where other languages either implement this as a keyword (as in Groovy's "with") or a framework (Spring's "templates"), here it can be added with one page of example code.

Beginning Scala also has a very thorough discussion of actors. I appreciate this, because actors were my main motivation for learning Scala in the first place.

Pollak separates the act of consuming a library from that of creating a library. He advises us to worry most about types, traits, co- and contravariance, etc. mainly when we are creating libraries. True to this notion, chapter 7 is called "Traits and Types and Gnarly Stuff for Architects". It doesn't sound like much fun, but it is important material. I find that Scala makes me think more about the type system than other languages. It's strongly, and statically, typed. (So much so, in fact, that it makes me realize just how loose Java's own type system is.) As such, it pays to have a firm understanding of how code turns into types. Scala has a rich set of tools for building an expressive type system, but there is also complexity there. Checking in at 60 pages, this chapter covers Scala's tools along with guidance on good styles and idioms.

Interestingly, although there is a Lift logo on the cover, there's nothing about Lift in the book itself. Considering that Pollak is the creator of Lift, it's curious that this book doesn't deal with it. Perhaps that's being left for another title.

Overall, I endorse Beginning Scala.

While I was at JAOO Australia this week, Amanda Laucher showed some F# code for a recipe that caught my attention. It used numeric literals with that directly attached units to quantities. What's more, it was intelligent about combining units.

I went looking for something similar in Scala. I googled my fingertips off, but without much luck, until Miles Sabin pointed out that there's already a compiler plugin sitting right next to the core Scala code itself.

Installing Units

Scala has it's own package manager, called sbaz. It can directly install the units extension:

sbaz install units

This will install it under your default managed installation. If you haven't done anything else, that will be your Scala install directory. If you have done something else, you probably already know what you're doing, so I won't try to give you instructions.

Using Units

To use units, you first have to import the library's "Preamble". It's also helpful to go ahead and import the "StandardUnits" object. That brings in a whole set of useful SI units.

I'm going to do all this from the Scala interactive interpreter.

scala> import units.Preamble._
import units.Preamble._

scala> import units.StandardUnits._
import units.StandardUnits._

After that, you can multiply any number by a unit to create a dimensional quantity:

scala> 20*m
res0: units.Measure = 20.0*m

scala> res0*res0
res1: units.Measure = 400.0*m*m

scala> Math.Pi*res0*res0
res2: units.Measure = 1256.6370614359173*m*m

Notice that when I multiplied a length (in meters) times itself, I got an area (square meters). To me, this is a really exciting thing about the units library. It can combine dimensions sensibly when you do math on them. In fact, it can help prevent you from incorrectly combining units.

scala> val length = 5*mm
length: units.Measure = 5.0*mm

scala> val weight = 12*g
weight: units.Measure = 12.0*g

scala> length + weight
units.IncompatibleUnits: Incompatible units: g and mm

I can't add grams and millimeters, but I can multiply them.

Creating Units

The StandardUnits package includes a lot of common units relating to basic physics. It doesn't have any relating to system capacity metrics, so I'd like to create some units for that.

scala> import units._
import units._

scala> val requests = SimpleDimension("requests")
requests: units.SimpleDimension = requests

scala> val req = SimpleUnit("req", requests, 1.0)
req: units.SimpleUnit = req

scala> val Kreq = SimpleUnit("Kreq", requests, 1000.0)
Kreq: units.SimpleUnit = Kreq

Now I can combine that simple dimension with others. If I want to express requests per second, I can just write it directly.

scala> 565*req/s
res4: units.Measure = 565.0*req/s

Conclusion

This extension will be the first thing I add to new projects from now on. The convenience of literals, with the extensibility of adding my own dimensions and units means I can easily keep units with all of my numbers.

There's no longer any excuse to neglect your units in a mainstream programming language.

Something came up last week, though, that just blew me away.

For various reasons, I've engaged Relevance to do a project for me. (Actually, the first results were so good that I've now got at least three more projects lined up.) They decided---and by "they", I mean Stuart Halloway---to write the engine at the heart of this application in Clojure. That makes it sound like I was reluctant to go along, but actually, I was interested to see if the result would be as expressive and compact as everyone says.

Let me make a brief aside here and comment that I'm finding it much harder to be the customer on an agile project than to be a developer. I think there are two main reasons. First, it's hard for me to keep these guys supplied with enough cards to fill an iteration. They're outrunning me all the time. Big organizations like my employer just take a long time to decide anything. Second, there's nobody else I can defer to when the team needs a decision. It often takes two weeks just for me to get a meeting scheduled with all of the stakeholders inside my company. That's an entire iteration gone, just waiting to get to the meeting to make a decision! So, I'm often in the position of making decisions that I'm not 100% sure will be agreeable to all parties. So far, they have mostly worked out, but it's a definite source of anxiety.

Anyway, back to the main point I wanted to make.

My personal theme is making software production-ready. That means handling all the messy things that happen in the real world. In a lab, for example, only one batch file ever needs to be processed at once. You never have multiple files waiting for processing, and files are always fully present before you start working on them. In production, that only happens if you guarantee it.

Another example, from my system. We have a set of rules (which are themselves written in Clojure code) that can be changed by privileged users. After changing the configuration, you can tell the daemonized Clojure engine to "(reload-rules!)". The "!" at the end of that function means it's an imperative with major side effects, so the rules get reloaded right now.

I thought I was going to catch them up when I asked, oh so innocently, "So what happens when you say (reload-rules!) while there's a file being processed on the other thread?" I just love catching people when they haven't dealt with all that nasty production stuff.

After a brief sidebar, Stu and Glenn Vanderburg decided that, in fact, nothing bad would happen at all, despite reloading rules in one thread while another thread was in the middle of using the rules.

Clojure uses a flavor of transactional memory, along with persistent data structures. No, that doesn't mean they go in a database. It means that changes to a data structure can only be made inside of a transaction. The new version of the data structure and the old version exist simultaneously, for as long as there are outstanding references to them. So, in my case, that meant that the daemon thread would "see" the old version of the rules, because it had dereferenced the collection prior to the "reload-rules!" Meanwhile, the reload-rules! function would modify the collection in its own transaction. The next time the daemon thread comes back around and uses the reference to the rules, it'll just see the new version of the rules.

In other words, two threads can both use the same reference, with complete consistency, because they each see a point-in-time snapshot of the collection's state. The team didn't have to do anything special to make this happen... it's just the way that Clojure's references, persistent data structures, and transactional memory work.

Even though I didn't get to catch Stu and Glenn out on a production readiness issue, I still had to admit that was pretty frickin' cool.