My first exposure to Monte Carlo methods -- at least the first time I remember using that name for the concept -- came from a relatively pedestrian source. In 2005, I read a paperback edition of Fooled by Randomness by Nassim Nicholas Taleb, which was first printed in 2001, and at last count had sold around half a million copies.
Taleb describes Monte Carlo well enough that when I wanted to tell my wife what I was working on, I asked her to read Chapter 3. I remember thinking two things after reading that chapter for the first time:
- I can think of something I'm working on <i>right now</i> where this would be useful, and
- Why don't I remember hearing about this before? Why is this not used all the time?
The project I was working on at the time, where I thought I could use this method, was called "Muster." In 2001, Fort Pedro had developed another project called Muster that was mostly a fork of the Roundup issue tracker, originally by Ka-Ping Yee. Ted Whalen and I (mostly Ted) modified it to use the Zope Object Database instead of whatever it had been written to use originally, and to support time tracking on issue resolutions so we could bill people by the hour.
My new Muster (I called it "Muster TNG") had very little to do with the old Muster, except that it was still vaguely a task manager, and I still liked the name. The idea for Muster TNG was born out of two human frailties: the pain of failure, and my inability to deal with things beyond a certain order of complexity.
Toward the end of my time as an actual software developer, around 2001-2002, I was dropping a lot of balls. As I saw it after the fact, there were two basic problems. First, I was usually working with immature tools and libraries, which meant there were a fair number of blind alleys and things that just weren't ready for prime time. Second, I could not successfully estimate how long it was going to take me to find something that did work. I'd read all the books, from The Mythical Man-Month up to the (at the time) latest books on extreme programming, but pair programming and test-first design seemed to make more sense for Kent Beck and Ward Cunningham than it did for me. I just couldn't seem to meet a schedule.
After moving home to Crystal City, I started a new career in the insurance agency that my father had founded the year before I was born. Prior to my arrival, the agency's technology was sadly behind the times, so I set myself to updating it. I also had sales initiatives that I wanted to manage. I had yard projects I wanted to do. I joined the Rotary club and served on the Park Board and wound up on the board of the Catholic high school I'd attended as a teenager. I had too much going on, and I often had difficulty deciding which hopelessly large project to work on first.
Before reading Taleb's book, I had already figured out based on earlier experience that uncertainty was my problem. I couldn't trust my own estimates and therefore I couldn't plan effectively. However, I had not yet realized that embracing uncertainty, and acknowledging complexity, were the keys to solving the underlying problems.
Muster TNG, at its core, was a fairly straightforward implementation of David Allen's Getting Things Done productivity method. Among the bells and whistles, there was to be a project management component that helped you estimate large projects by breaking them down into manageable tasks. For each task, you would specify some estimates for how long you thought it might take, plus a "best case" and "worst case" figure. And then somehow, poof, the system would compute the probable duration of the project, and whether you could finish it on time or should start scrambling for Plan B.
I realized after reading Taleb's book that a Monte Carlo method would work to provide the "poof" I needed.
That was just my first itch. It would take another few bites before I would answer its call.
Up Next: Citizen Becker and the Future of Journalism.