Superprogrammers

Of all the damaging ideas and bad practices that plague the software industry -- the beliefs that are responsible for the appalling statistics about how buggy software is, and how many projects are abandoned without ever being deployed -- I think the most pernicious must be the concept of the superprogrammer.

The notion comes to us from the accurate, common, undeniable observation that on any given project, some programmers are more productive than others. Not just more productive: much more productive. Studies by Boehm and others consistently show that our best programmers are 10 to 20 times more productive than the average.

We have come to rely on superprogrammers to help us meet impossible deadlines, to untangle complex problems, to track down resistant bugs. Programming's tall tales, myths, and legends are all about superprogrammers. We lionize them. They are at the top of the software business's caste system, and managers treat them as qualitatively different from the mere mortals who make up programming's hoi polloi. The existence of superprogrammers is integral to the software economy, an unquestioned part of the foundation that supports modern software development.

I think it is high time to question our belief in the value and importance of superprogrammers.

THE TROUBLE WITH SUPERPROGRAMMERS

I have three main complaints with our industry's reliance on superprogrammers.

First, talk of superprogrammers is destructive to other workers. Superprogrammers get privileges, benefits, and recognition. The achievements of regular and above-average programmers pass unrecognized. Instead, these hard workers are reminded, day in and day out, of all that they are not. Morale and motivation plummet as developers are presented with a fait accompli. "Too bad you weren't born with Jerry's talent. If you were a genius you could have a faster computer too." Everyone accepts the elitist notion that the gulf separating the mental gifts of superprogrammers from the mundane brainpower of mere mortals is too wide to cross.

Second, we lack longitudinal studies that would allow us to say that superprogrammers retain their gifts over time. My observations suggest that developers acquire superprogrammer talents for short periods -- perhaps just for a single project. I remember superprogrammers from the days of MS-DOS programming; some became leading developers on the Windows platform, while others found the transition leveled the playing field, and in the new ballgame they were average players. Superior performance comes and goes as developers encounter problems that fit the interiors of their skulls. Today's average programmer is tomorrow's superprogrammer, and vice-versa.

Finally, I think reliance on superprogrammers is a shortsighted management strategy that delivers short-term gains at best, at the cost of overall organizational development and widespread improvements in productivity. Instead of setting up systems to recruit superprogrammers and funnel the hardest problems to them, organizations should set up training programs and incentives that bring out the best performance in everyone. A 10 percent productivity improvement for the whole staff is better -- for deadlines, morale, and the work environment -- than finding and hiring a new superprogrammer. It's far better to set up a work environment that supports and encourages professional growth among every member of the staff.

I have seen average workers, given the appropriate encouragement, training, and resources, rise to unexpected levels of performance. They become more confident as their skills grow, and it is a joy to see their potential blossom. It is this process -- the nurturing of all staff members so they can do their best work and extend the limits of their competence -- that management is all about. In contrast, superprogrammer doctrine is corrosive to the spirit. For most workers, it emphasizes what they cannot do and the talents they lack, rather than reinforcing their strengths and establishing expectations about their growth.

BEING A SUPERPROGRAMMER

Programmers are not in general afflicted with excess modesty. If statistics say two programmers in 100 work at the superprogrammer level, that will not prevent fully 75 percent of developers from identifying themselves as superior. If self-confidence is a virtue, then software developers are among the world's most virtuous workers.

I don't count myself among programming's elite. I struggled hard to write applications for my clients, back when there was a population of users tolerant enough to accept me as a consultant. I always found that writing good software was hard work.

And yet...I remember occasional periods when I shifted into a hyperproductive mode, when I worked with something like a superprogrammer's skill. There were minutes, hours, days when I was able to decompose tricky problems effortlessly and quickly map them into properties, events, and methods. The world seemed transparent to me, and I was able to see through to the bones. For glorious hours at a time I was one with the computer. I could do no wrong. It is one of the most glorious, joyful feelings I know.

Jon Bentley, a Bell Labs researcher who deserves the title "superprogrammer" if anyone does, has told me that he experiences similar periods. On a day-in, day-out basis, Jon approaches programming problems with skill and knowledge that humble those of all but a handful of the world's programmers. Yet he too experiences "hack attacks" -- periods, typically late at night, when he is capable of wielding uncommon insights with uncommon speed and precision.

I think we all experience periods of performing like superprogrammers. We all have times when code flows effortlessly, when we write code that is better than we are ordinarily capable of writing.

Artists talk about periods of being "in the flow," of tapping into a muse's power and finding that they have access to creative powers far beyond their own. The phenomenon is not talked about as much in our business, but I'm convinced that it is just as real. I've been in the flow; I bet you have too.

Wouldn't it be something to create a software-development organization in which we spent less time lionizing superprogrammers and focused less on who was responsible for which schedule slippages...and concentrated instead on techniques for helping every member of the staff spend more time in the flow?

IN THE FLOW

I'm serious about this. I think we could make huge progress in making development a more repeatable, reliable, affordable process if we gave the notion of flow some formal recognition. What can we, as programmers and development managers, do to make "in the flow" periods more frequent -- commonplace, even? Could we even turn every programmer into a superprogrammer?

Of course we could. In fact, we're already working on this goal. Software development methodologies, RAD environments, development paradigms, individual tools...we spend a lot of time talking about the tasks they automate for programmers and the time they save, but what they really do is shift our awareness into new modes, give us new ways of thinking about programming problems. Eventually, by turning the problem over and around in our minds, we find an alignment that clicks. We understand the problem in a new way that trivializes it, makes the solution simple and obvious.

The secret truth of the software development industry is that all of our tools are really designed to alter programmers' minds and shift them into hypercreative, hyperaware, hyperproductive states.

Think about the tools you feel most comfortable using. Maybe it's Delphi, whose marriage of objects and components offers a privileged perspective for decomposing problems. Don't your favorite tools give you moments when inspiration and ultrafast thinking are the norm? Isn't that how they became your favorite tools?

So go ahead and read the spec sheets. Justify your tool choices and development methods according to whatever objective criteria are necessary to satisfy the pointy-haired managers who sign the purchase orders. But don't kid yourself. What you're looking for, ultimately, is technology that empowers you with breathtaking new perspectives that lay new worlds of programming prowess at your feet. You're looking to be in the groove as often and as long as possible.

In the end, that's what separates superprogrammers from mere mortals. And because we've all experienced it, we can all be superprogrammers. Why not?

J.D. Hildebrand is a former programmer and a mediocre jazz musician. He currently serves as content director of the Borland Developer Community.