The Invisible Software Process: Preface

By: Randy Miller

Abstract: This article contains the preface for an upcoming book, The Invisible Software Process.

This is a book about the business of software engineering. We have entered an era of globalization in the software industry. By no means is this a surprise to anyone. Ed Yourdon predicted this globalization in Decline & Fall of the American Programmer over a decade ago [Yourdon 1992]. The fact is our software industry can be found in all parts of the world. I have been proud to work along side some very good software developers in the United Kingdom, Canada, The Czech Republic, Russia, and Singapore.

For some, this globalization means developing projects at a lower cost. This is perhaps the biggest fallacy of the twenty-first century. For lowest cost does not necessarily mean greatest return on investment. If it did, you would almost certainly invest in penny stocks rather than those found in the more prestigious stock markets of the world. Instead, developers and development teams have a wide range of skills and offer a high degree of variation in productivity. A working and delivering team is like a high yield stock.

Developing good software requires much more than just than human capital. It requires skill, good tools, domain knowledge, and productivity. If all software developers were able to deliver software at the same rate, you could see a direct relationship between salary and ROI. But as we all know, the productivity of some developers is far greater than others [DeMarco 1982]. This productivity gap can be as great as forty times.

In other words, in order to achieve a complete picture of the return on a software project investment, we must look at the productivity of the development team. We must also look at other costs as well. For some projects, opportunity costs will factor into the total cost. If we can get these projects done sooner, we actually save money. It may be worthwhile to utilize a more expensive team to deliver a project faster. Paying more and delivering the software sooner saves this customer money. But spending more does not guarantee that a project is going to be delivered.

The opportune word is delivered. An undelivered project provides no value and its return on investment is nothing. And there are plenty of undelivered projects in the software world. However, there are also quite a lot of delivered ones as well. Some companies have a proven track record of delivering time and time again. As my good friend Joe Mesa reminded me once, there are also some companies that simply should not be making software.

Agile Software Development

What separates the successful companies from the unsuccessful companies? This book looks at some of the successful techniques that have been used at those companies that successfully make software. It describes an Agile Process Framework, a set of techniques that may be used to help deliver a software project. It provides a toolbox of tools that you can use when the need arises.

But tools are not enough to help you compete in this global software market. Wherever you may be living, achieving higher productivity requires a dedication to the software craft. The agile movement goes beyond a set of tools to a philosophy of succeeding and placing the core component of building software, the human element, first. It looks to deliver software projects through collaboration and teamwork. Indeed, achieving the highest levels of software productivity requires a level of honesty and motivation that must be nurtured and cultured.

Agile software engineering recognizes that software development is a unique combination of art and engineering. For gathering requirements and conceptualizing systems is an art much like oil exploration or nuclear plant design. But the translation of this concept into a working system is every bit of an engineering discipline. Herein lies the problem of software development and a challenge to the modern software engineer.

The twenty-first century software engineer must develop the skills to be able to do many things in addition to coding. The must be able to accurately estimate the amount of time that it takes to develop a system and report progress on that same development effort. They must be able to gather requirements and discuss those requirements with the various stakeholders. And they must be able to architect and build the system in conjunction with their team members.

What is proposed is the end of the software development silos where one group throws the spec over the wall to another. In this world, only the development and the quality assurance (testing) team are responsible for the delivery of the system. This is no longer acceptable. Instead, all of the folks must have skin in the game. The people capturing requirements must be able to communicate, write the requirements in the appropriate form, and understand what it means to build such a system.

Figure P-1: The Over the Wall approach

Modern software tools are also poised to help increase productivity. As development teams learn that collaboration is the method of embracing change, tool vendors have added more and more features to support a collaborative environment. The modern integrated development environment and change management systems are no longer separate entities. They work together to provide quick access to the information needed to build better software faster.

In short, agile software engineering is the combination of culture, techniques, and tools to provide a highly productive software development environment. Unfortunately, for many, software engineering has ceased to be something that is fun and engaging. As the development and delivery of software projects is knowledge work, motivation plays a key role in ensuring that a high quality system is delivered. The humanistic aspect of software engineering must be reintroduced to restore the fun in this profession.

Holonic Software Development

The goal of this book is provide an understanding of what we can do as software professionals to be more productive. It begins by discussing why most developers dont seem to follow a process. We also explore what a process can and cannot accomplish to make their lives easier. We provide a toolbox for developing all classes of systems, from the two-person team with a short duration to the multi-site, multi-year bet-your-business project. This toolbox is called the Agile Process Framework. Finally, we look at the cultural aspects that enhance the productivity of software teams.

Holonic Software Development is a new method of software development based upon principles from many fields. It stresses the role of the individual and the team in successfully delivering software projects. Arthur Koestler coined the term holon by combining holos (the whole) and on (the part) [Koestler 1967]. Koestler observed that elements of living organisms and social organizations behave independently while forming part of a larger whole. As a result, they make highly efficient use of resources, are highly resilient to disturbances, and are adaptable to changes in their environment.

I presented Holonic Software Development at a meeting of TogetherSoft mentors in early 2001. The meeting gathered some very gifted people who worked for the company at that time and the presentation was well received. I also enjoyed the opportunity of posting some of the philosophical articles on this subject on IBM developerWorks. I am convinced that there is a better way to build software that is evolutionary, rather than revolutionary.

Much of this book builds on the work of Kent Beck, Alistair Cockburn, and Jim Highsmith. These folks began to question the way that the field was going. Most of all, they shied away from the incorrect notions that one could install a process and make people interchangeable cogs in software delivery. To compliment their work, the techniques in this book are borrowed from successful software development organizations, projects, and teams from many different companies including Borland.

There are also techniques borrowed for other engineering and non-engineering disciplines as well. The nuclear power industry, for example, has been looking at the role of process in its activities for quite some time. This industry has built an extensive framework for the work that it performs. A failure in this domain can have pretty drastic consequences.

My discussions with Civil Engineers have shown that the major projects such as highway design must be done under the guidance of experienced engineers. In fact, you cannot advance in that field without being apprenticed by a more senior engineer. Strangely enough, I learned about cooking and jazz from some experts in these areas and found some of their quite ideas relevant as well. Finally, the military was also a source of inspiration. They have perfected the element of training which is important in our constantly changing industry.

Who Should Read This Book

Holonic Software Development is a natural evolution in the agile community. Agile software development processes such as the adaptive process [Highsmith 2000], Crystal [Cockburn 2002], Feature Driven Development [Palmer 2002], Scrum [Schwaber 2002], and eXtreme Programming [Beck 2000] provided fertile growth in the area of software development process.

These works provide a great starting point for those wishing to explore agile software development. Holonic software development is for all who are involved in the delivery of software, from requirements engineers to software developers to quality assurance and management. In fact, creating a common understanding between all of these groups is necessary to creating the type of software development process that we are espousing.

The idea is to create a way of delivering solutions that is understood by everyone but adaptable to meet the specific needs of each contributor. When the entire team is on the same page, you have created the invisible software development and delivery process. For the invisible software development process is a common understanding of what is needed to deliver a system.

However, there must be room for diversity beyond what is needed and called for by the invisible software development process. Holonic Software Development allows the diversity for team members to maximize their individual productivity on each specific task. The result is maximum team and organizational productivity. Most of all, Holonic Software Development is all about building the high yield, high performance software development organization capable of competing on any level in the modern, competitive world of delivering software systems.

I conclude this preface with a safe harbor statement. Ultimately, let common sense be your guide in building and delivering systems. There are many techniques that are tried and true in this industry. We do not suggest that you abandon your past. We suggest instead that you look at some of the ideas presented in this book to better understand what you are doing right and what may be improved. Making these improvements work is a complicated matter but one that will ultimately lead to higher productivity wherever you live.

Randy Miller

6/6/03

Sunset Beach, NC

Bibliography

[Beck 2000] Kent Beck, eXtreme Programming Explained, Addison Wesley (2000).

[Cockburn 2002] Alistair Cockburn, Agile Software Development, Addison Wesley (2002).

[DeMarco 1982] Tom DeMarco, Controlling Software Projects, Prentice Hall (1982).

[Highsmith 2000] James A. Highsmith III, Adaptive Software Development, Dorset House (2000).

[Koestler 1967] Arthur Koestler, The Ghost In The Machine, Random House (1967).

[Palmer 2002] Stephen R. Palmer and John M. Felsing, A Practical Guide To Feature-Driven Development, Prentice Hall (2002).

[Schwaber 2002] Ken Schwaber and Mike Beedle, Agile Software Development with Scrum, Prentice Hall (2002).

[Yourdon] Edward Yourdon, Decline & Fall of the American Programmer, Prentice Hall (1992).



Server Response from: ETNASC04