Chaos fears transparency
Oct12,2013
I guess this has been a long time coming. My cumulative frustration with dysfunctional management and engineering practices and my own experiences creating larger and larger software systems has led me to some sobering beliefs.
Chief among my beliefs is that Chaos fears transparency more than anything else.
Hey, I said it was a belief! I might as well be creative and imagine an adversary (Chaos) that is ingenious in its ability to bring disorder. I view my role as this:
Sometimes it feels like an epic confrontation: I don’t always win, but often I do, at least within reasonable limits.
I’ve learned how to design software the hard way: by not doing it. Then, by trying to do it and refining my technique, project after project. I’ve been writing software almost every working day since 1981. All but the first two years was work on Unix systems. I’m quite sure I’ve personally written well over a million lines of code and almost as much documentation and between one and two thousand separate programs.
Only about half of that had the level of design that it deserved. So, I’ve suffered. I’ve learned. I’ve taken some hard looks at my own work. My approach to software engineering has evolved over the years from entirely ad-hoc to very well planned. I have a lot more to learn. I’m certain I shall die before I’d ever feel like I understood software engineering sufficiently.
Creating a software system is nothing less than the deliberate creation of a highly or spectacularly ordered, multidimensional entity. Not a person, but more complicated than a layman could appreciate unless they also make some kind of equally complicated artifact. Software has no monopoly on complexity, and Chaos has time to provide personalized service to everyone who dares to create order.
What’s an entity? Well, it is a program running on one or more computers. You could be poetic and say it is hosted in the body of the computers, whose various physical systems support the needs of that program and many others that all share an ecosystem. The computers allow these entities (running programs) to function within that ecosystem. The server consumes energy and feeds information to the entities and accepts output from them for delivery to people or other programs. No, it isn’t Tron inside. But these entities do real things. They matter. Everyone who uses software of any kind relies on the reality of these entities. When the power is shut off they cease to exist. Their information lays dormant on a motionless hard drive or other media until the computer is turned on and the program is run again.
The entity is whatever it was created to be. It could be smart to not. In any case, if it is capable of doing something useful, it is probably complicated inside, and the more useful, the more complicated it is likely to be.
Chaos loves the smell of poorly understood complexity.
Ceating software is like cooking in some ways… procedures, iteration, parallelism, quality assurance. A good meal is no accident. Try making a dinner for 20 people without planning and see how it goes… but try getting a client to pay for the requirements and functional specifications phase of a project and they will look at you like food magically comes from the kitchen without any effort, or building software systems is like stacking coins and the sooner you begin stacking them the sooner you will finish.
I wish I was actually in a position to negotiate with them in good faith, but I am not. To negotiate in good faith either outcome must be a possibility: to design a large system first or just get coding. But, while it might seem possible to just get coding, the end result is not the same. One cannot have both the benefits of a well-designed system and also not spend any effort on requirements and functional specifications. That kind of hypocrisy is a shining beacon that calls Chaos from whatever dismal circumstance it was enjoying to come and kick down the house of cards and then laugh.
No, I can’t actually negotiate about whether my software will be designed before it is written. It will be. We can talk about what it costs, and it isn’t much, but if a client sees no value in participating in the requirements and functional specifications portions of the project, then I am tempted to refuse participation in the project from the outrage and abandonment I feel. A client like that is essentially saying “Do what I want, but I won’t really explain what it is. I’ll be sure to tell you when it is due and pester you to keep changing it as often as I change my mind, and then complain when it isn’t what I wanted.”
Instead, of negotiating I try to help them see the benefits of the essential requirements and functional specifications phases. While they are reviewing the functional specifications I am working on a software architecture and then elements of the software design. When the stakeholders approve the requirements I continue work on the software design.
It is almost always the case for me that the functional specs reveals a missed essential requirement or requires clarification of one or more essential requirements.
It is almost always the case that the software design reveals details missed in the functional specifications, such as feature interactions, boundary case handling or data maintenance or visibility tools that might be needed to manage what Chaos can do to a large system.
Each change to the functional specs or essential requirements must be negotiated with the client. It is usually that nobody thought about it and they are grateful that I brought it up. But, new requirements usually means additional work. Changed requirements sometimes undermines the functional specifications, which undermines the software design.
I call this design reverberation. Obviously we’d like to minimize this. Clear, thought communication and collaborative idea formation helps minimize this. People who limit access to, delay, withhold or obfuscate information, or who dissemble, cause design reverberations with their behavior and their choices.
So, what is the opposite effect? What reduces design reverberation, and its much larger cousin implementation reverberation? The opposite effect is transparency.
I’ve seen transparency thoroughly kick Chaos’s slightly askew ass right out of the system. I have to say that after having my ass kicked hard by Chaos it feels pretty good to watch transparency have its effect on a project and the people within it. Both are transformed by it to the extent that they all operate at a higher degree of functionality and efficiency.
Transparency reduces stress. It does. It levels the playing field of information and gives everyone a chance to contribute their ideas. Good ideas can come from anyone. Making good choices and having a shared vision are crucial to success. Success is stressful in its own way, but it is still better than the stress of project failure. It might not be so crucial if you are making something small. But large software systems are not incrementally more complicated than small ones. The ad-hoc, unplanned approach works adequately when building a one-room shack from spare materials but try building a sky scraper that way, or a hospital. Transparency reduces stress because it reduces dysfunction and the frustration and lost time that goes with it. It is the looming deadline that feels stressful, because if time is not a factor at all, then it is certainly time for a break.
So, I’ve become enamored of this thing called transparency. Maybe I’m just in that honeymoon phase with this new philosophy, so forgive me if I gush. But, I’m impressed with how effective transparency is as a tool to cope with some very difficult problems, like:
I am delighted to say that the solution in some of these cases has been to introduce more transparency. Engineers on my teams thrive on transparency. It gives them faith in management’s ability to make decisions based on actual information. The peer review that transparency enables strengthens healthy practices. The spirit of openness helps support an environment where people feel free to share ideas. We give by producing information for each other and we get because everyone else does too and we can read what they produced. We can help each other if we can understand each other.
My life has always been about software engineering since I first had a chance to play with a programmable calculator in the year of our country’s bicentennial.
It has been a wonderful journey, but I had no idea that would lead me here.
Chief among my beliefs is that Chaos fears transparency more than anything else.
Hey, I said it was a belief! I might as well be creative and imagine an adversary (Chaos) that is ingenious in its ability to bring disorder. I view my role as this:
I am someone who designs and builds a software system while Chaos mercilessly toys with me, but by the time I am done I have sought to banish Chaos from the kingdom of my software system.
Sometimes it feels like an epic confrontation: I don’t always win, but often I do, at least within reasonable limits.
I’ve learned how to design software the hard way: by not doing it. Then, by trying to do it and refining my technique, project after project. I’ve been writing software almost every working day since 1981. All but the first two years was work on Unix systems. I’m quite sure I’ve personally written well over a million lines of code and almost as much documentation and between one and two thousand separate programs.
Only about half of that had the level of design that it deserved. So, I’ve suffered. I’ve learned. I’ve taken some hard looks at my own work. My approach to software engineering has evolved over the years from entirely ad-hoc to very well planned. I have a lot more to learn. I’m certain I shall die before I’d ever feel like I understood software engineering sufficiently.
Creating a software system is nothing less than the deliberate creation of a highly or spectacularly ordered, multidimensional entity. Not a person, but more complicated than a layman could appreciate unless they also make some kind of equally complicated artifact. Software has no monopoly on complexity, and Chaos has time to provide personalized service to everyone who dares to create order.
What’s an entity? Well, it is a program running on one or more computers. You could be poetic and say it is hosted in the body of the computers, whose various physical systems support the needs of that program and many others that all share an ecosystem. The computers allow these entities (running programs) to function within that ecosystem. The server consumes energy and feeds information to the entities and accepts output from them for delivery to people or other programs. No, it isn’t Tron inside. But these entities do real things. They matter. Everyone who uses software of any kind relies on the reality of these entities. When the power is shut off they cease to exist. Their information lays dormant on a motionless hard drive or other media until the computer is turned on and the program is run again.
The entity is whatever it was created to be. It could be smart to not. In any case, if it is capable of doing something useful, it is probably complicated inside, and the more useful, the more complicated it is likely to be.
Chaos loves the smell of poorly understood complexity.
Ceating software is like cooking in some ways… procedures, iteration, parallelism, quality assurance. A good meal is no accident. Try making a dinner for 20 people without planning and see how it goes… but try getting a client to pay for the requirements and functional specifications phase of a project and they will look at you like food magically comes from the kitchen without any effort, or building software systems is like stacking coins and the sooner you begin stacking them the sooner you will finish.
I wish I was actually in a position to negotiate with them in good faith, but I am not. To negotiate in good faith either outcome must be a possibility: to design a large system first or just get coding. But, while it might seem possible to just get coding, the end result is not the same. One cannot have both the benefits of a well-designed system and also not spend any effort on requirements and functional specifications. That kind of hypocrisy is a shining beacon that calls Chaos from whatever dismal circumstance it was enjoying to come and kick down the house of cards and then laugh.
No, I can’t actually negotiate about whether my software will be designed before it is written. It will be. We can talk about what it costs, and it isn’t much, but if a client sees no value in participating in the requirements and functional specifications portions of the project, then I am tempted to refuse participation in the project from the outrage and abandonment I feel. A client like that is essentially saying “Do what I want, but I won’t really explain what it is. I’ll be sure to tell you when it is due and pester you to keep changing it as often as I change my mind, and then complain when it isn’t what I wanted.”
Instead, of negotiating I try to help them see the benefits of the essential requirements and functional specifications phases. While they are reviewing the functional specifications I am working on a software architecture and then elements of the software design. When the stakeholders approve the requirements I continue work on the software design.
It is almost always the case for me that the functional specs reveals a missed essential requirement or requires clarification of one or more essential requirements.
It is almost always the case that the software design reveals details missed in the functional specifications, such as feature interactions, boundary case handling or data maintenance or visibility tools that might be needed to manage what Chaos can do to a large system.
Each change to the functional specs or essential requirements must be negotiated with the client. It is usually that nobody thought about it and they are grateful that I brought it up. But, new requirements usually means additional work. Changed requirements sometimes undermines the functional specifications, which undermines the software design.
I call this design reverberation. Obviously we’d like to minimize this. Clear, thought communication and collaborative idea formation helps minimize this. People who limit access to, delay, withhold or obfuscate information, or who dissemble, cause design reverberations with their behavior and their choices.
So, what is the opposite effect? What reduces design reverberation, and its much larger cousin implementation reverberation? The opposite effect is transparency.
I’ve seen transparency thoroughly kick Chaos’s slightly askew ass right out of the system. I have to say that after having my ass kicked hard by Chaos it feels pretty good to watch transparency have its effect on a project and the people within it. Both are transformed by it to the extent that they all operate at a higher degree of functionality and efficiency.
Transparency reduces stress. It does. It levels the playing field of information and gives everyone a chance to contribute their ideas. Good ideas can come from anyone. Making good choices and having a shared vision are crucial to success. Success is stressful in its own way, but it is still better than the stress of project failure. It might not be so crucial if you are making something small. But large software systems are not incrementally more complicated than small ones. The ad-hoc, unplanned approach works adequately when building a one-room shack from spare materials but try building a sky scraper that way, or a hospital. Transparency reduces stress because it reduces dysfunction and the frustration and lost time that goes with it. It is the looming deadline that feels stressful, because if time is not a factor at all, then it is certainly time for a break.
So, I’ve become enamored of this thing called transparency. Maybe I’m just in that honeymoon phase with this new philosophy, so forgive me if I gush. But, I’m impressed with how effective transparency is as a tool to cope with some very difficult problems, like:
- corporate partner relationship issues
- rapid growth of an engineering team
- the need to manage large projects efficiently and minimize costs
- the need to work work with remote developers
- the need to build alignment around and adopt an engineering process
- incomplete/inaccurate functional specifications
I am delighted to say that the solution in some of these cases has been to introduce more transparency. Engineers on my teams thrive on transparency. It gives them faith in management’s ability to make decisions based on actual information. The peer review that transparency enables strengthens healthy practices. The spirit of openness helps support an environment where people feel free to share ideas. We give by producing information for each other and we get because everyone else does too and we can read what they produced. We can help each other if we can understand each other.
My life has always been about software engineering since I first had a chance to play with a programmable calculator in the year of our country’s bicentennial.
It has been a wonderful journey, but I had no idea that would lead me here.