
Product development is an inherently messy business. I can say that from my own experience but it seems to be true almost everywhere. Search for "product development uncertainty" on your favorite search engine and you will get millions of results, John Cutler calls his blog "The beautiful mess"1 and Google makes the ability to “thrive in ambiguity” a key characteristic of their expectations2, to name only a few indications.
I think this is somewhat inevitable. There's the latent uncertainty about the business value of what you're doing with your product. This uncertainty is often compounded by divergent personal preferences and perhaps even some hidden agendas. At the same time external or internal factors cause upper management to change the strategy from time to time. Add the normal organizational complexity of larger organizations and the omnipresent technical debt into the mix, spice it up with some constraints such as backwards compatibility requirements and you really have a big big mess that everyone is struggling to deal with somehow. Development is slow, there's never enough capacity to do all the things the company wants/needs to do, people are frustrated about seemingly inefficient processes and intransparent management processes, and so on...
They have the master plan that will solve many of your problems at once, if only they are given the necessary time and resources.
And then the white knight comes along. They may come in from outside the company or step up from the inside3. They have the master plan that will solve many of your problems at once, if only they are given the necessary time and resources. They are confident, they genuinely believe in their idea. They paint the future in bright colors. And because they project competence and are a good communicator it's easy for them to quickly convince a lot of people. The plan gets traction, people talk about it. Story telling trap and confirmation bias kick in. Critical voices find it harder and harder to be heard, because no one wants to see their new hope destroyed. Chances are the new plan gets the green light.
But slowly, stealthily at first, then more and more noticeably it happens: reality strikes back.
Fast forward a few months. The initial excitement is gone, but all in all things seem to be going well. Yes, there are a few unforeseen complications and delays, and maybe also a bit of collateral damage to other projects. That's nothing to worry about though, it'll all be resolved once the new thing is released, right? But slowly, stealthily at first, then more and more noticeably it happens: reality strikes back. The project takes way longer than expected, you realize that the new solution is somewhat incompatible with your existing business or infrastructure, the customers don't love the new project as much as you thought they would - there are many ways this can play out.
Eventually you may find yourself confronted with a bad case of second-system syndrome. Resource conflicts arise. Discussions now often revolve around the two opposing poles: "We need to put more resources into it" vs "We should stop this thing right now". But escalation of commitment is a powerful force, so the project gets dragged on for a few more months before it is finally cut down drastically or stopped altogether. The company may even see it through to the end, only to find that the results are not at all what everyone expected. By this time the white knight has often moved on to new endeavors already, and you're back to square one - only that the situation is even messier than before.
Many engineers are easily seduced by the prospect of working on a greenfield project that will allow them to 'do it right from the start'.
I have seen this pattern (with some variations) play out often enough to believe that it is a common fallacy. And to be clear: I strongly believe that following an imperfect plan is still much better than analysis paralysis, constantly changing the strategy or simply working without a big picture at all. I also believe that sometimes you have to take risks to succeed. But just like in politics and pretty much everything else in life: I think one is well advised to be very skeptical about seemingly simple and straightforward solutions to complex problems. And this advice goes out to everyone, not just senior management. Many engineers are easily seduced by the prospect of working on a greenfield project that will allow them to 'do it right from the start'. The same seems to be true of many designers. Being the product manager who saves the company with a bold strike is obviously an attractive vision as well, as is the promise that many problems with your own product will be solved if you just support your PM colleague's new initiative for a while.
I suggest that, whenever you're presented with such a master plan, make sure you don't forget your professionalism and do your due diligence. Make the underlying assumptions explicit and try to verify them. Ensure that everyone understands them. Create opportunities for critical opinions to be heard and addressed. Document these. Think big and work small4. Create as many checkpoints as possible that allow you to re-evaluate the viability of the plan and adjust course if necessary. That may not feel as good as 'just doing it', but I think it will help you a lot in the long run. It certainly would have helped me in several cases.
TL;DR: Be very skeptical of seemingly simple solutions to complex problems. Those are very rare. Be careful not to fall into the storytelling trap or become a victim of confirmation bias and do reality checks on your plans regularly.
Winters et al: “Software Engineering at Google”, O'Reilly 2020 (Chapter 2)
To be clear, I’m not primarily talking about senior management here. I’ve observed this pattern mainly in the mid-level of the org.
