I failed to transform the enterprise

A lesson I learned the hard way over the last years, a story of increasing successes, ending abruptly.

Prologue

It's been exactly 5 years since I was hired to help "digitalize" an enterprise travel company, give or take a few days. Some "Agile Coaching" happened in the months before, so existing leadership had a rough idea that they will need a few of those "developers" in-house, collaborating directly with business experts, to start replacing their (really) dated online services.

Upto this day, every bit of technology was bought from vendors, and than massively customized by additional external implementation partners to fit into the specific business niche and the existing zoo of systems accumulated over decades. The amount of customization by far exceeded the initial offering of the services, and had to be done within the constraints of each individual technology stack.

This accumulated in hundreds of partially overlapping systems, wired together with brittle ad-hoc integrations. Dozens of frontend systems just to assemble the current website, and critical dependencies on external partners where one single dispute could collapse the house of cards.

The Beginning

So, this was the situation I found myself in on my first day, together with few other developers. We were tasked to re-implement the most critical piece of the website: the actual booking funnel. Not only should it be more reliable, but also customized for the specific business needs and providing a superior UX to increase sales. An achievable goal, just enough manpower to actually get this done, and a clear success metric. So far so good.

The problem: to achieve the desired level of UX, we needed to convince stakeholders that a single-page-app approach would be needed, including some APIs to fuel them. Both things were completely unheard of internally, but after several emotional meetings we got the required "go ahead.". To shortcut discussions, I built the (working!) prototype by myself, since seeing is believing.

The next months was an intense learning and coding period within a crossfunctional team and finally we delivered exactly what was expected: the new booking frontend. A/B-tests showed a significant increase in online conversion rates, translating into millions of dollars per year.

Gaining Momentum

Because this specific project was a runaway success internally, we were given additional topics to "modernize", like all other parts of the eCommerce funnel (product listings, ...). Additional developers were hired and even supplied from external partners to work on-site. Major products got shipped, like the B2B frontends or a new HR career portal. My job was to coordinate development efforts, leading by hands-on prototyping and coaching team members to do the same.

We had track records of successes, grew in manpower, got more/bigger tasks to tackle.

But there also was a downside, clearly visible for me: some existing management and a few external partners got increasingly worried about our success. It was obvious for everyone that the "old way" of treating IT as a costly bunch of blue-collar drones didn't work as well as the "agile things" we did. Some traditional managers lost influence and power, things they tend to hate. With every feature we shipped, this situation got worse.

Success Peak That Wasn't

To settle these growing internal disputes, I started talks with upper leadership to move all the people currently working on building/improving products internally into a dedicated business unit, so that internal hierarchies and reporting lines are clearly defined and integrated into the enterprise. I even suggested who the new VP could be, a charismatic domain expert who even had leadership roles in the company in the past and was part of the initial "agile team" that started it all.

Finally, all these things happened, exactly like that. One could think that this is where someone has "won" the battle, now everything else would be just smooth sailing towards a bright future, just discussions about roadmaps and teambuilding events left.

At least, my thoughts at this time were something like this. This is what peak achievement must be like. But in reality, this was where I failed badly with every single decision.

Establishing a dedicated business unit from scratch is hard. Being a "digital" business unit is even harder, since it is basically an overlap with all other units, without having a clearly owned thing on its own. This screams identity crisis.

Other executives were afraid of the new VP. Not because he was inherently evil, quite the contrary, but the pure existence of his new role meant intermeddling with their own topics to them. Something executives tend to dislike.

Without any real support from any other executive, our poor VP was tasked the impossible: digitalize the enterprise... but without any decision making power outside of his own department. How would you replace old things other use, when they really do not want your help (or even are afraid to lose their job due to "automatization")?

Even the very first weeks mirrored exactly what went wrong. It was effectively impossible to even formulate something like an internal mission statement. Not only was the distant target unclear ("digitalize!"), but also how to achieve anything worthwile when key executives are hostile.

Many months of internal disputes within our unit followed, bitterness about the situation and how to (not) solve it.

We stopped delivering results. Just right after we got our hard-won autonomy.

This kind of paradoxic situation breaks even the best executive over time: when you just can't say what you intend to do or what has been done, because there is nothing, and no easy way to change this. Our VP gave up internally.

Good fodder for all the management folks and external vendors that opposed us. Schadenfreude, qualms, jokes. On our side, some people slowly became arrogant smartypants. This was like adding kerosene to a fire.

In retrospective, the plan I pitched could not work. This was clearly my fault, since key people trusted me based on previous successes that this next step would work. Executives did what actually is the best possible way of leading such changes: trusting internal experts, giving them the ressources to succeed, and, listen to their suggestions for the next steps and finally implement them.

There is no one other to blame than myself here, I vastly underestimated the social aspects of such changes, especially separation of development from business divisions. I should have seen this, but probably I already was too arrogant from the successes leading to the plan.

A Bumpy Deathmarch

Then 2020 finally there was this pandemic thing hitting us, the whole enterprise (travel business!) instantly had to cut expenses as much as possible, since all income streams became zero abruptly. Guess what a good axing target was? Yupp.

Only a few key developers were integrated into the old IT department to keep things running, everyone else was gone with short notice. The writing was on the wall, but I just could not find a way to prevent the meltdown.

But there was still hope. While everyone was paralyzed with the identity crisis, right after the creation of our new department, I actually used the new autonomy for good. Until then, we only built frontends and "some APIs". But we also had to firefight daily because some backend system was broken again, or there were network failures (hundreds of system ad-hoc connected with each other, remember?).

So I developed a real fullstack layer to shield us (and the customers) from systemic failures. As it turns out, this new tech stack was also much better to develop some complex frontends with (Phoenix LiveView). So while many team members were arguing about theoretical things, I onboarded important developers on to the new system and they were productive refactoring the old things we produced into the new technology stack.

Executives went full enrage when they heard about what we did there. A system popping up out of nowhere, without explicit business case and so on. But before they could capitalize on this sacrilege on its fullest, the pandemic shut us all down and things became silent.

But during these hard months, a much smaller team managed to get urgent features done to keep the business (mostly through the website) alive, things that required many more people before. And then the obvious thing happened: a complete breakdown of all important systems, like a house of cards. And: most vendors were not working due to lockdowns or other important issues during the pandemic.

The only system not affected by the complete meltdown was: our new one, which was designed to withstand outages from backend systems below. We were tasked to implement emergency-functionalities to cover basic business needs immediately with the few resources we still had, because there were zero other options available. And we did, in record time no one deemed possible. It is still a mistery for all executives how we managed to pull this off (it's actually simple).

But this was an intense time of measurable successes again. So, we teeth-gnashingly got a minimal increase in development capacity again. With the new motivation and capacity we tried to be a "helpful IT", and solved acute problems for people all over the enterprise, regardless of their business unit. Goodwill towards our small dev team within IT increased substantially.

The Abrupt End

While the upwards trend continued, I thought about how to prevent such wide system failures in the future. There needed to be some API Gateway instead of everything talking with everything, and there must be some central glue code within the AGW to map the customized interfaces of each system to something reusable. Oh, and we`re at it, this is the perfect situation to consolidate redundant systems and components, since less systems have less connections and finally less error potentials.

But nobody asked for that. IT was still dutifully recovering the hundreds of systems with the help of external partners, which took months. Essentially Building up the exact same house of cards again. And we talked about how to make the situation actually better.

Even worse, I built a working prototype that could do just that: Connecting external systems securely, at scale, but also could handle very custom things on it's own.

Just pitching the idea of it, with noting that it's already useable and we could start right now to fix the mess, was just too much. Why?

Because no executive had decided that we should do those things. It didn't matter that:

• we often did such things after working hours, in our free time.
• they wouldn't even come up with such ideas on thir own (all nontechnical people).
• we actually solved business problems and earned/protected cash flows.
• middle management often exactly knew what we we`re doing and helped us.
• we delivered much better final results than the external partners.
• we proved loyalty to the enterprise even when we were hated and degraded.
• there are sometimes no other options than doing it as we did.

Once the business started to ramp up again (albeit slowly), executive action seemed neccessary.

Big solutions to the big problems. Exactly what consultancies sell. It was obvious to them that having any inhouse-developers is the biggest business risk at all. They should just buy the biggest standard solution available (cough adobe cough) and merge everything possible into that. This can be done with external partners that are, by definition, much more reliable than the own employees, also much cheaper and there is just top-down management with budgets and no more disturbance.

This is something executives can actively decide. And so they did. And even actively declined that we remaining developers helped kickstart the platform, knowing how to get things running between the existing landscapes.

We lost.

Closing Thoughts

In retrospective, I made several critical mistakes.

• It is problematic to have real success without including all people affected by this. At least when you might depend on them later on.
• It is a very bad idea to separate development from business units. At least without giving them enough power to move things.
• Without toplevel commitment, best efforts and even results don't matter. Taylorism-style top-down managers are still there, and still powerful.
• When there just is no interest in actually improving things, I should leave it at that. Somehow deciding bad things or nothing at all is still more acceptable than just fixing the problem.
• Even with the best intents we had, when new systems are created, this must not surprise management, period. There might be a bigger plan we don't know about which just got at risk. Or budget planning got jeopardized and people get in trouble.
• Always stay humble, no matter what stunt you managed to pull off. The very next step could fail independently of previous successes, but chances are even higher when ignorance creeps in during planning.

So, here it is: my fuckups learnings of the last years. On the plus side, I improved a lot on my technology hard skills and got truly deep into LEAN philosophy of shipping results while mentoring dozens of colleagues.

Long Story Short

I am open for new opportunities, preferentially hybrid.

If your office is physically based near Vancouver, BC then that would be a major plus for me, since immigrating there is a distant life goal.

Be it frontend, backend, team lead, automatization: I just want to help teams achieve their goals, ideally without playing political games again.

update 2023: I am now very happy in Hamburg after relocating there, but thank you all for the overwhelming responses!