Two Development Team Configurations I Lobby Against

Product management doesn’t run Engineering; Engineering runs Engineering. And at least in public, Engineering and Product leadership need to be shoulder-to-shoulder, actively supporting each other at every turn.

But there are some engineering team configurations that I see as problematic. So behind closed doors and where it won’t create political/organizational issues, I lobby my engineering counterparts away from:

• Dedicated bug-fixing teams (usually proposed by Engineering)
• Special Projects or ‘Overflow’ Teams (usually proposed by Sales/Go-To-Market)

Ultimately not my call, but worth spending some social capital on. Here’s my thinking.

[1] Dedicated Bug Fixing Teams

Sometimes there’s a push to create development teams specifically to close out bugs and defects, especially after frequent outages or to address long-term system neglect. This is anchored on an assumption that existing development teams can (should) continue to do their regular work while a new team can clean up what’s broken. (I mostly see this proposed by technical leaders who come from non-revenue development organizations — those building software for internal use by companies in another business such as banking or airlines).

Some of my concerns:

• This arrangement creates an incentive for ‘core’ Engineering teams to ignore issues or kick the can down the road. Knowing that someone else will clean up our work can drive bad behaviors. In contrast, I believe in cradle-to-grave ownership: “you build it, you break it, you fix it.” Code creators own quality.*
• Bug teams are generally treated as second-class citizens: ignored or under-appreciated by the ‘real’ engineering teams doing ‘more important’ work. (This is doubly true for offshore or remote teams.) That leads to morale issues, lack of context/support from bug creators, and turnover. Transfers into the Bug Team can be viewed as a demotion, and few folks get transferred back onto new feature/function.
• The original developers are usually 5x faster at finding and fixing their own issues because they know what was written, where it lives, and why they thought it might be correct. A bug team that works across a product’s whole surface area won’t build the deep knowledge about any one part — or be included in design/architecture discussions and decisions.
• Bug fixers work in the same codeline as core engineers. We need excellent communication so that one group doesn’t step on the other’s changes.
• Bug or feature? Not everything that’s reported as a bug is a bug — or should be fixed. Customers and support teams often identify things that don’t work the way they like… tagged as bugs even if the system is behaving as intended. Product managers and engineers have long arguments about whether (and how) to fix something that doesn’t excite users. But bug teams tend to be shut out of these conversations: they hear “fix as many as you can, and leave us alone.” Naively counting closed tickets as a success metric can encourage fixing what’s not actually broken.

What’s an alternative? If quality is suffering or we’ve shorted repairs for a few quarters of “just this once, let’s work only on new features,” then we should own up to that. Pull the red handle, shift the mix of our existing teams from mostly new features to mostly quality and infrastructure repairs for a month (or two or six), then rebalance each team with a sustainable blend of shiny new stuff and unsexy-but-stay-in-business repairs.

If we’re just noticing a lack of automated tests, let’s allocate 10%-20% of each sprint to test creation until we catch up. And insist that we build tests for every new capability, at least by the time we’re checking it in, if not before we write the code (that is, test-driven development, aka TDD).
Hiring an outside team for a little while to build out test coverage might be smart. But fixing what’s broken still belongs to whoever broke it.

[2] Overflow or Special Project Teams

Overflow or Special Project Teams tend to be pushed from the Sales/Go-To-Market side of the organization out of general frustration with perceived Engineering (and Product) throughput or velocity problems. “We’re simply not getting enough done, and we can put more revenue on the board by addressing unmet demands from our biggest customers. We need an Engineering team directly responsible to Sales.” There’s the sense that we’re just a handful of features away from hitting quota, and that Engineering needs to be pushed harder. Priorities are top-down, needed this quarter, driven by the size of the overall deal, and based on Sales’ notes of what clients said they need.

Some of my concerns:

• Demand is infinite. As soon as our newly formed Overflow or Special Project Team takes on the top 2 projects for our 2 largest customers, there are 60 more customers and 120 more projects right behind them. And when word gets out that we have an Overflow Team, another 1000 requests come in. In the enterprise software space, every of our customers and execs will always have another dozen unmet needs, amplified by our enterprise sales teams. It’s turtles all the way down. So our first Overflow Team is immediately booked through 2024 and there’s demand for a second (third, fourth, fifth) team. We can accidentally (but rapidly!) become a professional services/custom consulting company.
• There’s no ongoing support model. When our Overflowers finish a custom integration with a client’s fascinating homegrown ERP system, we move them on to the next project. And the next. But all software inevitably breaks or needs updates or uses some soon-to-deprecated third party products. The Overflow model usually assumes one-time-projects: our smart folks move on to the next new thing, and no one maintains code or detailed information about the last one. Outages and political hot potatoes get tossed back to core Engineering, which has neither context nor time. Queue the finger-pointing.
• Many of these projects are off strategy, not well spec’d, already rejected by Product/Engineering as bad ideas, or incrementally pull us into a problem space we’ve carefully avoided. Current quarter deals override longer-term concerns. Overflow Teams tend to work on the biggest revenue deals, not on the ones that fit our Ideal Customer Profile or product strategy or overall corporate priorities. High likelihood that we’ll regret doing this work after the Overflow Team has moved on.
• There’s no rigorous testing. Unlike core Engineering, which (should have) invested in code reviews and automated test harnesses and code coverage and nightly regression runs, Overflow Teams usually skip repeatable/automated testing in favor of once-through manual smoke tests and one-time customer acceptance. (Not their fault: they are beaten up hourly for soonest-possible delivery, and quickly moved onto the next hot-hot item.)
• It’s hard to build expertise. Overflow Team members bounce from one issue/project to the next: report writing today, ERP API integrations tomorrow, sort algorithms next week. That’s great training for field sales engineers or support teams, but shallow for application developers.

What’s a partial alternative? With good intuition and planning, we can architect our products to make specific kinds of overflow easier to handle.

For example, we anticipate that lots of our enterprise customers will need custom connectors to their unique data pipelines. So we intentionally design a data pipeline API as part of our core product (with full market analysis, specs, testing, sample code, support, training…) and open that up to paying customers or third-party developers. Then they can build as many custom connectors as the market wants… through a well-supported interface. We’ve gotten out of our own way.

Or partner with a real professional services firm which explicitly sells the time and expertise of its technical staff. When our customers need something we won’t (or shouldn’t) do, we refer them outside. Customers are welcome to pay a third party for whatever special wonderfulness they desire — plus an annual support contract — built on our existing tools and published interfaces.

Sound Byte

There’s nothing more valuable than the time and focus of our engineering teams. So let’s share our strongly-held viewpoints about how to make their days better — be their champions. While we’re not in charge… as product leaders and product managers, we care deeply about our teams’ happiness, productivity, and whether they are organized to succeed.

___________

Originally posted to https://www.mironov.com/team-configs/

* Thanks to Ron Lichty for this reminder (and so many other great ideas).