Strategy, Alignment, and Organizational Entropy

A given “tops-down” or “bottoms-up” strategy is already a failure

DALL-E

Building an effective strategy is hard for large engineering teams. Leaders up and down the org chart often argue over whether the strategy development process should be “tops down” or “bottoms up”. The reality is that any strategy produced by just one side is likely to be ineffective, because both sides are accountable for completely different sets of responsibilities. A tops-down strategy is out of touch. A bottoms-up strategy is a science project. Sometimes, the strategy exists only in some leaders’ heads.

Hundreds or thousands of people need to know what they’re supposed to be doing; can we be certain they’re working on the right things?

There was a period of time at Twitter when we did not have an organized and effective cascading strategy process within Engineering. Company leadership broadly declared the direction the product and company should go, but Engineering teams were not mobilized against those directives in a consistent way. This was true for teams reporting to first-level managers, and true for entire orgs reporting to VPs. Directed and urgent progress was made primarily through two means:

• Direct executive attention and pressure on a highly-specific program or set of isolated deliverables, and/or:
• Code Reds, which were the periodic forceable movement of people, teams, projects and technology to support those goals

This operational style caused a number of problems:

• Teams left out of the small set of focal projects had no clear direction, and no objective method to understand the relative importance of their work or cross-team dependencies, leading to constant argument and escalation (true again for VP-sized teams)
• Teams included within the set of focal projects faced morale issues from team shuffling, executive interference and micromanagement, loss of autonomy, and lack of executive accountability when those projects didn’t move the needle for the company
• Leaders committed to various technically infeasible outcomes on planning boundaries which resulted in both throwaway strategies and loss of trust with the org
• The process of deciding what to roadmap was a high-effort, often haphazard process for the eng teams, based on either localized priorities or educated guesses
• The platform and infrastructure orgs did not understand how to most effectively support the longer-term needs and goals of the product teams
• It was unclear how to organize against multi-year outcomes

From 2019–2022 Twitter refined a fairly complex cascading strategy and prioritization system, which combined high-level direction with low-level details and tradeoffs from the teams. It’s hard to say whether this resulted in objectively better outcomes for the business, but it did result in the following improvements:

• All teams understood, and could help frame their role in supporting the long-term objectives of the business
• Executive attention gradually began to focus on the plan and its outcomes rather than its implementation
• The understanding and reconciliation of relative project priorities and timelines were backed by objective, public strategy and stack ranking rather than hand-waving or escalation
• The path from roadmap to strategic outcome became a more linear and stable process, reducing planning entropy across teams

In short, teams felt included, leadership felt informed, Engineering understood its obligations, and planning overhead was bounded.

Without alignment, autonomy is squandered. A 1 degree deviation in course of a rocket heading to the sun means it will miss the sun by 1.2 million miles. A lack of alignment compounds quickly.

Jean-Michel Lemieux: Alignment > Autonomy

The Principal-Agent Problem and Agency Costs

Often overlooked in strategy formulation is the concept of information asymmetry, from contract theory. This occurs when one party involved in an agreement (say, a team roadmap approval) possesses more — or better — information than the other. This can be bidirectionally true for both leadership and engineering teams, as reluctant as they may be to admit it.

This asymmetry feeds a moral hazard called the Principal-Agent Problem. In an organizational setting, the principal (leadership) delegates work to the agent (engineering teams). However, due to information gaps, the agent makes decisions that aren’t aligned with the principal’s objectives, and the principal likely won’t know the difference.

Alternately, leadership may be working with incomplete or unsurfaced information from the engineering teams, which results in suboptimal or under-informed decisions. For example, a lack of information about existing technical capabilities of a company’s technology stack may result in either an unachievable, or under-ambitious business strategy.

This misalignment creates what’s known as agency costs— the cost of ensuring that agents’ actions align with the principals’ goals. These costs cascade; middle management can act both as principals and agents, and costs can manifest in practice through roadmap churn, micromanagement, wasted resources, loss of team autonomy, or failed objectives.

This overhead isn’t avoidable, but it can be managed.

Strategic Processes for Alignment of Long-Term Outcomes

A well-designed strategy development process works to minimize these agency costs, as an outcome. The goal should always be to arrive at the best course of action for the organization at any point in time, with the best possible information and lowest possible effort.

A good strategy should be clear, transparent, defensible, and able to steer cascading levels of strategy. With the appropriate accountability hooks, it ensures both leadership and teams on the ground operate with better long-term alignment, with less thrashy overhead. Teams understand the direction; leadership better understands the costs, technical constraints and possibilities.

A good strategy represents the common, scaled interface layer between leadership and the teams expected to do the work. A good strategy process ensures this layer is always in place and effective.

Desiderata:

1. Clarity, Transparency and Accountability: Leadership must articulate the long-term strategic vision and goals clearly in written form, ensuring that all levels of the organization understand the broader objectives and how they map into them. The intentional continuance or deprioritization of earlier strategic efforts to support the updated goals should be clearly called out and justified.
2. Cascading Strategy: The strategy should cascade down the organizational layers, becoming more specific and actionable at each level. This process helps translate high-level goals into practical, team-specific objectives.
3. Feedback Loops: Mechanisms which ensure insights, challenges and realities from the ground level are communicated back to leadership and their plans. This bidirecetional information flow helps in continuous strategy refinement, the inclusion of technical experts in organizational approaches, and closing of information gaps.
4. Regular Course Correction: Periodic but consistent check-ins and strategy reviews help ensure both the principal’s vision and the agents’ execution stay aligned.

There are probably an infinite number of reasons why written strategy outperforms implicit strategy, but a few that I’ve seen matter in particularly important ways are:

- You can get feedback on it

- You can make updates to it

- You can explain why you made updates to it!

- You can clarify points of confusion

- Nuance is important, and almost impossible in unwritten strategy

- It democratizes technical decision making beyond a small caste of architects

- You can hold people accountable for not following it

- New hires can learn proactively rather than “fail their way into learning”

Will Larson: Solving the Engineering Strategy Crisis

Agency costs can’t be eliminated, and even if they could, it’s probably not a desirable outcome. It would mean the principals are involved in all work and every decision. But what’s important is that the cost is managed, and kept sub-linear relative to the size of the organization.

Clear alignment, reached quickly and consistently, compounds in value. This is where the benefit of cascading org strategy comes in — any team on the ground should be able to map its local strategy into a broader and credible strategy for the Engineering org or the business. If it can’t, then the entropy in the organization is too high.