Jeet Ka Gyan

Things that make us slow; thoughts on engineering productivity

Jitendra Agrawal -
  • "We started small, and when we had 15 engineers, the company had a different vibe. We worked hard; things moved fast. Most stuff got delivered much faster than I expected. But now we are at size 200 and nowhere as fast as we were at 15." I have often heard this from friends, founders, engineers, and product managers. I have experienced this slowdown myself.
  • While I want to write about the vibe and perception of speed in the future, I will focus on the mechanical aspects here. As we add more engineers to the team, productivity doesn't precisely scale linearly. To take a parallel from distributed computing, the work done by computers reduced as I added more nodes to my cluster (this is called efficiency)
  • Here are some things that I have noticed with my teams:
  • Lack of context and documentation
    • In a super-small team, everyone knows everything; the product and code are small. As teams grow, this tribal knowledge becomes a problem if not documented.
  • Lack of common goal understanding
    • In a small team, goals are well understood, and everyone moves in the same direction. As teams grow, even if everything is giving their best efforts, they may not be trying to move in the correct direction. It's like some rowers in a boat race row in the opposite direction.
  • Large code
    • Large code comes with its challenges. Developers and PMs can't remember all the behavior/functionality.
    • Compilation times increase as code becomes large but is not modular.
    • Code review cycles become bigger
    • Confidence in code goes down unless unit test and automation discipline is maintained
    • Merge conflicts themselves can become tricky and time-consuming.
  • Impact area and iterations
    • I think about this as degrees of freedom as joins increase or the number of possible states as variables increase. If you have a single variable with boolean values, you need to test for two possible values. If you have two variables with boolean values, I need to test for four combinations. With three boolean variables, I am looking at eight different combinations. The system becomes increasingly complex as we add more variables and possible values.
    • For large software, the impact area for even minor changes may sometimes be large, and hence, occasionally, multiple iterations are required to implement this small change properly.
  • Approvals
    • Everyone doesn't have access to everything—an approval process is needed for many resources, 'and though it's required,' it slows everyone down.
  • Dependencies
    • For the larger team, the whole organization is only as fast as the slowest team if everyone else is dependent on them
    • A good organizational structure limiting daily team interaction can work wonders here.
  • Synchronization
    • Multi-threaded applications spend a lot of time synchronizing (or locking). The same is true for teams that grow bigger.
    • This was also the case for distributed applications. Synchronization and communication reduce efficiency.
  • Debt
    • A lot of debt is accumulated because of various reasons
    • Teams take "known" debt because of timelines.
    • Unknown debt that only becomes obvious as business requirements change
  • Skill levels
    • In some startups, the first few engineers are co-founders. Sometimes, the founding engineers have worked with founders before. These teams see a major slowdown as they expand.
    • The primary reasons for slowing down are different skill levels and familiarity with the exact tech stack.
  • Motivation levels
    • While it's a controversial take, everyone in a startup is not as motivated as the founders or the first few team members. The risk-to-reward ratio for early joiners and late joiners is different. Reasons for joining a startup pre-PMF and post-PMF are different.
  • MLP and not MVP
    • A mentor used the phrase minimum lovable product (MLP) in a discussion, and I think this gives a more emotional meaning to minimum viable product (MVP). We all know that the definition of MVP changes based on the size of the business. An A/B test's success depends not only on the basic hypothesis about customer behavior but also on how well the variant experience is implemented.
    • As companies grow, they can not ship MVPs they are ashamed of. Instead, they need to put in more effort to deliver MLPs.
  • My approach to handling these obvious issues is the following:
    • 1. Find when one or more of these issues have started affecting your teams badly. I have often used engineering metrics (DORA and others), team 1-on-1s, surveys, etc. to find when each of these issues starts affecting my teams.
    • 2. Set up tooling and systems to mitigate some of the issues.
    • 3. Big teams can't match small team velocities, but knowing about these reasons allows me to set systems and culture accordingly, which also helps us manage our stress levels for team members who still yearn to see the velocity of yesteryears.
  • I believe that people running in different directions and having too many interdependent teams are the most significant pieces of the puzzle that need to be solved and can have a considerable impact. In a future blog post, I will write about some solutions for these issues.