I. Session Takeaways
Date: April 2024
Speaker Overview
Jean-Denis
- Jean-Denis joined Plaid when it had 20 engineers and grew the team to about 400 engineers.
- He joined as Head of Engineering. Initial responsibility was recruiting and EPD effectiveness.
- Over time, he accumulated responsibility. All of the GMs eventually reported to him (100% of revenue)
Amjad
- Started Replit in 2016. Currently around 120 people with 60 to 70 being engineering
- They run on a big scale with millions of users, so they do a lot of work to keep the service up and running
- He’s run engineering since 2016. He’s in process of handing off EPD to a internal promote at Replit
Engineering will naturally want to slow down as communication costs increase
- As you scale, there is an entropy-like force you will push against. When you are small, everyone is in a room and everyone understands the entire stack. When you have 100 engineers, new people only know what they work on and you get hit with massive communication costs. You must mitigate these costs to maintain speed.
Jean-Denis on why shipping velocity decreases as you scale
“Any business that is scaling there is an entropy-like force you are pushing against. Things will want to get slow. You need to have some meta way to identify what is slowing you down. Often it is not intuitive. When you are a team of 15 eng everyone knows every part of the stack… but when you are 100 eng the only people that know the entire stack are the early people and the new people who work on one part of the stack dont know it. You then hit a ton of communication costs.“
- Speed is relative to your competition and to your engineering goals. As an example, Plaid hired engineers that were used to working at social media companies. When these people joined, they believed Plaid was slow. The reality is Plaid had to go slower because they operated with financial institutions and sensitive data. Relative to their peers (Finicity, etc), they were 10x faster. This is what matters.
- Taste and slow decision making are often the root causes of slowness, not your CI/CD pipeline. Teams spend weeks debating what to build versus executing. The ultimate version of “taste” at scale is Parker Conrad at Rippling. At 3,000 people, he’s very deep in the product and helps reduce decision-making entropy. Technical founders can stay in the details for longer than you think.
How to ship fast as you scale
- There is no quick way to address speed. It requires consistent investment. We outline tactics/ideas below.
Creating a culture that rewards speed
- If you want to be fast, it needs to be part of your culture. You need to talk about speed, reward people in the organization, and push champions in the organization to take responsibility.
Tactics to embed speed in the culture
- Talk about speed in every relevant 1-1 and in all-hands. Plaid focused on “ambition, urgency, and scrappiness”
- Promote people who demonstrate speed consistently. Show the organization you reward it
- Take actions that show speed matters. For example, Plaid had focus days to enable work to get done. On these days, they’d run a script to remove meetings from calendars. This reiterated that speed mattered to the company
System architecture matters
- Making good architectural decisions is the job of the CTO. Bad architectural decisions compound. You need to revisit them because they inform how your team works and communicates when making product decisions.
- When Replit first built the AI team, they made the decision that AI would be a platform team that built AI tools for other teams in the organization. This had two advantages.
- 1) Replit created a robust set of AI primitives that product teams at the company could build on top of.
- 2) It reduced communication costs. Teams can leverage AI primitives without needing to talk to the team
Amjad on the importance of good architecture decisions and primitives
“System architecture is very important. If you have systems that have good primitives and are programmable. Where people get their work done and are documented quickly without needing to reach out to someone. This is the job of the CTO, you have to make the large architectural decisions…those are not just technology decisions but how your organizations works and communicates“
Come up with meta processes to identify what is slowing you down
- Design processes to identify what is slowing you down and how to act on it.
- The faster you can make the identification and resolution loop, the better your engineering organization will run.
- Plaid used a weekly meeting called the “Key Project Meeting” to understand structural issues driving slowness
Jean-Denis on the key project check in meeting
“I used to have a meeting called the key project check in. Every quarter I would pick 3-5 of projects and they would come in weekly and would tell me how the project was doing. The format was this is the goal, the current status, this is what we recently shipped, and the last, which is the only one we discussed, is what’s blocking. The point of the meeting is not talk about product, not about customers, but about what is blocking”
Create abstractions to reduce communication overhead
- As you scale, communication costs increase. One of the big reasons communication costs increase is engineers no longer understand the full-stack, and are only exposed to what they are working on. Below are tactics Plaid and Replit use to reduce communication costs and maintain speed.
Tactic 1: Move high-performers so more of the team understands the stack
- Plaid forced engineers to switch teams regularly to learn more about the stack
- They would aim to move strong performers with product sense to learn other parts of the business
- Managers are not incentivized to move their best people, so Jean-Denis would force this to happen
- If someone great was on a team for >2 years, he would try to move them
Jean-Denis on moving engineers around to maintain knowledge of the stack
“We didn’t move everyone. We just wanted a higher percentage of people who had touched different parts of the stacks. It was part of a code review every 6 months. We would ask the question especially for really strong performers with really strong product sense- does it make sense to expose this person to other parts of the business. Managers are not always incentivized to move their best people and the best people are not always aligned with the right projects. I would basically force hands to get the movement to happen “
Tactic 2: Product engineering versus front end engineers to reduce communication overhead
- Replit’s product is designed as several products on top of a set of platform primitives
- Instead of employing front-end engineers, Replit hires product engineers that are full-stack
- This means product engineers can sprint really fast without much guidance or assistance from other teams
- This only works if you have a solid set of platforms/primitives that product teams can leverage
Amjad on creating abstractions to maintain speed
“We don't have front end engineers. We have product engineers, and we have lots of platforms. We have product engineering that sits on top of all these platforms, and product engineering is supposed to be full stack. So they have to write server code and client side code. And they can sprint really fast in order to build a product. Now, you know, that slows down when the primitives are not there, or there's bugs in them that they have to go talk to the team involved. But again, your mission is to reduce crossing the abstraction boundary too much.”
Scale “taste” through promotions
- Taste is the ability to make good decisions that lead to great outcomes for your product
- Eventually, you will need to trust other leaders to make decisions within the product and engineering organization
- Identify people in the organization that have a track record of making the right calls and promote/reward them
- At Plaid, this sometimes would “break” seniority. There were several examples where they promoted people to “Staff Engineer” with just a few years of experience but a track record of getting things done in the business
Jean-Denis on scaling “taste”
“When I was younger I really believed in interviewing the perfect people. I have this square shaped hole in my org. Lets go fill it. What I found out overtime, you can have the best interview process in the world, you are not getting world class people out of it every time. You can’t evaluate in 10-15 hours if someone is really within your culture is going to crush. I identify people in the company who have a track record of making the right calls and the right judgment. Especially on product and on eng. I reward and promote those people. Sometimes that means a new grad after 3 years will be a staff engineer and they clearly have gaps but they have a track record of getting things done in the business“
- Don't just keep track of people shipping things, when rewarding people for taste. Keep track of the product and if people are still using it 6 months later. One way to do this is to put a note to review what a person has shipped 6 months ago at their next performance review. This will help you ensure you reward people that execute well.
Jean-Denis on tracking projects after launch
"Something will ship on time, and it will meet your business goals. But then it turns out the abstraction or the API is terrible. And it slows your engineering team down for like two years. Don't just reward shipping. Six months afterward, put a reminder for the next person to review the things that they shipped. Are they still working?”
Metrics in Engineering
- Jean-Denis and Amjad have different perspectives on metrics. You need to pick what is authentic to you.
Metrics at Replit
- Replit is a metrics focused organization. They use metrics to track shipping speed and performance.
- Amjad has a dashboard that tracks pull requests, pull requests per contributor, and pull requests by day
- They use this dashboard to identify trends and to implement fixes to improve or maintain shipping speed
Example of how Amjad uses the dashboard
- Amajd recently noticed pull requests declining relative to historical trends
- He looked into the data and saw that pull requests on Friday were down over historical periods
- This helped him identify the issue. They used to have meetings on Monday and all hands on Friday. In the last few months, they moved all hands to Thursday to accommodate scheduling. After this change, they noticed that Friday became “part of the weekend” and activity on Friday declined. They are now changing in-office days to Monday, Wednesday, and Friday to encourage work on Friday. They are also moving the meeting back to Friday.
Amjad on identifying trends with data
“There is a bit of a concerning trend. If you look at PRs by day of the week. You can tell Tuesday and Thursday are where a lot of the work is getting done. Monday we have a lot of individual meetings. We moved the Friday weekly meeting to Thursday and there is a concerning downward trend on Friday. We believe it's because Friday has become part of the weekend. We are now flipping office days from Tuesday and Thursday to Monday, Wednesday, and Friday and we are moving the meeting back to Friday.“
- Replit believes being prolific is important. Metrics are essential for identifying whether someone is prolific.
- They incorporate metrics into reviews. Each quarter, they ask managers to rank and score their reports.
- Metrics help managers be more objective. If you see someone you like produce very little, you are less likely to score them highly, especially if you need to defend this score in front of others.
- They leave it up to the managers for what metrics they want to show - typically PRs merged, interviews done.
- It is also crucial to understand the “story” behind the metrics. External factors sometimes drive performance (e.g., a family member is sick).
Metrics at Plaid
- Jean takes an extreme view. He just cares about the business impact and whether goals are hit. He doesn’t care if one line of code is shipped or if one thousand lines of code are shipped. He doesn’t focus on input metrics.
- He focuses on process metrics and optimizing the build and deployment process. In his view, this is where time gets eaten up and you can drive throughput in the engineering organization. Metrics he cares about:
Example metrics Jean-Denis tracks to optimize process
Build, test, deploy times
Look at your long-term graphs and try to get this under 5 minutes.
Code review SLA
Monitor whether teams are meeting code review SLAs. If code reviews are taking too long, this can impact velocity.
PRD Spec to PRD Sign-Off Time
This signals whether you have projects that are off-rails or if you are shifting to a consensus driven culture
Subjective Engineering Survey Responses
Ask your engineers what works well, what is slow, etc. This will help you identify issues that need to be addressed.
Interview/meeting times
They did regular meeting purges. During times of hyper growth, interview times become a disaster. They solved this by creating interview days for individual teams, so that less time was taken up by interviews and recruiting candidates.
Engineering Structure
- There is not a single “right” structure for building your engineering organization. Ask yourself what is wrong with the current structure and what are the problems I am trying to solve. This will inform organization structure and technical architecture decisions (e.g., monolith, microservices, etc).
Replit Engineering Structure
- Pod’s didn’t work for Replit. On a quarterly basis, they would spin up teams to focus on projects. These teams would typically include a few engineers and a designer. These operated like small companies within Replit.
- They found this process was “top-down” heavy and robbed people of autonomy. Everyone became focused on scope and teams lost ambition to go the extra mile. It created a scarcity mindset and reduced shipping speed.
Amjad on pods and how Replit is currently organized
“Our experience felt like a very top down heavy process. It robbed people from a lot of autonomy. People became very conscious of time and it became a scarcity thing. You would go into a pod and talk about goals and they would be constantly thinking of scope. Scope became this supreme thing, there wasn’t any ambition. Everyone was trying to hit the minimum viable product, goals, or tasks. The culture changed, Reppelit is generally ambitious, people are trying to go the extra mile. You introduce this thing that created this scarcity of mindset. That was very negative on our shipping velocity and the way we work in general. Today how we work is a mix of top down and bottom up.”
- They’ve shifted to a process of top-down and bottoms-up organization. The management team sets goals and metrics they want to move. Teams come up with individual projects and organize themselves into projects that they would be interested in working on. This functions similarly to pods, but people feel more autonomy as they help decide projects and team formation
- They begin the year with a “hack week” where Amjad sets the strategy. After that week, they look at what was built by teams and construct a quarterly or biannual roadmap. People like this because they have agency in the decisions even though there is top down direction.
- Amjad spends a lot of time setting goals and strategy. This is what drives the success of their roadmap process
Amjad on “hack weeks”
“We start the year with a hack week, I give the strategy and the goals. Everyone ideates and builds things. And then we look at what everyone builds, and we start constructing the roadmap. By the third week of January, we have some kind of roadmap for at least a quarter and maybe a half. And that's something that everyone had agency in. But then we apply some top down input such as these are the timelines and this is the roadmap. Then we put everything in linear and we start to track these things.”
Plaid Engineering Structure
- At 20 engineers, they would go into a room, pick a few projects, and put 3-4 people on 5 projects. Jean-Denis would give goals and the teams would work on these projects for the next month. Eventually, they realized they had mature services that needed long-term ownership.
- To drive long-term ownership, Plaid switched to long-lived, horizontal teams focused on different aspects of the product. These teams have their own roadmaps and customers. They generally replicate the things they used to do when they were small (e.g., break down into smaller teams to hit horizontal team goals).
Jean-Denis on Plaid’s team structure
“The story of Plaid for teams. When I joined, 20 engineers, no PMs, a couple designers. We would get in a room, we would argue, we would pick a few projects and put 3-4 people on 5 projects and that's what we would get done in the next month. I would give direction of what the goals were. At some point, we realized the problem is that every month different people roughly own different features but no one permanently owns key things that need to happen. We eventually organized into horizontal teams. These horizontal teams own a clear set of systems and customer problems. They are able to determine their own roadmap internally and most of those teams replicate the processes from when we were smaller.”
Accountability in Engineering
Plaid: Optimize for outcomes and whether teams hit “goal posts”
- Whenever possible tries to understand “goal posts” and measure performance against these outcomes
- If you don’t have a goal post, it is difficult. Jean-Denis relies on putting people with “taste” in abstract projects
- For these, they would set “ship it” dates and trust the person in charge to figure out the how to launch
Jean-Denis on tracking outcomes for undefined projects
“The more either undefined the outcome was or the harder it was to decide in the short term if something met its business goals, the more we trusted the judgment of putting the right person in place. We would set ship goals, like it would do these things by this date, and just trust their judgment as to the how. “
Replit: Optimize for performance, not outcomes.
- Amjad focuses on performance (e.g., the inputs, the work) and not the outcome. In his experience, the outcome flows naturally from good performance. He doesn’t like only looking at outcomes because outcomes can either be enhanced or corrupted by the real-world. He doesn’t want to tax someone for strategic errors at leadership levels or factors that are external to the company
- This is particularly true for Replit where they launch new products frequently. In their case, it can be hard to define the right metrics. Therefore, they try to learn frequently through user research to understand what’s working
Amjad on tracking outcomes, not performance
“You focus on the performance and not on the outcome, the outcome flows naturally from the performance. The world is fairly complex, you can have someone working on a project and that can be incredibly successful but from something totally out of your control… There are also ways in which the world dictates the outcome in a negative way…Outcomes tend to be corrupted by the real world or enhanced“
Hiring a VP of Engineering
Why hire a VP of Engineering?
- When post product market fit, having people who can look around corners is valuable
- Engineering or sales/GTM are the largest functions and typically break because they scale first
- In general, the responsibility of a VP are now the highest leverage responsibilities for founders:
- VPs help with EPD execution, accountability, goal setting, recruiting, culture
- Hiring a great VP of Engineering lets founders do only what they can do. This includes hiring execs with singular talent, understanding customers, running product and product strategy, business strategy, GTM, the next 0-1 big thing for the business. This also includes thinking through the details of the technical architecture.
- VPs are important but are not true multipliers for the business. They give founders back time for .5-1.4% equity
When to hire a VP of Engineering?
- 15-30 engineers. The search will take 6-18 months, so start conversations when you have 10-15 engineers.
What does a VP of Engineering do? How to figure out what you want?
- This largely depends on the company and the founder. As the founder, you need to figure out what you want to own and what you want to give up. As an example, Marcelo at Faire kept recruiting because he cared deeply about hiring and he wanted to own technical architecture decisions. He hired a VP of Engineering to manage people management and the engineering process.
- To figure out what you want, talk to VPs of Engineering who run 150 person teams. This will help you understand what you want/what to look for when hiring a candidate.
- A fail case is hiring a VP of Engineering that clashes with the founding team or that wants to own pieces of the system that the founders are not willing to give up. This upfront exploration process helps you avoid that.
The anti external profile(s)
- There are three anti-profiles outlined below.
Profile #1
- In Amjad’s opinion, there is a class of “mercenary” VP of Engineering that jump from company to company
- He has a preference for people that rise quickly within their companies versus the professional “VP of Eng”
Profile #2
- The person who had various jobs as an IC at large tech companies but never reached aleadership position
- Looks good, but they’ve never had the experience of building 0 to 1. You can’t afford them learning on the job
Profile #3
- Became a VP of Engineering at a small startup with 20-30 engineers. Startup scales, but fails before real scale.
- This candidate is appealing because they’ve managed at your scale, but they’ve never seen scale done well
- If you hire this person because they are smart, you need to provide coaching to help them scale with the company
The ideal external profile
Profile #1
- Joined as IC at 10 engineers
- Became manager via internal promotion at 30 engineers
- Worked closely with VP of Eng from 30 to 150 engineers
- Went from managing 5 people to 50 people at the company
- Before being at a startup was at a larger organization with a well-run engineering team
- Companies like Plaid have a lot of people with these trajectories
Profile #2
- Saw company go from 50 to 450 engineers as a manger
- Before that worked at a early stage startup as an IC and at a big organization as a manager
Jean-Denis on why his profile worked at Plaid
“One of the things that made me successful at Plaid was that I joined Dropbox at 100 engineers. I was number 101. I saw dropbox go from that to 600. I wasn’t in charge but I saw all the pain points that happened through that journey. When I could get the VP of Eng at Plaid, it was a huge step up for me, I was so excited to imprint my vision of engineering excellence somewhere. But, I had seen what could go wrong. I had a network of people of great ICs that I could recruit into a place, that were the kind of talent you would want to bring into a new company ”
Internal versus External
- Internal candidates can work. They understand your culture and have respect within the organization
- You need to support them with the right coaching, especially if they haven’t led large teams before
- Replit recently promoted someone to run engineering because they fit the culture and were exceptional
How to find a great VP of Engineering?
- It is difficult. There are a lot of these executives for hire that fall into the “mercenary bucket”
- The best VP’s of Engineering are only on the market for 1 or 2 stints - meaning they only do it once
- The best profile is to find someone 3x-5x your size who wants to step up into the VP role to prove themselves
Jean-Denis on finding a great VP
“There are some people that jump from company to company as senior director/VP who are good at their job but are not good for a 15-30 person engineering company. Those folks, the best ones, are only on the market for 1-2 stints as VP of engineering. I would count myself there, I was a director of Dropbox, and VP of Eng at Plaid, I don't want to do it again. The really great people will not be VPs again, and again, again. The best profile is to find someone at 3-5x your size that saw the growth you are going through even if they weren't VP but because they have seen all the ways things can break and be fixed.”
- Ask your network for referrals. Do the work. You can also work with search firms to supplement your network
- Search firms only have 30 profiles available at any given time. Don’t fully rely on search firms because of this.
- This also helps you build the muscle to know what is great and will help you evaluate candidates better
Jean-Denis on how to find your VP of Engineering Candidates
“There is no secret. You have to network. Go through every director at a firm and find people who worked at a couple up and coming companies and have the right amount of management scope and before that worked at well run bigger shops. You will find folks. You can definitely work with agencies. Rivera is very good, but the thing to keep in mind is that they have 30 people in their pocket who they know are looking. Initially, there is a big hit of talent and after that you don't get that much of a hit of talent. If that initial group doesn’t have the right person, they have to find more candidates and it takes more time. ”
Evaluating a VP of Engineering
Calibration
- Calibrate by talking to non-candidate VPs to get a sense of what great looks like. Ask them questions you plan to ask candidates and observe how they answer the prompt. Additionally, talk to a lot of candidates, both good and bad. This also helps you build a picture of what “great” looks like.
Characteristics
- Flexible and willing to adapt. They need to be willing to adapt their approach to managing and engineering. If they are not a flexible thinker, and are going to use an existing playbook, it's not going to work. It is rare that you can apply what worked exactly from one company to the next as cultures and approach are too different
- Self awareness. This is very important because subordinates do not give good feedback. They own up to their mistakes and can reflect on their failures and faults.
- You see how they can contribute beyond engineering. They need to be additive to the company. In the process, you find yourself wanting to hear their voice on business problems. This is a sign that they will be a contributor beyond engineering processes.
- You’re excited about them. Meeting the “bar” is not enough. You need to be excited about the prospect of working with them and collaborating on building out the engineering function. They also need to match your culture. If you are analytical, you should probably hire a more analytical VP of Engineering.
- They have frameworks AND real life experience. They should have frameworks for how they want to set up processes and also real-world experiences. The reality is real-world constraints often impact what you can implement, so you want someone who can adapt their frameworks to the real-world environment.
Jean-Denis on the importance of frameworks and real-world experience
“A good candidate will have frameworks..they might say I [to design an interview system] I would understand our cultural values and our roadmap. I would map out a set of interviews that aligns with both of those things. And this is the process that I would use to create a perfect interview…’ (tell me in practice how that works) ‘This is how I used parts of this framework [because of constraints] to create the interview… They have to be able to apply the framework in a resource constrained environment”
Onboarding a VP of Engineering
VPs of Engineering work only 60%-70% of the time. When it doesn’t work it's typically because the wrong person was hired or because the person was not onboarded properly. There are five goals for onboarding:
- 1) Build trust with the team
- 2) Build trust with the founder
- 3) Quickly identify if it won’t work out
- 4) Maximize chance of success
- 5) Make sure the person doesn’t screw up the company
Tactics to drive successful onboarding
Tactic 1: Create opportunities for easy wins
- Easy wins help new executives gain confidence and build trust in the organization
- The more trust and understanding they have, the more likely they are to be able to solve hard problems
- Asking a new executive to solve a really hard problem in their first week is a mistake
Jean-Denis on creating easy wins for his Head of Design
“I hired a Head of Design at Plaid a lover over a year ago. The first thing I asked him to do was to reformat our design review. We had a design review it was okay, but I wanted him to look at it and make some tweak. The changes were easy, the attendance felt random, there was no crossteam design review, we didn’t celebrate the outcomes of design reviews. It didn’ tell him how to fix it but these four things seemed off… I knew he would win, its listening to seven people and taking the intersection of what they say and that would be much better than we we have today”
Tactic 2: Stay close, but do not intervene
- Stay close, but intervening can undermine their ability to build trust. It will also make them doubt autonomy
- This doesn’t mean that you let them operate independently, but publicly give them the space to operate
Tactic 3: Fire quickly. Set up accountability check-ins to know if you need to make a decision.
- Holding yourself accountable to make hard decisions. Most bad VPs are fired within 12-18 months
- This is often too late. Ideally, you let go of a VP within 4-6 months
- To facilitate accountability Jean-Denis puts an invite on the calendar with his CEO to review performance of new leaders. This helps them objectively measure whether they are doing a good job or if they need to be fired
Tactic 4: Once they are doing well, signal publicly that they are the leader. This helps create trust.
- Once confident the VP is good, leadership needs to signal to the team they are doing well
- The more signaling, the more that person can take on as the organization will publicly acknowledge them
Plaid’s Onboarding
Jean uses a 30/60/90 plan and evaluates performance at 6-9 months and 12 months
30/60/90 days
- 30 days: Meet people on the team
- 60 days: First quick win
- 90 days: Give feedback, deliver second quick win
3 months
- 3 months: Perform their first medium fix
6-9 months
- 6-9 months: Evaluate their fit
- This is another escape hatch
12 months:
- Evaluate some of the bigger things they have done
Comments