September 19, 2004
Hiring: Sabermetrics for startups?
If you pitch a venture capitalist for money, be prepared to hear sports analogies. "When are you going to put the puck on the ice" (when are you going to launch). "You need to move the ball downfield" (self explanatory). So, when I saw boxes and boxes of Michael Lewis' book Moneyball stacked up in Accel Partners' offices, I figured it was merely an attempt to brush up on new baseball analogies for their next crop of startups.
However, when I read the book, I was fascinated. On the surface, Lewis' book is a baseball book. But, the more I read, the more I was convinced that there was a great startup lesson contained in its pages. And that's when I understood why Accel was distributing the book.
As I mentioned, the book on its surface, is about baseball. Specifically, it asks the question of how the Oakland As, with the second lowest payroll in baseball can consistently make the playoffs against competition that is far better funded. (Poor entity competing against a rich entity? Sounds like a startup problem to me.)
To set the stage, Lewis explains just how great a discrepency exists between the highest and lowest payrolls in baseball by contrasting baseball with other sports. In football, the ratio of highest to lowest payroll is 3 to 2. This means that the highest paid team pays out 1.5 times what the lowest paid team does. In basketball, that ratio is 1.75 to 1. In baseball, it's 4 to 1. 4 to 1!? I couldn't believe it. Talk about an inherent disadvantage.
So, how could such a financially poor team like the As fare so well? The answer, it turns out is by exploiting a baseball market inefficiency. According to Moneyball, most baseball scouts, the guys who pick talent, look for the same things; things that "feel" right. How fast a player can run? How powerfully can they hit? How fast can they throw? How well can they field? How much do they have the "look" of a baseball player? It's an old game, with lots of traditions and common wisdom. Everyone just "knows" that these things matter.
With all the scouts looking for the same thing, those players who exhibit the commonly looked-for traits are (of course) highly priced. No poor team could afford them, including the As.
But, what if everyone was looking for, and pricing highly, the wrong characteristics? As it turns out, in baseball, the scouts were doing just that. They were highly *overvaluing* players based on a set of "common knowledge" that had very little correlation with winning. Turns out, if you do the math and run the stats, how well you run the bases, how fast you throw and how hard you hit have very little impact on winning baseball games. In fact, the math (apparently) shows that fielding doesn't matter in the slightest.
So, did the scouts believe the math? Of course not. They had tradition and years of experience and wisdom on their sides. They continued to value the players that "felt" right or looked right. The only teams that did believe the math were the teams that couldn't afford not to believe it.
The As were one of those teams. They couldn't afford to pay what the market was highly valuing. So, they set about doing a fact-based statistical analysis and picked players that no one else wanted, but which the math suggested would highly influence the team's ability to win games.
And win they did.
So, what does this have to do with anything startup related? Well, I started to wonder if there was a similar market inefficiency in the people market for technology startups. Were certain characteristics in people highly overvalued relative to their impact on a company's success? Conversely, were there characteristics undervalued in the people marketplace -- hidden gems waiting to be found that could have a radically positive impact on a company's trajectory? If this was the case, how could I identify these so that I could avoid the overvalued people and zero in on the undervalued people?
Well, I don't have an answer just yet, but I'm thinking a lot about it. In general, I think that the technology people market overvalues certain VP-level jobs, typically in marketing and business development relative to these positions impact on a company's success (how often have you seen the "killer" VP of marketing get brought in with some huge salary and equity package to save a company and end up not having much effect?) On the other side, I think the market generally undervalues key engineering hires relative to their contributions.
Why does this happen? For a few reasons, I think. First, most CEOs are not technical (I'm not either). So, they tend to highly value the things they understand (marketing, business dev, sales) and undervalue the things they don't (engineering). Second, people are more attracted to people like themselves. Come from marketing and you'll probably pay more for marketing people. Third, and perhaps most importantly, is the fact that there are no identified fact-based metrics that help CEOs understand how to value engineers. Let me see if I can explain.
In Moneyball, Lewis describes the search for the statistics that matter - those that are highly correlated with scoring runs and thus winning games. And, it turns out that there are two in baseball -- slugging percentage and on-base percentage. Find these characteristics and you've got a strong likelihood that the player will contribute positively to the team.
What are those stats for evaluating a potential engineering hire? People might be willing to pay more for a great engineer (pay what they're actually worth for example) if they could believe with a high degree of confidence that this person will actually make the difference they are paid to make.
I've been wondering if there are such engineering "stats" that are likely indicators of significant future contribution to a startup. Anyone out there got any ideas? For example, does being a commiter to a large open source project have a strong correlation with likely success inside a company? How about a MS degree in Computer science from a top 20 school. Is that a leading indicator? What about coding for fun? If someone codes for fun (as opposed to just for work) is that person more likely to make an unusual positive contribution to the success of a startup?
I'm really curious if there are items that correlate. I'm tired of guessing...
The study of baseball statistics is called Sabermetrics. Has anyone done Sabermetrics for startups?
September 19, 2004 | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference Hiring: Sabermetrics for startups?:
Tracked on Sep 20, 2004 7:46:29 PM
Tracked on Sep 20, 2004 7:47:03 PM
» Sabermetrics in tech biz from Troutgirl
As Joe pointed out recently, running your business like the Oakland A's is the latest fad. Only in America could a book about a non-winning baseball front-office guy become a bus [Read More]
Tracked on Oct 2, 2004 8:25:59 PM
Tracked on Aug 28, 2005 11:00:05 PM
Since none of the startups I've been with have been successful, I guess I'm the anti-candidate ;)
Posted by: Brian Yamabe | Sep 19, 2004 10:33:21 PM
Metrics are hard in engineering. There's a line of thought that collecting them doesn't even make sense -- that you start to negatively affect the group the moment you (explicitly) begin a measurement.
I once measured things like checkins, lines of code, lines of documentation, test cases, and so on, and found that the two engineers I intuitively considered to be the best in the group were at the extremes -- one had the highest number of each of these measures, and the other had the lowest. One of them was blazingly fast, the other was slow and deliberate and just as useful. I tend to think of these measurements as fairly meaningless as a result.
One measurement I have found useful is bug reopen rates during a single release. If as a developer you mark a bug as resolved and the verifier (QA person or whomever else) later has to reopen it, in theory you should have been able to catch that and save both of you some time. Bug reopen rates seems to me to map well to the attitude developers take towards QA and bugs in general.
There is one measure that people talk about as "productivity," and the general wisdom (I think from Fred Brooks?) is that the best developers are ten times as productive as the next best. Certain people are off the scale -- you don't even need to measure because it's so obvious. Reference checks on these people are so much more forceful than on others. As you say in your "No False Positives" post on hiring, you have to wait, in this case until the references are screaming in your ear about how great this person is. Coding tests during interviews also bring this out pretty well, although nothing beats a trusted report from someone who's worked with the candidate for a long time.
Unlike in Moneyball, engineering groups don't tend to have freaky fans collecting extra statistics about them from the bleachers. The stats aren't public. Open source projects are easier to gauge, and as a result people do instrument them quite a lot. But I don't think those measurements are any more useful than those you can make against a closed source group.
Posted by: Marc Hedlund | Sep 20, 2004 12:17:10 AM
For our new hires, it's all that "feel" thing. Luckily, that's worked for us so far, but that goes back to the A's hiring A's thing.
Once hired though, we look at reopen rates as well, with input from their up and down stream coworkers. If I have a buddy I go to lunch with all the time, I won't say anything bad about them. However, if this coder sends me "fixed" things that aren't really fixed, I know it, and I'll say something because they are making more work for me with late nights and weekends near a release.
You can't take feel completely out of the equation (or maybe I'm just an old scout myself) because I can spot great folks a mile away. The problem is, not everyone's born a great judge of people. I can even tell it on the phone, but always bring them in just to be sure. It would be great if I could quantify it or replicate it in others, but for now I'm in on every interview with every candidate just in case. Wish I could sell this scouting skill, if anyone figures that out, let me know.
Posted by: Scott Palmer | Sep 20, 2004 6:09:58 AM
Great site. Some interesting posts already. I love this post. I love hearing stories where science takes on conventional wisdom and wins. I found Scott's comment interesting,
"You can't take feel completely out of the equation (or maybe I'm just an old scout myself) because I can spot great folks a mile away. The problem is, not everyone's born a great judge of people. I can even tell it on the phone, but always bring them in just to be sure."
It reminded me of some studies in a famous psychology book called, 'Uses and Abuses of Psycholoygy'. I think it was by Hans Eysenck..one of the big hitters. There are some chapters on the history of personnel selection by interview. One experiment involved taking a set of successfull, experienced MDs, who claimed to be good judges of character and therefore good at hiring. The study showed that there was virtually no correlation between the ratings the different MDs gave to the candidates in the study. In other words, they all rated the candidates differently. They all said they could pick good candidates, yet they all rated them differently. So...draw your own conclusions.
Also, the book reviewed studies that showed that there was little correlation between interview success and job success. The best predictor of candidate success was merely aptitude to do the appointed task (as judged by either academic aptitude, interview tasks that mimic the job tasks closely - the best predictor it turns out - or previous, proven experience).
I'd love to find the statistics Joe wants too. Let me know if you get any! I love it when you can win by bucking conventional 'wisdom'.
Posted by: mindful_learner | Sep 20, 2004 6:19:39 AM
Great post. To me, quiet productivity is the On Base Percentage of startups - the attribute undervalued by the market. Harvard MBAs, McKinsey alums and published "thought leaders" are the base stealers of the entrepreneurial world - important, but extremely overvalued by the market. I'll take one quietly efficient engineer over the bluster and self-promotion of 20 "top-tier" candidates any day.
Posted by: Ken Norton | Sep 20, 2004 9:06:42 AM
Re: overvaluing certain skill sets - in the world of public companies, I would bet there is a direct correlation between the growth of CEO salaries to the growth of the salaries of professional athletes, especially relative to worker bee salaries. This is because salaries are publicized, either in an annual report or the daily sports pages.
An athlete's agent would not go for $10 million bucks a year if he didn't know someone got $9.5 million last year. CEOs are the same - they look at what other CEOs get. Meanwhile, worker bees don't have this kind of information. This is a classic information assymetry problem. http://en.wikipedia.org/wiki/Joseph_E._Stiglitz
I'm not trying to start a class war, but this is the kind of thing that creates it. If public companies are required to report officer compensation, maybe they should report all compensation. There would likely be a huge upward lunge in salaries for the worker bees that make things happen, and maybe even downward pressure on the CEOs, especially those that suck at evaluating the talent that makes up their core competencies.
Out here in Denver, former Qwest CEO Joe Nacchio once infamously justified his outrageous compensation by saying he creates more economic value than a second baseman. Of course, all that moolah would be very helpful in paying Qwest's estimated $250 milliion in SEC fines right now. And just like a second baseman who blows out his knee in training camp, Nacchio has declined to give the money back when it turns out he wasn't so helpful after all.
Posted by: Derek Scruggs | Sep 20, 2004 9:26:29 AM
As they say, Great post son! I find in evaluating prospective tenants for vacant apartments that "feel" is more important than the application. I get more out of talking with an applicant on a personal level about things in general than anything he/she puts on the app.
Thanks for the tip on Moneyball, it was a great read.Love your Blog.
Posted by: Dad | Sep 20, 2004 10:04:51 AM
In startups, technical people need to have the ability to be self-starters, self-organized, communicate well, and have a broad set of skills. Ability to be a 10x producer is also a good thing, but sometimes being 10x as productive means understanding the business enough to know whats a waste of time and what isn't.
So, if I'm doing a startup, I'm looking for these "measurable" attributes:
a) have been in early stage startups before
b) can communicate well
c) understands the core focus of the business and understands what sort of technology adds value and what is a waste of time (e.g. can apply buy vs. build decisions at the most technical levels while still understanding the context of the business)
d) has a wide variety of hands-on technical skills
e) comes with glowing recommendations from both technical and non-technical/business people
That last one is key.
I think you can put up with certain personality foibles - this is, in many cases, where the market discards people who can thrive in certain dynamic environments, but do poorly in more structured jobs. Short attention span, for example, can be a *good* thing at startups, as long as people are self-organized. Those folks can put up with the constantly changing priorities and the curveballs coming from every direction.
I think in general, however, you have to go by gut and the recommendation of people you trust. Doing startups, IMHO, is much more complicated than playing a sport. Success is defined differently, and the rules of the game are different for every startup...
Posted by: Gabe Wachob | Sep 20, 2004 10:11:52 AM
Welcome to the blogosphere, I've subscribed. great stuff so far.
I think one attribute which is fairly easy to measure which correlates well with success as an engineer is IQ. It would be great if you could give candidates an IQ test, but you can't, so the technical puzzles are a reasonable proxy. In baseball they say "you can't teach speed" and in programming we say "you can't make someone smarter".
A great failure of many interview processes is that they score candidates based on knowledge rather than aptitude. This is where having recruiters screen resumes based on keywords is such a bad thing.
Anyway it is a tough problem and an important problem. The best hires I've made have nearly always been people I already knew or people recommended by people I already knew.
Posted by: Ole Eichhorn | Sep 20, 2004 5:44:14 PM
Just came upon your post (and blog), by way of Jim Thompson's (http://smallworks.com). Interesting post, and the book seems to be a pretty interesting read as well.
With your questions in the post, are you looking for engineering leadership roles (VP Engg) - someone who can guide contruction of your system, leading a team of people, or are you looking for people who can actually go build the stuff? There may be marginally different criteria for evaluation there.
The engineering/development community has always been pretty conflicted about metrics - on the one hand, people talk about metrics being bad since they might be used to reflect on performance/salary, and of course you have the usual "beware of the single guy in a garage" stuff. On the other hand, you also have everyone quoting (without empirical data) about how the best developers are 10 times as productive as the average.
The most striking outcome of this is that there are no real quantitative "developer productivity" metrics around. Can some metrics be derived from activities like involvement in open source projects? I'm not so sure. Over the last few years, there seems to be a concerted effort to categorize Stallman and the GNU bunch as fanatics, and open source development as communistic. Even the vast majority of developers who don't take this extreme viewpoint still seem to fear "coding for free". I'd also argue that a large number of the more visible developers on open source projects are the ones that are more adept at marketing themselves?
So, are there any other, more qualitative, metrics that might be used? Is there a "great engineer" profile, so to speak? Personally, I'd be inclined to go with the same principles as one of your other commenters - namely, look for people who got good peer reviews from people who they worked with previously. And I'd look for feedback from both technical as well as non-technical collegues. I'd look for some mix of personality traits (plays well with others, is driven to do a good job) and core engineering (nee critical thinking) skills. After all, in the final analysis, person A who is able to crank out 10 times more code than person B may still not be 10 times as valuable/critical to the success of the startup?
How would you distinguish a good VP BizDev from a bad one? Or a good VP Marketing from a bad one?
Posted by: Venki Iyer | Sep 21, 2004 6:21:01 AM
Haven't read Moneyball, but isn't it correct that the Yankees have the best record over the last 6 or 7 years? Haven't the Yankees won more pennants than the A's? Won the World Series more?
Lewis is a financial reporter and knows nothing about baseball. Last year's World Series was exceptionally close; just a few Yankee fielding errors gave the Marlins the edge they needed to win.
The Yankees had a long dry spell some years ago, but it had nothing to do with the scouts. Steinbrenner was a mean, meddling, micromanaging SOB, and this hurt the team. He took a forced sabbatical, hired a great manager, let the guy work, and the rest is history.
Among the players the scouts (those ignorant dolts) found were Derek Jeter and Bernie Williams, who are two of the greatest fielders of modern baseball. Williams is also probably the most graceful player since Willie Mays at his peak, and has been a joy to watch.
I imagine Joe Torre will soon be calling SHR VCs for advice, who will undoubtedly tell him that since fielding doesn't matter, he should equip his players with potholding mitts, and thus cut costs, which will enable the Yankees to win as much as the A's.
Posted by: ned | Sep 21, 2004 10:30:07 PM
WRT your comment on top marketing spots often not contributing: I've been at several technology startups and find that this spot is often populated by people without the technical background to really do this job. They often are described as 'top-notch brand-creators' and come armed with and MBA from a top marketing focused school. While brand management is important, I think without the tech background, it is often useless and the spot would be better be served by someone with a lot of marcom experience and not necessarily the degree.
I'd recommend looking for someone with a strong tech base (education and experience) who picked up a minor in fine-arts or design or who has well developed hobbies in this area.
Posted by: Johan | Sep 22, 2004 7:49:23 AM
Straight out of college (with a MS and BS in Applied Math) I became a (very good) software engineer. About five years later I made one of the worst mistakes of my life and went into (product) marketing. Though I managed to work in (relatively senior) software marketing positions at both Oracle and Sun, some life errors are difficult to reverse!
I completely agree that VP Marketing and Biz Dev types are overvalued and to this day it beggars belief that most (and I do mean most) of these bozos ever got hired.
I also completely agree that engineers are completely undervalued. After all, Silicon Valley was built by engineers.
Of the great engineers and technologists that I have known in my 15 plus years in the business the one attribute that stands out is that they were all extremely articulate. Whether in conversation or written communication they all displayed the most wonderful grasp of language, grammer and sentence construction. They communicated lucidly and with clarity.
Being articulate is my first measure of someone's technical ability.
Posted by: Dinesh Vadhia | Sep 22, 2004 1:12:04 PM
I've been recenlty pondering if you could predict engineering output just based on time spent in a shell window (or perhaps an IDE). My best developers spend more time in front of their computers, and less time in meetings. I realize it's somewhat circular, but the people who really enjoy development are drawn to it, and avoid meetings as much as they can.
Also negatively correlated with output is amount of time spent producing paperwork: plans, schedules, docs, etc. That's the way I feel. The more time you spend in a startup trying to justify the work you do, the less you will get done. Make a prediction and GO. Failure is a prerequisite to success. Avoid process-lovers.
Posted by: Chris Lunt | Sep 23, 2004 11:52:08 AM
Read the book Ned, all the answers are in there.
Starting with 1995, the Yankees rankings in overall wins (both leagues) have been: 5, 3, 3, 1, 3, 9, 3, 1 (tie), 1 (tie) and currently 2. Not a bad run, but for spending four times as much money, you'd expect them to be blowing Oakland out of the water. Didn't happen.
Playoffs can be a bit of a crapshoot, but the season doesn't lie. And the season, without a doubt, belongs to Atlanta. If their string of division championships lasts much longer, it'll be eligible to vote.
Posted by: Doug | Sep 24, 2004 2:46:28 PM
I just came out of one of those top 20 CS Master's programs, and into co-founding a fairly hot startup. The upshot is that I have found myself doing a boatload of hiring, and having to come up with some metrics for picking out good engineers.
My metrics are based on my experience at MIT, where I found that a large portion of the students weren't very good engineers. After spending 5 years hanging out with a group that was (though not neccessarily the only group), I have come up with a few metrics for identifying at least one class of really good engineers:
* Speaking articulately. Nerds who can't communicate are B-league nerds.
* Understanding systems they have worked on, from top to bottom. I get people to describe something they have built (I let them pick), and then I drill down. What was the network protocol? Did you run into locking problems? But how does that interact with the windows threading model? Why not asynchronous IO? etc.
* Enjoying talking about things they have worked on. With the best people, you can just let them keep talking.
Now, I don't have hundreds of hires under my belt, or even all that much experience, but that is the best I have come up with thus far.
Great thread. I'll have to check out the book.
Posted by: Richard Tibbetts | Sep 26, 2004 4:38:42 PM
I used to say that I was looking for people who "get it, and get it done." That more or less translated into people who understood the business, understood the technology, could apply the technology to the business, and were reliable in execution and attitude. Unfortuantely I haven't found a good way to interview for all of that, which is why I tend to rely very much on personal references. This has the downside of making the business a bit of an old boys club (which implies you do need to hire across all of your networks, not just one location). It has the upside that you can actually measure for the ability to do these things based on having seen people do them before. It has the risk that you will have people in your organization who look at this as a touchy-feely judgement rather than metric ("I think candidate X is competent" v. "in the 2 years I worked with candidate x we had 6 deadlines and the candidate hit all of them").
The bug reopening thing is a good one, although you have to be close enough to the QA process to make sure the initial dynamic is correct (e.g. I worked in an organization where we had sub-par QA engineers who kept reopening the same bug to report new and completely unrelated problems which often turned out to be user error. One of the most effective engineers was handed maintenance of one of the hairiest parts of the code when the original author left, and this part of the code was very poorly understood by QA. As a result, his reopen rate was disproportionately high.)
Posted by: Faisal N. Jawdat | Sep 27, 2004 2:18:06 PM
Much like in baseball you measure slugging and on base percentage as a key indicator, I think for engineers one should look for measure of successful completion of tasks. For example, how many products shipped, quality metrics of products produced, and overall success of the product. Results is the only thing that matters as in baseball.
Posted by: SWong | Sep 28, 2004 2:47:30 PM
A great resource I use for sabermetrics as a Software CEO is the research reports from Software Success (www.softwaresuccess.com). Included is the "Sales & Marketing Super book" and the "Financial Handbook" which include trend analysis of some of the most successful, and profitable software businesses.
It coorilates all the stats for you so you can get a clear indidication how the leaders in different areas run their business. Is it perfect? No. But its a good indication of how the market is running... lean and mean trying to be a profit machine.
Consider checking that out the next time you want to do sabermetrics for a new software startup.
Posted by: Dana Epp | Sep 29, 2004 10:24:30 AM
Paul Graham's "Great Hackers" might give you ideas -
One counterview: http://software.ericsink.com/entries/No_Great_Hackers.html
Posted by: Seb | Oct 7, 2004 3:22:48 AM
There really are two kinds of programmers. Those who went into it for the money, work 8 hours and clock out.
Then there are the passionate ones. They're at the user group, writing articles, contributing to open source projects, always trying out new stuff and learning.
They are part of the programming community and when a problem arises they have personal contacts and can reach out to people that can help.
Given two coders of equal skill I will always hire the passionate one.
You also have to look into personality, whether they will mesh well with your existing group.
Biggest hiring mistake I ever made was hiring a rock star coder whom I knew would antagonize everyone else. It doesn't matter how good someone might be, if they're a disruptive force they best be avoided.
Posted by: Rick | Nov 4, 2004 10:29:45 PM
Very interesting. I just came across your site today.
I'll apologize in advance since I don't have any specific metrics for you.
I am not enough of a veteran that I feel confident to go by "gut". I think "gut" is used too often and it doesn't work. Yes, I've been conned by "good salesmen" candidates.
I don't believe that you can just take some global metric for a position like "developer" etc. and use these global metrics to automate a search, but there is a process that might help you develop specific metrics for your situation.
The CVCA ran a seminars on hiring called "top grading". It was very valuable. There is also a book with the same title. (Haven't read it though.) It isn't rocket science. It is a process.
I believe that "top grading" has gotten a bad rap, but the process of systematically figuring out the key skills or traits that define success for your company and for the specific situation makes the process well worth the effort. These traits and skills become a matrix for what you are looking for in the interview process.
You will have more consistent data to reference check against too.
I use it for every hiring situation now.
Posted by: Nuke | Nov 11, 2004 1:33:16 PM
This is one of the more interesting blog threads I've read in a long time. Howver I think you're all missing the real point about what Billy Beane has done.
I realize I'm standing on a landmine with my point here but what the heck. I'll be bold.
Posted by: Bill | Nov 20, 2004 2:08:58 PM
Guess what? The As didn't hire A players.
I first came across Moneyball at the Management by Baseball Blog: http://cmdr-scott.blogspot.com/.
Posted by: David Locke | Jan 16, 2005 7:13:34 PM
Johan, Software marketers that mention "Brand" worked for post-IPO institutionalized companies that use advertising for investor relations purposes.
You do not become a market dominator by advertising. For a pre-institutionalized software startup, advertising is a waste of money. Stick to word of mouth, publicity, press releases, events, SEM, just about anything, but advertising and brand.
So if you get a marketer that talks brand, move on to the next guy.
Pre-institutionalized software startups typically have NO MBAs. Gates never got his. Ellison never got his.
Sales people can also be a problem if they worked for the huge institutionalized companies, because they demand marketing support and can't build the business from zero.
As for having a technical background, everyone except line managers have technical background, just not in software, and a background in software is not required to be a great marketer.
If you want to be a great tech company, only hire techs. But, this will gurantee that you end up being number two in the industry, provided you are creating a new industry. If you want to be number one, you have to hire marketers, and some of those marketers will not have technical backgrounds.
People with the job title technical marketers, are not necessarily the best marketers. It will depend on the person and the specific job.
By asserting that tech is required, you are disrespecting everyone else in your company. That won't be a healthy culture. If you don't respect them, even an A player with a technical background will fail.
Posted by: David Locke | Jan 16, 2005 7:24:49 PM
The comments to this entry are closed.