Respect the Silent Evidence

I have been reading The Black Swan : The Impact of the Highly Improbable by Nassim Nicholas Taleb (a.k.a. NNT) and been thinking about its impact on some of the assumptions that I have made.  One of the main points is how often accept as proven fact theories which ignore silent evidence.  Software professionals fall for this kind of logical failure all the time.

Let’s consider a theory that ignores silent evidence.  I plan on voting for Barack Obama for president.  If you were to make conclusions from a poll that consisted ONLY of asking me who I would vote for, it would be logical for you to conclude that Obama is going to overwhelmingly win both Pennsylvania and the national popular vote. These predictions may very well prove to be accurate and you might be hailed as a tremendous prognosticaltor.  But most serious analysts would mock you.  They would be correct to do it.  Because of the miniscule sample size, your poll does not accurately reflect the size of the entire population that will be participating in the election.  The "silent evidence" are the opinions of all the voters who have NOT been polled.  A good pollster mitigates ignoring the silent evidence by trying to come up with a representative sample; the idea being that if you can get a large enough sample broken down by representative demographics, you can make a reasonable prediction.

So what’s the point?  Let’s consider another example.  In a famous essay, Paul Graham talks about the advantage of using functional programming languages when creating your startup.  If you haven’t read the post, I highly recommend it, it is well written and thought provoken.  However, it clearly ignores the silent evidence.  Graham takes his singular experience and makes a conclusion based off it.  But consider the silent evidence:

  • How successful of functional language based software startups versus all software startups.

or a slightly different cohort

 

  • How successful were functional language startups during 1995 (when ViaWeb was started) versus all software startups founded during that year.

With both these cohorts, we should probably include in the sample those kitchen table companies that started but never shipped a product.  Maybe many functional language startups fizzle quickly out and only the ones with really great ideas survive.

Of course, I could just as easily make the argument that statups that have two really, really smart co-founders who have impressive technical and business savvy and are starting their company in an exploding market have a much better probability of success than companies without those advantages.  My theory is no more right or wrong than the one posited above, we both are ignoring the silent evidence in positing our

My point is not to say Paul Graham is wrong about startups and functional languages.  He is a smart guy and has accomplished much more than I.  The Beating the Averages post was just the first one to popped into my mind while reading Black Swan, I’m sure I can find countless other technical articles.  But I do want to introduce some skepticism into the reading of his post and many other software legends.  When critically reading a book, article, blog posting, consider the silent evidence before making judgement on the theory.