Solid State UX

Proofiness is the mathematical version of "truthiness." It lurks in business, politics, media, and yes - user research.

If Charles Seife’s Proofiness has a lasting contribution to offer those in the fields of user experience, design, or even business, it will be in the elegant branding of its own subjectivist epistemology.  This, in itself, is no small victory.  It involves taking a complex debate on the origin of knowledge and in a single catchy word, turning it into a meme.  Picture a future where somebody whips out a clever piece of marketing research in a design or business meeting, something with lots of correlations and a confident sounding sample.   Maybe there is a scientific looking visualization,  like a scatterplot diagram with one of those Jackson Pollack splatters of microscopic pinpoints, something that screams data was collected here.  Its presenter starts speaking with the cajoling air of someone trying to impress the truth, with a capital ‘T,’ upon their audience.  And then suddenly, the attendees stand in protest and accuse their tormenter, in unison, of proofiness.

Then try to grasp what a profound departure that is from today’s climate of info digestion, where almost nothing is spit back if it smells and tastes like it was cooked up from numbers.   Jakob Nielsen, the founding father of pragmatism in HCI research, has condemned number fetishism in our field periodically since he came to prominence in the early 1990s – most exasperatingly in this 2004 post, The Risk of Quantitative Studies.   He writes, “…most statistical research is less credible than qualitative studies.  Design research is not like medical science.” In a 2009 post, five years later, little has changed.  He writes, “People still pay far more attention to questionable quantitative studies than they do to simpler qualitative studies that have much greater validity.”

If those of us in the innovation game are struggling with number fetishism, that is saying quite a lot. User and design research is about human cognition and emotion, pretty squishy topics really.  And design, which is generally carried out in the service of business, is an applied field.  What business person has the time to wrestle with taking the uncertainty factor below a certain threshold of instinctive confidence?  (The genius of Nielsen’s discount usability method was in demonstrating that closing in on the uncertainty factor has a long, expensive tail as it approaches zero – which, of course, it never does.  n=3 represents a lot of bang for the buck, and n=2000 a lot less so.) Why, then, are we so persuaded by numeric evidence?

Unlike the output from qualitative research studies, usually called insights, numbers have a primordial pull on us.  They are more valuable entities in social exchange, a fact which our lizard brains are fully aware of.  Numbers can be transmitted with less nuance, that is to say, less noise, and are therefore choice currency.   Seife provides evidence that human beings are capable of reacting to numbers from somewhere biologically deep, processing them somewhere in the muscles and tissue and nerves but not necessarily with the intellect.  “No matter how idiotic, how unbelievable an idea is, numbers can give it credibility.” (p.8)  He cites an example of MSNBC host Deborah Norville reporting with a straight face that 58% of all exercise done in America is broadcast on television.  3.5 billion situps were done in 2003, she reported.  Two million and 300,000 of those on exercise shows.  “The numbers had short-circuited Norville’s brain,” Siefe notes, “rendering her completely incapable of critical thought.”

Our lust for numbers is like our lust for sweet and fatty foods. Two million years of evolution have taught us to crave them.  But in the modern world, where they are not only in abundance, but easily manipulated and processed with the explicit goal of satiating us, they are dangerous.   The author lays out, in clear language, the heart of his epistemic argument.  Once a number tries to describe the real world, or “acquires a unit” in Seife’s language, it loses its purity.   There is always a measurement bias of one sort or another and therefore it can no longer inhabit the “platonic realm of absolute truth.”(p.10) He writes this without apologizing for it or acknowledging its provocativeness. But this is no mainstream view. This is a rejection of the very idea of structure or universal truth, a return to the Dionysian notion that the sublime lies in closeness of experience and not in critical distance. This is the sort of postmodern thinking that was radical in intellectual circles as recently as the 1960s.  This is Piaget.  This is Derrida.  And maybe because of that inaccessibility, this sort of thinking is still far from being absorbed into the fabric of our daily thoughts and culture in business life.

Enter proofiness.  A great term, a self-descriptive masterpiece of nomenclature. Its dubious etymological structure (with its comic closeness to being a real word, a science word) carries its actual semantic argument.  Of course, the author, Charles Seife, had some help in this.  The word is a clear homage to television comic, Stephen Colbert’s, famous neologism “truthiness.”  But it is Seife who brings it to the concept of proof, not truth.  Truthiness, which applies to information that has the patina of truth, is pure social criticism.  Proofiness, which also applies to information that has the patina of truth, is an attack on status quo views of ontological reality.  Truthiness, like the Bushism ‘strategery,’is aimed at easy targets and misanthropes- liars, manipulators, unilateralists, oversimplifiers.  Proofiness is aimed at all of us with the instinct to prove, and therefore at the natural condition of mankind itself.  “Proofiness has power over us because we’re blind to this impurity (of numbers).  Numbers, charts, graphs all have an aura of perfection.”  Seife’s argument, in short, is the more important one.

The book’s tagline, “The Dark Arts of Mathematical Deception,” implies that the book is about a sort of numeric version of truthiness, some sort of number trickery done with intent to mislead.  In fact, the description of various mathematical anti-patterns ends by about page 40.  These are interesting in their own right.  Potempkin numbers are those which are built out of data that only looks like real data, data based on nonsensical or made-up measurements, such as IQ or the crowd estimates taken at free outdoor concerts.   Disestimation is the act of taking numbers more literally than the uncertainty surrounding them would seem to warrant.  (Seife’s example is the museum docent who dates the brontosaurus bones to 65,000,038 years. What’s the point of including the final 38 years when the error margin on such an estimate is likely in the tens of millions or hundreds of millions of years?)  By page 26, we’re on to fruit-packing and cherry-picking, which refers to the act of ignoring or obscuring the data that fails to support a hypothesis or argument.  The book is already drifting away from mathematics and is onto basic research ethics with 275 pages to go.

Despite the fact that there are the usual warnings here about the pitfalls of overlooking co-variants and making specious correlations that one would see in any tome on statistics, the book isn’t about math.  It’s about numbers. And to Seife, numbers represent powerful disinformation, unfairly persuasive rhetoric.   There is very little statistical depth or discussion of mathematical techniques in this book, other than to say, what is the point of doing statistical slicing and dicing on something that is fictional to begin with?  When Seife explains the concept of margin of error in political polling, which uses statistical formulas to harness some pretty nifty laws of nature in order to determine the amount of random weirdness likely to be in any sample, he doesn’t criticize the math, or even accuse the journalists who cite the polls of being innumerate.  He is more offended by those who accept the mathematical veracity in such a way that they don’t question basic systematic errors in the logic behind the poll’s construction and execution, such as who was sampled or what were they asked and in what way?(p.102)

Particularly revealing of Seife’s lack of concern with actual number manipulation is in his principle of causuistry.  “Casuistry – without the extra ‘u’ – is the art of making a misleading argument through seemingly sound principles.  Causuistry is a specialized form of casuistry where the fault in the argument comes from implying that there is a causal relationship between two things when in fact there isn’t such a linkage.”  He breaks this out as a separate idea from the statistical concept of regression analysis, used to prove causation between multiple independently correlated values. When Seife accuses someone of causuistry he isn’t concerned by the bad math on display, he’s offended by the sheer nerve that someone would try and build an argument out of it.

This gets back to my opening statement.  If the book is to make a lasting contribution outside of journalism or civics (Seife teaches journalism and the book is unfortunately heavy-handed with examples about electoral polling, elections, law and politics.  There are virtually no examples from the world of business or private life), it will require people to start taking this proofiness thesis to heart.  The concept instantly resonated with me.  Numbers, the coldest, hardest facts of all, are twisted and manipulated in order to add an air of proof to some act of data collection.   This touches right to the core of what are the least obvious but most insidious sources of user research failures: those that are related to epistemological hubris and the act of using research to validate ideas rather than to enrich them.

Where research fails most often is when the intent of it is misused.   The more you know about research methodologies, the more aware you are of their inevitable flaws.  The bad researchers are invariably the objectivists, the ones who arrogantly presume to be reporting on reality.   It is in this spirit of hubris that virtually all unforgivable research mistakes are made.     For instance, if you have just conducted a survey and concluded that 85% of potential customers liked your idea for an online information website, I will tell you that you have probably wasted your time.   Most likely your survey was flawed, your sample was flawed, or you created some other sort of systematic error and you cannot make that claim.   You have committed an act of proofiness.   And in the meantime, instead of using that opportunity to talk to your customers to learn something that might enrich your idea (which you always knew you were going to do anyways), you have squandered it trying to convince yourself and others that the idea is less risky than it probably is.   Concentrate on delivering insights, not validation.

Once, I worked as a usability researcher for two arch competitors in the same retailer category at roughly the same time.  Their contrasting styles will always stick with me.  One had a usability manager that emphasized insights.  The other had an old school HCI guy that emphasized Truth (capital T is no accident here) and experimental control.   When interviewing for the first client, I would tell the participant that I didn’t know where this interview was going but I was interested in partnering with them to try and understand what it is like to use this website, how it fits into their life, and how it could be improved.  When interviewing for the second client, I would give the participant a pre-filled out card with a task on it and watch them try and complete the task while somebody from the retailer’s staff timed them in the back room.    I barely opened my mouth for the second client, lest I contaminate the experimental conditions.    The first client had a progressive stance about the research, understood its limitations and was focused on collecting inspiration to refine their design in ways that might resonate with its customer base.  The second client wanted validation only for its existing “agreed upon” design, and set up a priori experimental conditions to get at it.   It is worth noting that for reasons unrelated to this particular usability study, the second client unexpectedly and spectacularly went out of business several years later.  The first one is thriving.   I think it goes to show that proofiness never prospers.