Debunking Klement’s Attack On ODI
Klement still believes ODI can’t work. Here we debunk Klement’s remaining claims so he and his followers can understand why ODI does work.
In “Klement — Ulwick Debate…,” Klement summarizes (after a lengthy debate) the 5 remaining reasons he still believes there are statistical flaws in the ODI process. This provided me with an opportunity to better understand how Klement believes ODI works and how it applies statistical principles. In this article, I consider and address each of Klement’s remaining concerns. Klement’s belief that desired outcomes are somehow transformed into preferences (when they are not) is the root cause of his concerns. He also misinterprets or misrepresents a number of basic statistical principles.
This article explains the statistics and rational we use to effectively uncover underserved and overserved desired outcomes — which is the key to predictable innovation.
In preparing this response, I conferred with a statistician on my staff who is armed with a PhD, decades of experience, and intimate knowledge of the ODI process. Below is my response to each of Klement’s 5 claims. Links to responses and Klement’s previous attacks/claims can be found at the end of the post.
Klement’s Claim 1
Anything with a value judgement is a preference. The color blue is not a preference. It’s a color. However, when I attach data to it such as importance, significance, satisfaction, desirability…, it becomes a preference.
A desired outcome (as you define it) by itself is not a preference. However, ODI transforms it into a preference when customers are asked to attach a judgment of satisfaction or importance to it.
Response To Claim 1
Klement’s claim is wrong on two fronts. First, simply attaching a “value” to something does not make it a preference. A preference is defined as “a greater liking for one alternative over another or others.” As such, a preference requires a comparison or a choice. For instance: do I prefer 2 oranges and 3 apples to 3 oranges and 2 apples. If, when offered the choice of the two different bundles of goods, I would choose the first bundle over second bundle it can be concluded that I prefer the first bundle. In addition, simply because one places a value of some sort on a bundle of goods does not necessarily mean that I prefer one bundle over the other. For instance, the cost of apples may be higher than the cost of oranges but this in no way determines which bundle I would prefer. Hence, attaching data to an outcome does not somehow turn it into a preference as Klement falsely claims. This is a misrepresentation of basic statistics.
Second, in our ODI research, desired outcomes are NOT in any way treated as preferences. Klement seems to imply that outcomes should be treated as preferences — but that is the opposite of what we are trying to do. We just want to know which individual outcomes are under- and overserved. So let’s be clear: in ODI research, customers are never asked to choose between desired outcomes, compare them against each other, or to choose one outcome over another. Nor are they asked which desired outcomes they prefer. So even if Klement wants to claim outcomes are transformed into preferences, his point is moot as they are not treated as preferences, nor should they be.
Klement’s Claim 2
Preferences are judgements of value, and are not zero-sum alternatives. I have a preference for $100 bills compared to $50 bills. However, if you offer me both, I’ll take both. A preference of a thing is separate from any choice associated with it.
Response To Claim 2
To suggest that preferences are not “zero-sum alternatives” is indicative of Klement’s lack of understanding of preference. Preferences inherently involve the concept of choice. When presented with the choice of 2 apples and 3 oranges I would choose this over the alternative of 3 apples and 2 oranges because I prefer the first bundle to the second. To suggest that preference doesn’t involve choice demonstrates a complete lack of understanding of what it means to prefer something. And yes, if offered both alternatives I would choose that bundle, but then I am being offered a third bundle. I can in fact rank the preferences that Klement offers as bundles. $50<$100<$150. While Klement’s claim is invalid, his point is also moot as desired outcomes are not preferences, nor are they treated as preferences.
Klement’s Claim 3
The top-two-box proportion method involves division. This is how a proportion is created. Proportion operations requires ratio data and cannot be done with ordinal data.
Response To Claim 3
Here again, neither statement is true. The top-two-box proportion method that we use does not involve division. We are not dividing 3 by 2 to create our proportion. What we are doing when we calculate a proportion does not involve any mathematical manipulation of the responses. It is merely a grouping of the categories. More specifically, we group together sample respondents that state an outcome is either very or extremely important or satisfied. If 80% of the sample says an outcome is very or extremely important, then the top-two-box proportion (80%) becomes the proportion we use in the Opportunity Algorithm — no division involved.
In addition, it is simply not true that only ratio data can be used to calculate proportions. I can also create a proportion based on nominal data if I wish. I can calculate a proportion of the population that owns a green car or is a male or I can calculate a proportion of the sample that weighs more than 100 pounds or lives a distance of more than 100 miles south of San Francisco. There is no restriction on the type of data that can be used to partition a sample into groups. Claiming otherwise is a misrepresentation of basic statistics.
All else that follows from Klement’s erroneous suggestion that you can’t look at the proportion of people who answered a question in a certain manner is meaningless since it is based on false assertions and a complete misunderstanding of how we calculate the top-two-box proportion.
Klement’s Claim 4
There is no way to guarantee that my rating of “extremely satisfied” is the same as your rating of “extremely satisfied”. It could be that my rating of “extremely satisfied” is equivalent to your rating of “satisfied”.
Without the guarantee that every respondent is interpreting constructs like “satisfied” or “extremely satisfied” exactly the same, they cannot be grouped together.
Response To Claim 4
While we agree that there is no way to guarantee that everyone who is surveyed defines importance in the same way, Klement’s conclusion that people rating them in a similar way cannot be grouped is completely off base.
Despite the reality that people may mean something different when they think of “very or extremely important,” it is reasonable to conclude that if 80% of the people in our sample say an outcome is very/extremely important and just 20% say they are very/extremely satisfied, then this reveals an opportunity for value creation. This is the purpose of the Opportunity Algorithm.
Klement’s Claim 5
Even if everything up to this point was statistically valid, ODI would breakdown here. Why? Because different dimensionless quantities cannot be subtracted from one another. I cannot do this:
· (ratio of dogs to cats) — (ratio of chairs to tables)
· Heat index — Humidity
· Pi — Golden ratio
Therefore, even if importance and satisfaction represent dimensionless numbers, one cannot be subtracted from the other because there is no guarantee that they share like units.
Response To Claim 5
Once again, Klement’s statement completely misrepresents basic statistics: dimensionless quantities (even if they are “different”) are simply numbers that don’t have units, and can be added and subtracted all you want.
The rules are straightforward: either values have units or they don’t. If something has units, then you can only add or subtract them if the units are the same. If there are no units, you can add and subtract them all you want. Since importance and satisfaction (which are both proportions of a sample and not “different”) are dimensionless quantities (i.e., have no units), it follows that they can be added or subtracted. Klement’s argument is totally flawed.
To us, the real question is why would you want to subtract two dimensionless quantities from one another? While subtracting Pi — Golden ratio may not produce a valuable insight, it is technically possible. In the Opportunity Algorithm, it makes sense to subtract satisfaction from importance because the math helps us uncover hidden opportunities — desired outcomes that are under-served and others that are over-served.
Lastly, it should be pointed out that in Klement’s first example, he chooses items that are NOT dimensionless quantities — which is the reason why they cannot be subtracted. In his third example, he claims you cannot subtract the two items (Pi — Golden ratio) when in fact you can because they are dimensionless quantities. His examples confuse whatever point he is trying to make.
Once Klement accepts these truths as they have been presented, he may begin to appreciate the thinking that went into making ODI work.
Links
Klement has deleted several articles on Medium where he makes other claims regarding ODI that have since been debunked. I have attached his deleted articles to my previous articles to preserve the record. Here are the links:
Klement’s Fallacy Misleads The Jobs-to-be-Done Community
Confusion Leads To Klement’s Illusion
Klement Reveals His Mistakes And Deletes The Evidence
Final Notes
As I have highlighted in previous articles, Klement has a long history of misrepresenting ODI and himself. Klement has made numerous libelous claims against me and Clayton Christensen that can still be found on social media. In addition, Klement claims he wrote the first book on Jobs-to-be-Done in 2016, while he continually mocks What Customers Want, the actual first book on the subject published back in 2005. He also continually misrepresents the history of JTBD Theory. Klement portrays himself as an expert ODI critic — in reality, he’s never actually used the ODI process; nor has he been trained on its use. Klement also portrays himself as a JTBD expert, even though his contributions to the theory (in my opinion) take the theory and its application a giant step backwards. Klement now portrays himself as a statistics expert as well, although he continually misinterprets and misapplies basic statistics in his arguments against ODI.
Why do I point all this out? As Klement says:
