Here is a bad research design I see way too frequently.* The author presents a model. The model shows that if sufficient amounts of x exist, then y follows. The author then provides a case study, showing that x existed and y occurred. Done.
Do you see the problem there? I removed “sufficient” as a qualifier for x from one sentence to the next. Unfortunately, by doing so, I have made the case study worthless. In fact, such case studies often undermine the exact point the author was trying to make with the model!
Let me illustrate my point with the following (intentionally ridiculous) example. Consider the standard bargaining model of war. State A and State B are in negotiations. If bargaining breaks down, A prevails militarily and takes the entire good the parties are bargaining over with probability p_A; B prevails with complementary probability, or 1 – p_A. War is costly, however; states pay respective costs c_A and c_B > 0.
That is the standard model. Now let me spice it up. One thing that the model does not consider is the cost of the stationery**, ink, and paper necessary to sign a peaceful agreement. Let’s call that cost s, and let’s suppose (without loss of generality) that state A necessarily pays the stationery costs.
Can the parties reach a peaceful agreement? Well, let x be A’s share of a peaceful settlement. A prefers a settlement if it pays more than war, or x – s > p_A – c_A. We can rewrite this as x > p_A – c_A + s.
Meanwhile, B prefers a settlement if the remainder pays better than war, or 1 – x > 1 – p_A – c_B. This reduces to x < p_A + c_B.
Stringing these inequalities together, mutually preferable peaceful settlements exist if p_A – c_A + s < x < p_A + c_B. In turn, such an x exists if s < c_A + c_B.
Nice! I have found a new rationalist explanation for war! You see, if the costs of stationery exceed the costs of war (or s > c_A + c_B), at least one state would always prefer war to peace. Thus, peace is unsustainable.
Of course, my argument is completely ridiculous–stationery does not cost that much. My theory remains valid, it just lacks empirical plausibility.
And, yet, formal theorists too often fail to substantively interpret their cutpoints in this way. That is, they do not ask if real-life parameters could ever sustain the conditions necessary to lead to the behavior described.
Instead, you will get case studies that look like the following:
I presented a model that shows that the costs of stationery can lead to war. In analyzing the historical record of World War I, it becomes clear that the stationery of the bargained resolution would have been very expensive, as the ball point pen had only been invented 25 years ago and was still prohibitively costly. Thus, World War I started.
Completely ridiculous! And, in fact, the case study demonstrated the opposite of what the author had intended. That is, if you actually analyze the cutpoint, you will see that the cost of stationery was much lower than the costs of war, and thus the cost of stationery (at best) had a negligible causal connection to the conflict.
In sum, please, please interpret your cutpoints. Your model only provides useful insight if its parameters match what occurred in reality. It is not sufficient to say that cost existed; rather, you must show that the cost was sufficiently high (or low) compared to the other parameters of your model.
* This blog post is the result of presentations I observed at ISA and Midwest, though I have seen some published papers like this as well.
** I am resisting the urge to make this an infinite horizon model so I can solve for the stationary MPE of a stationery game.
William Spaniel 