The Great Debate: Quantitative vs. Qualitative

[wordshadow]

So this has been discussed in various tidbits throughout forum posts for this application cycle, but nowhere in any real systematic detail. Here, young torch-holders of both methodological camps (and for those quirky mixed-methods people like myself), is your thread to hash out pros-cons of each methodological approach. The back and forth can perhaps lead to a more nuanced debate and generate answers more telling than the polarizing cliches of "obviously quantitative is superior as most departments/jobs which hire look for quantitative/statistical sorts of skills", or "qualitative is superior because it provides a narrative left un-captured by dry empiricist analysis" or the diplomatic, timeless answer of "it depends." Or perhaps these cliches capture it all...

I believe it is pretty evident that this really is a matter of individual preference and the nature of the questions one wants to investigate and ask. I think it also evident that which method one thinks works for them does not work and should not work for everyone. Furthermore, it definitely seems to me to be the case that the field largely preferences quantitative over qualitative methodology in general in an attempt to 'be scientific.'

But the question becomes why did you personally choose your methodological route? Are you more mathematically-inclined? Do you do whichever route you do because that is the predominant route employed by the schools you want to go to and you just absorb whichever approach your department has? Does choosing a more qualitative of a path over a quantitative path really disadvantage you that much in the job market outside academia? Does this debate really matter? Will it matter twenty years down the road? What are the dilemmas each approach (including mixed methods) encounters?

Edited March 4, 2012 by wordshadow

[Eudaimonia]

Great topic, wordshadow!

A nice article on this: http://www.uvm.edu/~dguber/POLS293/articles/smith.pdf

I'm definitely inclined towards the qualitative side of this discussion, although of course both methods are important. I believe the name for rebellion against the current quantitative trend in political science is called "Perestroika" --http://en.wikipedia.org/wiki/Perestroika_Movement_(political_science) I'm not sure how much success it is having and would love to hear more about the current state of the debate.

[PDCU]

I guess I'm more of a mixed-method person. In terms of "-ism", I am a real-constructivist. But if I had to choose a camp, I'd side with the qualitative. I've heard both sides, and I find both of them useful, but my biggest problem with quantitative methods is that it assumes that all actors are rational. When it comes to human behavior, there are too many unknowns and inexplicable and unpredictable factors only qualitative analysis can even attempt to answer. So, in my opinion, quantitative methods has the most explanatory power in the general sense, but qualitative methods can provide the most normative power in case by case basis. In my research, I try to combine both methods to use qualitative analysis to conceptualize the identity and interest of actors and then use quantitative methods (game theory) to explain their interactions.

Then again, I think it's also a value judgment. Where is the value of "good" Political Science? It tries to explain a political phenomena, but is it ultimately trying to predict the future? Or is it trying to make a normative criticism and provide a new order?

[RWBG]

Ugh, I just wrote something and then accidentally pressed a button that went back to the previous page and deleted everything I wrote. This version has been written more haphazardly.

First, I think it's useful to divide formal theory and stats, because they have different advantages. Formal theory helps to ensure logical consistency (though is not necessary to do so), and allows you to play around with the model to find contingencies. It also allows one to have a rigorously defined justification for the specification of a statistical model, helping to avoid the "Garbage Can regressions" that were discussed in another thread of this forum, in which one includes a laundry-list of variables in the hope of increasing statistical significance. However, the advantages of formal theory are distinct from stats, and sometimes game theory work accompanies case studies.

Stats are different, and are more about getting the most out of limited data (which we all face as political scientists), and ensuring a clear and well-defined framework for causal inference. Case studies can be useful, but it can be hard to pick cases that control for the right variables. People sometimes talk about "special cases", e.g. the case that falsifies a deterministic theory (which I don't really think exist in the social sciences, most are probabilistic), or cases that are particularly persuasive because they are the case which you wouldn't expect a hypothesis to hold up in. The second kind of case I think is fine, but we should be careful about making too many strong inferences from that, given that we're still dealing with only one instance of something occurring in a particular way when trying to uncover probabilistic hypotheses. There's also process-tracing, which I know much less about, but I imagine it would be very difficult to "observe" causal processes in a complex social system; difficulty in observing biological mechanisms is part of what led to a move towards evidence-based medicine from more mechanistic approaches, and I think social systems are just as complex and biological systems, if not more.

So finally, the article posted above writes of a trade-off between empirical rigor and substantive importance, favouring the latter. I don't think that such a trade-off is particularly fair, and I think much of the formal theory and stats work has been very important substantively. However, if we do accept this trade-off, then I think there's a real question about the value of work that is based on broad, important questions, but produces very little in terms of empirically evaluating competing hypotheses. In these instances, I think there's a danger of work just becoming a way for people to justify previously held viewpoints. That being said, I tend to think attempts to answer big questions are valuable, and it's good for people to put forth the effort to answer those questions as rigorously as possible. Still, there would be no reason not to use statistical analysis as part of a toolkit to answering big questions; maybe you get a broad-based regression that's suggestive of a particular relationship, and support that work with case-studies and process tracing. Sure, quant work sometimes encourages work that focuses on smaller questions that can be proved more conclusively (e.g. look in economics at a lot of the natural experiment and instrumental variable stuff), but it's not a necessary distinction, and I think the article above is criticizing a "normal science" approach more than a "science" approach.

Ultimately, the question is less about cosmetic similarity to "science," and more about making the most of the data we have to make the best inferences we can. That may involve quant work, may involve qual work, and may involve some combination of the two. One thing I definitely think is true though is that both quant and qual people should spend more time thinking about the epistemology behind their research designs, and possibly spend more time reading philosophy of science.

Edited March 5, 2012 by RWBG

[RWBG]

A couple of added notes I forgot about: the article above also talks about using tools that are "easily understood." I don't really think that's a fair critique; if social scientists can explain their evidence and how the methods work, then I think you can use methods that are, themselves, not broadly understood. Economists have been doing it for a long time, and probably have more influence on policy than political scientists.

Also, @McMuffin, stats work certainly does not assume rationality. There's been a fair work done that is actually meant to test hypotheses that contradict rational choice; see, for instance, work done to test theories of bounded rationality. Moreover, rationality as it is described by formal theorists is far less narrow than most people assume; it basically just means that people will try to do what they view as in their interest. It also is not a requirement that all formal work be based on rationality assumptions; see, for instance, some of the work being done by Arthur Lupia that engages with work in neuroscience, or a lot of work done in complex systems modelling.

Finally, on normative work, I don't have that much of an opinion, except to say that I think you can divide positive work from its normative implications. Also, from what little I know of normative theory, sometimes concepts of welfare analysis derived from economics can be useful in structuring one's thought (e.g. I think Rawls has utility curves in one of his books?). I think John Roemer does some stuff on normative formal theory? In any event, I certainly don't think normative work should be dominated by statistics

Edited March 5, 2012 by RWBG

[balderdash]

I agree that really it's a complementary relationship. In the roughest of outlines: quants answer "littler" questions (not in a pejorative sense!) with great rigor, while quals go all meta but are more liable to err and allow their biases to creep in. Obviously, there are exceptions to both. But I think the conversation among those who mobilize divergent methods is really useful for shaping interests and ultimately reaping better research. That might be part of the reason why mixed methods analysis is up-and-coming.

[Bdeniso]

I think there is clear evidence that the field benefits from both. I think the problem is some in the field are resistant to quant and some are trying to be economists and feel quant only is what is needed. Both of these biases needs to be rectified and hopefully our era of scholarship can help lead the fight for mixed methodology