This paper is receiving a fair amount of attention, primarily as ammunition in academic culture wars. The basic exercise is an LLM-based assignment of social science extracts to one of 10 ideology levels (right to left) that are defined by views of a set of political figures plus a Monthly Review writer.
The analysis does *not* provide credible evidence on ideological bias/imbalance in social sciences. It is disappointing to see academics tout the paper. I frankly doubt most of them have even bothered to read it and certainly have not thought about the empirical methods.
What are the problems?
1. Arbitrariness of ideology scale.
The exercise reifies a set of subjective choices about ideological categories and treats their putative rank order from 1 to 10 as cardinal measures of ideological orientation. Why is Tom Cotton and not Mitt Romney the choice for center right? Why is Joe Machin the anchor for the measurements, i.e. defined as a moderate? One can go on. There is no reason to believe the results are robust to different choices of figures for the various ideological categories.
2. Cardinal measures are inappropriate for ideological positions as used in the paper. Conditional on the choices of figures to rank order different positions on a right left scale, it is not meaningful to say the ideological distance between Cotton and Manchin is the same as between Manchin and Biden, etc. By implications the ideology scores for papers are not meaningful.
3. The ideology measurements are not interpretable even in an ordinal sense. The analysis assumes a scalar notion of ideology which is not tenable.
4. The exercise is a black box as to *why* an abstract is associated with the ideology of a particular figure. Without understanding how the association is done, substantive claims the papers under study cannot be made.
The article gives an example that illustrates the problem. Ronald Oaxaca's classic 1973 paper "Male-Female Wage Differentials in Urban Labor Markets" is ranked as 7, a match to Elizabeth Warren and thus "progressive". The abstract contains this sentence
"Culture, tradition, and discrimination tend to make restrictive the terms by which women may participate in the labor force."
which presumably is the source of the match. But anyone who knows the article also knows it is an exploration of how to identify discrimination versus other sources of Male/Female labor market differences. The method Oaxaca developed in fact provide a way to remove nondiscriminatory reasons for M/F differences. The only sense in which it is "progressive" is if one thinks that interest in the role of discrimination in labor market outcomes is sufficient to label a paper as progressive. This is obviously nonsensical.
More generally the association of a paper's abstract to a political positions does not distinguish between 1) the subject matter is putatively of interest to left versus right figures versus or 2) the paper's analysis has been affected by ideology.
This identification failure also undermines the paper's claim that social science has drifted to the left. Consider economics. It is certainly true that interest in inequality has increased in economics. This reflects, in my opinion, an intellectual enrichment of the field. I would go so far as to say inequality was understudied 40 years ago. Further, the openness of economists to sociological and psychological factors in understanding inequality has changed. If, as suggested by then Oaxaca example, the LLM used here codes papers on inequality as "left" then the purported ideological drift is scientifically desirable.
If one wants to criticize the role of ideology in social science, it is necessary to delve into the details of particular research programs and specific arguments and analyses. The type of exercise conducted in this paper is not informative.
New in
@Theory_Society, the first systematic, cross-disciplinary, assessment of ideology in social science, drawing from ~600,000 social science abstracts across ~60 years.
Check it out here (open access!):
link.springer.com/article/10…