Replicability Rankings of Eminent Social Psychologists

Social psychology has a replication problem.  The reason is that social psychologists used questionable research practices to increase their chances of reporting significant results. The consequence is that the real risk of a false positive result is higher than the stated 5% level in publications. In other words, p < .05 no longer means that at most 5% of published results are false positives (Sterling, 1959). Another problem is that selection for significance with low power produces inflated effect sizes estimates. Estimates suggests that on average published effect sizes are inflated by 100% (OSC, 2015). These problems have persisted for decades (Sterling, 1959), but only now psychologists are recognizing that published results provide weak evidence and might not be replicable even if the same study were replicated exactly.

How should consumers of empirical social psychology (textbook writers, undergraduate students, policy planners) respond to the fact that published results cannot be trusted at face value? Jerry Brunner and I have been working on ways to correct published results for the inflation introduced by selection for significance and questionable practices.  Z-curve estimates the mean power of studies selected for significance.  Here I applied the method to automatically extracted test statistics from social psychology journals.  I computed z-curves for 70+ eminent social psychologists (H-index > 35).

The results can be used to evaluate the published results reported by individual researchers.  The main information provided in the table are (a) the replicability of all published p-values, (b) the replicability of just significant p-values (defined as p-values greater than pnorm(2.5) = .0124, and (c) the replicability of p-values with moderate evidence against the null-hypothesis (.0124 > p > .0027). More detailed information is provided in the z-curve plots (powergraphs) that are linked to researchers’ names. An index less than 50% would suggest that these p-values are no longer significant after adjusting for selection for significance.  As can be seen in the table, most just significant results are no longer significant after correction for bias.

Caveat: Interpret with Care

The results should not be overinterpreted. They are estimates based on an objective statistical procedure, but no statistical method can compensate perfectly for the various practices that led to the observed distribution of p-values (transformed into z-scores).  However, in the absence of any information which results can be trusted, these graphs provide some information.  How this information is used by consumers depends ultimately on consumers’ subjective beliefs.  Information about the average replicability of researchers’ published results may influence these beliefs.

It is also important to point out that a low replicability index does not mean researchers were committing scientific misconduct.  There are no clear guidelines about acceptable and unacceptable statistical practices in psychology.  Zcurve is not designed to detect scientific fraud. In fact, it assumes that researcher collect data, but conduct analyses in a way that increases the chances of producing a significant result.  The bias introduced by selection for significance is well known and considered acceptable in psychological science.

There are also many factors that can bias results in favor of researchers’ hypotheses without researchers’ awareness. Thus, the bias evident in many graphs does not imply that researchers intentionally manipulated data to support their claims. Thus, I attribute the bias to unidentified researcher influences.  It is not important to know how bias occurred. It is only important to detect biases and to correct for them.

It is necessary to do so for individual researchers because bias varies across researchers.  For example, the R-Index for all results ranges from 22% to 81%.  It would be unfair to treat all social psychologists alike when their research practices are a reliable moderator of replicability.  Providing personalized information about replicability allows consumers of social psychological research to avoid stereotyping social psychologists and to take individual differences in research practices into account.

Finally, it should be said that producing replicabilty estimates is subject to biases and errors.  Researchers may differ in their selection of hypotheses that they are reporting. A more informative analysis would require hand-coding of researchers’ focal hypothesis tests.  At the moment, R-Index does not have the resources to code all published results in social psychology, let alone other areas of psychology.  This is an important task for the future.  At the moment, automatically extracted results have some heuristic value.

One unintended and unfortunate consequence of making this information available is that some researchers’ reputation might be negatively effected by a low replicability score.  This cost has be be weighted against the benefit to the public and the scientific community of obtaining information about the robustness of published results.  In this regard, the replicability rankings are no different from actual replication studies that fail to replicate an original finding.  The only difference is that replicability rankings use all published results, whereas actual replication studies are often limited to a single or a few studies.  While replication failures in a single study are ambiguous, replicability esitmates based on hundreds of published results are more diagnostic of researchers’ practices.

Nevertheless, statistical estimates provide no definitive answer about the reproducibility of a published result.  Ideally, eminent researchers would conduct their own replication studies to demonstrate that their most important findings can be replicated under optimal conditions.

It is also important to point out that researchers have responded differently to the replication crisis that became apparent in 2011.   It may be unfair to generalize from past practices to new findings for researchers who changed their practices.  If researchers preregistered their studies and followed a well-designed registered research protocol, new results may be more robust than a researchers’ past record suggests.

Finally, the results show evidence of good replicability for some social psychologists.  Thus, the rankings avoid the problem of selectively targeting researchers with low replicability, which can lead to a negative bias in evaluations of social psychology.  The focus on researchers with a high H-index means that the results are representative of the field.

If you believe that you should not be listed as an eminent social psychologists, please contact me so that I can remove you from the list.

If you think you are an eminent social psychologists and you want to be included in the ranking, please contact me so that I can add you to the list.

If you have any suggestions or comments how I can make these rankings more informative, please let me know in the comments section.

***   ***   ***    ***    ***

REPLICABILITY RANKING OF EMINENT SOCIAL PSYCHOLOGISTS
[sorted by R-Index for all tests from highest to lowest rank]

Rank   Journal R-Index (all) R-Index (2.0-2.5) R-Index (2.5-3.0) #P-vals H-Index #Pub #cit(*1000)
1 Steven J. Heine
81 44 55 197 41 83 10
2 James J. Gross
80 35 58 360 82 413 34
3 Constantine Sedikides
76 45 53 884 52 263 10
4 Bertram Gawronski
73 29 51 1717 37 113 6
5 Kathleen D. Vohs
73 36 53 452 49 158 11
6 Paul Rozin
73 39 57 155 65 218 16
7 Alice H. Eagly
72 42 50 384 61 161 18
8 Anthony G. Greenwald
72 28 51 273 64 175 26
9 David Dunning
72 30 50 674 40 105 8
10 Richard E. Nisbett
72 57 59 190 62 119 20
11 Shinobu Kitayama
70 37 44 545 35 103 15
12 Timothy D. Wilson
70 30 52 327 44 85 14
13 Mahzarin R. Banaji
68 52 53 651 60 133 22
14 Marilynn B. Brewer
68 43 46 193 49 107 15
15 Patricia G. Devine
67 44 52 1098 35 83 9
16 Susan T. Fiske
67 26 41 419 66 213 22
17 Thomas Gilovich
67 23 44 754 44 104 8
18 Daniel T. Gilbert
66 26 46 357 43 107 8
19 Hazel R. Markus
66 48 53 348 38 96 12
20 Mark P. Zanna
66 26 49 565 57 167 11
21 Wendy Wood
66 33 40 373 42 112 7
22 Brad J. Bushman
65 33 50 247 48 227 13
23 E. Tory. Higgins
65 27 44 887 72 274 25
24 Jeff Greenberg
65 30 42 679 60 154 13
25 Nira Liberman
64 25 37 1578 46 115 10
26 Caryl E. Rusbult
63 27 34 171 36 68 9
27 Dacher Keltner
63 38 43 903 60 159 15
28 Harry Reis
63 21 41 470 37 83 6
29 John F. Dovidio
63 28 37 2323 56 206 12
30 John T. Cacioppo
63 23 41 256 101 422 41
31 Nalini Ambady
63 24 42 675 51 171 10
32 Philip R. Shaver
63 24 42 675 60 184 13
33 Richard E. Petty
63 30 37 1428 69 190 20
34 Robert B. Cialdini
63 45 49 258 51 121 11
35 Michael Ross
63 38 51 631 42 245 7
36 Lee Ross
62 23 44 952 49 218 12
37 Roy F. Baumeister
62 32 44 1015 100 363 46
38 S. Alexander Haslam
62 34 41 289 52 234 9
39 Tom Pyszczynski
62 33 40 1101 60 149 13
40 Philip E. Tetlock
62 29 39 158 58 189 11
41 Arie W. Kruglanski
61 23 42 1140 50 228 13
42 Galen V. Bodenhausen
61 18 43 465 39 80 8
43 Norbert Schwarz
61 36 42 2524 49 138 13
44 Jonathan Haidt
60 16 37 98 42 84 14
45 Shelly Chaiken
60 15 36 288 46 86 10
46 Ap Dijksterhuis
59 17 36 456 42 118 8
47 Eddie Harmon-Jones
59 25 37 343 57 212 10
48 Fritz Strack
59 22 43 588 51 149 11
49 Joseph P. Forgas
59 29 43 176 37 142 4
50 Yaacov Trope
59 20 39 1957 57 135 12
51 Charles M. Judd
59 28 33 666 37 142 6
52 Craig A. Anderson
59 26 34 265 51 117 11
53 C. Nathan DeWall
58 30 41 1099 37 135 7
54 Eli J. Finkel
58 24 35 1921 38 109 4
55 Jeffry A. Simpson
58 18 28 261 42 95 7
56 Peter M. Gollwitzer
58 29 44 1711 45 158 11
57 Mario Mikulincer
58 29 34 104 68 272 15
58 Russell H. Fazio
57 23 29 464 51 134 14
59 Thomas Mussweiler
57 24 38 1128 35 85 5
60 Daniel M. Wegner
56 27 36 699 53 130 14
61 John A. Bargh
56 20 36 755 61 140 22
62 Robert S. Wyer
56 33 44 283 40 203 8
63 Carol S. Dweck
54 17 29 458 59 147 20
64 Michael Inzlicht
54 32 36 156 36 124 5
65 John T. Jost
52 21 33 249 46 132 11
66 Shelly E. Taylor
51 15 28 198 73 169 28
67 Claude M. Steele
48 38 42 376 29 49 13
68 Gerald L. Clore
45 20 35 200 37 84 9
69 Adam D. Galinsky
43 24 28 585 57 198 11
70 Robert Zajonc
39 11 26 67 40 114 16
71 Jennifer Crocker
22 5 5 99 38 68 7

 

Advertisements

2 thoughts on “Replicability Rankings of Eminent Social Psychologists

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s