Mundane corrections to the dishonesty literature – Beragampengetahuan

There is a good deal of coverage of the more shocking reasons that papers on the psychology of dishonesty by Dan Ariely and Francesca Gino need to be corrected or retracted. I thought I’d share a more mundane example — in this same literature, and in fact in the very series of papers.

There is no allegation of further fraud here, the errors are mundane, but maybe this is relevant to challenges in correcting the scientific record, etc.

Back in August 2021, Data Colada published the initial evidence of fraud in the field experiment in Shu, Mazar, Gino, Ariely & Bazerman (2012). They were able to do this because Kristal, Whillans, Bazerman, Gino, Shu, Mazar & Ariely (2020), which primarily reported failures to replicate the original lab experimental results, also reported some problems with the field experimental data (covariate imbalance inconsistent with randomization) and shared the spreadsheet with this data.

So I clicked through to the newer (2020) paper to check out the results. I came across this paragraph, reporting the main results from the preregistered direct replication (Study 6):

We failed to detect an effect of signing first on all three preregistered outcomes (percent of people cheating per condition, t[1,232.8] = −1.50, P = 0.8942, d = −0.07 95% confidence interval [CI] [−1.96, 0.976]; amount of cheating per condition, t[1,229.3] = −0.717, P = 0.7633, d = −0.04 95% CI[−1.96, 0.976]; and amount of expenses reported, t[1,208.9] = −1.099, P = 0.864, d = −0.06 95% CI[−1.96, 0.976]). The Bayes factors for these three outcome measures were between 7.7 and 12.5, revealing substantial support for the null hypothesis (6). This laboratory experiment provides the strongest evidence to date that signing first does not encourage honest reporting.

A couple things jumped out here. First, this text says the point estimate for the effect of signing at the top on amount of cheating is d = −0.04, but Figure 1 in the paper says it is d = 0.04:

Figure 1 of Kristal et al. (2020), where Study 6 is the pre-registered direct replication.

So somehow the sign got switched somewhere.

Second, if you look at that paragraph again, there are some unusual things going on with the confidence intervals. They are all the same and aren’t really on the right scale or centered anywhere near the point estimates. In fact, it seems like it seems like a critical value (which would be ±1.96 for a z-test) and a cumulative fraction (which would be .025 and .975) got accidentally reported as the lower and upper ends of the 95% intervals. I imagine this  could happen if doing these calculations in a spreadsheet.

So in August 2021 I emailed the first author and Francesco Gino to report that something was wrong here, concluding by saying: “Seems like this is just a reporting error, but I can imagine this might create even more confusion if not corrected.”

Professor Gino thanked me for bringing this to their attention. I followed up in October 2021 to provide more detail about my concerns about the CIs and ask:

This line of work came up the other day, and this prompted me to check on this and noticed there hasn’t been a correction issued, at least that I saw. Is that in the works?

First author Ariella Kristal helpfully immediately responded with the correct information (the correct point estimate is positive, d = 0.04, and so the correct value seems to have already been used in the random effects meta analysis in Figure 1), and said a correction had not yet been submitted by they were “hoping to issue the correction ASAP”. OK, these things can take a little time — obviously important to make these corrections with care!

But still I was a bit disappointed when, in February 2022, I noticed that there was not yet any correction to the paper. So I emailed the editorial team at PPNAS, where this paper was published, writing in part:

I notified the authors of these problems in August.

I’m wondering if there is any progress on getting this article corrected? Have the authors requested it be corrected? (Their earlier response to me was somewhat ambiguous about whether PNAS had been contacted by them yet.)

I’m a bit surprised nothing visible has happened despite the passage of six months.

Staff confirmed then that a correction had been requested in October, but that the matter was still under review. (In retrospect, I can now wonder whether perhaps by this point this had become tied up in broader concerns about papers by Gino.)

In September 2022, with over a year passed since my initial email to the authors, I thought I should at least post a comment on PubPeer, so other readers might find some documentation of this issue.

As of writing this post, there is still no public notice of any existing or pending correction to “Signing at the beginning versus at the end does not decrease dishonesty”.

Of course, maybe this doesn’t really matter so much. The main result of the paper really is still null result, and nothing key turns on whether the point estimate is 0.04 or –0.04. And there is open data for this paper, so anyone who really wants to dig into that could figure out what the correct calculation is.

But maybe it worth reflecting on just how slowly this is being corrected. I don’t know whether any of my emails after the first helped move this along, so maybe really anything beyond the first email, which was easy for me to write, did nothing. Perhaps my lesson here should be to post publicly (e.g. on PubPeer) with less of a delay.

[This post is by Dean Eckles.]