Study: No race or gender bias seen in initial NIH grant reviews

Examinations of National Institutes of Health grants in the last 15 years have shown that white scientists are more likely to be successful in securing funding from the agency than their black peers.

A new study from the University of Wisconsin–Madison shows that bias is unlikely to play out in the initial phase of the process NIH uses to review applications for the billions of federal grant dollars it apportions annually to biology and behavior research, even though the reviewers at that early stage in the process are aware of each grant applicant’s identity.

“Absence of bias here does not mean there is no bias in the entire review process,” says psychology professor Patricia Devine, who secured NIH funding to assess the agency’s review process. “But we’re confident that this is a strong and valid result showing no evidence of bias against female and black scientists in this first review of grant applications.”

Their findings were published this week in the journal Nature Human Behavior.

A team of UW–Madison psychologists selected 48 actual grant proposals sent to NIH — half of which were awarded funding — and stripped them of any identifying information. Each study was reproduced four times with new fictitious names, information implying the applying scientist was a white man, white woman, black man or black woman.

They recruited more than 400 scientists with credentials qualifying them to serve as reviewers for grant applications submitted to NIH’s four largest institutes — most of whom had served as NIH reviewers, applied to NIH for funding, or both. Each volunteer reviewer received three of the experiment’s grant applications: two ostensibly written by white men, and one with names reworked to appear as authored by a black woman, black man or white woman.

The reviewers read the applications and returned detailed critiques as they would in an actual, initial NIH review, including scores in several areas (the most consequential being an “impact” score).

There was no consequential difference in the impact scores or the reviewers’ use of descriptive language that can be consequential for how grants are perceived — how they apportioned words like “diligent,” “fails,” “limits,” “convincing” or “remarkable” —  in their reports.

“That will be surprising to people. This is a place where people will assume there is bias,” says Devine. “But the reviewers were focused on the actual grants that were in front of them, and not the social categories of the applicants.”

The initial review phase may not be a likely step to favor a race or gender. The first assessment is an in-depth affair, with long, written justifications for judgments that don’t lend themselves well to the usual trappings of bias.

“We know from other areas of research related to bias that when people have a lot of information, take more time to think, and are more accountable for how they act, bias is less likely to show up,” says William T. L. Cox, a UW–Madison scientist. Cox and University of Arkansas professor Patrick S. Forscher were members of Devine’s lab and are co-authors of the study with Devine and UW–Madison psychology professor Markus Brauer.

“We have captured the part of the review process where reviewers are taking the most time and paying the most attention,” Cox says. “So, it may be the area where we would least expect bias to appear.”

The researchers also found no difference in the treatment of high-scoring grants that NIH had actually funded and lower-scoring grants that missed out.

“The social science literature tells us that when things are ambiguous, men tend to get a bit of a bonus while women and black people are kind of downgraded,” Devine says. “We didn’t see that in the reviews supplied in this experimental study.”

The painstaking research took five years to complete, and included confirming the findings by applying more than 4,500 possible differences in analytical emphasis to the data. More than 97 percent of the time, the results showed no significant difference in treatment of applicants based on race or gender.

“But we don’t want to be understood to suggest that we don’t think there is any bias in the process,” says Devine, who admires NIH for supporting the kind of examination that could have uncovered a bias among reviewers evaluating grant proposals. “If there is bias, I don’t know yet where it is and how it manifests.”

In later steps in the grant review process, which can involve shallower analysis, sometimes brief debate and less individual accountability for reviewers, Cox points out, it’s more likely bias could creep in.

The bias seen in the apportionment of funding may not be built into the review process at all. Grant applications submitted by women and black scientists may be somehow different than those written by white men, the researchers suggested.

Letters of recommendation written on behalf of black scientists may undermine them in subtle ways relative to peers, for example. Editing help from overcompensating white or male colleagues may be less constructively critical. Women may be more cautious than men in their writing, including more qualifying statements and fewer bold claims.

“Those things would be important to know,” Devine says. “If anybody isn’t receiving careful training on how to engage effectively in this process and write grants in a way that increase their chances for funding, that’s something you could train people on and a way to turn the tide.”