Economic
and Game Theory
Human Subject Protection in Economics
Back
Email from Tom Palfrey and David Levine, co-directors of CASSEL to Dan Newlon, NSF Economics Program Director dated May 10, 2001. This has been edited to remove the names of specific individuals, but not otherwise.
Dear Dan:This is not a happy letter. We need to let you know what is happening with CASSEL, and seek your advice on the situation. The summary of the situation is this: we were prepared to begin doing research in early February. It is now May. To date, however, we have conducted only a single one hour session at CASSEL and there is little prospect that we will be able to do much experimental economics for the remainder of this quarter. The reason is simply that we have been unable to get approval from our human subjects committee to conduct any economics experiments under conditions that are scientifically acceptable. There some hope that we may be able to resolve this problem in the next couple of weeks, but given the events so far, there is a real prospect that the crisis will continue to drag on and seriously hinder a number of NSF-funded research projects.
Here is a brief chronology: The issue of whether experimental economics is ordinarily exempt was first informally discussed with the the UCLA OPRS in late July 2000. Anticipating that there might be issues to be resolved with the IRB, David submitted two requests to the IRB for exemption in October and November 2000, before construction on the lab was completed. Both were approved as exempt. Shortly before CASSEL was scheduled to open, and just as we were preparing to conduct our first experiments, problems with the IRB commenced. We have a website for recruiting subjects. On February 1 we received a notification from the IRB that they would have to approve our website before it could be used for recruiting. We did not argue with the IRB, but attempted to comply. Never-the-less it took until March 23 for the website to be approved. We again prepared to conduct the experiments that we had received approval for in October/November 2000, and simultaneously submitted several other applications for exemption.
On April 16 we received notification from the UCLA OPRS that no experiments conducted at CASSEL could ever be granted exemption. At this point we became quite concerned, since this seemed capricious, and we sent a letter to the OPRS complaining about their change of course, and raising with them three issues: whether they had jurisidiction over non-UCLA researchers; whether written and signed consent forms are needed in this kind of research; and whether this research is properly exempt.
At the same time, we raised these issues with the UCLA administration, and pointed out that the clock was ticking on conducting research during the Spring quarter. It was agreed that we would have a meeting with the OPRS to discuss these issues the following week. However, instead of meeting with us, the OPRS made themselves unavailable by phone and by email, and without consulting with us claim to have gone to the OPRR in Washington DC over the issue of exemption. They claim the OPRR ruled that in fact this type research cannot be exempt. We learned of this by mail on May 1.
On May 1, they responded to our letter in a 6-page memo citing various regulations that: (1) they do have jurisdiction over non-UCLA experiments if they are conducted using UCLA facilities; (2) experimental economics research is not exempt; and (3) they will not waive written consent forms. We agree with their verdict on (1), but disagree with both their conclusions and their logic concerning (2) and (3). We were also quite upset with the tone of the memo, which was hostile and condescending. This, in spite of the fact that they agreed the experiments pose no risk.
Since that time we have been working nonstop trying to explain our research protocols and satisfy the various requirements for non-exempt research, which can be pretty onerous. The first step was trying to write detailed consent forms in a way that will satisfy them. The next step involved them rewriting and expanding our consent forms in totally unacceptable ways. The biggest problem (but not the only one) was the specificity with which they wanted us to describe the purpose of the research. We do not feel as a rule we can reveal the purpose of an experiment with any specificity, since it creates framing effects. The IRB now recognizes this as a legitimate argument, and at least will not require us to disclose our hypotheses prior to conducting the experiment. But that turned out to be a hollow victory. Instead, they want us to debrief the subjects after the experiment in detail about the purpose of the experiment and the kinds of hypotheses we are trying to test. Of course we cannot do this in a large lab like CASSEL where many people are running many related experiments. Individual subjects participate in several experiments, and so the subject pool would quickly become contaminated. We had extensive discussion with our board over the weekend, and believe we all agree on this basic point. David has explained this to the UCLA OPRS. They seem sympathetic, but we will need approval from the IRB itself not the staff of the OPRS, and given OPRS/IRB behavior in the past, it is hard to have any confidence in anything they say, or even be certain something has been approved. Moreover, we have never been able to speak to anyone on the committee.
Here is what we are requesting and why:
(1) Exemption: As standard practice, our projects should be classified as exempt. Note that at UCLA this requires us to specifically apply to the OPRS for exemption for each proposal. Instead, the OPRS indicates that they do not regard our projects as exempt, but only as qualified for expedited review. We are now discovering that this category is subject to all of the cumbersome human subject regulations that are costly and compromise the science.
(2) No Signed Consent Forms: As a standard practice, our subjects should not be required to sign written consent forms. As you know, we instruct them (without deception) in the fine details of the experiment verbally, and inform them when they sign up that their participation is entirely voluntary and their decisions are kept confidential. This is consistent with Federal Regulations regardless of whether or not it is exempt.
Both of these positions are disputed by the IRB. We are having trouble figuring out why they are intransigent. While we still hold out some hope that they can be reasoned with, but we are running out of arguments and time.
(3) No Debriefing: As a standard practice, research hypotheses should not be revealed before or after the experiment.
The OPRS has indicated that they are sympathetic on (3), but have not issued a formal ruling.
It may be useful to give a bit of the politics. Our administration is supportive, but is reluctant to be seen as taking action against the OPRS on the basis of complaints from investigators whose applications have been rejected. This seems especially the case at UCLA. In addition both the OPRS and IRB themselves seem very jittery. All of this probably stems from actual malfeasance by some of our medical investigators, combined with the fact that the OPRR and NIH seem to be on the warpath at the moment. However, the UCLA OPRS and IRB have gone far beyond assuring themselves that our research is harmless, a point which they seem to concede. And it is hard to see how making it difficult to conduct harmless research projects, or subjecting them to exacting scrutiny, protects subjects from harmful research.
There is also a broader social problem: it is unclear to us whether the UCLA OPRS went to the OPRR and obtained a general ruling that that economics experiments are not exempt. In their letter to us, they say
"DHHS/OPRR clarified for the UCLA IRBs that experiments that involve manipulation of the subjects’ environment, such as asking subjects to come to a laboratory in order to participate in a computer experiment does not qualify for exemption under 46.101(b)(l)."
There is fear by us and by our board that this or other actions by the UCLA OPRS might take could have a negative impact not only on CASSEL, but on experimental work in economics more broadly. Regardless, one area where the NSF can be particularly helpful is in clarifying for us and for the OPRS at UCLA current interpretations by the Federal Government, including both the NSF and OPRR, concerning economics experiments.
Finally, we want to tell you the saga of our visiting graduate student, because it illustrates pretty clearly the problems we have had with our OPRS/IRB. We had a graduate student from another University who was visiting in Los Angeles, and asked to do some research in our lab. Since part of our charter is to be open to outsiders, and as he was proposing some interesting work on threshold effects in public good contribution games, we felt that this would be a good place to start. We invited him to conduct his research at CASSEL, and have also provided him with some programming support so he could do his experiments. Along with us, he had to wait for our website to be approved before doing anything. After receiving approval to use our website for recruiting on March 23, we submitted his application for exemption to the IRB. On April 16 we were told that none of our applications could be exempt. Given the lateness in the quarter and his need to get his research done, we decided we should simply have him reapply as nonexempt, and try to satisfy the IRB as quickly as possible. Here is what actually happened. We resubmitted the application for expedited review more or less immediately. On April 23, David sent an email reminding them that we had resubmitted for expedited review, indicated our willingess to comply with reasonable changes, and urging quick action. On April 26, not having heard anything David called the OPRS. He was told that the application had been reviewed, looked fine, and was just awaiting the signature of the IRB chair. On April 27 the application was rejected on the grounds that he listed earning money as a benefit to the subjects, and provided an incomplete script for the second half of his experiment. We immediately fixed this and resubmitted. Approval was received on May 1. On May 2 we received notification that the approval had been withdrawn. We have gone through some more rounds with them since then, but at this point, as he has no affiliation with UCLA, we have simply moved his research to Caltech. By way of contrast, the UCLA human subject website indicates in several places that application for exempt research will be evaluated within three working days.
Any advice you have to offer would be most welcome. Perhaps you can suggest a convenient time for us to call you.
Regards,
David Levine and Tom Palfrey
CASSEL co-directors