In 2014, a groundbreaking study was released. Researchers had followed activists from the Los Angeles LGBT Center as they knocked on doors of California voters. The study showed that gay canvassers were able to significantly increase support for gay marriage by simply having a single, face-to-face conversation. And, the study claimed, not only were those conversations surprisingly effective, the impact was also long lasting.
The study got a lot of attention from national media and made waves in the research world where the prevailing thought is that changing beliefs is notoriously difficult.
Among the researchers who took notice were Josh Kalla at Berkeley and David Broockman at Stanford. Kalla and Broockman were fascinated by the study and wanted to run a version of it themselves. They partnered with the LA LGBT Center to launch a new study in Florida to test if canvassers could shift opinion on transgender rights. To design their own study, they dove into the original research and methodology in detail. And as they did, they made a startling realization: the results of the original study were completely fabricated. Kalla and Broockman found inconsistencies and inaccuracies across the methodology and the results. When they released their findings, Donald Green, co-author of the original study immediately retracted the paper. Michael LaCour, the paper’s other co-author, has stood by his research.
The retraction resulted in a frenzy of corrections by media outlets. While the collapse of the original research grabbed headlines, Kalla and Broockman continued to work on the LGBT Center’s Florida canvassing campaign — essentially doing the research the original study claimed to do. In a fascinating twist, their study revealed that, while important details from the original “findings” weren’t accurate, the idea that a conversation can make a significant and lasting impact on a person’s opinion on an issue, is backed up by the numbers.
We sat down with Josh Kalla to talk through their research and how they think it fits into the current conversation on how people’s opinions evolve on some of most hotly debated issues.
Michael O’Shea: Your story related to the work that you did with Dave Fleischer and the LA LGBT center has a lot of interesting twists and turns, but let’s start by focusing on the research that you published about the canvassing effort. What exactly did you set out to find with your research?
Josh Kalla: Attitude change is something that psychologists and political scientists, and the social sciences more broadly, have really been interested in exploring since the rise of modern social science, in the early 20th century. It’s something that’s been debated, whether it’s easy, whether it’s hard, what can cause it, when can it happen.
For the longest time, the general academic viewpoint, not to oversimplify things, was that it’s really hard to reduce prejudice.
One area where it’s been hotly researched is on prejudice, really beginning with studies around racism, but you also have a lot of interest in prejudice reduction against handicapped people, against Muslims, against transgender people. For the longest time, the general academic viewpoint, not to oversimplify things, was that it’s really hard to reduce prejudice in American society, or society more generally. These are deeply ingrained beliefs that are personally held. You tend to acquire them in childhood. You’re socialized into them. Your religion, your social status, all reinforce them. These are things that are hard, if not impossible, to change. Our best hope to reduce prejudice in society is generational replacement. Maybe our children will be better, but people are born with these prejudices. They develop them. They’re hard to change. That’s the academic background.
Along comes Dave Fleischer with this innovative technique and story to try to reduce prejudice. David Broockman, my colleague, and I were fascinated by this work. Before we got started, we went to canvas with Dave and his team and thought there was really something special going on there, something worth studying. We set out to conduct this academic experiment to see whether or not the work that the Leadership LAB was doing was actually effective at reducing prejudice towards transgender people.
MO: How do you go about actually measuring that?
JK: Measuring prejudice is hard. Most people don’t want to openly admit that they’re prejudiced, particularly if they know that they’re being studied. They tend to give the “socially desirable” answer. So we conduct this academic experiment where we really separate out the research side of things from the work that Dave Fleischer and his group were doing.
First we send around 60,000 people in South Florida a piece of mail inviting them to participate in a survey sponsored by our universities, Berkeley and Stanford. This survey had a whole range of questions in them. Everything from “What’s your favorite beach in Miami?”, to “What do you think of the latest films?”, to, at the time, I think there were questions about the early 2016 presidential election. Sprinkled in there were a few questions about transgender rights. There might have been half a dozen questions, out of 50 or so questions in that survey, to make it not seem like this was a survey about transgender rights. Then, among the people who took this survey, we got their baseline measures, and we randomly assigned half of them to be canvassed by Dave and his group and to get the full, 10 or so minute conversation about transgender people and reducing prejudice. The other half of the people got what we call a placebo. This is a group that we’re not trying to change their attitudes about transgender people or prejudice, we just want to know, who are the people we could have talked about that, but let’s talk to them about something else. Then, afterwards, we follow up with an additional survey that lets us compare whether the change over time in the group that got the trans rights conversation is different than the group that got the other, unrelated conversation. We do it both immediately and then over time, to see whether this was a one-off effect (which is a big fear in the prejudice reduction literature) or whether this is a longer term, deeper change in attitudes.
MO: So what did you guys find?
JK: We found very large and long lasting effects. We found that the conversations reduced the amount of prejudice towards transgender people. It’s hard to put it on a scale that’s comprehensible, but the most comprehensible scale is the average reduction that occurred in society, more generally, towards gay men and lesbians from, I want to say, 1996 to 2012. That general reduction is what we see happening in that Miami study, with just a 10 minute conversation.
MO: It seems a lot of people grasped onto the “findings” of the original study because it makes sense on the surface. Talking to somebody of a particular group could reduce prejudice towards that group. So since that’s not the whole story, what’s the insight here? It’s just simply that having a face to face conversation and presenting people with ideas that are counter to what they already believe, regardless of who it comes from, increases empathy?
The mechanism that we think might be reducing the prejudice is this perspective taking. It’s the fancier way of saying empathy.
JK: The mechanism that we think might be reducing the prejudice is this perspective taking. It’s the fancier way of saying empathy. The conversation was around getting the canvasser to share a story of a time that they faced prejudice and discrimination. If they were transgender, they may talk about it in terms of that. If they were LGB, they may talk about it in terms of that. If they were an ally, they might talk about all the other prejudice that exists in society. Whether you’re a woman, whether you’re a minority, whether you are friends with people who face prejudice. Prejudice is so widespread that almost everyone can talk about a personal experience. Getting the person being canvassed to share a similar story and trying to, through analogy, make that connection between, “Hey, here’s a time when you felt discriminated against or when a loved one felt discriminated against. It feels really lousy. We all, collectively, think this is bad and we want to avoid this. Here’s a group that faces a whole bunch of prejudice in their lives. You probably haven’t learned much about this group. You probably haven’t heard much about this group.” This might have been the first conversation that people had ever had about transgender people or transgender rights. I think part of the effect might have just been educational, of demystifying this group you had never heard of before, then the other part is this empathy building.
MO: It seems like a lot of things that I’ve read about research that’s adjacent to your work seems to take a much more pessimistic view of our ability to change our minds. There’s a lot of talk about confirmation bias and people seeking out opinions that are the same as their own. Often, the feeling that you’re left with is, and as you said this was the prevailing thought for a long time, that it is very hard to change people’s minds. Your research goes in direct conflict with that. Can you talk about how you think this research fits into that larger discussion about how people form and change their opinions?
It’s always hard in science to try to explain why something happens.
JK: I think it’s important to keep in mind the issue of transgender rights, where it is a fairly new issue for a lot of people, as opposed to racism, or abortion, or whether or not you’re a Democrat or Republican. Those, I think, are probably conceptually very different issues for many people, particularly at the time when we were doing the study. I think that’s one more straight forward explanation as to why this study was different. It’s just this particular issue was not something that people had much exposure to before. It was not tied to other parts of their identity in a way that other issues are. David and I have another paper with the LA LGBT Center, trying to change attitudes around abortion and reducing stigma towards women who have had abortion and there we find no effect. We find that similar strategy, it’s the same group, same leadership, they were not able to change attitudes around abortion. Again, there are a whole lot of explanations you could give. Maybe it wasn’t the right script. Maybe if you were to do it again, it would be more effective. It’s always hard in science to try to explain why something happens, but at least the two data points that we have is trans rights, big effects, abortion, currently no effects. Abortion is this old issue that’s been on the political agenda for a long time. Most people in American society have thought about abortion, or pregnancy, and sex, and birth control, and that constellation of issues at some point in their lives. But trans rights are different. Me just defining what it means to be transgender might be the first time that has ever happened for someone, in a way that abortion, when you’re talking to a middle aged person, is probably not the first time that’s ever happened. I think that’s one way to reconcile the broader literature with these findings, but I also think Dave Fleischer and his group is potentially on to something with their canvassing. I spend a lot of time with various political groups, and go canvassing with them, and look at their persuasive efforts, and no one really does what LA LGBT Center does.
Related video:
