Measuring healthy conversation

By and
Monday, 30 July 2018

Earlier this year, as part of our global health initiative, we committed to serving the public conversation and working to increase the collective health, openness, and civility of the dialogue on our service. We want everyone’s experience on Twitter to be free of abuse, harassment, and other types of behaviors that can detract or distort from the public conversation. Work has been underway for months to make progress in these areas, such as making changes so you can be confident in your follower counts, introducing new signals into how we present and organize Tweets, sharing more on our expanded efforts to fight malicious automation, and acquiring Smyte to address safety, spam, and security challenges more quickly and effectively.

If you want to improve something, you must be able to measure it. Inspired by our friends at Cortico, a nonprofit research organization that has taken the time to more deeply understand the concept of measuring conversational health, we wanted to examine what this would look like for Twitter. However, we knew this was not something we should address alone, and put out a request for proposals to help us develop a measurement framework. Partnering with outside experts who can provide thoughtful analysis, external perspective, and rigorous review is the best way to measure our work and stay accountable to those who use Twitter. Since March, we’ve reviewed more than 230 proposals from around the world that challenged us to think critically about how we can define and measure the health of public conversation on Twitter.

Our teams reviewed each proposal in depth. To start, we organized a review committee with representatives from diverse groups across Twitter, including Engineering, Product, Machine Learning, Data Science, Trust & Safety, Legal, and Research. In the first round, each proposal was reviewed and scored by two different people. The 50 proposals that made it to the next round were each reviewed by a total of four different people, and then the 16 semifinalists were further evaluated by a small committee of subject-matter experts. Finally, we interviewed a handful of finalists by video conference before making the final selection. By the end, we had written more than 350 reviews in total and examined the finalists in depth; as a result, we’re confident in the process that led us to the two extraordinary partners we’ve selected.

Examining echo chambers and uncivil discourse

Led by Dr. Rebekah Tromble, assistant professor of political science at Leiden University, along with Dr. Michael Meffert at Leiden, Dr. Patricia Rossini and Dr. Jennifer Stromer-Galley at Syracuse University, Dr. Nava Tintarev at Delft University of Technology, and Dr. Dirk Hovy at Bocconi University, this project will develop two sets of metrics: how communities form around political discussions on Twitter, and the challenges that may arise as those discussions develop.

“In the context of growing political polarization, the spread of misinformation, and increases in incivility and intolerance, it is clear that if we are going to effectively evaluate and address some of the most difficult challenges arising on social media, academic researchers and tech companies will need to work together much more closely. This initiative presents an important and promising opportunity for Twitter and our team of researchers to share expertise and work on solutions together,” Tromble said.

The Leiden-led project will primarily focus on two key challenges: echo chambers and uncivil discourse. Based on their past findings, echo chambers, which form when discussions involve only like-minded people and perspectives, can increase hostility and promote resentment toward those not having the same conversation. The project’s first set of metrics will assess the extent to which people acknowledge and engage with diverse viewpoints on Twitter.

The second set of metrics will focus on incivility and intolerance in Twitter conversations. The group has found that while incivility, which breaks norms of politeness, can be problematic, it can also serve important functions in political dialog. In contrast, intolerant discourse — such as hate speech, racism, and xenophobia — is inherently threatening to democracy. The team will therefore work on developing algorithms that distinguish between these two behaviors.

The project brings together scholars with different backgrounds and expertise, bridging political science, communication, and computer science. The six scholars working together across four universities will combine deep social scientific subject expertise with strong technical expertise to develop and adapt measures of healthy conversations on Twitter.

Bridging gaps between communities on Twitter

Professor Miles Hewstone and John Gallacher at the University of Oxford, in partnership with Dr. Marc Heerdink at the University of Amsterdam, will be studying how people use Twitter, and how exposure to a variety of perspectives and backgrounds can decrease prejudice and discrimination.

“We’re very excited about the opportunity to work with Twitter on investigating the important social challenges of a digitally connected world,” said Miles Hewstone, Ph.D., professor of social psychology at Oxford University. “Evidence from social psychology has shown how communication between people from different backgrounds is one of the best ways to decrease prejudice and discrimination. We’re aiming to investigate how this understanding can be used to measure the health of conversations on Twitter, and whether the effects of positive online interaction carry across to the offline world.”

The project builds on Professor Hewstone’s longstanding work to study intergroup conflict. When the communication between groups contains more positive sentiments, cooperative emotions, and more complex thinking and reasoning from multiple perspectives, prejudice is reduced and relations can improve. This was previously demonstrated in a variety of contexts, including in post-conflict community building efforts in Northern Ireland as well as in online communities. As part of the project, text classifiers for language commonly associated with positive sentiment, cooperative emotionality, and integrative complexity will be adapted to the structure of communication on Twitter.

What’s next?

Ensuring we have thoughtful, comprehensive metrics to measure the health of public conversation on Twitter is crucial to guiding our work and making progress, and both of our partners will help us continue to think critically and inclusively so we can get this right. We know this is a very ambitious task, and look forward to working with these two teams, challenging ourselves to better support a thriving, healthy public conversation.

Thank you to everyone who submitted an application — we are grateful to all who took the time and initiative to help shape this critical work. We look forward to working with these two organizations, and the broader academic community to counsel and support our teams to create healthier experiences on Twitter.

This post is unavailable
This post is unavailable.