Year: 2023

When US cognitive scientist Joshua Hartshorne was investigating how people around the world learn English, he needed to get tens of thousands of people to take a language test. He designed ‘Which English?’, a grammar game that presented a series of tough word problems and then guessed where in the world the player learnt the language. Participants shared their results — whether accurate or not — on social media, creating a snowball effect for recruitment. The findings, based on data from almost 670,000 people, revealed that there is a ‘critical period’ for second-language learning that extends into adolescence¹.

This sort of ‘gamification’ is becoming a powerful research tool across fields that study humans, including psychology, neuroscience, economics and behavioural economics. By making research fun, the approach can help experiments to reach thousands or millions of participants. For instance, experiments embedded in a video game demonstrated that the layout of the city where a child lives shapes their future navigational ability². Data from a digital word search showed that people who are skilled at the game do not necessarily give better advice to those trying to learn it³. And a dilemma game involving millions of people revealed that most individuals have reliable moral intuition⁴.

Gamification can help to avoid the pitfalls of conventional laboratory-based experiments by allowing researchers to study diverse populations, to conduct more-sophisticated experiments and to observe human behaviour in naturalistic environments. It can improve statistical power and reproducibility, making research more robust. Technical advances are making gamification cheaper and more straightforward, and the COVID-19 pandemic has forced many labs to move their human experiments online. But despite these changes, most have not yet embraced the opportunities gamification affords.

To reach the full potential of this approach, researchers must dispel misconceptions, develop new gamification technologies, improve access to existing ones and apply the methods to productive research questions. We are researchers in psychology, linguistics, developmental science, data science and music who have run our own gamified experiments. We think it’s time for science to get serious about games.

Games theory

Gamification motivates people to participate in experiments by incorporating point-scoring, competition, feedback about performance and the opportunity to learn about oneself through play (see ‘Take part in gamified research’). In early forays in 2005 and 2008, cognitive neuropsychologist Laura Germine, now at Harvard Medical School in Boston, Massachusetts, adapted psychological assessments and placed them on her citizen-science website, TestMyBrain.org. Her work showed that self-selected samples can produce high-quality data even when participants are unsupervised and unpaid⁵. As gamification has developed, it has united developmental psychology with computer science, web development and user-experience research to create exciting, immersive encounters for participants.

Gamification can include transforming experiments into bespoke games, embedding experiments in existing games and extracting data from ongoing ones. The popular game Wordle — in a sense, the world’s largest psycholinguistics experiment — has already inspired investigations on topics such as optimization problems in active learning⁶ and the contexts in which people cheat⁷.

Reproducibility: expect less of the scientific paper

Participants in conventional lab-based studies of human behaviour are often few in number and WEIRD (that is, from Western, educated, industrialized, rich and democratic societies). This leads to results that are statistically imprecise or irreproducible, or that cannot be generalized to other groups.

The massive data sets enabled by gamified science can help to address questions about reproducibility and generalizability. For example, small studies had shown that a person’s experience of speaking a tonal language — one that uses pitch, or tones, to distinguish between words, as in Mandarin — alters their ability to perceive musical pitch. However, such work had been conducted largely in Mandarin or Cantonese. To explore tonal languages that are less commonly studied, Jingxuan Liu, now a graduate student at Columbia University in New York City, and one of us (S.A.M.) used data from a popular web-based quiz, ‘Test your Musical IQ’, to replicate these findings in half a million speakers of such languages, including Ewe, spoken in West Africa, and Burmese, used in Myanmar⁸.

Issues of reproducibility and generalizability are particularly acute for scientists who work with hard-to-reach study populations, such as children. Gamified experiments have the potential to encourage participation from children in settings such as schools or museums, rather than requiring a special trip to a lab, which only some families have the time for or interest in doing. For example, one of us (B.L.) installed a ‘Let’s Draw!’ kiosk at the Children’s Discovery Museum of San Jose in California. Children visiting the museum were asked to draw various things — such as a watch or a tiger — and then play games trying to recognize each other’s drawings. Over the next 18 months, the kiosk collected more than 37,000 drawings from some 8,000 children aged between 2 and 10, creating the world’s largest corpus of children’s drawings and showing how object recognition changes with age⁹.

Games played in the lab can also improve the robustness of data. Two of us (A.B.-L. and D.G.W.), working with psycholinguist Joe Toscano at Villanova University in Pennsylvania, wanted to study how people communicate through prosody — the patterns of intonation, pausing and rhythm that convey what information is important, as well as sarcasm and humour. The stilted conversations that participants usually produced in the lab limited these acoustic cues, and previous results on prosody were inconsistent. We designed puzzles in the game Minecraft that players had to solve by communicating while absorbed with playing. This produced more-natural dialogue and higher-quality data, showing that participants used speech intonation and duration to convey subtle information, such as whether a word was new to the conversation¹⁰.

Invest 5{af0afab2a7197b4b77fcd3bf971aba285b2cb7aa14e17a071e3a1bf5ccadd6db} of research funds in ensuring data are reusable

Compared with lab-based research, online gamified experiments make it easier and cheaper to recruit diverse participants, including those from groups that are under-represented in scientific research. In a study published last year, a gamified experiment on infant-directed speech and song recruited hundreds of people who self-identified across three gender categories and a wide range of household incomes and ethnicities in the United States¹¹. More broadly, translating a gamified experiment and distributing it globally on the Internet can help to reduce the deeply English-centric nature of behavioural-science research¹².

The barriers to gamification are receding. Coding an experiment to run online used to require a lab to maintain and develop custom code. But the past few years have seen an explosion in free, open-source tools in mainstream programming languages, such as JavaScript and Python. One of the most common tools for basic human experiments on the Internet, the JavaScript library jsPsych¹³, was launched in 2012 by Josh de Leeuw, a cognitive scientist at Vassar College in Poughkeepsie, New York. It has been used in nearly 1,000 published papers, according to Google Scholar.

The jsPsych tool provides a means for researchers to show their participants stimuli (text, images, audio, video); to ask questions about them; to collect responses in various ways (multiple choice, free text, clicking or tapping an image); and to output data in a structured format. Most importantly, de Leeuw maintains an active user forum and GitHub repository where researchers can discuss questions and collaborate on its codebase. This and similar tools — such as lab.js, Open Sesame and psychTestR — make it straightforward for researchers to start creating behavioural experiments online (see ‘Get into gamification’).

Ups and downs

Gamified experiments have clear weaknesses. Many scientists are used to having total control over their lab environments: they can observe participants’ behaviour directly during experiments and check that people are who they say. Critics might be wary of losing this control, or might worry that people will not engage fully with the tests or will warp results by faking their identities, completing games multiple times or participating maliciously using Internet bots.

These criticisms can be partly assuaged. Gamified studies have the potential to engage participants better than lab-based ones can, because they are intrinsically motivating. Indeed, many psychologists have observed all manner of disengagement with lab experiments, such as participants looking at social media or even taking a quick nap, because tasks are often boring and low paid. The fact that many gamified experiments have gone viral on YouTube, Reddit and other social-media platforms provides a proof of concept that participants can become deeply engaged in research — perhaps more so than with conventional approaches, although we need more data to be sure.

Creative approaches to experimental design and data analysis can also give researchers more control over gamified science than might be expected. For example, in a music study aimed at children, cognitive scientist Courtney Hilton at the University of Auckland, New Zealand, and his colleagues (including S.A.M.) studied when participants played a game and compared the distribution of local times with the times adults played. The participants were less likely to play at night, implying that they were indeed children, who are more likely to be asleep at those times¹⁴. Concerns that participants could be bots can be mitigated by comparing the timing of spikes in recruitment to known events, such as a social-media post promoting a game, and by the use of manipulation checks, which are designed to be easy for a participant to answer, but hard for a scammer or bot. Ultimately, however, some research questions are better suited to the lab. The less-controlled environments used in web-based gamified science can and should complement those of lab work.

Another justified critique is that gamified science, when delivered online, leaves out people who lack smartphones, computers and Internet access. But it still offers an improvement on the status quo. Under-resourced communities are already at risk of being left out of lab experiments because labs are concentrated in a small number of places, and participants must spend time and money to visit them. By contrast, smartphones are becoming ubiquitous worldwide. In the United States, three-quarters of adults with household incomes below US$30,000 a year have a smartphone, according to a 2021 survey by the Pew Research Center in Washington DC. The fact that people from lower-income communities are more likely to connect to the Internet using smartphones than with desktop computers means it is a high priority for researchers to develop mobile-friendly experiments that work when Internet connections are slow or patchy.

Gamified experiments, especially large online ones, must also address issues concerning copyright, privacy, data storage and data accessibility. Data-protection and privacy laws differ across countries, and some research studies collect information (such as that on gender or income) that falls under protected categories. To preserve privacy and avoid legal issues, researchers can avoid collecting person-related data, such as IP addresses or other identifying information.

If data are collected by a third-party company in an existing game, the information can be proprietary. A data-use agreement is needed for research and to avoid conflicts of interest. Many universities have legal and ethical teams that can advise on gamified experiments, just as they do for lab ones. For example, when gamifying an experiment involving published music recordings, one of us (S.A.M.) received helpful guidance about US copyright law from Harvard lawyers¹⁵.

Four future steps

For gamification to reach its full potential, four steps need to happen. First, researchers already in the community need to coordinate their development activities to widen access to technology and share technical skills. Labs should make their software publicly available, so that scientists starting out don’t need to reinvent the wheel. Although open-science practices are becoming more widespread, thanks to platforms such as GitHub, Zenodo and the Open Science Framework, they are not yet ubiquitous. The community would benefit from pooling resources to create centralized tools and best practices for technology development, as well as running conferences and workshops aimed at increasing access to gamification and citizen-science expertise.

Everyone should decide how their digital data are used — not just tech companies

Second, existing tools should be improved. Many researchers use commercial software such as Qualtrics or SurveyMonkey to conduct online surveys, but no analogous software is freely available for gamified experiments. Front-end tools such as jsPsych are generally not yet optimized for mobile-friendly experiments. Back-end tools that are accessible to scientists without substantial technical expertise are uncommon — but these would jump-start scientists’ ability to produce massive-scale gamified experiments, which are still relatively rare. The scientific community should work to develop these resources, and funders worldwide should make it a priority to support these efforts. The US National Science Foundation has already called for digital infrastructure projects as part of efforts to advance reproducibility (see go.nature.com/3xi6bae).

Third, scientists need to look to professions outside research to broaden their technical toolkits. Gamified experiments that produce lots of data could benefit from the use of software and practices that are standard in the technology industry but still underused in science. These include version control using Git (a way to systematically track different versions of code) and storing data in relational databases (such as those using SQL, a language for database management that is ubiquitous in tech companies). Research labs should consider consulting or recruiting artists, graphic designers, educators and others from diverse fields who can increase the robustness of experiments by improving game design. And hiring or contracting dedicated software developers ensures that technical skills are not lost when graduate students and postdocs leave a lab group. Funders should support these essential positions.

Finally, the community needs to improve the way it dispels inaccurate impressions about gamification, which by its nature risks sounding like a diversion rather than a serious research tool. We think the best way to do so is through science itself. If gamified research continues to enable substantive scientific discoveries, the results will speak for themselves. Scientists should try out gamification on their most creative, pressing and exciting research questions.

The onus is on the community to ensure that these experiments are credible and valuable. Researchers can do so by designing studies that attract deep engagement from the public, by using robust data-management practices, by adopting analytical strategies that demonstrate the validity of gamified data and by sharing data for colleagues to scrutinize. Editors and reviewers can support these efforts by encouraging the publication of high-quality experiments.

In 1986, the sociologist Lee Sproull suggested that researchers consider “a new tool for data collection — electronic mail”. Decades after her work on what might be the earliest web-based study¹⁶, collecting data on the Internet is powerful and routine. Gamified science has not yet reached such widespread acceptance, but we think it can, should and will.