In the Learning Games Lab, research is an essential component of creating meaningful educational media, addressing the field and users’ needs. During the design and development of an educational product, the lab uses two types of research: formative and summative. We use formative research to inform what our products should be and to test prototypes with users while we are developing them. Additionally, once the product is finished, we use summative research to evaluate and assess the product’s impact on the user, as well as to evaluate our process and learn how to improve our efforts.
This post focuses on three different kinds of Impact and Evaluation studies we are currently conducting on our products to measure impact.
Outbreak Squad Study: How Can Games Change Students’ Knowledge?
Outbreak Squad is a game about health outbreaks designed for learners in grades 5 and above. The game addresses food safety through a social studies/civics/government lens. During gameplay, players fight against food outbreaks using powers from a team of heroes to keep the community from getting sicker.
In this online study, we aim to learn what youth understand after playing the game, as well as to get their feedback on the gameplay experience. In this study, youth play the game for at least 20 minutes and answer a short survey immediately after. The survey covers players' experience, including questions such as, “what do you think about the game?” and content-related questions such as, “what is true about low-risk people in this game?”
We anticipate conducting an additional study in classrooms, to better define how the game may change students’ understanding of government careers and roles. When finalized, we’ll publish findings on the game website: outbreaksquad.org
The development of Outbreak Squad was supported by the SPECA Challenge Grant, National Institute of Food and Agriculture, U.S. Department of Agriculture, under award number 2015-38414-24223, led by the University of Tennessee – Knoxville. Research was approved by NMSU Institutional Review Board: #21655 Outbreak Squad Study Impact Measurements.
4-H online study: Identifying the potential of virtual labs to lead to other kinds of learning
Online tools often facilitate learning that can’t be done in person. We’re curious: can online programs build users’ familiarity with content and foster their interest in new, self-directed learning? Working with insects can sometimes reveal biases about touching, handling or killing insects: this can be a barrier to giving kids hands-on experiences. We are investigating the impact of an online virtual lab on users’ knowledge, their comfort with online virtual labs, and the potential of labs to counter bias about insects and encourage additional learning.
The Virtual Insect Collection Lab is an interactive that allows learners to practice pinning, pointing, and spreading insects to preserve them for scientific study. Designed to foster learners' interest in entomology, the interactive covers a topic that might get overlooked by learners. The lab provides highly detailed insect illustrations and accurate scientific pinning practices. We wanted to explore the potential of labs like this to give youth a chance to explore something they may have previously not had access to, and see if it encourages them to explore other related concepts. Virtual Insect Collection Lab is designed for youth (ages 8–18). Adults (parents, agents, leaders) will play a large role in sharing the lab with youth, so we sought to identify the impact of the lab on both youth and adults.
Participants in this study take an initial survey about their experiences with insect science and virtual labs. They then receive a link to a page with the Virtual Insect Collection Lab and other materials related to insect collections curated for this research (videos, animations, web pages). After two weeks, participants complete a final survey about their interest in science and experience with insect pinning and entomology, including questions such as: “what interests you most about learning about insects?; What would you want to avoid?” As a way to thank participants for their time, we mailed them an insect pinning kit (if they wanted one).
With this work, we hope to inform development of additional new labs to lead users to other kinds of content and additional self-guided learning. Research findings will be published on the site: insectcollectionlab.nmsu.edu
The Virtual Insect Collection Lab is based upon work supported by the SPECA Challenge Grant, National Institute of Food and Agriculture. U.S. Department of Agriculture. U.S. Department of Agriculture, under award number 2018-38414-28610. Research was approved by NMSU Institutional Review Board: #19976 Insect Pinning Testing.
Summer sessions program evaluation: What do youth learn from designing games?
Every year the Learning Games Lab provides summer sessions for kids and youth from the community. In these week-long sessions, called Think Tanks, youth have a chance to interact with professionals in the technology and design field; are exposed to careers related to technology; and build skills in creating digital tools, storytelling, collaborative work, critical thinking, and game design. In addition to providing outreach opportunities, these labs serve an important purpose: all participants are actively engaged in user testing products in development.
Youth helps us evaluate how well we did in the sessions, and reflect on what they learned. We invite youth who attend the sessions to participate in product testing and program evaluation research through a survey at the end of each session.
Findings from the previous summer session with 12 youth suggests that youth described the sessions as fun and educational and found the skills taught in session to be useful. They described meeting new people, playing/testing new games, and learning through cool challenges. The sessions made youth feel more interested in learning about game design, computer skills, and playing new games. Parents/guardians saw an improvement in their kids’ technology skills, presentation skills, and critical thinking.
The studies presented in this post show the variety of impact research conducted on our products. Our impact studies vary from product to product, but as Extension specialists, we always select one or two products annually and try to assess the impact on specific audiences.
Research is approved by NMSU Institutional Review Board: #19877 Learning Games Lab Impact.
To learn more about our research projects contact:
Barbara Chamberlin, PhD
Interim Department Head
Extension Educational Media Specialist
bchamber@nmsu.edu
575-646-2848
Written by Matheus Cezarotto, Post-doctoral Researcher, matheus@nmsu.edu