Posts Tagged ‘Throwback Thursday’

Whole Garden and West Gallery exhibition [2013]

(Read the full report)

The United States Botanic Garden (USBG) contracted RK&A to study visitors’ experiences in the current West Gallery exhibition. However, after an initial meeting, USBG recognized that any changes to the West Gallery should be intentional and done in the context of staff’s aspirations for the whole Garden experience; thus, the study evolved into a more holistic endeavor with two main goals: (1) collect data about visitors’ experiences in the West Gallery exhibition to inform redesign of the Gallery; and (2) study visitors’ experiences in the whole Garden in the context of the newly-articulated visitor impact statement: Inspired by the welcoming, sensory, and restorative experience, visitors appreciate the diversity of plants, value the essential connection between plants and people, and embrace plant stewardship.

How did we approach this study?

RK&A facilitated a series of planning workshops with USBG staff to help them articulate the impact they aspire to achieve with their audiences. An Impact Framework resulted from these workshops. The Framework articulates the impact statement above, as well as audience outcomes and indicators which make the impact statement concrete and measurable. Guided by the Impact Framework, RK&A conducted an audience research study, employing four methods to explore West Gallery experiences and the Garden’s intended impact: (1) a standardized questionnaire; (2) in-depth interviews; (3) focused observations and interviews in the West Gallery; and (4) focus groups with teachers. Following the audience research study, RK&A facilitated two Using Evaluation Results workshops to help staff reflect on findings and develop action steps moving forward.

What did we learn?

The audience research study revealed many rich findings related to the Whole Garden, its audiences, and the West Gallery exhibition specifically, including visitor types that the Garden can use to inform their decision making (see full report for details). Study findings revealed that visitors’ experiences are, in some ways, well aligned with the Garden’s desired impact and, in other ways, not as well aligned. Specifically, staff used study findings to brainstorm more cohesive interpretive themes for the Whole Garden and West Gallery exhibition. Looking forward, USBG staff has two great opportunities to leverage these themes for an upcoming Garden-wide interpretive planning project and Conservatory Room evaluation.

What are the implications of the findings?

This project highlights the all-important link between planning and evaluation. Too often, evaluation is conducted in a vacuum (one program or exhibition at a time) as opposed to considering the organization’s aspirations for impacting the visitor. USBG staff recognized the need to consider changes to the West Gallery exhibition in the context of their intentions for the Whole Garden experience. In doing so, they now have baseline information about their audiences in the context of the impact they hope to achieve. This information helps USBG staff understand the alignment between their aspirations and visitors’ experiences and how they might need to change their practice to achieve greater impact.

Read Full Post »

Summative Evaluation of Cyberchase: The Chase is On! [2008]

(Read the full report)

Children’s Museum of Houston (CMH) contracted with Randi Korn & Associates, Inc. (RK&A) to evaluate the National Science Foundation-funded exhibition Cyberchase: The Chase is On! The exhibition used a popular children’s television show as an entry point to convey two key exhibition messages—“Math is a way of thinking and everyone can be successful at it,” as well as “We use math every day.” The Museum developed the exhibition for travel; thus, data for the summative evaluation were collected in two locations: Children’s Museum of Houston and the New York Hall of Science in Queens, New York.

 How did we approach this study?

RK&A combined three data collection strategies in order to generate both quantitative and qualitative data for analysis. Evaluators used these strategies to assess visitors’ experiences in the exhibition, including visitors’ use of the exhibits, visitors’ understanding of the exhibition as a whole, and the level of engagement fostered by select exhibits. Methodologies included: timing and tracking observations of visitors between the ages of 5 and 10, cued exit interviews of both adults and children, and stationed observations at two exhibits.

 What did we learn?

Data from the timing and tracking observations showed that the median time visitors spent in the exhibition was seven minutes. However, while the time spent was relatively short, observational data revealed that the exhibits fostered interactive experiences for both adults and children. For instance, timing and tracking observations showed that more than two-thirds of all visitors were coached by an adult. Further, stationed observations demonstrated that activities at the two observed exhibits were often shared or collaborative experiences, with adults either participating in or leading the activities. Findings from the interviews further contextualized this data, revealing that both children and adults were able to extract meaning from these experiences. One-half of adult interviewees spoke about the exhibition’s main idea in terms of math; similarly, slightly less than one-half of children interviewed recalled doing something math-related in the exhibition (without prompting from the interviewer).

What are the implications of the findings?

Overall, Cyberchase successfully promoted interactions between adults and children and effectively conveyed proposed themes. Math is a difficult concept to incorporate in an exhibition; however, the familiarity of the Cyberchase television show coupled with the highly interactive nature of the exhibits allowed visitors to actively engage with these demanding concepts. In addition to incorporating videos, computers, and low-tech interactive components, most exhibits created challenges for children to complete, allowing children to leave the exhibition with a sense of accomplishment for having finished a task. Notably, adults often participated in these challenges, increasing children’s understanding of and interaction with the exhibits. Both children and adults understood the theme of Cyberchase and enjoyed the interactive experience, indicating that using a familiar concept as an entry point and incorporating simple, user-friendly components in an exhibition can nurture meaningful learning experiences in the museum.

Read Full Post »

The case study below highlights two summative evaluations RK&A did at the California Academy of Sciences (CAS).  Both exhibitions debuted in the new CAS building that opened in San Francisco’s Golden Gate Park in September 2008.  Although the two exhibitions use different interpretive methods and have different learning outcomes, the two projects together highlight the importance of exhibition introductions.

Water Is Life and Altered State: Climate Change in California [2010]

Summative evaluations of two exhibitions for a natural history museum

The California Academy of Sciences (CAS) contracted Randi Korn & Associates, Inc. (RK&A) to evaluate two exhibitions debuting in the CAS’s new facility. One exhibition, Altered State: Climate Change in California, uses fossils, interactive technology, and live animals “to explore the science of climate change, the effects we may expect to see in our own backyard, and the steps that can be taken to mitigate these dramatic changes,” while the other exhibition, Water is Life, explores the importance and diversity of water using the Steinhart Aquarium’s Living Collection.

How did we approach this study?

We believe that each evaluation study is unique and should strongly consider the goals and objectives of the exhibition, program, or other endeavor. As such, RK&A worked closely with CAS to identify its goals and objectives for each exhibition. Water is Life was focused on visitor learning. In response, RK&A conducted a remedial evaluation to identify operational or conceptual shortcomings early in exhibition development, followed by a summative evaluation that employed a rigorous, modified pre-test/post-test design to measure visitor learning. For Altered State, CAS sought to understand what visitors did in the exhibition, as well as what they took away from their experiences; thus, RK&A conducted timing and tracking observations and in-depth exit interviews.

What did we learn?

While many findings were exhibition-specific, there were also two larger trends. First, both evaluations show that visitors had strong affective experiences in the exhibitions, although learning objectives were challenging to meet. For instance, in the Water is Life evaluation, findings show that visitors who went to the exhibition demonstrated much greater interest in and concern for the natural world than visitors who did not see the exhibition. However, there were few differences in the knowledge of visitors who went to the exhibition and those who did not.

Second, visitors need a strong physical and conceptual introduction to exhibitions. In the Water is Life remedial evaluation, some interviewees described way-finding issues, and a few interviewees specifically asked for a better introduction. Further, in Altered State, the open space exhibition design, with its multiple entry and exit points, likely contributed to low dwell times.

What are the implications of the findings?

 The evaluations are a keen reminder that exhibition introductions are imperative. They can set the conceptual stage for visitors, and if they are well-conceived and executed, they can also convey overarching concepts, connect subthemes, and present the intent of the exhibition. On a similar note, while the open exhibition design offers visitors a free-choice learning environment, introducing some structure and direction might help those seeking to understand the exhibition’s “big idea” without compromising the free-choice quality. Additionally, the studies reaffirmed that the unique value of exhibitions are the strong affective experiences they prompt.

 

Read Full Post »

The case study below is from a summative evaluation RK&A did with the Wildlife Conservation Society.  The Madagascar! exhibit at the Bronx Zoo is an indoor exhibit that allows visitors to come face-to-face with wildlife from this island habitat.  The exhibit also features a film, Small Wonders, Big Threats, that addresses environmental challenges the island is facing.

Madagascar! [2009]

A summative evaluation with a zoo

The Wildlife Conservation Society (WCS) contracted Randi Korn & Associates, Inc. (RK&A) to evaluate its new exhibition, Madagascar!, located at the Bronx Zoo. Madagascar! showcases the wildlife and landscapes of the world’s fourth largest island. Built in the historic Lion House, the exhibition transformed the interior, while preserving the historic building’s Beaux-Arts beauty. The exhibition offers opportunities to see the island through the eyes of a conservationist at various interactive stations.

How did we approach this study?

RK&A worked with WCS to clarify its goals and objectives for Madagascar! and to identify criteria to measure visitor outcomes. We conducted a summative evaluation that employed a rigorous, modified pre-test/post-test design to measure visitor learning and attitudinal changes. Through in-depth open-ended interviews, we explored visitors’ attitudes toward and understandings of threats to Madagascar and its animals as well as knowledge of WCS’s conservation efforts on the island. We then scored the interview data using rubrics and compared the achievement of eight objectives by visitors who had not seen the exhibition to visitors who had seen the exhibition.

What did we learn?

Findings demonstrate that the exhibition was extremely successful at achieving its goals. Statistically significant findings showed that visitors who experienced the exhibition gained the following new knowledge, ideas, and beliefs, including: 1) enhanced interest in the animals of Madagascar based on knowledge of their habits, environment, and endangered status (versus interest based solely on novelty); 2) knowledge that Madagascar’s environment and animals are threatened, especially by the loss of trees; and, 3) an understanding of why conservation scientists (including those from WCS) are in Madagascar: to study the animals and environment so that they can implement appropriate conservation strategies toward its protection.

What are the implications of the findings?

Even though recent public discourse on global warming has grown substantially, the general public’s familiarity with environmental issues still tends to be vague or even ill-conceived. Yet, findings demonstrate that Madagascar! shifted visitors’ knowledge of conservation science toward a more accurate, specific, and concrete understanding. These positive findings are remarkable when one considers how difficult it is to change people’s knowledge and attitudes, particularly in one relatively short visit to a single exhibition. Through experiences in exhibitions like Madagascar!, visitors assimilate new ideas and perceptions with their pre-existing ideas and perceptions and create new meaning. The exhibition effectively utilized simple low-tech interactive exhibits, large-scale video walls, live interpretation, and intimate, close-up looks at animals to connect visitors to the environments and wildlife of Madagascar. Evaluation results have shown that zoos can be appropriate environments for moving visitors beyond the novelty of seeing wild animals to developing an understanding of where the animals come from, why they are important, and how conservation efforts can protect them.

Read Full Post »

The case study below is from a project RK&A did with the Museum of Science and Industry in Chicago, IL and highlights the importance of iterative testing.

Future Energy Chicago Simulation [2013]

An evaluation of a multimedia simulation for a science museum

Between 2012 and 2013, RK&A conducted four rounds of formative evaluation of the Future Energy Chicago simulation for the Museum of Science and Industry in collaboration with the design firm Potion. In the simulation, up to five teams compete against each other in five games: Future House, Future Car, Future Neighborhood, Future Power, and Future Transportation. In the games, players have to make decisions that challenge them to think about energy production and usage, and they receive badges as rewards for selecting energy-efficient choices.

How did we approach this study?

RK&A included primarily middle school youth (home school groups, etc.) in testing, as they are the target audience for Future Energy Chicago. Each round of evaluation explored unique issues relevant to a particular design phase. In the first round of evaluation, RK&A tested three-dimensional paper prototypes of each game to explore middle school youth’s understanding of the concepts presented. In the next two rounds (alpha and alpha prime), RK&A tested the games on touch-screen monitors to explore each game’s functionality as well as youth’s motivations and learning, including a badge system aimed at rewarding youth’s energy-efficient choices. In the last round of evaluation, RK&A tested the games using a combination of multi-touch and projection technology that closely mirrored the final simulation environment. For each round of evaluation, RK&A staff conducted observations and interviews with middle school youth who played the games.

What did we learn?

Each round of evaluation revealed successes and challenges of the Future Energy Chicago games that MSI staff and Potion designers used to improve the games’ functionality and messaging. Throughout testing, findings revealed three key characteristics of the game that were compelling to middle school youth—variety of energy choices, opportunities to design aspects of their energy environment, and challenging energy problems to solve. Findings also revealed that youth’s prior knowledge and experiences with energy choices highly influenced the choices they made and the messages they took away from each game. A consistent challenge throughout testing was helping youth understand the idea of trade-offs in energy choices (comfort or cost versus saving energy). A badge system was implemented to address this issue, as well as to incentivize youth to select energy-efficient choices.

What are the implications?

This study underscores the importance of iterative testing when evaluating a complex digital learning environment. Not only did MSI staff and Potion designers need to understand barriers to effectively using the games, including the intuitiveness of the technology, the Museum needed to understand what about the simulation motivated youth’s game play and effectively empowered them to make smart energy choices as the future residents of Chicago. Further, RK&A facilitated reflective discussions between rounds of testing that enabled MSI staff and designers to apply the study findings and recommendations to the next round of testing, ultimately improving the overall functionality and effectiveness of Future Energy Chicago.

Read Full Post »

Emily’s last blog post (read it here) talked about when evaluation capacity building is the right choice.  When we think about building capacity for evaluation, we think about intentional practice.  This does not necessarily involve teaching people to conduct evaluation themselves, but helping people to ask the right questions and talk with the right people as they approach their work.  RK&A has found this to be particularly important in the planning phases of projects.

The case study below is from a project RK&A did with the Museum of Nature and Science in Dallas, TX (now the Perot Museum of Nature and Science) and involved an interdisciplinary group of museum staff thinking intentionally about the impact the Museum hoped to have on the community.  With a new building scheduled to open a year after this project took place, it was a wonderful time to think intentionally about the Museum’s impact.

Building Capacity to Evaluate [2012]

An evaluation planning project with a nature and science museum

The Museum of Nature and Science (MNS) hired RK&A to develop an evaluation plan and build capacity to conduct evaluation in anticipation of the Museum’s new building scheduled to open in 2013.

How did we approach the project?

The evaluation planning project comprised a series of sequential steps, from strategic to tactical, working with an interdisciplinary group of staff across the Museum. The process began by clarifying the Museum’s intended impact that articulates the intended result of the Museum’s work and provides a guidepost for MNS’s evaluation: Our community will personally connect science to their daily lives. Focusing on the Museum’s four primary audiences that include adults, families, students, and educators, staff developed intended outcomes that serve as building blocks to impact and gauges for measurement. Next, RK&A worked with staff to develop an evaluation plan that identifies the Museum’s evaluation priorities over the next four years, supporting the purpose of evaluation at MNS to measure impact, understand audiences’ needs, gauge progress in the strategic plan, and inform decision making.

The final project step focused on building capacity among staff to conduct evaluation. Based on in-depth discussions with staff, RK&A developed three data collection instruments, including an adult program questionnaire, family observation guide, and family short-answer interview guide, to empower staff to begin evaluating the Museum’s programs. Then, several staff members were trained to systematically collect data using the customized evaluation tools.

What did we learn?

The process of building a museum’s capacity to conduct evaluation highlights an important consideration. Evaluating the museum’s work has become more important given accountability demands in the external environment. Stakeholders increasingly ask, How is the museum’s work affecting its audiences? What difference is the museum making in the quality of people’s lives?

Conducting systematic evaluation and implementing a learning approach to evaluation, however, require additional staff time which is a challenge for most museums. MNS staff recognized the need to create a realistic evaluation plan given competing demands on staff’s time. For example, the evaluation plan balances conducting evaluation internally, partnering with other organizations, and outsourcing to other service providers. Also, the plan incrementally implements the Museum’s evaluation initiatives over time. The Museum will begin with small steps in their efforts to affect great change.

Read Full Post »

Today’s Throwback Thursday comes from deep in the RK&A vault – a study we did in 2002 for the National Museum of American History (NMAH) in Washington, DC.

For Which It Stands: The American Flag in American Life [2002]

Study Context

The National Museum of American History, Bering Center (NMAH) asked RK&A in 2002 to concept test ideas for For Which It Stands: The American Flag in American Life, a new exhibition that would feature the Star-Spangled Banner, the flag that inspired the national anthem. The study’s objectives were to examine:

  • The range of meanings people attach to the flag
  • How participants think their meaning of the flag has been shaped
  • Participants’ tolerance for different meanings of the flag
  • Whether seeing objects and images selected for the exhibition causes people to see the flag in new ways
  • Whether people understand the changing, complex meaning of the flag.

Approach

NMAH is part of the Smithsonian, and, as such, it tries to accommodate visitors from all backgrounds and of all ages. Like many other museums, NMAH wants to attract teens and appeal to adults. Therefore, we tested the exhibition’s concepts using focus groups. We conducted three focus groups with teens and three focus groups with adults. Sixty individuals participated in the six groups.

Findings

Through discussions about the exhibition panels, participants were exposed to new ideas and stories about the American flag, which broadened their understanding of the flag. Participants freely shared their personal stories about what the American flag means to them and everyone said they enjoyed hearing other people’s stories and ideas and thinking about the American flag in new ways. However, individuals’ personal meanings of the flag were not altered as a result of their experience.

Conclusion

Although exhibition planners initially had wanted the exhibition to change people’s meaning of the American flag, they learned that participants’ beliefs about the flag were shaped by unique experiences deeply rooted in their identities. RK&A’s audience research demonstrated that visitors bring valuable experiences to museums and their stories can add depth to other visitors’ museum experiences.

Read Full Post »

Older Posts »