Archive for November, 2014

25th Anniversary ButterflyAs Randi has shared in some of her posts, we at RK&A value the concept and four actions associated with Intentional Practice—Plan, Align, Evaluate, and Reflect. A few weeks ago, Randi wrote about Align, which she noted is the most complex. Today I write about Reflect, which is probably the most alluring of the four actions. At the same time, it is also the most easily dismissed—the one swept aside as a luxury. In workshops we facilitate, we usually show museum staff the Cycle of Intentional practice and ask them what percentage of time they spend on each of the fours actions. Inevitably, reflection falls short, usually garnering between 5 and 10 percent. Staff tell us this isn’t for a lack of desire or need; they wish they had more time for reflection. But as is so common in our modern world, we tend to get stuck in the continual act of “doing.” Take, for example, how difficult it has been for me to sit down and write this blog post…. Certainly I am not immune.

I love the literal manifestation of reflection, which is when light strikes a surface and bounces around in unusual ways, making us see something we didn’t see before. For me, the allure of reflection in its literal form is not that different from what happens when we talk about reflection in evaluation. When it comes to evaluation, reflection leads to insights, ah-ha moments, and new ways of seeing, thinking, and knowing. This happens when I invariably ask one of my favorite question, “What does it mean?” and even when reflection is very difficult—for instance—when I ask, “What does failure mean?” the pay-off is usually worth the pain.

Billboards at Night

Billboards at Night (Detroit), Knud Lonberg-Holm, 1942

Despite its allure, we can’t avoid the fact that reflection is easily dismissed, postponed, and overlooked. Why is this? The answer may lie partly in how hard and sometimes downright painful it is to reflect. At times it is too difficult to consider those important questions; it feels easier to ignore them and continue “doing.” In the same way, reflection in its literal form can sometimes be painful, such as the way light reflecting off the hood of a car is blinding or when light bouncing around becomes disorienting. But I don’t think discomfort is the barrier—from my experience, it seems the demands (both internal and external) to produce thwart our intentions of taking the time to reflect.

In our work with evaluation, reflection is critical. Without taking the time to reflect on the meaning of data, evaluation results fall flat and hollow. As evaluators it is our duty and privilege to ask and try to answer hard questions about what data means, what it tells us. And we do that. But even though we possess a valuable outsider perspective and can offer significant insights about evaluation findings, the insider perspective is equally important. And, our work is at its best when reflection happens collaboratively between the client and us.

So, we try as often as we can to facilitate a Reflection Workshop at the end of a project. In a Reflection Workshop we meet with staff from across the museum to collaboratively explore the question, What have we learned? from the evaluation. We don’t simply present findings; rather, we pose questions and facilitate discussion to help staff explore the meaning of the evaluation findings. And, we don’t shy away from negative findings; rather, we use those as opportunities for understanding and growth. The purpose of the workshops is to come to some conclusions about ways to improve the effectiveness of a program. But more than anything, the purpose is to simply take the time to ask challenging questions and think deeply.

Read Full Post »

25th Anniversary ButterflyAt RK&A, we think a lot about intentional practice and we encourage our clients to do the same. In planning meetings and reflection workshops, we ask clients to think about which elements of their work align with their institutional mission and vision (check out Randi’s blog post for more about the challenges of alignment). We push them to consider who might be the right audience for their program or exhibition, and we ask them to talk about the intended outcomes for their projects. Posing these kinds of questions is much easier for an “outsider” to do because we don’t have institutional baggage or a personal connection to a problem project. As consultants, we aren’t beholden to the way things have always been done. I get it – it can be hard to let go; but seeing clients seek information to make informed decisions is a powerful, exciting process. These clients want more information. They are willing to try new things, to change old (and sometimes new) programs to see if they can improve upon the results. These are museum professionals who want the very best experiences for their visitors.

We recently completed a project with a history museum and the results were, well, not as rosy as one might hope. Change is HardAfter explaining the challenges of changing students’ perspectives in a short, one-time museum visit, we started talking about what could be done to increase the effectiveness of the program. One of our suggestions was to increase the time allotted for the program and rather than spending that extra time in the exhibition, use that time to facilitate a discussion with students so they can process and reflect on what they had seen. Changing a program’s format and duration is a difficult task for the museum to undertake – it may require extra staff and certainly a different schedule – but it could make a difference. A few days later, our client asked us if there are any studies that show that longer programs are more effective. After failing to come up with any examples (if you know of any such studies, please leave a comment), the client asked for another study to see if a longer program leads to a different outcome.

As an evaluator, I want to support museums as they change the way they do their work. Evaluation can provide the necessary information to see if new ideas work. It can give clients the data-based push they need to let go of the way things have always been done and to try something new. If nothing else, the evaluation process can be a forum to remind people that even when you are changing course, there is a place for you on the Cycle of Intentional Practice: Plan, Align, Evaluate, Reflect.

Read Full Post »

The case study below is from a summative evaluation RK&A did with the Wildlife Conservation Society.  The Madagascar! exhibit at the Bronx Zoo is an indoor exhibit that allows visitors to come face-to-face with wildlife from this island habitat.  The exhibit also features a film, Small Wonders, Big Threats, that addresses environmental challenges the island is facing.

Madagascar! [2009]

A summative evaluation with a zoo

The Wildlife Conservation Society (WCS) contracted Randi Korn & Associates, Inc. (RK&A) to evaluate its new exhibition, Madagascar!, located at the Bronx Zoo. Madagascar! showcases the wildlife and landscapes of the world’s fourth largest island. Built in the historic Lion House, the exhibition transformed the interior, while preserving the historic building’s Beaux-Arts beauty. The exhibition offers opportunities to see the island through the eyes of a conservationist at various interactive stations.

How did we approach this study?

RK&A worked with WCS to clarify its goals and objectives for Madagascar! and to identify criteria to measure visitor outcomes. We conducted a summative evaluation that employed a rigorous, modified pre-test/post-test design to measure visitor learning and attitudinal changes. Through in-depth open-ended interviews, we explored visitors’ attitudes toward and understandings of threats to Madagascar and its animals as well as knowledge of WCS’s conservation efforts on the island. We then scored the interview data using rubrics and compared the achievement of eight objectives by visitors who had not seen the exhibition to visitors who had seen the exhibition.

What did we learn?

Findings demonstrate that the exhibition was extremely successful at achieving its goals. Statistically significant findings showed that visitors who experienced the exhibition gained the following new knowledge, ideas, and beliefs, including: 1) enhanced interest in the animals of Madagascar based on knowledge of their habits, environment, and endangered status (versus interest based solely on novelty); 2) knowledge that Madagascar’s environment and animals are threatened, especially by the loss of trees; and, 3) an understanding of why conservation scientists (including those from WCS) are in Madagascar: to study the animals and environment so that they can implement appropriate conservation strategies toward its protection.

What are the implications of the findings?

Even though recent public discourse on global warming has grown substantially, the general public’s familiarity with environmental issues still tends to be vague or even ill-conceived. Yet, findings demonstrate that Madagascar! shifted visitors’ knowledge of conservation science toward a more accurate, specific, and concrete understanding. These positive findings are remarkable when one considers how difficult it is to change people’s knowledge and attitudes, particularly in one relatively short visit to a single exhibition. Through experiences in exhibitions like Madagascar!, visitors assimilate new ideas and perceptions with their pre-existing ideas and perceptions and create new meaning. The exhibition effectively utilized simple low-tech interactive exhibits, large-scale video walls, live interpretation, and intimate, close-up looks at animals to connect visitors to the environments and wildlife of Madagascar. Evaluation results have shown that zoos can be appropriate environments for moving visitors beyond the novelty of seeing wild animals to developing an understanding of where the animals come from, why they are important, and how conservation efforts can protect them.

Read Full Post »