25th Anniversary ButterflyThe most challenging evaluation report I’ve written consisted of 17 PowerPoint slides. The slides didn’t pull the most salient findings from a larger report; the slides were the report! I remember how difficult it was to start out with the idea of writing less from qualitative data. While I had to present major trends, I feared the format might rob the data of its nuance (17 PowerPoint slides obviously require brevity). The process was challenging and at times frustrating, but in the end I felt extremely gratified. Not only was the report thorough, it was exactly what the client wanted, and it was usable.

As evaluators, we toe the line between social science research, application and usability. As researchers, we must analyze and present the data as they appear. Sometimes, especially in the case of qualitative reports, this can lead to an overwhelming amount of dense narrative. This acceptable reporting style in evaluation practice is our default. Given the number of reports we write each year, having guidelines is efficient and freeing. We can focus on the analysis, giving us plenty of time to get to know and understand the data, to tease out the wonderful complexity that comes from open-ended interviews. As researchers, the presentation takes a backseat to analysis and digging into data.

However most of the time we are writing a report that will be shared with other researchers; it is a document that will be read by userspaper-stack-300x251museum staff who may share the findings with other staff or the board. Overwhelming amounts of dense narrative may not be useful; not because our audience can’t understand it, but because often the meaning is packed and needs to be untangled. I would guess what clients want and need is something they can refer to repeatedly, something they can look at to remind themselves, “Visitors aren’t interested in reading long labels,” or “Visitors enjoy interactive exhibits.” As researchers, presentation may be secondary, but as evaluators, presentation must be a primary consideration.

As my experience with the PowerPoint report (and many other reports since then) taught me, it can be tough to stray from a well-intentioned template. A shorter report or a more visual report doesn’t take less time to analyze or less time to write. In fact, writing a short report takes more time because I have to eliminate the dense narrative and find the essence, as I might with a long report. I also have to change the way I think about presentation. I have to think about presentation!

At RK&A, we like to look at our report template to see what we can do to improve it – new ways to highlight key findings or call out visitor quotations. Not all of our ideas work out in the long run, but it is good to think about different ways to present information. At the end of the day, though, what our report looks like for any given project comes from a client’s needs—and not from professional standards. And I learned that when I wrote those 17 PowerPoint slides!

Emily’s last blog post (read it here) talked about when evaluation capacity building is the right choice.  When we think about building capacity for evaluation, we think about intentional practice.  This does not necessarily involve teaching people to conduct evaluation themselves, but helping people to ask the right questions and talk with the right people as they approach their work.  RK&A has found this to be particularly important in the planning phases of projects.

The case study below is from a project RK&A did with the Museum of Nature and Science in Dallas, TX (now the Perot Museum of Nature and Science) and involved an interdisciplinary group of museum staff thinking intentionally about the impact the Museum hoped to have on the community.  With a new building scheduled to open a year after this project took place, it was a wonderful time to think intentionally about the Museum’s impact.

Building Capacity to Evaluate [2012]

An evaluation planning project with a nature and science museum

The Museum of Nature and Science (MNS) hired RK&A to develop an evaluation plan and build capacity to conduct evaluation in anticipation of the Museum’s new building scheduled to open in 2013.

How did we approach the project?

The evaluation planning project comprised a series of sequential steps, from strategic to tactical, working with an interdisciplinary group of staff across the Museum. The process began by clarifying the Museum’s intended impact that articulates the intended result of the Museum’s work and provides a guidepost for MNS’s evaluation: Our community will personally connect science to their daily lives. Focusing on the Museum’s four primary audiences that include adults, families, students, and educators, staff developed intended outcomes that serve as building blocks to impact and gauges for measurement. Next, RK&A worked with staff to develop an evaluation plan that identifies the Museum’s evaluation priorities over the next four years, supporting the purpose of evaluation at MNS to measure impact, understand audiences’ needs, gauge progress in the strategic plan, and inform decision making.

The final project step focused on building capacity among staff to conduct evaluation. Based on in-depth discussions with staff, RK&A developed three data collection instruments, including an adult program questionnaire, family observation guide, and family short-answer interview guide, to empower staff to begin evaluating the Museum’s programs. Then, several staff members were trained to systematically collect data using the customized evaluation tools.

What did we learn?

The process of building a museum’s capacity to conduct evaluation highlights an important consideration. Evaluating the museum’s work has become more important given accountability demands in the external environment. Stakeholders increasingly ask, How is the museum’s work affecting its audiences? What difference is the museum making in the quality of people’s lives?

Conducting systematic evaluation and implementing a learning approach to evaluation, however, require additional staff time which is a challenge for most museums. MNS staff recognized the need to create a realistic evaluation plan given competing demands on staff’s time. For example, the evaluation plan balances conducting evaluation internally, partnering with other organizations, and outsourcing to other service providers. Also, the plan incrementally implements the Museum’s evaluation initiatives over time. The Museum will begin with small steps in their efforts to affect great change.

As evalua25th Anniversary Butterflytors, we are often asked to help our clients build evaluation capacity among staff in their organization. The motivation for these requests varies. Sometimes the primary motivator is professional development; other times it is perceived cost savings (since conducting professional evaluations can require resources that not all organizations have at their disposal). We welcome when an organization values evaluation enough to inquire about how to integrate it more fully into their staff’s daily work. If an organization has a true interest in using evaluation as a tool to learn about the relationship between its work and the public, building an organization’s evaluation capacity may be quite successful. On the other hand, if the primary motivator is to save costs associated with evaluation, often the outcome is much less successful, mostly because evaluation takes considerable time and invariably there is a trade-off; when the evaluation is being done, something else is being ignored.

Evaluation capacity building can take a variety of forms. It can range from building staff’s capacity to think like an evaluator, perhaps by helping staff learn how to articulate a project’s desired outcomes (I think this is the most valuable evaluation planning skill one can learn), to training staff to conduct an evaluation from beginning to end (identifying outcomes, creating an evaluation plan, designing instruments, collecting data, conducting analyses, and reporting findings). Even among the most interested parties, it is rare to find museum practitioners who are genuinely interested in all aspects of evaluation. As an evaluator, even I find certain aspects of evaluation more enjoyable than others. I’ve noticed that while practitioners may be initially intimidated by the data collection process, they often find talking with visitors rewarding and informative. On the other hand, they have much less enthusiasm for data analysis and reporting; I’ve only encountered a handful of museum practitioners who enjoy pouring over pages and pages of interview transcripts. We lovingly refer to these individuals as “data nerds” and proudly count ourselves among them.

There is yet another challenge, and it has to do with the fact that most museum practitioners are required to wear many mountain-data-mining33hats. Conducting evaluations is my one and only job; it is what I am trained to do and have intentionally chosen for my vocation. While a practitioner may be intrigued by what evaluation can offer, often it is not the job he or she was trained or hired to do, which means that evaluation can become a burden—just one more hat for a practitioner to wear. Some organizations have addressed an organization’s evaluation needs by creating a position for an in-house evaluator and the individual who might fill that position is usually someone who is schooled in evaluation and research methodologies, much like all of us here at RK&A. I would caution organizations to be very realistic when considering building their organization’s evaluation capacity. Does your staff have the time and skills to conduct a thoughtful study and follow through with analysis and reporting? What responsibilities is your organization willing to put aside to make time for the evaluation? And, do you want your staff to think like evaluators or become evaluators?—an important distinction, indeed. Otherwise, even those with the best of intentions may find themselves buried in mountains of data. Worse yet is that what was once an exciting proposition may be perceived as an annoyance in the end.

25th Anniversary ButterflyLike me, you have probably seen dozens of friends and/or celebrities participating in the ALS Ice Bucket Challenge, may have participated yourself, or at least heard about it on the news. While the challenges have slowed since peaking about a week ago, they have reached broadly through my network. I was particularly surprised to see some of my non-American friends from the United Arab Emirates and Thailand participating just yesterday. An article from The Guardian noted in its headline, “The Ice Bucket Challenge for ALS is a gimmick—and it’s good.” The Challenge was intended to raise dollars for ALS research, and by all accounts, the ALS Challenge rings of success: as of August 26 date, the Challenge brought in 88.5 million dollars! True—raising dollars is an obvious output of the Challenge; however another is to raise awareness of ALS. The evaluator in me asks, “How much awareness has the Challenge actually raised (and how do we know)? Does raising awareness mean that people know the acronym ALS (but do they know what ALS stands for)? Does raising awareness mean that people know one fact about ALS? Do people know that it is a horrible disease that scientists are still trying to understand (and thus, that is why research dollars are needed)?

 

Raising awareness is often something museums desire as an end result for visitors who have seen an exhibition or attended a program. But raising awareness about something is an elusive concept—unless staff articulate what they mean by raising awareness. I have often heard museum professionals say that awareness is something that is hard to measure. But what I think is really difficult about measuring awareness is defining a realistic and appropriate lens through which to articulate awareness. For instance, if an exhibition wants to raise awareness of issues of social justice, is it successful in raising awareness only if everyone comes away with a thorough understanding of social justice issues in their community or strong feelings of advocacy for social justice issues? While I, curators, and exhibition designers may want to see everyone walk away from an exhibition about social justice take immediate action, it would be erroneous to say that is the only measure of success in raising awareness. I believe success is seeing any movement along a continuum from pre- to post-campaign, post-program, or post-exhibition. Certainly it is exciting and to see those big jumps across a continuum of awareness, but what I have learned along the way is that many people making little jumps or baby steps along a continuum are big successes.

My niece and daughter taking their baby steps together.

My niece and daughter taking their baby steps together.

Today’s Throwback Thursday comes from deep in the RK&A vault – a study we did in 2002 for the National Museum of American History (NMAH) in Washington, DC.

For Which It Stands: The American Flag in American Life [2002]

Study Context

The National Museum of American History, Bering Center (NMAH) asked RK&A in 2002 to concept test ideas for For Which It Stands: The American Flag in American Life, a new exhibition that would feature the Star-Spangled Banner, the flag that inspired the national anthem. The study’s objectives were to examine:

  • The range of meanings people attach to the flag
  • How participants think their meaning of the flag has been shaped
  • Participants’ tolerance for different meanings of the flag
  • Whether seeing objects and images selected for the exhibition causes people to see the flag in new ways
  • Whether people understand the changing, complex meaning of the flag.

Approach

NMAH is part of the Smithsonian, and, as such, it tries to accommodate visitors from all backgrounds and of all ages. Like many other museums, NMAH wants to attract teens and appeal to adults. Therefore, we tested the exhibition’s concepts using focus groups. We conducted three focus groups with teens and three focus groups with adults. Sixty individuals participated in the six groups.

Findings

Through discussions about the exhibition panels, participants were exposed to new ideas and stories about the American flag, which broadened their understanding of the flag. Participants freely shared their personal stories about what the American flag means to them and everyone said they enjoyed hearing other people’s stories and ideas and thinking about the American flag in new ways. However, individuals’ personal meanings of the flag were not altered as a result of their experience.

Conclusion

Although exhibition planners initially had wanted the exhibition to change people’s meaning of the American flag, they learned that participants’ beliefs about the flag were shaped by unique experiences deeply rooted in their identities. RK&A’s audience research demonstrated that visitors bring valuable experiences to museums and their stories can add depth to other visitors’ museum experiences.

25th Anniversary ButterflyAs we move further and further into the digital age, museums hold something that is becoming a rare commodity—real objects and artifacts. It may be hard to believe, but one day, many tangible objects may be obsolete, the way that printed photographs and airplane tickets are becoming scarce items. Instead of going on “digs,” future archaeologists may primarily use computer-driven devices to search for clues of our ancestors. In the distant future, I can imagine that museums will be magical places where people can see “the real thing.” …But wait, maybe they already are?

 

This is my third reflection—informed by what I have learned about museum visitors in all my years studying them. I have found that there are many reasons people visit museums, but I believe the primary reason, one that we may take for granted, is that they want to see “the real thing.” A common question heard in museums is, “is it real?” especially in regard to bones and historical objects. I have heard it in our research, but you have probably heard it too, or said it yourself while walking through a museum. Why do people ask this question? What underlies the need to know if something is “real.” As museum goers, can’t observing a replica of a dinosaur skeleton or a 17th century Dutch ice skate tell us just as much as the real thing? Maybe so. But there is just something about being in the presence of authentic artifacts and objects that is thrilling; maybe it has to do with feeling connected to other people, to the past, or to other parts of the world. Professors David Thelen and Roy Rosenzweig say it best in their landmark 1998 national study, The Presence of the Past: “approaching artifacts and sites on their own terms, visitors could cut through all the intervening stories, step around all the agendas that have been advanced in the meantime, and feel that they were experiencing a moment from the past almost as it had originally been experienced—and with none of the overwhelming distortions that they associated with movies and television, the other purveyors of immediacy.”

 

Whenever studying museum visitors, I come face to face with their sense of wonder about and desire to get clDinosaur Sueose to (even touch) real objects. Whether evaluating text panels, interactive exhibits, touch tables, or ideas and concepts, visitors will usually keep coming back to the objects. As museum professionals, it is sometimes too easy to forget the centrality of the object when you are knee-deep in trying to interpret and contextualize something. We can get lost in these various mediums of interpretation, but visitors will usually remind us what they are really there for. For example, I was doing a study for a museum and historic site last year in which we were testing ideas for high-tech touch tables intended to convey information about the historic building the museum is housed in. I had gotten so wrapped up in testing all the information, that I had mentally pushed aside the museum’s biggest asset, the historic building it resides in. But visitors brought me back when they practically skipped over my questions about the touch tables and rather, kept circling back to the building itself—its authentic and tangible sense of history. Of course that is what they wanted to talk about and why they were there. This isn’t to say that interpretation of any kind is futile. But I believe it is important to keep reminding ourselves, as museum professionals, that interpretation should be used primarily to help visitors make sense of the objects and artifacts they are there to see—it’s really that simple.

 

The very reason I work with museums is because of my own sense of wonder and astonishment when it comes to objects and artifacts. Yes, I love studying people and how they learn and make sense of experiences, but I could do that in many different settings. I chose to do it in museums because of my own belief that we can learn so much from studying “the real thing.”

25th Anniversary ButterflyFor me, intentionality, a concept I view as essential to museum planning, emerged from two core experiences: results from hundreds of exhibition and program evaluations; and observing museum staff wanting to put too many concepts into an exhibition. Intuitively I knew there was a connection between exhibitions that didn’t fare too well (at least according to the evaluations) and staff not letting go of ideas that are near and dear to their hearts—regardless of whether those ideas supported the thesis of the exhibition.

When I have the good fortune to attend planning meetings, I always find myself thinking critically about what should be included in the exhibition under discussion and what could be saved for another time. My consideration always includes the big idea of the exhibition, what the museum would like to achieve with the exhibition vis-à-vis the public, humans’ capacity to process new ideas when in unfamiliar environments (like that of an exhibition hall), evaluation results from other projects that show what leads to quality visitor experiences and what might move visitors away from having quality experiences, and my utmost respect for scholars’ knowledge and passions. While passionate individuals love their subject matter (and really, I love their subject matter, too), one’s willingness to recognize that not all good ideas (or even great ones) belong in an exhibition and then exercising follow through are traits of intentional practice.

Embedded in intentional practice is the concept of alignment—ensuring that project concepts, components, and elements are present because they support the impact the team wants to achieve. If there are concepts, components, elements that do not contribute to the Intentionalitycore idea of the exhibition and its potential impact on audiences, they need to be omitted. I certainly don’t mean to sound ruthless, but I am acutely aware of how easy it is to keep putting more and more into an exhibition plan and how painfully difficult it is to take anything away. I am also aware of how challenging it is to stay focused on the exhibition’s big idea and have the discipline to say no to ideas because they do not support the intended impact of the exhibition. Learning to say “no” is a necessary survival skill and saying “no” is deeply connected to intentional practice. When practitioners are intentional, they are focused on the impact they want to achieve; they exercise discipline and restraint when determining how to best move forward; and their decision making is egoless and for the sake of achieving the results the team envisions.

Intentional practice represents the culmination of my experiences to date, and my passion for it is directly is tied to my evaluation experience. Over the years I started to realize that when exhibitions tried to do too much, visitors’ experiences didn’t amount to much—from their perspective; their heads were full but descriptions of their experiences were nebulous. Sense-making seemed futile. Nudging me was my memory of exhibition development discussions and tensions about what to put in (everything) and what to take out (nothing). Clarifying the intent of the exhibition and then staying focused on the intent of the exhibition is hard work—not likely to end soon, which is okay. Seeking clarity—whether in thought or action—is a never-ending pursuit.

Follow

Get every new post delivered to your Inbox.

Join 130 other followers