Posts Tagged ‘measuring impact’

I follow Max van Balgooy’s blog Engaging Places.  Last week he posted “Rethinking the mission statement” (http://engagingplaces.net/2013/02/19/rethinking-the-mission-statement/) and it caught my attention because I, too, have written about museum mission statements, raising some of the same points that Max raises (see “A Case for Holistic Intentionality” http://randikorn.com/docs/the_case%20for_holistic_intentionality_042007.pdf).  In Max’s words, “most (mission statements) are mild mannered”; I note that museums’ missions are interchangeable as most describe what museums do (“collect, preserve, and interpret” according to Max). But in today’s impact-driven, evidenced-based non-profit world, the work of a museum isn’t as important as the result of a museum’s work—on people.  Max and I have been in touch this past week and we agreed to continue the conversation through our blogs.

Mission Statement Cartoon

Museums may need to change how they do their daily and strategic work as the (funding) environment in which they reside is quickly changing; museums can’t afford to stay stuck in a world that looks inward.  Mission statements, while important, grounding statements for any organization, focus only on what museums do; if museums collect, preserve, interpret, then what is the outward result of this work?  In a city that has a dozen museums, for example, how is that city benefiting from those museums’ assets and staff members’ diligence?  What impact do museums intend to create by doing their work?  What evidence is there that museums are making a difference in the quality of people’s lives?  What might those results look like, sound like?

I grapple with these questions every day as someone who wants to help museums collect evidence that demonstrates museums’ value.  The challenge is, before I can document the ways in which a museum has made a difference in people’s lives, first museums need to take the time to describe (in painstakingly concrete—and dare I say, measurable, terms) what their hard work affords a community.  Thus, I believe that mission statements need companion statements.  In addition to mission statements, I suggest museums also develop impact statements to describe the intended result of a museum on museum audiences—most notably those who live within the community where the museum resides.

Future posts will delve into what comprises an impact statement, but for now, I want to further explain why writing an impact statement is a necessary step moving forward.  Max notes that he hopes to encourage readers to “rethink their mission, vision, and strategy to become more relevant and engaging in their communities.”   For museums to achieve Max’s hope, museums may need to balance their thinking about mission with thinking about relevance and for whom museums are relevant, because it appears that collecting, etc., are a means to an end.  To what greater end are museums doing their work?

In today’s world, museums may need to start thinking about and acting on a much larger purpose—one that adds value for the common good.  Many museum staff members believe that their work already serves a larger purpose; however the connection between a museum’s work and the public good is not always transparent, especially to city and government officials and even funders who opt to support other kinds of non-profits (the Wallace Foundation, for example, no longer funds museums).  To make a difference in people’s lives, doing good work is no longer enough; museums will need to overtly, explicitly, and relentlessly connect the dots.  As museum geeks, it is our responsibility to help others see the value of a museum experience; we just can’t say that museums are great and leave it at that.  We have to back up claims with evidence.  But first museums need to clarify what they will measure.  And that will require debate and discussion among museum leadership and staff about what they hope their museum achieves by doing their work—in terms of the public good.  Museum boards and leadership can no longer afford to remain silent or arrogant about the topic.  If they want their museum to make a difference in people’s lives, they will need to articulate precisely what they mean—and then someone can measure it.

Read Full Post »

This week, I’d like to begin to hone in on the idea of measuring impact that Randi raised in our first blog post.  We define impact as the difference museums can make in the quality of people’s lives, and measuring it can be both exciting and intimidating.  Exciting because just about every museum professional I’ve ever met believes museums have the potential to affect people in deeply powerful ways.  Stories abound from people who have distinct and palpable memories of museum visits from childhood—memories that became etched in their being and identity (for many, it is the giant heart at the Franklin Institute, for others it may be a beautiful Monet water lily, and for the nine-year old me, it was a historic house, My Old Kentucky Home).  It’s these kinds of experiences that draw museum professionals to their field.  On the other hand, the idea of measuring impact can be intimidating because some think it is impossible to evaluate, measure, or assess something as intangible as a personal connection, engagement, identity growth, a lasting memory, an aesthetic experience, or an “ah-ha” moment.  When these fears emerge, we try our best to allay them and try to move them towards an important first step in measuring impact—describing what impact looks like or sounds like.  Evaluators are accustomed to figuring out how to measure something—once the impact is described.

The Giant Heart at The Franklin Institute

The Giant Heart at The Franklin Institute

I, as a researcher and evaluator, become excited at the thought of being tasked with measuring impacts, such as “engagement” or “creativity.”  I relish the idea of studying something so I can explain the unexplainable, of drawing meaning from and describing unique human experiences.  As long as I can remember, I have been interested in the complexities of the human experience, especially in how it plays out in specific contexts, within the social realm, and in relationship to material culture, such as art, artifacts, and natural history specimens.  These interests led me to the field of anthropology and to work in social science research and museum evaluation, where I have the pleasure of spending my days exploring the ways people make meaning in museums and other similar institutions.

Sometimes when museums cite their impact, they fall back on the common practice of reporting visitation numbers.  While not unimportant, numbers indicate only that people came—they do not indicate the quality of visitors’ experiences.  Imagine hearing that a museum attracted a million visitors and then hearing about the qualitative difference a museum has made in people’s lives—wouldn’t that sound more meaningful? This brings me to discussing an often overlooked methodology in museum evaluation—case study research.  Its low rate of use is interesting in light of museums’ desire for evidence of impact, as a case study can provide rich details of a person’s or entity’s (e.g., a school) experience.  Case study research is “an in-depth description and analysis of a bounded system”—that “system” could be any number of things: an individual museum visitor, a school partner, or a community.  It provides a focused, in-depth study of one particular person or entity.  Practically speaking, what we do is follow several participants (or “cases”) over time (during and after a program for example), by interviewing them repeatedly, observing them in the program, and interviewing others within their sphere of influence (such as a parent, spouse,  museum professional, student, or community member) who can comment about their experience. The outcome is a concrete, contextualized, nuanced understanding of a particular phenomenon (for example,  a person’s growth over time, a relationship between a museum and school, or a museum’s affect on a community) that can explain not only what happened as a result of the program, but how it happened.   Knowing the “what” and the “how” are invaluable to museums; the “what” can offer indictors of impact and the “how” tells you what the museum might have been doing to create the “case” experience.

An example of case study research comes from an evaluation we did for an art museum that was launching a new multi-visit program in middle schools.  We began our work by helping the museum define its intended impact, which is articulated as: “Students are empowered to think and act creatively in their lives, their learning, and their community.”  We then worked with staff to operationalize the impact statement by developing a series of concrete, measureable outcome statements.  We identified our “cases” as three middle schools.  Each “case” study included a series of interviews with students, classroom teachers, a few parents, and program staff, as well as an observation of program activities in the school and in the museum—all over the course of several months.  The data were rich, specific descriptions of what happened to participants and how the program functioned in each school—all in relationship to the impact statement.  Not surprisingly, each school had a slightly different experience, with one school more closely meeting the impact statement than the others.  The case study approach unveiled the complex interplay of variables at each school to help explain why one school was more successful than the others.  Both the successes and challenges provided great insight as the museum considered its second year of program implementation.

I know what you are thinking.  How can we measure impact by focusing on a few individuals or one or two schools?  What about generalizability?  While these questions are reasonable, they miss the point of case studies, which I believe strongly aligns with what we know about museum experiences—case studies account for differences in people’s unique dispositions, life experiences, and knowledge; they value distinctiveness; and they recognize the complexities of life and situations and do not try to simplify them.  If a museum really has trouble accepting the essential value of what a case study can afford, plenty of museum programs are small enough to warrant conducting a case study without worrying about generalizability.  The example above is a relatively small program serving six schools, three of which we examined through our study.  In another example, we used case study research to assess the impact of a museum-based summer camp serving 20 or so teens.  Conducting case studies to demonstrate the impact of museums’ small programs might be just the perfect baby step towards museums’ beginning to measure impact.

I sense urgency in museums’ need to have evidence of the value of museums in the American landscape.  I think it is time we stopped worrying about what might be immeasurable and instead begin describing what success looks like.

Read Full Post »

Welcome to the Intentional Museum Blog, an all-staff endeavor of Randi Korn & Associates.  We intend to share our thoughts and questions twice monthly about compelling ideas we come across in our work and readings.  The platform for our practice and the inspiration for our postings is the Cycle of Intentional Practice.

Cycle of Intentional Practice

This cycle has emerged slowly, evolving over the last 10 to 15 years, as I started to think about all I had learned as an evaluator and researcher while pursuing my passion of studying people’s experiences in informal learning environments.  In our work as planners and evaluators in museums, we have witnessed how difficult it can be to achieve results that reflect a museum’s original intentions.  Likewise, we have witnessed that results surface when organizations focus their energy, decisions, actions, and dollars towards well-articulated ends.  Thus, increasingly, intentionality is becoming a driving force in our work with clients, as we recognize how vital it is as an idea, action, theory, and practice.

Intentionality is not a new concept, as several well-known authors have written about it.  For example, business author Jim Collins notes in Good to Great that the work of a great organization must “attract and channel resources directed solely . . .” to their intentions and “reject resources that drive them away from” their passion and unique value.  Collins’ concept applies to museums, too, as described by museum scholar Stephen Weil in Making Museums Matter.  He wrote, “The only activities in which the museum can legitimately engage are those intended to further its institutional purpose.”  Collins’ and Weil’s concepts embody the essence of intentionality, where all actions are purposeful, deliberate, and focused on achieving the intended impact or results.  With funders requiring evidence that their dollars are being used in the way they intended, intentionality rings of relevance.  Likewise, with fewer dollars available to museums, how museums use those dollars is increasingly important.

The Cycle of Intentional Practice places “impact” in the center of the cycle and assumes museums want to make a positive difference in people’s lives, which is how I define “impact.”  Intentional practice requires that the museum articulates the impact it would like to achieve (by writing an impact statement), align a museum’s practices and resources to support the impact it would like to achieve, measure the ways in which the museum is achieving impact, and reflect on the results to learn from them for the purpose of improvement.  Intentional practice may seem insular, but it acknowledges a museum’s responsibility to its external community through evaluation and other efforts.  The tension between a museum and its public is real and the museum may need to work hard to balance its internal aspirations and resources with its community’s needs.  Balancing potentially conflicting ideals, though challenging, demonstrates that the organization is striving to be true to itself and its audience and community.   I have learned in my 30 or so years of experience that when a museum applies laser focus to the center of the cycle, it creates an opportunity to achieve meaningful, measurable results that an evaluator can detect.

We invite you to share your thoughts—agreements or disagreements—in the spirit of our collective learning, as learning has always been the motivational force behind our work.

Read Full Post »