Posts Tagged ‘impact statement’

Emily’s last blog post (read it here) talked about when evaluation capacity building is the right choice.  When we think about building capacity for evaluation, we think about intentional practice.  This does not necessarily involve teaching people to conduct evaluation themselves, but helping people to ask the right questions and talk with the right people as they approach their work.  RK&A has found this to be particularly important in the planning phases of projects.

The case study below is from a project RK&A did with the Museum of Nature and Science in Dallas, TX (now the Perot Museum of Nature and Science) and involved an interdisciplinary group of museum staff thinking intentionally about the impact the Museum hoped to have on the community.  With a new building scheduled to open a year after this project took place, it was a wonderful time to think intentionally about the Museum’s impact.

Building Capacity to Evaluate [2012]

An evaluation planning project with a nature and science museum

The Museum of Nature and Science (MNS) hired RK&A to develop an evaluation plan and build capacity to conduct evaluation in anticipation of the Museum’s new building scheduled to open in 2013.

How did we approach the project?

The evaluation planning project comprised a series of sequential steps, from strategic to tactical, working with an interdisciplinary group of staff across the Museum. The process began by clarifying the Museum’s intended impact that articulates the intended result of the Museum’s work and provides a guidepost for MNS’s evaluation: Our community will personally connect science to their daily lives. Focusing on the Museum’s four primary audiences that include adults, families, students, and educators, staff developed intended outcomes that serve as building blocks to impact and gauges for measurement. Next, RK&A worked with staff to develop an evaluation plan that identifies the Museum’s evaluation priorities over the next four years, supporting the purpose of evaluation at MNS to measure impact, understand audiences’ needs, gauge progress in the strategic plan, and inform decision making.

The final project step focused on building capacity among staff to conduct evaluation. Based on in-depth discussions with staff, RK&A developed three data collection instruments, including an adult program questionnaire, family observation guide, and family short-answer interview guide, to empower staff to begin evaluating the Museum’s programs. Then, several staff members were trained to systematically collect data using the customized evaluation tools.

What did we learn?

The process of building a museum’s capacity to conduct evaluation highlights an important consideration. Evaluating the museum’s work has become more important given accountability demands in the external environment. Stakeholders increasingly ask, How is the museum’s work affecting its audiences? What difference is the museum making in the quality of people’s lives?

Conducting systematic evaluation and implementing a learning approach to evaluation, however, require additional staff time which is a challenge for most museums. MNS staff recognized the need to create a realistic evaluation plan given competing demands on staff’s time. For example, the evaluation plan balances conducting evaluation internally, partnering with other organizations, and outsourcing to other service providers. Also, the plan incrementally implements the Museum’s evaluation initiatives over time. The Museum will begin with small steps in their efforts to affect great change.

Read Full Post »

I love a good story.  Who doesn’t?  It’s how we humans make meaning—we construct narratives to explain and interpret events both to ourselves and for others.  Think about the number of stories you tell or hear in a day, even the mundane ones.  It’s a way to form and sustain connections with others and to understand ourselves.  So I was intrigued to see that this year’s AAM theme was “The Power of Story.”  I remembered that the 2012 AAM keynote address included a couple storytellers from The Moth (the tagline is True Stories told Live, and it features everyday people telling very personal stories on stage and is broadcast on National Public Radio).  The Moth was one the highlights of the 2012 AAM conference for me, so I was especially disappointed that I was unable to attend AAM this year.  But I talked to several people who did attend and read some blogs and, not really surprisingly, its sounds like panelists wove the theme into their presentations in interesting and appropriate ways, (which certainly isn’t always the case with conference themes).

It got me thinking.  Storytelling isn’t something I consider on a daily basis in my work, at least not in a literal, explicit way.  But the more I think about it, the more I realize storytelling permeates my work in nearly every way and has even become a tool for helping museums think about and define their impact.

To begin with, I am a qualitative researcher.  I was drawn to the field as a way to understand the world, in particular, people and groups of people—how they live, experience life, make meaning, and why and how they do what they do.  Of course one can study all this through quantitative research as well, but I am interested in the messiness and ambiguities inherent in qualitative research.  Qualitative data is narrative, and more specifically, I’ve noticed the best data often results when, for example, an interviewee or focus group participant tells a story to illustrate an idea.  And in fact, a strand of qualitative research called narrative research explicitly uses storytelling as a methodology.  Stories as data are powerful because they resonate and illuminate truths about the human experience.

Bed Curtain

Bed Curtain: England (1690-1710), artist unknown, V&A Museum.

Secondly, I was drawn to work in museums because of my love of objects—whether art, natural history specimens, or historical artifacts; to me, objects embody stories.  Objects are the physical evidence demonstrating that something was here; something happened here!  Objects stir the imagination and stimulate storytelling, whether fantastical stories (just listen to a child explain a work of art) or stories based on interpretation and deductive reasoning.  And, based on all my years of conducting research and evaluation in museums, I can tell you visitors feel the same way I do about objects—authentic objects evoke stories for visitors, and as I mentioned earlier, stories are how we construct meaning and connect with others—objects help us bridge a gap between ourselves and another (whether an artist, a dinosaur, or the mysterious person who used this 17th century bed curtain shown to the right).

The final, perhaps more subtle way that stories are important to my work is when we help our museum clients clarify and define impact (as defined by Stephen Weil is “making a difference in the quality of people’s lives”).  Randi has written a lot about impact on our blog so I won’t say too much about it here.  But I will say that one of the best ways for museums to begin thinking about impact is by telling stories about their work and why they love it.  I’ve never explicitly sat down with a client and said, “Okay, tell me a story about your visitors’ experiences.”  But invariably, that is what happens when we ask questions to help museums articulate their impact—they start telling stories (and at least twice, I’ve watched the telling of those stories lead to tears).  These stories are a starting place for museums to think authentically about the affect they have on their audiences.  As I discussed in my previous blog post, Explaining the Unexplainable, it is daunting to sit down and try to articulate impact and outcomes (particularly if you worry about measuring them), but starting with stories grounds you in what’s real and meaningful and can lay the foundation for articulating a distinct and authentic impact statement.

Read Full Post »

Mission ImpactIn my last post, I refrained from sharing examples of impact statements because I wanted readers to ponder the idea for themselves (always a very useful activity) but promised I would delve into what comprises an impact statement and provide examples in future posts.  I believe museums need impact statements because if they aren’t clear about what they want to achieve, how will they make decisions to get there?  Museums need impact statements to guide their planning and decision making, but more importantly, they need to clarify (to themselves and stakeholders) why their work has value—public value.

Simon Sinek states with enormous clarity the importance of understanding why an organization does what it does in this Ted Talk:  (http://www.ted.com/talks/simon_sinek_how_great_leaders_inspire_action.html) He makes the distinction between what organizations do and why they do their work.  He notes that people care about the why much more than they care about the what.  While all of his examples are from the business world, his point is clear, well articulated, and relevant to museums and other non profits. I see a clear connection between answering the why question and articulating intended impact, as impact describes how the museum will make a difference in the quality of people’s lives, and presumably, that quality is of tremendous public value.

There are three ingredients or building blocks to creating an impact statement:

  1. Passion: discovering the collective passions of staff—why you do what you do.  What about your work are you most passionate, and why that work is important (ask the why question three times to arrive at people’s deepest passions)?  As Sinek notes, talking about the why behind your work will help others know why they should care.
  2. Distinctiveness: identifying a museum’s distinctiveness—what does your museum do better than any other organization for the people in your community?  Distinctiveness is of vital importance because if you can describe what is distinct about your museum, you begin to suggest your museum’s value—its public value.
  3. Relevance: is about exploring the intersection among staff passions, the museum’s strengths and greatest assets—both of which suggest its distinctiveness, and what is relevant to the public. What the museum presents and how it presents it must be responsive to the museum, stakeholders, and the public.

In our last post Amanda was writing about the director at the Tate who believes “that art is a vital force for civic good . . .” (https://intentionalmuseum.com/2013/03/27/returning-to-your-core-radically-rethinking-the-museum/) The concept—for the civic good—comes close to what an impact statement might embody, although some unpacking may be required to fully understand what he means by those words.

Here are a few museum mission and impact statements—both statements work together to convey what the museum does and the result of what the museum does on audiences served:

Mission:

The Burke Museum of Natural History and Culture creates a better understanding of the world and our place in it. The Museum is responsible for Washington State collections of natural and cultural heritage and sharing the knowledge that makes them meaningful. The Burke welcomes a broad and diverse audience and provides a community gathering place that nurtures life-long learning and encourages respect, responsibility and reflection.

Vision (the Burke calls its impact statement its vision statement):

People value their connection with all life—and act accordingly

Mission:

The Baltimore Museum of Art seeks innovation and excellence in an artistic program that focuses on art of the modern era, from the 19th century to the present.  The Museum is committed to creating an environment that inspires creativity, encourages learning, and fosters human understanding in a place where everyone feels welcome as a place for personal learning and civic engagement.

Vision:

Visitors will expand their creative thinking, deepen their understanding of human experiences, and value the museum as a place for personal learning and civic engagement.

Mission:

The mission of Mid-America Science Museum is to stimulate interest in science, to promote public understanding of the sciences, and to encourage life-long science education through interactive exhibits and programs. The Museum also serves as a premier tourism attraction in Arkansas.

Impact:

Inspired by discovery, visitors are encouraged to investigate the world around them and realize science impacts everyone and everything.

As you and your staff explore their passions, the museum’s distinctiveness, and what is relevant to the public, you will begin conceptualizing an impact statement. With passion and focused attention on what you do best—in other words, playing to your strengths—and a deep understanding of your public, your value will be felt by all who experience your museum’s work.

Read Full Post »

I follow Max van Balgooy’s blog Engaging Places.  Last week he posted “Rethinking the mission statement” (http://engagingplaces.net/2013/02/19/rethinking-the-mission-statement/) and it caught my attention because I, too, have written about museum mission statements, raising some of the same points that Max raises (see “A Case for Holistic Intentionality” http://randikorn.com/docs/the_case%20for_holistic_intentionality_042007.pdf).  In Max’s words, “most (mission statements) are mild mannered”; I note that museums’ missions are interchangeable as most describe what museums do (“collect, preserve, and interpret” according to Max). But in today’s impact-driven, evidenced-based non-profit world, the work of a museum isn’t as important as the result of a museum’s work—on people.  Max and I have been in touch this past week and we agreed to continue the conversation through our blogs.

Mission Statement Cartoon

Museums may need to change how they do their daily and strategic work as the (funding) environment in which they reside is quickly changing; museums can’t afford to stay stuck in a world that looks inward.  Mission statements, while important, grounding statements for any organization, focus only on what museums do; if museums collect, preserve, interpret, then what is the outward result of this work?  In a city that has a dozen museums, for example, how is that city benefiting from those museums’ assets and staff members’ diligence?  What impact do museums intend to create by doing their work?  What evidence is there that museums are making a difference in the quality of people’s lives?  What might those results look like, sound like?

I grapple with these questions every day as someone who wants to help museums collect evidence that demonstrates museums’ value.  The challenge is, before I can document the ways in which a museum has made a difference in people’s lives, first museums need to take the time to describe (in painstakingly concrete—and dare I say, measurable, terms) what their hard work affords a community.  Thus, I believe that mission statements need companion statements.  In addition to mission statements, I suggest museums also develop impact statements to describe the intended result of a museum on museum audiences—most notably those who live within the community where the museum resides.

Future posts will delve into what comprises an impact statement, but for now, I want to further explain why writing an impact statement is a necessary step moving forward.  Max notes that he hopes to encourage readers to “rethink their mission, vision, and strategy to become more relevant and engaging in their communities.”   For museums to achieve Max’s hope, museums may need to balance their thinking about mission with thinking about relevance and for whom museums are relevant, because it appears that collecting, etc., are a means to an end.  To what greater end are museums doing their work?

In today’s world, museums may need to start thinking about and acting on a much larger purpose—one that adds value for the common good.  Many museum staff members believe that their work already serves a larger purpose; however the connection between a museum’s work and the public good is not always transparent, especially to city and government officials and even funders who opt to support other kinds of non-profits (the Wallace Foundation, for example, no longer funds museums).  To make a difference in people’s lives, doing good work is no longer enough; museums will need to overtly, explicitly, and relentlessly connect the dots.  As museum geeks, it is our responsibility to help others see the value of a museum experience; we just can’t say that museums are great and leave it at that.  We have to back up claims with evidence.  But first museums need to clarify what they will measure.  And that will require debate and discussion among museum leadership and staff about what they hope their museum achieves by doing their work—in terms of the public good.  Museum boards and leadership can no longer afford to remain silent or arrogant about the topic.  If they want their museum to make a difference in people’s lives, they will need to articulate precisely what they mean—and then someone can measure it.

Read Full Post »

This week, I’d like to begin to hone in on the idea of measuring impact that Randi raised in our first blog post.  We define impact as the difference museums can make in the quality of people’s lives, and measuring it can be both exciting and intimidating.  Exciting because just about every museum professional I’ve ever met believes museums have the potential to affect people in deeply powerful ways.  Stories abound from people who have distinct and palpable memories of museum visits from childhood—memories that became etched in their being and identity (for many, it is the giant heart at the Franklin Institute, for others it may be a beautiful Monet water lily, and for the nine-year old me, it was a historic house, My Old Kentucky Home).  It’s these kinds of experiences that draw museum professionals to their field.  On the other hand, the idea of measuring impact can be intimidating because some think it is impossible to evaluate, measure, or assess something as intangible as a personal connection, engagement, identity growth, a lasting memory, an aesthetic experience, or an “ah-ha” moment.  When these fears emerge, we try our best to allay them and try to move them towards an important first step in measuring impact—describing what impact looks like or sounds like.  Evaluators are accustomed to figuring out how to measure something—once the impact is described.

The Giant Heart at The Franklin Institute

The Giant Heart at The Franklin Institute

I, as a researcher and evaluator, become excited at the thought of being tasked with measuring impacts, such as “engagement” or “creativity.”  I relish the idea of studying something so I can explain the unexplainable, of drawing meaning from and describing unique human experiences.  As long as I can remember, I have been interested in the complexities of the human experience, especially in how it plays out in specific contexts, within the social realm, and in relationship to material culture, such as art, artifacts, and natural history specimens.  These interests led me to the field of anthropology and to work in social science research and museum evaluation, where I have the pleasure of spending my days exploring the ways people make meaning in museums and other similar institutions.

Sometimes when museums cite their impact, they fall back on the common practice of reporting visitation numbers.  While not unimportant, numbers indicate only that people came—they do not indicate the quality of visitors’ experiences.  Imagine hearing that a museum attracted a million visitors and then hearing about the qualitative difference a museum has made in people’s lives—wouldn’t that sound more meaningful? This brings me to discussing an often overlooked methodology in museum evaluation—case study research.  Its low rate of use is interesting in light of museums’ desire for evidence of impact, as a case study can provide rich details of a person’s or entity’s (e.g., a school) experience.  Case study research is “an in-depth description and analysis of a bounded system”—that “system” could be any number of things: an individual museum visitor, a school partner, or a community.  It provides a focused, in-depth study of one particular person or entity.  Practically speaking, what we do is follow several participants (or “cases”) over time (during and after a program for example), by interviewing them repeatedly, observing them in the program, and interviewing others within their sphere of influence (such as a parent, spouse,  museum professional, student, or community member) who can comment about their experience. The outcome is a concrete, contextualized, nuanced understanding of a particular phenomenon (for example,  a person’s growth over time, a relationship between a museum and school, or a museum’s affect on a community) that can explain not only what happened as a result of the program, but how it happened.   Knowing the “what” and the “how” are invaluable to museums; the “what” can offer indictors of impact and the “how” tells you what the museum might have been doing to create the “case” experience.

An example of case study research comes from an evaluation we did for an art museum that was launching a new multi-visit program in middle schools.  We began our work by helping the museum define its intended impact, which is articulated as: “Students are empowered to think and act creatively in their lives, their learning, and their community.”  We then worked with staff to operationalize the impact statement by developing a series of concrete, measureable outcome statements.  We identified our “cases” as three middle schools.  Each “case” study included a series of interviews with students, classroom teachers, a few parents, and program staff, as well as an observation of program activities in the school and in the museum—all over the course of several months.  The data were rich, specific descriptions of what happened to participants and how the program functioned in each school—all in relationship to the impact statement.  Not surprisingly, each school had a slightly different experience, with one school more closely meeting the impact statement than the others.  The case study approach unveiled the complex interplay of variables at each school to help explain why one school was more successful than the others.  Both the successes and challenges provided great insight as the museum considered its second year of program implementation.

I know what you are thinking.  How can we measure impact by focusing on a few individuals or one or two schools?  What about generalizability?  While these questions are reasonable, they miss the point of case studies, which I believe strongly aligns with what we know about museum experiences—case studies account for differences in people’s unique dispositions, life experiences, and knowledge; they value distinctiveness; and they recognize the complexities of life and situations and do not try to simplify them.  If a museum really has trouble accepting the essential value of what a case study can afford, plenty of museum programs are small enough to warrant conducting a case study without worrying about generalizability.  The example above is a relatively small program serving six schools, three of which we examined through our study.  In another example, we used case study research to assess the impact of a museum-based summer camp serving 20 or so teens.  Conducting case studies to demonstrate the impact of museums’ small programs might be just the perfect baby step towards museums’ beginning to measure impact.

I sense urgency in museums’ need to have evidence of the value of museums in the American landscape.  I think it is time we stopped worrying about what might be immeasurable and instead begin describing what success looks like.

Read Full Post »