Posts Tagged ‘goals’

Emily’s last blog post (read it here) talked about when evaluation capacity building is the right choice.  When we think about building capacity for evaluation, we think about intentional practice.  This does not necessarily involve teaching people to conduct evaluation themselves, but helping people to ask the right questions and talk with the right people as they approach their work.  RK&A has found this to be particularly important in the planning phases of projects.

The case study below is from a project RK&A did with the Museum of Nature and Science in Dallas, TX (now the Perot Museum of Nature and Science) and involved an interdisciplinary group of museum staff thinking intentionally about the impact the Museum hoped to have on the community.  With a new building scheduled to open a year after this project took place, it was a wonderful time to think intentionally about the Museum’s impact.

Building Capacity to Evaluate [2012]

An evaluation planning project with a nature and science museum

The Museum of Nature and Science (MNS) hired RK&A to develop an evaluation plan and build capacity to conduct evaluation in anticipation of the Museum’s new building scheduled to open in 2013.

How did we approach the project?

The evaluation planning project comprised a series of sequential steps, from strategic to tactical, working with an interdisciplinary group of staff across the Museum. The process began by clarifying the Museum’s intended impact that articulates the intended result of the Museum’s work and provides a guidepost for MNS’s evaluation: Our community will personally connect science to their daily lives. Focusing on the Museum’s four primary audiences that include adults, families, students, and educators, staff developed intended outcomes that serve as building blocks to impact and gauges for measurement. Next, RK&A worked with staff to develop an evaluation plan that identifies the Museum’s evaluation priorities over the next four years, supporting the purpose of evaluation at MNS to measure impact, understand audiences’ needs, gauge progress in the strategic plan, and inform decision making.

The final project step focused on building capacity among staff to conduct evaluation. Based on in-depth discussions with staff, RK&A developed three data collection instruments, including an adult program questionnaire, family observation guide, and family short-answer interview guide, to empower staff to begin evaluating the Museum’s programs. Then, several staff members were trained to systematically collect data using the customized evaluation tools.

What did we learn?

The process of building a museum’s capacity to conduct evaluation highlights an important consideration. Evaluating the museum’s work has become more important given accountability demands in the external environment. Stakeholders increasingly ask, How is the museum’s work affecting its audiences? What difference is the museum making in the quality of people’s lives?

Conducting systematic evaluation and implementing a learning approach to evaluation, however, require additional staff time which is a challenge for most museums. MNS staff recognized the need to create a realistic evaluation plan given competing demands on staff’s time. For example, the evaluation plan balances conducting evaluation internally, partnering with other organizations, and outsourcing to other service providers. Also, the plan incrementally implements the Museum’s evaluation initiatives over time. The Museum will begin with small steps in their efforts to affect great change.

Read Full Post »

Recently, Christine Castle asked readers of her Museum Education Monitor for their “words to live by”—pithy phrases and bon mots that help [them] make it through the museum education day.  This got me thinking about the words I live by as a museum evaluator.  Three little words easily popped into my mind—less is more.  These words epitomize themselves; they are beautiful in their simplicity yet they embody our whole philosophy as an evaluation firm and my own personal approach to evaluation.  What is so interesting to me about the concept of “less is more” is how incredibly hard it is to achieve.  Doing less seems so simple; but to truly live by those words is extraordinarily difficult.

Less is MoreLet me give an example.  We often facilitate planning workshops for our clients.  As we have probably said in many a blog post, planning and evaluation are inextricably linked.  Evaluators are true believers in planning with the end in mind.  Otherwise, how are we going to know that our clients have achieved the effect they desire on the audiences they serve?  The ultimate goal of these planning workshops is to help our clients articulate their desired public impact.  They can use the end result—an Impact Planning Framework—to guide their decision making, and we can use it to guide audience research and evaluation.  In these workshops, we facilitate exercises for museum staff, and one of the exercises asks staff to select a finite number of audiences for which they will envision impact.  You may not be surprised, but often a key sticking point for museum staff is the very notion of limiting the number of target audiences.  At times, it almost feels like we have asked them to remove an appendage; the resistance can be palpable.

It’s touching on many levels that it is so difficult for museum staff to prioritize their audiences.  It speaks volumes about the passion they have for the public dimension of the work they do.  However, and this is a big however, museums cannot be everything to everyone.  It’s just not possible no matter how hard museums try.  I do not say this to sound negative or glass half empty.  I want museums to succeed in achieving their desired impact.  But, here’s the thing.  Impact is really hard to achieve (we know this from countless evaluations).  The rationale for prioritizing audiences is to help the museum focus resources and actions towards achieving results on those audiences.  Trying to be everything to everyone may result in the opposite of what a museum is striving for—nothing meaningful for anyone.  And, while difficult to believe, focusing one’s efforts and resources on a few doesn’t usually lead to others feeling excluded.  So often, what a museum might do for a few will have meaning for so many more.

The beauty of “less is more” is that if you try, you will feel liberated.  Focusing one’s efforts to achieve impact on three or four audiences (instead of “everyone”) is scary, but once a museum bites the bullet, staff may feel like they just received a “get-out-of-jail-free” card.  Finally, staff will have an excuse to focus their efforts on those few audiences where they feel they can make a difference.  Pursuing “less is more” is an ongoing process, which means that it takes a while to embrace it, and, once you do, you have to continue to work at living by those words because everything around us screams “more.”  It’s not easy, but worthwhile pursuits never are.  For me, knowing that doing less will actually help our clients achieve more is worth it in the end.  So, that’s why “less is more” are my words to live by.

Read Full Post »

In June, The Association of Science and Technology Centers (ASTC) invited professionals to respond to these questions for an upcoming issue of Dimensions magazine: When are evaluation and other visitor feedback strategies the most useful for helping advance a science center’s mission?  When are such strategies less successful?  We pondered this at a staff meeting and decided that a small but important tweak may be needed to begin addressing the questions.  First, let’s clarify that mission describes what a museum does and impact describes the result of what a museum does—on the audiences it serves.  We believe that anything a museum does—collect, exhibit, educate—is meaningless unless it is done in the pursuit of impact.  So, when is evaluation most useful for advancing a science center’s tree_fallsmission?  When it is done to advance impact not mission.  It’s a little like that old adage: If a tree falls in the forest and no one is around to hear it, does it make a sound?  With regard to mission and impact, we take a slightly different angle—if a museum does work or evaluation that does not lead to impact, are they really doing the work?

Evaluators are in the same boat as some museum practitioners.  Evaluation is a means to an end, just as a museum’s collections are a means to an end.  Unless evaluation is placed in a meaningful context, such as helping a museum pursue impact, evaluation doesn’t serve a purpose.  As an evaluator, I suppose I should say evaluation is always valuable.  But, that’s just not true.  I’m a self-proclaimed data nerd.  I love the minutia of evaluation—pouring over pages and pages of interview transcripts and pulling out those five key visitor trends.  I can get lost in data for days and find myself pulled in many seemingly fruitful directions.  “Oh, how interesting!” I will say to no one in particular.  I often find myself lost in the visitors’ world, chuckling to myself about a quirky response to an exhibit or wondering who someone is and why he or she responded to a museum experience in a particular way.  Getting lost in your work can be fun and, lucky me, happens to those of us who are passionate about what we do.  So, while pursuing tangents in evaluation data is fun for me, there is a flip side to this coin—a lack of focus that can be detrimental to the pursuit of a larger goal.  This is why we, as evaluators, push our clients to articulate what it is they want to achieve to keep us (and them) on track.

We consistently find museum practitioners to be among those most passionate about their work.  Thus, these moments of losing oneself in one’s work, whether researching or examining an object, designing an exhibition, or creating a program, are frequent occurrences.  When it comes to pursuing impact, this passion is both a joy and a burden.  It is a joy because most practitioners can easily articulate what they do for their audiences.  But, they often get lost in what they do and may not think about why they do what they do.  A practitioner articulating the “why” is similar to the entire museum articulating its intended impact.  Articulating impact provides a laser focus for all the work that museum practitioners do and helps keep them on track toward pursuing that larger goal.  So, our response to ASTC’s second question, When are evaluation strategies less successful in helping advance a science center’s mission?  When a science center and its collective staff have yet to articulate the impact they hope to achieve on the audiences they serve.  Otherwise, we can all do evaluation until we are blue in the face but those reports will continue to collect dust on hundreds of science centers’ shelves.  Of this I am certain—just like death and taxes.

Read Full Post »

Insider Out

I’m not a museum evaluator, but I play one on television; at least that’s what I tell people who ask me what I do for a living.  As the Business Manager at Randi Korn & Associates, I don’t have the educational or employment background of my colleagues, but I do have a long-standing love of museums – I had to name my favorite museum (and why) when I interviewed here.

New Yorker Cartoon SteinbergSo, in order to do my job effectively, I had to learn a few things about how museums operate and what museum evaluation is in order to function. Just as Steinberg cleverly shows here how New Yorkers view the world, those in the museum field may also have a skewed perception of their domain from the inside; this poster hangs on our office wall to remind us to “think outside the museum.” Here are my top 5 “surprises” from the outside.

Surprise #1: Who knew that museums had goals and objectives when they put up all that cool stuff? Laugh if you will, but before I came here, I didn’t realize there is more to an exhibition than putting like things together. Now that I know, it’s given me a whole new dimension to explore when I visit a museum.

Surprise #2: There’s a whole world of people conducting research on/in museums.
Being a process person and a big believer in constructive criticism, it’s good to know that, not only did the museum have a purpose when it put this stuff together, but someone is actually gathering information to make it better!

Surprise #3: Wow, I work with really smart people!
Not really surprising, because the museum field is loaded with lifelong learners and people who are naturally curious. And evaluators are really curious, otherwise they would not have the desire to probe and probe further, and question how and why.

Surprise #4: I was evaluating museums, I just didn’t know it.
Invariably, I always had a lot to say after visiting an exhibition: short little me (4’ 10”) couldn’t see through the crowd; I didn’t realize I was going in the wrong direction until I found the orientation panel at the end; this interactive was really cool; etc. etc. How refreshing to learn someone might actually be interested in my opinion!

Surprise #5: Gee, people have some odd opinions to share!Narwhal Horn
I keep a file of amusing interview and survey responses. The winner so far is, when asked if they had anything to add about their experience in the museum that day, the interviewee said “There were no beets in the beet soup!”  You can have the best exhibition ever, but visitors will remind you how they value every aspect of their museum encounter.

Oh, and in case you’re wondering, my favorite museum is the Cluny in Paris. Because it has a narwhal horn on display. Go figure.

Read Full Post »