25th Anniversary ButterflyAs evaluators, we work with museum professionals to collect data around problems they are facing, and not so surprisingly, museums often face similar problems.  In my six years with RK&A, I have definitely seen trends, and certainly in RK&A’s 25 years, the company has as well.  For this reason, I sometimes find myself wondering whether collecting more data around an issue is worthwhile.  As someone who considers herself a life-long learner, the instinct is to say, “No, we don’t know enough; there is always more to learn.”  But then I consider that, if there is enough existing and reliable information out there, our clients can save time and money but still make informed decisions.  This consideration gives me pause as my intention is for the work we do to help museums do their work better.

I was recently feeling this way while conducting focus groups with teachers about barriers to fieldtrips and their needs for teaching resources. We have worked on many evaluations of museum-school programs lately in which we collected data from teachers about museum programs and professional development, including for Kentucky Historical Society, the National Air and Space Museum, and the Corcoran Gallery of Art. Indeed, during the recent teacher focus groups, I heard a lot of familiar trends—cost of field trips, curriculum links, lack of time due to testing. But as I listened to these teachers, I gained a new appreciation for the phrase “the devil is in the details.” For, while some of the barriers were the same as those I was expecting, there were nuances and specifics unique to the context of the Museum and its community that make a familiar issue particularly challenging—which I have found to be true with every evaluation.

Pieter Bruegel the Elder The Seven Deadly Sins or the Seven Vices - Pride 1558

Pieter Bruegel the Elder
The Seven Deadly Sins or the Seven Vices – Pride
1558

So to the question, have we heard it all before when it comes to barriers to fieldtrip experiences? No. While there are certainly cases when existing research in the field can sufficiently answer a museum’s questions, more often than not, there are situational challenges unique to a museum and its community that are crucial to helping a museum address these challenges. Sometimes our work is about helping museums see the forest for the trees—identifying the big trends. But in the case of identifying barriers to fieldtrip experiences, I need to unpack every detail to help the Museum truly understand the barriers and identify recommendations.   Like this Bruegel painting, it can appear messy and confusing but inspecting each detail is necessary for making meaning.

Randi Korn & Associates, Inc., a consulting firm specializing in evaluating museum programs and exhibitions, is seeking a Research Associate in its Alexandria, VA office.  Under the management of a mentor/supervisor, the Research Associate will be responsible for implementing diverse evaluation projects and services; coordinating contractor data collection teams; collecting and analyzing qualitative and quantitative data; and preparing reports and presentations.

The ideal candidate will have 1-2 years experience conducting evaluation in museum-like settings and a strong a desire to join a work environment that is client-centered.  A master’s degree in applied anthropology, evaluation, educational psychology, sociology, educational measurement, museum education, museum studies, or a related field is required.  Qualified candidates will be familiar with current evaluation methods and practices.  Qualitative data analysis experience is required, and quantitative data analysis experience is a plus.  The qualified candidate must have excellent writing skills, be able to juggle multiple projects and work both independently and as part of project teams.  A passion for museums or other kinds of informal learning environments is a preferred, and knowledge of HTML coding is a plus.

RK&A’s strength is that we are a team.  In joining our team you will work with people who love what they do and are driven to learn from each other, our clients, museum visitors, and our museum colleagues around the country.  Our casual office environment is comfortable and adaptable.  If you think this is the job for you, we welcome your application.

RK&A offers a competitive compensation and benefits package.  For information on RK&A, please visit www.randikorn.com.  Interested applicants should forward a resume with cover letter, salary requirements, and two independently written and edited writing samples to: skidmore@randikorn.com, or to: RK&A, 2417 B Mount Vernon Avenue, Alexandria, VA  22301.  Resumes will be accepted until April 30, 2014.

I25th Anniversary Butterflyn Reflection #3, Emily Skidmore talked about how you can’t rush measuring outcomes and advocated for slowing down and conducting front-end and formative evaluation to improve exhibitions, programs, and experiences prior to jumping into measuring outcomes.  I’d like to piggy-back on the slow movement and talk about Institutional Review Board (IRB) and school district review, which is Slow with a capital ‘s’—for better or worse.

IRB is a formally designated board that reviews social science, biomedical, and behavioral research to determine whether the benefits of the research outweigh the risks for the participants in the study.  To be blunt, IRB can be a real thorn in our side.  It requires extensive, tedious paperwork for something we may consider innocuous (e.g., interviewing teachers about their program experience).  Given the many forms and thorough explanations of research procedures required, we spend a lot of time preparing for IRB, and then there is the fee to the external IRB to review the paperwork and methodology.  In addition to the budgetary implications IRB has for our clients, IRB procedures also can significantly delay the research well past when the museum may have expected its research to take place.  Not all of our work requires IRB review, but generally, most research projects where we measure outcomes do.

When our work includes collecting data from students and teachers, we sometimes have to submit our protocols to 1609_Color_Nit-Picking_IRBschool districts for formal review too.  School district review is separate from IRB review, although a school district’s criteria for reviewing research protocols are normally akin to IRB criteria.  Nevertheless, it is yet another required process that can really put the brakes on a project.  For instance, one school district took five months to review our project—much to the chagrin of our client and its funder (understandably so).

At times, IRB and school district reviews can feel like ridiculous hoops that we have to jump through, or bureaucracy at its worst.  As Don Mayne’s cartoon portrays, sometimes the IRB feels like a bunch of nitpicky people who exist solely to make our lives more difficult when we and our museum clients simply want to improve experiences for museum visitors.  So as I justify our sampling procedures for the fifteenth time in the required paperwork, I may shake my head and curse under my breath, but I truly do appreciate the work that IRB and school districts do (I swear there aren’t IRB reviewers holding my feet to the fire as I type!).

When I take a moment and step back, I realize that the process of submitting to IRB forces me to think through all the nitty-gritty details of the research process, which ultimately improves the research and protects museum visitors as research participants.  The extreme assumptions any given IRB makes about our research—(no, I will not be injecting anyone with an unknown substance)—I try not to take them personally and simply respond as clearly and concisely as possible.  And I have gotten pretty good at navigating the system at this point.  Then, I hold our client’s hand, try to protect them from as much of the paperwork and tedium as I can, and tell them, ever so gently, that their research may be delayed.

25th Anniversary ButterflyIntentional Museum is pleased to announce its first student blogging competition!  In honor of RK&A’s 25th anniversary, we want to hear from tomorrow’s museum professionals about their intentional practice and the impact it can have on the visitor experience. 

For our first blogging competition, we ask you to reflect on the following question: Through your intentional practice, how do you help museums enrich the lives of others?  Perhaps you are focusing on collections or exhibitions, using objects and artifacts to tell stories.  Maybe your love is museum education or visitor services, ensuring visitors have positive museum experiences.  Whether front-of-house or behind-the-scenes, we want to hear from you (check the guidelines for more information)! 

We often reflect on our professional experiences on Intentional Museum, but we appreciate the personal connection.  We want your blogs to tell a story, to speak about your experience, and to highlight your unique insight into the museum field.

Guidelines:

  • Bloggers must be currently enrolled in a museum studies, museum education, library sciences or similar degree or certificate program.  Undergraduate and graduate students are welcome to apply.
  • Blogs should be no longer than 500 words and written in a conversational style.  Avoid jargon and academic language to ensure clarity.
  • You are welcome to share how the work of others has influenced your practice, but this isn’t required.  If you include quotes, be sure to include citations.
  • We have no idea what the winning blogs will look like – if you look through our past posts, you will see we tell stories, share academic insights, and sometimes we are funny.  We want to hear your story, so let your passion show.
  • Check your work carefully for spelling and accuracy.  While no one is perfect, winning blogs will be error free.
  • Email your entry to craig@randikorn.com by 5:00pm (EST), Friday, April 4, 2014.

RK&A staff will review all entries, pick the top 3 and publish them on the Intentional Museum blog.  Winners will be notified and announced at the end of April.  Winning blog posts will be shared with our readers in May and June 2014.  Winners will also receive a copy of one of our favorite museum books, Stephen Weil’s Making Museums Matter, with a personalized note from Randi.

How to Enter:

  • One (1) entry per blogger, please. 
  • Send your blog as a Word document attached to an email. 
  • Include your name, school, degree program and expected graduation date in the body of the email, with the subject line “Intentional Museum Blog Competition.”
  • Please do not include your name/identifying information as a header to your blog entry.  Each entry will be assigned a number to ensure unbiased review. 
  • Email your entry to craig@randikorn.com by 5:00pm (EST), Friday, April 4, 2014.

Other Important Information:

  • RK&A reserves the right to edit winning blog entries for content and length.
  • Winners will be notified via email and will have 48 hours to respond with their contact information for book delivery.
  • Books will only be mailed to those in the United States and will be sent via the US Postal Service no later than May 30, 2014.
  • If a winner does not respond in the allotted timeframe, an alternate winner will take his/her place.
  • Winners will be asked to submit a picture of themselves for publication with their blog.

Still have questions?  Contact us at craig@randikorn.com, or post a comment in response to this post on our blog!

Welcome to our new Throwback Thursday series, where we take a moment to look back at projects from our archives.  Today we’ll be sharing a case study about our planning and evaluation work with the Science Museum of Virginia and their Sphere Corps Program.  You might recall this particular Science On a Sphere program from one of our prior posts, Learning to Embrace Failure, and today we’ll share a bit more about how we approached the study, what we learned, and the implications of those findings.

Sphere Corps Program [2012]

For this planning and evaluation project with The Science Museum of Virginia (SMV), RK&A evaluated Sphere Corps, a Science on a Sphere program about climate change developed by SMV with funding from the National Oceanic and Atmospheric Administration (NOAA).    

How did we approach this study?  

The study was designed around RK&A’s belief that organizations must be intentional in their practice by continually clarifying purpose, aligning practices and resources to achieve purpose, measuring outcomes, and learning from practice to strengthen ongoing planning and actions.  To this end, the Sphere Corps project included five phases of work—a literature review, a workshop to define intended program outcomes, two rounds of formative evaluation, and two reflection workshops.  Formative evaluation data were collected using naturalistic observations and in-depth interviews.  Each phase of work allowed staff to explore their vision for the Sphere Corps program and how it changed over time as they learned from and reflected on evaluation findings.

What did we learn?SOS

SMV staff’s goal was to create a facilitated, inquiry-based Science on a Sphere program about climate change.  RK&A first completed a literature review that revealed a facilitated Sphere experience was in keeping with best practices and that using inquiry methods in a 20-minute program would be challenging but worth exploring further.  Staff then brainstormed and honed the outcomes they hoped to achieve in Sphere Corps, which guided planning and script development.  The first round of formative evaluation identified implementation barriers and an overabundance of iClicker questions, all of which created a challenging environment for educators to effectively use inquiry.  Upon reflection, staff reduced the number of iClicker questions and added visualizations and questions that required close observation of the Sphere.  Following a second round of formative evaluation, staff made additional changes to the program script and began to reflect on the reality of using inquiry in a single 20-minute program.  Since the script covered a range of topics related to climate change, staff wondered if they should instead go deeper with one topic while encouraging more visitor observation and interpretation of Sphere data.  Out of this discussion arose the idea of “mini-programs”—a series of programs that would focus on communicating one key idea about climate change, such as helping people understand the difference between weather and climate.

What are the implications of the findings?

Central to the idea of the “mini-program” is the idea of doing less to achieve more.  Impact and outcomes are incredibly difficult to achieve and trying to achieve too much often results in accomplishing very little.  Through a reflection workshop and staff discussion, the SMV team was able to prioritize and streamline the outcomes and indicators originally written for the Sphere Corps program.  Staff also recognized that their primary goal with the Sphere Corps program is to encourage visitors to think more critically about the science behind climate change.  By scaling down the number of topics covered in the presentation, each program could intentionally focus on: (1) one key idea or question related to climate change; (2) achievement of only a few intended outcomes; and (3) implementation of specific facilitation strategies to achieve those outcomes.  Intentionally covering less content also opens up opportunities to more effectively use inquiry methods.

25th Anniversary Butterfly

So often we evaluators are asked to measure outcomes or results, which of course align with our expectations.  When we conduct an evaluation and the results are positive, an organization can wave its flag; and ideally the whole museum field benefits from learning why a particular exhibition or program is so successful at achieving its outcomes.  During my time as an evaluator, I have learned that there is enormous value in walking before running.  Because measuring results sounds compelling to museums and their funders, museums often jump over important evaluation processes and rush into measuring results.  Accordingly, staff, in a moment of passion, forgo front-end and formative evaluation—those early stages of concept testing, prototyping, and piloting a program—that help staff understand the gaps between the intended outcomes for their audience and the successes and challenges of implementing a new project. 

So, when we are asked to measure results, we always ask the client if the project has ever been evaluated.  Even then, we may pull the reins to help slow down our clients enough to consider the benefits of first understanding what is and is not working about a particular program or exhibition.  More often than not, slowing down and using front-end and formative evaluation to improve the visitor experience increases the likelihood that staff will be rewarded with positive results when they measure outcomes later.  In fact, when an organization’s evaluation resources are limited, we often advocate for conducting a front-end and/or formative evaluation because we believe that is where all of us will learn the most.  It is human nature to want to jump right in to the good stuff and eat our dessert first.  We, too, get excited by our clients’ passion and have to remind ourselves of the value of taking baby steps.  So, one of the many lessons I’ve learned (and am still learning) is that when it comes to evaluation, encouraging practitioners to walk before they run (or test before they measure) is key to a successful project and their own personal learning.

RK&A’s Stephanie Downey will moderate a session, Getting the Most from Evaluation, for the New York City Museum Educators Roundtable on Wednesday, February 12th at 6:00 pm.  Panelist from The Bruce Museum of Arts and Science, The Wildlife Conservation Society, and The Metropolitan Museum of Art will join her to share their lessons learned and best practices from evaluation projects.

For more information, please click HERE.

Follow

Get every new post delivered to your Inbox.

Join 105 other followers