Archive for January, 2014

25th Anniversary ButterflyOver the years there have been pivotal moments in which we at RK&A tried something out-of-the- ordinary to meet the needs of a particular project that then became a staple in how we do things.  It wasn’t always clear at the time that these were “pivotal moments,” but in retrospect I can see that these were times of concentrated learning and change.  For me, one of these pivotal moments was the first time we used rubrics as an assessment tool for a museum-based project.  I had been introduced to rubrics in my previous position where I conducted educational research in the public school system, which sometimes included student assessment.  Rubrics are common practice among classroom teachers and educators who are required to assess individual student performance.

Rubrics had immediate appeal to me because they utilize qualitative research methods (like in-depth interviews, written essays, or naturalistic observations) to assess outcomes in a way that remains authentic to complicated, nuanced learning experiences; while at the same time, they are rigorous and respond to the need to measure and quantify outcomes, an increasing demand from funders.  They are also appealing because they respect the complexity of learning—we know from research and evaluation that the impact of a learning experience may vary considerably from person to person. These often very subtle differences in impact can be difficult to detect and measure.

To illustrate what a rubric is, I have an example below from the Museum of the City of New York, where we evaluated the effect of one of its field trip programs on fourth grade students (read report HERE).  As shown here, a rubric is a set of indicators linked to one outcome.  It is used to assess a performance of knowledge, skills, attitudes, or behaviors—in this example we were assessing “historical thinking,” more specifically students’ ability to recognize and not judge cultural differences.  As you can see, rubrics include a continuum of understandings (or skills, attitudes, or behaviors) on a scale from 1 to 4, with 1 being “below beginning understanding” to 4 being “accomplished understanding.”  The continuum captures the gradual, nuanced differences one might expect to see.

Museum of the City of New York RubricThe first time we used rubrics was about 10 years ago, when we worked with the Solomon R. Guggenheim Museum, which had just been awarded a large research grant from the U.S. Department of Education to study the effects of its long-standing Learning Through Art program on third grade students’ literacy skills.  This was a high-stakes project, and we needed to provide measurable, reliable findings to demonstrate complex outcomes, like “hypothesizing,” “evidential reasoning,” and “schema building.”  I immediately thought of using rubrics, especially since my past experience had been with elementary school students.  Working with an advisory team, we developed the rubrics for a number of literacy-based skills, as shown in the example below (and note the three-point scale in this example as opposed to the four-point scale above—the evolution in our use of rubrics included the realization that a four-point scale allows us to be more exact in our measurement).  To detect these skills we conducted one-on-one standardized, but open-ended, interviews with over 400 students, transcribed the interviews, and scored them using the rubrics.  We were then able to quantify the qualitative data and run statistics.  Because rubrics are precise, specific, and standardized, they allowed us to detect differences between treatment and control groups—differences that may have gone undetected otherwise—and to feel confident about the results.  For results, you can find the full report HERE.

Solomon R. Guggenheim Museum RubricFast forward ten years to today and we use rubrics regularly for summative evaluation, especially when measuring the achievement of complicated and complex learning outcomes.  So far, the two examples I’ve mentioned involved students participating in a facilitated program, but we also use rubrics, when appropriate, for regular walk-in visitors to exhibitions.  For instance, we used rubrics for two NSF-funded exhibitions, one about the social construction of race (read report HERE) and another about conservation efforts for protecting animals of Madagascar (read report HERE).  Rubrics were warranted in both cases—both required a rigorous summative evaluation, and both intended for visitors to learn complicated and emotionally-charged concepts and ideas.

While rubrics were not new to me 10 years ago (and certainly not new in the world of assessment and evaluation), they were new for us at RK&A.  What started out as a necessity for the Guggenheim project has become common practice for us.  Our use of rubrics has informed the way we approach and think about evaluation and furthered our understanding of the way people learn in museums.  This is just one example of the way we continually learn at RK&A.

Read Full Post »

25th Anniversary ButterflyThe quarter-century mark feels like the right time to take stock of where RK&A is, and at the very least, think about what we have learned along the way.  Reflecting on the past is a task that feels comfortable; we know where we have been, and we are familiar with the present.  Making sense of the past and forging new ideas from the past is far more difficult—yet, as a staff, that is what we have decided to do for our 25th anniversary celebration—which will take the entire year!  As some of you know, we have been blogging for exactly a year—at  What better platform is there to share what we have learned over the last quarter century?

Evaluation, visitor studies, audience research—that is the work I set out to do and it remains our traditional work.  RK&A has carefully grown to seven people, including a small satellite office in NY.  Along the way all of us have learned so much—about visitors, about cultural organizations, and about the relationship between the two.  As a staff that has always strived for excellence, we try hard to apply new knowledge to our practice.

About 10 years ago I gravitated towards the notion of intentionality as a concept I wanted to explore.  Conducting evaluations had shown me that it would be worthwhile to figure out a way to help cultural organizations focus their passions, skills, and resources towards their vision of impact.  Helping organizations determine what impact they wanted to achieve seemed like the first step, and so was born our intentional practice workshops (which took about two years of R&D).  Achieving impact with audiences is harder than one thinks, so in order for cultural organizations to achieve impact, they need to be intentional in how they carry out their work.

The demand for our intentional practice workshops continues to grow, and because our intentionality work emerged from our evaluation work, it was only a matter of time until we would begin to weave what we have learned back into our evaluation practice.  We now offer intentionality-like workshops as part of our evaluation services to help staff understand and apply evaluation data to their planning.  Rarely do staff from across the organization get together to discuss and debate their visitors and work, but when they do, the results are inspiring.

However, inspiration doesn’t always lead to action.  While most people and organizations want to change, saying so is easier than doing so.  I have learned the virtue of taking baby steps towards change, and sometimes baby steps give people time to learn and internalize a new way of thinking and working.  It’s that way at RK&A. Sometimes our intentionality work feels organic and sometimes we need to be more deliberate and forthright in our decision making in order to sustain a change in our practice.  It is human nature to gravitate to old ways of doing things until new ways become comfortable; it takes conviction and focus to continue to move forward.  To sustain our learning and help RK&A maintain its momentum with our new intentionality work and traditional evaluation practice, we will share 25 years of learning with you over the next 25 blog posts.  We hope our 25th year celebration of RK&A’s learning inspires you to learn along with us.

Read Full Post »