Posts Tagged ‘learning’

25th Anniversary ButterflyI have the honor of writing the last blog post of our 25th year—which just so happens to coincide with the end of the calendar year. It was New Year’s Eve in 1989 when my husband and I arrived in DC after driving from LA. RK&A was born soon thereafter. When I reflect on the last 25 years I find it impossible not to think about the changes that have taken place in our little evaluation world and the larger museum world.   Seriously, a whole new world order has emerged. And all of us at RK&A have tried very hard to move along with those changes so we could continue living our passion—working with museums to help them achieve impact in their communities.

Our intent for this celebratory year was to share our learning, and I hope we have done that for you. Our learning isn’t always linear, obvious, or easy to describe. The very act of writing these 25 posts has helped us process and internalize what we have learned, which helps us continually apply our learning to our practice. Honestly, sometimes we struggled to find a learning topic that we were ready to share. Sometimes the things we were learning felt too new or raw to share; other times we weren’t far enough along in our thinking where we had a handle on exactly what we had learned; and sometimes, if we were lucky, through the process of writing, we clarified our thinking and learning. Learning can be a funny thing; new ideas can feel scary—especially if they go against what we are accustomed to thinking or take us out of our comfort zone (see Reflection 24! https://intentionalmuseum.com/2014/12/17/reflection-24/).

I know learning can be fun (or so I am told), but sometimes learning can be really hard—like the times when we (Okay, I) wrote circles around an idea because the learning hadn’t quite jelled, where I didn’t quite have the words to express my thought, or when my writing sounded murky—obviously not my intent. More and more, though, with each passing year, I have come to respect and take advantage of time—that thing we never seem to have enough of. Time can be my friend if I let it; if I patiently let an idea simmer or if I deliberately take the time to become acquainted with a new way of thinking I can begin to ease into the new idea until it feels a tiny bit more comfortable—comfortable enough for me to begin playing with it. Without any self-imposed pressure (take note—that’s the important part), I just let it roll around in my head until it feels more familiar.

A past New Year’s Day walk; it was chilly but lovely.

A past New Year’s Day walk; it was chilly but lovely.

So I am working on the obvious emergence of 2015; a seemingly familiar idea because a new year emerges every 365 days or so—and I’ve lived through enough of them that this shouldn’t be a surprise. The unknown (e.g., the future), like learning, can be scary. And the weather isn’t cooperating either—at this writing it is dreary and gray—not the way I want to end one year and not very welcoming as the start of another. My vegetable garden is soggy and dormant; my front garden lacks interest. But I have hope because on New Year’s Day I will take my 10-mile walk as I have done for the past many years and ready myself for all kinds of new experiences and learning. I’m getting kind of excited just thinking about it. I do not know what 2015 will bring, but I know my glass will be half full and my learning will be rich. I can just feel it.

Happy holiday to all and a very healthy new year!

Read Full Post »

25th Anniversary Butterfly

For the last few months, I’ve been on maternity leave with my second child. My first little girl is three years old and I had forgotten many of the nuances of caring for an infant. Through the process of getting to know my new little one, I began to reflect on a drawing that we often show to our clients in our planning or reflection workshops.

 

panic learning comfortThe drawing consists of three concentric circles. In the center circle, we write “comfort zone.” This, we tell practitioners, is where most of us operate on a daily basis during our work day.  We feel safe in this zone because it consists of routines and interactions with colleagues we know well. In the middle circle, we write “learning zone.” This is the zone where we have “ah-ha” moments and take in new ideas from our own work or interactions with colleagues. Perhaps we go to a meeting where different points of view are shared, and we have a breakthrough moment about a project we are working on or see something familiar in a new light. We tell clients that this is, ideally, where we want them to be during our workshops—open to sharing and receiving new ideas. Then, of course, there is the outside circle which we label the “panic zone.” This is the zone where we shut down because we are too uncomfortable to take in new learning or ideas. When this happens, we long for the “comfort zone” and, until we find our way back there, we are unlikely to be receptive to our colleagues’ ideas. Instead, we often put up walls or use defense mechanisms to deflect what we find uncomfortable.

I’ve operated in all of these zones the past few months. Before going on maternity leave, I was in my “comfort zone” with my first daughter. Even though she changes every day, we have a daily routine that works pretty well. With her, I happily and regularly enter the “learning zone” as well. She is constantly learning new things and, now that she is in school, she is learning at a rapid rate. As I’m sure many of you who are parents know, this is challenging and surprising in a pleasant sort of way. Then, my second daughter was born, and I entered the “panic zone.” She is amazing but, although I remembered the newborn phase in the abstract, I’d forgotten all the little challenges of caring for a very small baby. Once you think you’ve mastered one thing, it changes, and you have to adapt all over again within a relatively short period of time. As someone who loves organization, schedules, and routines, I find this an uncomfortable state of being. She is slowly shifting into more predictable patterns as she grows but I’ve decided that the newborn phase is my “panic zone.” I’m much more comfortable with older babies and toddlers.

Comfort ZoneWhen we facilitate meetings or workshops, we encourage our clients to invite a diverse group of relevant stakeholders to sit around the table. Many times, those who attend have interacted with one another on a limited basis. Thus, the reason we present the learning zone graphic is because we know from experience that facilitating a conversation among colleagues who rarely come together to discuss and reflect on ideas can create a challenging environment for some. And, as we tell everyone, we all have different thresholds for these three zones. What is comfortable for some might cause others to panic. For example, I am more likely to hit the “panic zone” when dealing with a newborn while another mother might hit this zone more readily with a toddler. Since we want everyone to operate in the “learning zone,” we remind practitioners to pay attention to how they and their colleagues are receiving the ideas being discussed so no one enters the “panic zone” where learning ceases to happen. So, at a time when we are all reflecting on the past year and forming New Year’s resolutions, I find myself thinking how important it is for us all to be honest with ourselves and one another about our thresholds for these different zones so we can spend more time learning and less time panicking.

Read Full Post »

25th Anniversary ButterflyWe are excited to introduce a guest blogger this week: Johanna Jones, former Managing Director of RK&A’s San Francisco office.  Having been with RK&A for 14 years, Johanna contributed greatly to the company’s learning, and thus, we are happy she agreed to reflect as part our 25 years of learning series.

 

As I think back on my work at RK&A, like Stephanie, I am struck by an unintended outcome of the evaluation process; namely, that asking visitors questions about their experience can serve an important interpretive role in museums and for visitors. We don’t often think about the direct value of evaluation on visitors—rather we focus on evaluation as an essential step in the institutional cycle of learning (see reflection 2 for the cycle of learning) and in creating an intentional organization which, of course, benefits visitors. I would argue that the evaluation process itself—the very act of interviewing visitors—serves a powerful interpretative function. When visitors are asked open-ended questions about their visit, they are afforded the time and space to reflect on their experiences. The questions become a framework for thinking about their visit—beyond what they liked and didn’t like—that prompts them to consider “What does this mean to me?”openended question

I didn’t always see the direct value of evaluation for visitors. When I first started building my evaluation skills, the educator in me worried about imposing on visitors’ time by asking questions. I had some rocky starts—people have been so saturated with market research that they are wary of someone approaching them for feedback. But I quickly learned that if you ask visitors meaningful open-ended questions, you are usually met with meaningful responses. I remember a Vietnam veteran who cried when telling me what the American flag meant to him and the young children who were pumped to save the Condors (not exactly warm and cuddly creatures). I recall avid art museum-goers who were amazed to realize that they could interpret works of art for themselves and Twenty-somethings who expressed civic pride in a once beleaguered natural history museum. The more I talked to visitors the more I realized that the evaluations were not only fulfilling the institutions’ need to understand visitors but also visitors’ need to process and make meaning from their museum experience.

Now, as a museum visitor, I still use meaty open-ended questions. I ask them of myself and my family members when we visit museums. And, I must say, I find our conversations richer for it.

Read Full Post »

25th Anniversary ButterflyI would like to dedicate this post to Alan Friedman, who passed away on Sunday. I wrote this blog post last week before I heard the news of Alan’s illness. In retrospect, it seems absolutely fitting that I honor him by telling the story of how I went from someone ambivalent about science to someone who now sees science as part of my everyday experience. I’m quite certain that what I describe below is at least partially the result of all the work Alan did in the field of informal science education, for which I am deeply grateful. 

 

As an evaluator, I am typically working with a museum around the idea of outcomes—the results they intend for their visitors. Sometimes, through the research process, we discover a museum has achieved outcomes it didn’t necessarily intend, but is delighted to have done so nonetheless. These are usually referred to “unanticipated outcomes.” In life, unanticipated outcomes happen all the time for lots of people in all kinds of circumstances.

 

I’ve experienced my own unanticipated outcome through my work at RK&A. As a consultant, I have the privilege of working with all types of museums all over the country. A typical result of my work is learning something new in the areas of art, history, or science. Over the years, I have learned the mostly unknown story of the hospitals at Ellis Island, how the book The Little Prince was written, and what a watershed is. These are just three of hundreds of examples. I guarantee, if it weren’t for my work, I wouldn’t have these learning opportunities. These examples are isolated and discreet, but one big unanticipated outcome has slowly become a part of me and how I see and experience the world—I have acquired an appreciation and understanding of science, especially the scientific process, that I didn’t have before and maybe never would have had, had I not pursued a career in museum evaluation and research.

 

It isn’t that I disliked or wasn’t curious about science before becoming a museum evaluator. My primary interests Stephblog7were more in the area of art, history, and culture. Prior to becoming a museum evaluator, I visited art museums and historical sites, not science centers or science museums. Science just wasn’t part of my everyday life. Thus, I’ll never forget my first project working for RK&A—I was to do a front-end evaluation for an exhibition about materials science, which I quickly learned involved concepts like atomic structure! I remember I stopped at Barnes and Noble on my way home that night and gave myself a crash course on atoms and how they affect the properties of different materials (this was in the Internet’s infancy, so I couldn’t simply Google “atomic structure and material science”). Through my preparation of the interview guide and activities, conducting interviews with visitors, and analysis of data, I learned more about atomic structure than I may have ever learned in school. I gained a true appreciation, based on understanding, for why certain materials are brittle and others are flexible, for instance. More than anything, and simply put, my work on various science projects over the years has sparked and nurtured a curiosity in observing the physical world around me and asking, “Why is that the way it is?”

 

The other day, I had a conversation with my nine-year old daughter that helped me realize that, over time, this unanticipated outcome has become deeply embedded with who I am today and has carried over to others in my life. I was telling my daughter about a new project at a natural history museum and casually asked her if she knows why scientists study fossils. She provided me with a pretty accurate and relatively complex response, saying that fossils are clues to the past and help us understand what happened thousands of years ago.   I was kind of blown away and asked with genuine curiosity, “How on earth do you know so much?” Her response: “Because I have a mommy who works with museums, of course.” I realized then that my (new) interest in science had become so much a part of who I am that it had, of course, rubbed off on my daughter (and my son too, actually). I now have two kids who beg me to stay up late Sunday nights to watch Cosmos. I’m sure they were probably born with an interest in science, but I am fairly certain their interest wouldn’t have been nurtured to the extent that it has if it weren’t for my own unanticipated outcome working as a museum evaluator.

Read Full Post »

25th Anniversary Butterfly

So often we evaluators are asked to measure outcomes or results, which of course align with our expectations.  When we conduct an evaluation and the results are positive, an organization can wave its flag; and ideally the whole museum field benefits from learning why a particular exhibition or program is so successful at achieving its outcomes.  During my time as an evaluator, I have learned that there is enormous value in walking before running.  Because measuring results sounds compelling to museums and their funders, museums often jump over important evaluation processes and rush into measuring results.  Accordingly, staff, in a moment of passion, forgo front-end and formative evaluation—those early stages of concept testing, prototyping, and piloting a program—that help staff understand the gaps between the intended outcomes for their audience and the successes and challenges of implementing a new project. 

So, when we are asked to measure results, we always ask the client if the project has ever been evaluated.  Even then, we may pull the reins to help slow down our clients enough to consider the benefits of first understanding what is and is not working about a particular program or exhibition.  More often than not, slowing down and using front-end and formative evaluation to improve the visitor experience increases the likelihood that staff will be rewarded with positive results when they measure outcomes later.  In fact, when an organization’s evaluation resources are limited, we often advocate for conducting a front-end and/or formative evaluation because we believe that is where all of us will learn the most.  It is human nature to want to jump right in to the good stuff and eat our dessert first.  We, too, get excited by our clients’ passion and have to remind ourselves of the value of taking baby steps.  So, one of the many lessons I’ve learned (and am still learning) is that when it comes to evaluation, encouraging practitioners to walk before they run (or test before they measure) is key to a successful project and their own personal learning.

Read Full Post »

25th Anniversary ButterflySometimes when learning surfaces slowly, it is barely visible, until one day the world looks different.  Responding to that difference is the first layer of that complex process often labeled as learning.  The Cycle of Intentional Practice was a long time coming—emerging from many years of conducting evaluations, where I worked closely with museum staff and leadership as well as with visitors.  The Cycle of Intentional Practice is an illustration of an ideal work cycle that started to form when I was writing “A Case for Holistic Intentionality”.  I am visually oriented and I often have to draw my ideas before I write about them; in this case, I was writing about my ideas and then I felt the need to create a visualization to depict what I was thinking—in part to help me understand what I was thinking, but also to help others.  I included the first iteration of the cycle in the manuscript to Curator, but the editor said the Journal does not usually publish that kind of illustration, so I put it aside.

That original cycle differs from the one I use today—it was simpler (it included “Plan,” “Act,” and “Evaluate”), and while I didn’t know it at the time, it was a draft.  There have been several more iterations over time (one was “Plan,” “Act,” and “Evaluate & Reflect,” for example); as I continue to learn and improve my practice, I change the cycle accordingly.  Most stunning to me was that the first draft of the cycle showed nothing in the center—nothing!  I feel a little embarrassed by my omission and I am not entirely sure what I was thinking at the time, but I hope my oversight was short-lived.  At some point I placed the word “Intentions” in the center, and as I clarified my ideas, with the hope of applying the cycle to our evaluation and planning work, I eventually replaced “intentions” with “impact.”  I recall how difficult it was to explain the concept of “intentions” so I eventually needed to remove the word from the center (as much as I loved having it there).  If my goal was to have museums apply the cycle to their daily and strategic work, the cycle needed to represent an idea people found comfortable and doable.  Soon I realized that intentionality was the larger concept of the cycle and what needed to be placed in the center was the result of a museum’s work on its publics–impact.  So was born our intentionality work with museums.  Then I realized the true power of intentionality—mission could go in the center as well as outcomes, or anything for that matter.  The artist’s rendition below demonstrates the versatility of intentionality as a concept.

Cycle of Intentional Practice

An artistic rendering of the Cycle of Intentional Practice by artist Andrea Herrick

What I find most amazing is that two crucial ideas—reflection and impact—were not present in the first iterations of the cycle, although they were discussed when I talked about intentionality.  Our intentional planning work (which we refer to as impact planning) would be rudderless without the presence of impact and our ability to learn from our work would be weakened without reflection.  And that brings me to another realization, which I am reminded of daily—the never-ending pursuit of achieving clarity of thought, followed by writing a clear expression of that thought.

Today I talk about the Cycle of Intentional Practice as a draft—it will always be on the verge of becoming, but these days I am more comfortable with the idea of the Cycle being a draft—an idea in process—than I was a decade ago; in fact, I have come to realize that all work is a draft and that if one is serious about learning and applying new ideas to work and life, then all ideas, all products, all knowledge are mere drafts because learning is continuous, right?

Humbling?  Yes indeed.

Read Full Post »

25th Anniversary ButterflyOver the years there have been pivotal moments in which we at RK&A tried something out-of-the- ordinary to meet the needs of a particular project that then became a staple in how we do things.  It wasn’t always clear at the time that these were “pivotal moments,” but in retrospect I can see that these were times of concentrated learning and change.  For me, one of these pivotal moments was the first time we used rubrics as an assessment tool for a museum-based project.  I had been introduced to rubrics in my previous position where I conducted educational research in the public school system, which sometimes included student assessment.  Rubrics are common practice among classroom teachers and educators who are required to assess individual student performance.

Rubrics had immediate appeal to me because they utilize qualitative research methods (like in-depth interviews, written essays, or naturalistic observations) to assess outcomes in a way that remains authentic to complicated, nuanced learning experiences; while at the same time, they are rigorous and respond to the need to measure and quantify outcomes, an increasing demand from funders.  They are also appealing because they respect the complexity of learning—we know from research and evaluation that the impact of a learning experience may vary considerably from person to person. These often very subtle differences in impact can be difficult to detect and measure.

To illustrate what a rubric is, I have an example below from the Museum of the City of New York, where we evaluated the effect of one of its field trip programs on fourth grade students (read report HERE).  As shown here, a rubric is a set of indicators linked to one outcome.  It is used to assess a performance of knowledge, skills, attitudes, or behaviors—in this example we were assessing “historical thinking,” more specifically students’ ability to recognize and not judge cultural differences.  As you can see, rubrics include a continuum of understandings (or skills, attitudes, or behaviors) on a scale from 1 to 4, with 1 being “below beginning understanding” to 4 being “accomplished understanding.”  The continuum captures the gradual, nuanced differences one might expect to see.

Museum of the City of New York RubricThe first time we used rubrics was about 10 years ago, when we worked with the Solomon R. Guggenheim Museum, which had just been awarded a large research grant from the U.S. Department of Education to study the effects of its long-standing Learning Through Art program on third grade students’ literacy skills.  This was a high-stakes project, and we needed to provide measurable, reliable findings to demonstrate complex outcomes, like “hypothesizing,” “evidential reasoning,” and “schema building.”  I immediately thought of using rubrics, especially since my past experience had been with elementary school students.  Working with an advisory team, we developed the rubrics for a number of literacy-based skills, as shown in the example below (and note the three-point scale in this example as opposed to the four-point scale above—the evolution in our use of rubrics included the realization that a four-point scale allows us to be more exact in our measurement).  To detect these skills we conducted one-on-one standardized, but open-ended, interviews with over 400 students, transcribed the interviews, and scored them using the rubrics.  We were then able to quantify the qualitative data and run statistics.  Because rubrics are precise, specific, and standardized, they allowed us to detect differences between treatment and control groups—differences that may have gone undetected otherwise—and to feel confident about the results.  For results, you can find the full report HERE.

Solomon R. Guggenheim Museum RubricFast forward ten years to today and we use rubrics regularly for summative evaluation, especially when measuring the achievement of complicated and complex learning outcomes.  So far, the two examples I’ve mentioned involved students participating in a facilitated program, but we also use rubrics, when appropriate, for regular walk-in visitors to exhibitions.  For instance, we used rubrics for two NSF-funded exhibitions, one about the social construction of race (read report HERE) and another about conservation efforts for protecting animals of Madagascar (read report HERE).  Rubrics were warranted in both cases—both required a rigorous summative evaluation, and both intended for visitors to learn complicated and emotionally-charged concepts and ideas.

While rubrics were not new to me 10 years ago (and certainly not new in the world of assessment and evaluation), they were new for us at RK&A.  What started out as a necessity for the Guggenheim project has become common practice for us.  Our use of rubrics has informed the way we approach and think about evaluation and furthered our understanding of the way people learn in museums.  This is just one example of the way we continually learn at RK&A.

Read Full Post »

Older Posts »