I know we promised you a new post in our Intentional Practice series today, but intentional thought takes time! Sorry for the delay – we promise to share a new Intentional Practice post soon!
Earlier this month, I had the opportunity to attend Stephanie Evergreen’s Presenting Data Effectively workshop at The Evaluator’s Institute. The workshop offered logical advice so your work will stand out from the rest (you know those Microsoft default colors? Ditch them!), and a chance to talk to evaluators who work outside the museum field. I learned so many things that I can’t wait to try in my work; each night I wanted to go home and practice what I learned.
However, the biggest take-away wasn’t a new type of graph to try or the websites that can help me pick complimentary colors for reports. It was a very simple reminder: Although we have five senses, we take in information first and foremost with our eyes. After realizing this, I reflected a bit on my work and my process. Yes, I take time to organize reports so that they make sense – the data dictates the most logical way to organize the report; for example, I often organize trends from most to least pervasive. But, working with qualitative data as often as I do, I haven’t spent a lot of time thinking about how to best visually present essential data or how to present information using graphics and colors so report results are easier for clients to see. And then I was reminded: We take in information first with our eyes.
So, since it is January and a time for resolutions, I resolve to make 2015 the year I keep report visuals in mind. Here’s to a more visually pleasant 2015!
With a new year upon us and all sorts of possibilities—most of them unknown at this time—our blog entries will take on a slightly different flavor. We intend to remain true to the name of this blog, the Intentional Museum, by presenting a monthly series on Intentional Practice in museums. Throughout this series, we’ll discuss how we see intentional practice emerging in our work with clients as well as investigate how professionals working in different areas of the museum field think about intentionality. In addition to reflecting on how we see intentional practice emerging in our clients’ work, we’ll interview people from a range of museums and areas of the profession and talk with them about how they infuse intentional practice into their thinking, actions, and aspirations. Perhaps we will be reaching out to some of you!
As we embark on 2015, it might be useful to recall the meaning of intentionality. When my interest in intentionality surfaced, I did a little background research to learn about its origins. Little known to me at the time, the word “intentionality” has deep philosophical roots. Timothy Crane, a professor of philosophy in Cambridge who is best known for his work on intentionality, credits Franz Brentano for reintroducing the concept in 1874; it derives from the Medieval Latin. Brentano’s Thesis, as it is known, “can be expressed by saying that one cannot believe, wish, or hope without believing or wishing something.” Webster’s Dictionary’s definition of “intend” is “to direct the mind on,” which harkens back to Brentano’s original explanation of intentionality noted as “the direction of the mind on an object.” These above definitions are considered scholastic definitions, although I feel like they suggest that when the mind is focusing attention on something (a concept or even intended impact) it is possible to move mountains, which doesn’t sound very scholastic. I can feel the intensity of Brentano’s Thesis, and it is with that intensity that I have come to appreciate intentionality and its power to help museums achieve their aspirations.
After digging a bit deeper, I discovered that the way I apply the term in my practice with museums is more similar to how the field of social cognition defines and uses the term. In 2001 Bertrum F. Malle, Louis J. Moses, and Dare A. Baldwin edited a book called Intentions and Intentionality: Foundations of Social Cognition. And according to the Psychology Wiki, which provides different uses of the term, the following use is offered under social cognition: “Human perceivers consider a behavior intentional when it appears purposeful or done intentionally—that is, based on reasons (beliefs, desires) and performed with skill and awareness.” The social cognition use of the term aligns well with all that I have attached onto intentionality, for example, the notion of intentional practice, the four quadrants—all comprised of actions, and (of course) Impact. I believe intentionality is required practice if museums are going to make a difference in the quality of people’s lives, which is how I define “impact.” Without an intense focus on taking actions to achieve a well-defined end result, the end result will be difficult to achieve; ahhhh, if only impact would just magically appear . . . .
We’ll publish a new post in our Intentional Practice series once a month, on the third Wednesday of each month. Stay tuned for the next post on January 21st!
The case study below highlights two summative evaluations RK&A did at the California Academy of Sciences (CAS). Both exhibitions debuted in the new CAS building that opened in San Francisco’s Golden Gate Park in September 2008. Although the two exhibitions use different interpretive methods and have different learning outcomes, the two projects together highlight the importance of exhibition introductions.
Water Is Life and Altered State: Climate Change in California 
Summative evaluations of two exhibitions for a natural history museum
The California Academy of Sciences (CAS) contracted Randi Korn & Associates, Inc. (RK&A) to evaluate two exhibitions debuting in the CAS’s new facility. One exhibition, Altered State: Climate Change in California, uses fossils, interactive technology, and live animals “to explore the science of climate change, the effects we may expect to see in our own backyard, and the steps that can be taken to mitigate these dramatic changes,” while the other exhibition, Water is Life, explores the importance and diversity of water using the Steinhart Aquarium’s Living Collection.
How did we approach this study?
We believe that each evaluation study is unique and should strongly consider the goals and objectives of the exhibition, program, or other endeavor. As such, RK&A worked closely with CAS to identify its goals and objectives for each exhibition. Water is Life was focused on visitor learning. In response, RK&A conducted a remedial evaluation to identify operational or conceptual shortcomings early in exhibition development, followed by a summative evaluation that employed a rigorous, modified pre-test/post-test design to measure visitor learning. For Altered State, CAS sought to understand what visitors did in the exhibition, as well as what they took away from their experiences; thus, RK&A conducted timing and tracking observations and in-depth exit interviews.
What did we learn?
While many findings were exhibition-specific, there were also two larger trends. First, both evaluations show that visitors had strong affective experiences in the exhibitions, although learning objectives were challenging to meet. For instance, in the Water is Life evaluation, findings show that visitors who went to the exhibition demonstrated much greater interest in and concern for the natural world than visitors who did not see the exhibition. However, there were few differences in the knowledge of visitors who went to the exhibition and those who did not.
Second, visitors need a strong physical and conceptual introduction to exhibitions. In the Water is Life remedial evaluation, some interviewees described way-finding issues, and a few interviewees specifically asked for a better introduction. Further, in Altered State, the open space exhibition design, with its multiple entry and exit points, likely contributed to low dwell times.
What are the implications of the findings?
The evaluations are a keen reminder that exhibition introductions are imperative. They can set the conceptual stage for visitors, and if they are well-conceived and executed, they can also convey overarching concepts, connect subthemes, and present the intent of the exhibition. On a similar note, while the open exhibition design offers visitors a free-choice learning environment, introducing some structure and direction might help those seeking to understand the exhibition’s “big idea” without compromising the free-choice quality. Additionally, the studies reaffirmed that the unique value of exhibitions are the strong affective experiences they prompt.
I have the honor of writing the last blog post of our 25th year—which just so happens to coincide with the end of the calendar year. It was New Year’s Eve in 1989 when my husband and I arrived in DC after driving from LA. RK&A was born soon thereafter. When I reflect on the last 25 years I find it impossible not to think about the changes that have taken place in our little evaluation world and the larger museum world. Seriously, a whole new world order has emerged. And all of us at RK&A have tried very hard to move along with those changes so we could continue living our passion—working with museums to help them achieve impact in their communities.
Our intent for this celebratory year was to share our learning, and I hope we have done that for you. Our learning isn’t always linear, obvious, or easy to describe. The very act of writing these 25 posts has helped us process and internalize what we have learned, which helps us continually apply our learning to our practice. Honestly, sometimes we struggled to find a learning topic that we were ready to share. Sometimes the things we were learning felt too new or raw to share; other times we weren’t far enough along in our thinking where we had a handle on exactly what we had learned; and sometimes, if we were lucky, through the process of writing, we clarified our thinking and learning. Learning can be a funny thing; new ideas can feel scary—especially if they go against what we are accustomed to thinking or take us out of our comfort zone (see Reflection 24! http://intentionalmuseum.com/2014/12/17/reflection-24/).
I know learning can be fun (or so I am told), but sometimes learning can be really hard—like the times when we (Okay, I) wrote circles around an idea because the learning hadn’t quite jelled, where I didn’t quite have the words to express my thought, or when my writing sounded murky—obviously not my intent. More and more, though, with each passing year, I have come to respect and take advantage of time—that thing we never seem to have enough of. Time can be my friend if I let it; if I patiently let an idea simmer or if I deliberately take the time to become acquainted with a new way of thinking I can begin to ease into the new idea until it feels a tiny bit more comfortable—comfortable enough for me to begin playing with it. Without any self-imposed pressure (take note—that’s the important part), I just let it roll around in my head until it feels more familiar.
So I am working on the obvious emergence of 2015; a seemingly familiar idea because a new year emerges every 365 days or so—and I’ve lived through enough of them that this shouldn’t be a surprise. The unknown (e.g., the future), like learning, can be scary. And the weather isn’t cooperating either—at this writing it is dreary and gray—not the way I want to end one year and not very welcoming as the start of another. My vegetable garden is soggy and dormant; my front garden lacks interest. But I have hope because on New Year’s Day I will take my 10-mile walk as I have done for the past many years and ready myself for all kinds of new experiences and learning. I’m getting kind of excited just thinking about it. I do not know what 2015 will bring, but I know my glass will be half full and my learning will be rich. I can just feel it.
Happy holiday to all and a very healthy new year!
For the last few months, I’ve been on maternity leave with my second child. My first little girl is three years old and I had forgotten many of the nuances of caring for an infant. Through the process of getting to know my new little one, I began to reflect on a drawing that we often show to our clients in our planning or reflection workshops.
The drawing consists of three concentric circles. In the center circle, we write “comfort zone.” This, we tell practitioners, is where most of us operate on a daily basis during our work day. We feel safe in this zone because it consists of routines and interactions with colleagues we know well. In the middle circle, we write “learning zone.” This is the zone where we have “ah-ha” moments and take in new ideas from our own work or interactions with colleagues. Perhaps we go to a meeting where different points of view are shared, and we have a breakthrough moment about a project we are working on or see something familiar in a new light. We tell clients that this is, ideally, where we want them to be during our workshops—open to sharing and receiving new ideas. Then, of course, there is the outside circle which we label the “panic zone.” This is the zone where we shut down because we are too uncomfortable to take in new learning or ideas. When this happens, we long for the “comfort zone” and, until we find our way back there, we are unlikely to be receptive to our colleagues’ ideas. Instead, we often put up walls or use defense mechanisms to deflect what we find uncomfortable.
I’ve operated in all of these zones the past few months. Before going on maternity leave, I was in my “comfort zone” with my first daughter. Even though she changes every day, we have a daily routine that works pretty well. With her, I happily and regularly enter the “learning zone” as well. She is constantly learning new things and, now that she is in school, she is learning at a rapid rate. As I’m sure many of you who are parents know, this is challenging and surprising in a pleasant sort of way. Then, my second daughter was born, and I entered the “panic zone.” She is amazing but, although I remembered the newborn phase in the abstract, I’d forgotten all the little challenges of caring for a very small baby. Once you think you’ve mastered one thing, it changes, and you have to adapt all over again within a relatively short period of time. As someone who loves organization, schedules, and routines, I find this an uncomfortable state of being. She is slowly shifting into more predictable patterns as she grows but I’ve decided that the newborn phase is my “panic zone.” I’m much more comfortable with older babies and toddlers.
When we facilitate meetings or workshops, we encourage our clients to invite a diverse group of relevant stakeholders to sit around the table. Many times, those who attend have interacted with one another on a limited basis. Thus, the reason we present the learning zone graphic is because we know from experience that facilitating a conversation among colleagues who rarely come together to discuss and reflect on ideas can create a challenging environment for some. And, as we tell everyone, we all have different thresholds for these three zones. What is comfortable for some might cause others to panic. For example, I am more likely to hit the “panic zone” when dealing with a newborn while another mother might hit this zone more readily with a toddler. Since we want everyone to operate in the “learning zone,” we remind practitioners to pay attention to how they and their colleagues are receiving the ideas being discussed so no one enters the “panic zone” where learning ceases to happen. So, at a time when we are all reflecting on the past year and forming New Year’s resolutions, I find myself thinking how important it is for us all to be honest with ourselves and one another about our thresholds for these different zones so we can spend more time learning and less time panicking.
This year, I was lucky to receive a full scholarship to attend the 42nd annual Museum Computer Network (MCN) conference in Dallas, TX. For those who don’t know, MCN’s a fantastic organization that focuses on digital engagement in the cultural sector. Here’s a video of some of the highlights from the conference:
I’d wanted to attend MCN for a long time after hearing many friends and colleagues rave about the amazing energy and talents of MCN-ers. Before I left, I set two (very broad) goals for myself for MCN2014:
- Deepen my own understanding of how digital is transforming museums
- Think about new ways to apply this understanding to my work as an evaluator
Luckily, this year’s conference theme—“Think Big, Start Small” —aligned perfectly with these goals. I figured I would “start small” by going to the conference, all the while remembering to “think big” about the relationship between evaluation and digital transformation in the cultural sector. And with that in mind, I dove headfirst into the MCN2014 madness.
I quickly realized that I was one of the only full-time evaluators there. Despite the high energy of Wednesday’s activities (including a workshop on dabbling with microcontrollers and a series of inspiring Ignite talks), I couldn’t shake the feeling that I was going to be out of my element among all of these “tech” people as I headed to the first official sessions on Thursday. Would anyone understand why an evaluator was at a conference that’s all about digital? Worries aside, I was curious to learn how other attendees were thinking about and creating digital experiences, and how, if at all, they were working to evaluate their impacts (even if we might use different vocabulary to talk about the evaluation process).
My worries were quickly alleviated. While there may not have been many full-time evaluators at MCN2014, I was blown away by the amount of evaluative thinking I observed in nearly every session I attended (and, frankly, in all of the side conversations I had throughout the conference). Not only are those who work at the intersection of the cultural and digital sectors a highly energetic and creative group of people, but they are also working hard to determine realistic goals for their projects and are thinking seriously about how to measure them. I was both inspired and amazed by the extent to which evaluation was part of the other attendees’ thinking and planning processes. This evaluative thinking showed throughout the conference tweets (#MCN2014 was actually trending on Twitter during the conference), so I figured I’d use a few of my favorites to talk about some of the many ideas I’ve been thinking about ever since I got back from Dallas:
These two perfectly sum up a few ideas that we are constantly thinking about at RK&A. To echo Simon Tanner, there’s no point in gathering data just for sake of having information. It’s essential to think about why you want to gather data and outline how you plan to use the data you collect. Without articulating a plan for using the data in the long-run, it becomes difficult to ensure that you’re gathering the types of data that will be most useful to you. And having a plan for how you will use the data will ensure that when it’s time for analysis you can align your analysis with your long-terms goals. At RK&A, we want to make sure our clients clearly understand that data are there to be used so that when it comes time to make changes based on the data, they are already prepared to do so. However, I think that data is primarily used to test assumptions rather than confirm them. The word “confirm” is misleading, because it presupposes positive assumptions, i.e. that a project is working well. If that’s the case then it’s understandable that people want to see those positive assumptions confirmed (which in turn would mean having to make few changes). Learning to accept what the data tells you, even when the results are negative, is no simple task. It’s very easy to become so attached to a project that you ignore the problems and only see what’s working well. But remaining “open to surprise” and letting the data shine new light on a project is the best way to develop a true understanding of what’s happening so you can adapt and make changes to help achieve your goals.
I have mixed feelings on drawing distinctions between testing for user experience and testing for content. In my opinion, the two are separated by a fine line—at what point in any museum interactive, mobile app, game, or other digital experience does the user experience become entirely separate from the content? All content matters in terms of the user experience because the content itself, no matter the particular subject, dictates the experience that visitors/users expect to have. In other words, I think that visitors’/users’ prior expectations of a (digital) experience and opinions of the experience after use inherently depend on the subject matter presented to them. Visitors’/users’ preconceptions about the particular content at hand are so much a part of their experience. While there are always smaller usability issues that can be addressed without giving much regard to content (the size of a button, for example), I ultimately think that the entire user experience can never be truly separated from the content that supports it. If you change the content, you can’t help but change the experience.
Those are just a few of the ideas discussed at MCN2014 that I am still thinking about weeks later. The conference evoked so many interesting issues and questions that I couldn’t possibly go into all of them in one post. Suffice it to say that I left MCN2014 feeling silly for ever being nervous about whether others would perceive the overlaps between the worlds of evaluation and digital. MCN turned out to be a fantastic experience that greatly expanded my own thinking on these issues, and I’m excited to put these new ideas to use in my work and to (hopefully) explore them further at MCN2015 in Minneapolis!
Didn’t make it to MCN2014 but want to view the sessions? Check out MCN’s YouTube channel. And don’t forget to check out the amazing (and short—9 minutes!) Ignite talks. You can also find all of the conference tweets using the hashtag #MCN2014.