Reflection 18: Weighing the Positive Visitor Experiences with the Negative

25th Anniversary ButterflyA recent Telegraph article announced that the chairman of Arts Council England thinks there should be a one-hour photo ban (on selfies in particular) in art galleries. My first reaction was: “This is an interesting and absolutely horrible idea.” I see how a photo ban could be conceived as a strategy to enhance the visitor experience—I have certainly muttered to myself in annoyance when there are so many people taking photos of an artwork that I can’t get close enough to see it (or if I feel brazen enough to make my way to the front so I can see it, I feel bad for ruining everyone’s photo). However, if this one-hour photo ban were to go through, I see it creating a lot more negative visitor experiences than positive ones when you put yourself in the shoes of the security guard—the person who has to enforce the rule.

Let me step back a moment and say that I owe my current professional career to my work as a security guard. In addition to many other roles as an intern at the Peggy Guggenheim Collection, I guarded galleries and certainly learned a lot about visitor experiences. As someone who wanted to work in a museum, I found that I, as a security guard, had the power to either make or break the quality of a visitor’s experience. Tell visitors about Peggy’s many artist lovers while standing in her bedroom—make their visit and even their day. Ask a visitor to leave her bag in a locker or coatroom—incite anger to the point of someone asking for a refund and never setting foot in the museum. It was a humbling experience to say the least and completely transformed my thinking about the work I wanted to do for museums.

Now jumping back to the policy at hand…when reading the article, I first imagined how this would work on the ground.  I immediately empathized with the poor security guards who would have to enforce this policy (as did a Hyperallergic author who commented on this policy). Yes, perhaps signage would alert visitors to the ban, but from evaluation we know that it would go largely unnoticed. Therefore, my predictor is that the first awareness a visitor would have of the policy is when a security guard tells him or her not to take a photo. No matter how friendly a security guard may be, being told not to do something can create an embarrassing situation. How the visitor then reacts to feeling embarrassed is another story. Does he argue with the guard? Just continue to take pictures anyway? Does he internalize it and feel awful for the rest of the day? Any way it plays out will generally result in a negative experience for the visitor as well as those around him.

The chairman’s comparison of this no-photo policy and the “quiet car” is a perfect analogy in my opinion. As a frequent train rider, I love the concept of the quiet car, and I choose to sit in it more often than not. It works well when everyone knows they are in the quiet car. The trouble is, typically, there is one person who doesn’t know, which puts a negative pallor on everyone else’s experience in the quiet car. Most quiet cars have a sign, sometimes the lights are dimmer than other cars, and sometimes the conductor will announce which car is the quiet car. Perfect—except non-regular riders do not notice the signs, subtle lighting cues, or are aware what car they are in (am I in the first car?). Therefore, when the un-knowing person is encountered by a fellow quiet-car rider or conductor about a rule-breaking cell phone conversation, the ensuing interaction often doesn’t go well. I have seen and heard about everything—from a New Jersey Transit rider being escorted off the train by police after starting a fight with a confronting rider, to an Amtrak rider construing the conductor’s request as him telling her she “has a big mouth.” For these reasons, I find myself avoiding the quiet car lately because I end up being more frustrated than relieved and feeling more negative than positive. From what I have seen as security guard, evaluator, and expert train rider, more negative than positive visitor experiences might result from this potential photo-ban policy.

Photo from the Fortune article, "The Cult of the Amtrak Quiet Car," an interesting read for quiet care devotees and those unaware: http://fortune.com/2014/09/17/amtrak-quiet-car/

Photo from the Fortune article, “The Cult of the Amtrak Quiet Car,” an interesting read for both quiet car devotees and those completely unaware: http://fortune.com/2014/09/17/amtrak-quiet-car/

 

Reflection 17: Presentation Matters

25th Anniversary ButterflyThe most challenging evaluation report I’ve written consisted of 17 PowerPoint slides. The slides didn’t pull the most salient findings from a larger report; the slides were the report! I remember how difficult it was to start out with the idea of writing less from qualitative data. While I had to present major trends, I feared the format might rob the data of its nuance (17 PowerPoint slides obviously require brevity). The process was challenging and at times frustrating, but in the end I felt extremely gratified. Not only was the report thorough, it was exactly what the client wanted, and it was usable.

As evaluators, we toe the line between social science research, application and usability. As researchers, we must analyze and present the data as they appear. Sometimes, especially in the case of qualitative reports, this can lead to an overwhelming amount of dense narrative. This acceptable reporting style in evaluation practice is our default. Given the number of reports we write each year, having guidelines is efficient and freeing. We can focus on the analysis, giving us plenty of time to get to know and understand the data, to tease out the wonderful complexity that comes from open-ended interviews. As researchers, the presentation takes a backseat to analysis and digging into data.

However most of the time we are writing a report that will be shared with other researchers; it is a document that will be read by userspaper-stack-300x251museum staff who may share the findings with other staff or the board. Overwhelming amounts of dense narrative may not be useful; not because our audience can’t understand it, but because often the meaning is packed and needs to be untangled. I would guess what clients want and need is something they can refer to repeatedly, something they can look at to remind themselves, “Visitors aren’t interested in reading long labels,” or “Visitors enjoy interactive exhibits.” As researchers, presentation may be secondary, but as evaluators, presentation must be a primary consideration.

As my experience with the PowerPoint report (and many other reports since then) taught me, it can be tough to stray from a well-intentioned template. A shorter report or a more visual report doesn’t take less time to analyze or less time to write. In fact, writing a short report takes more time because I have to eliminate the dense narrative and find the essence, as I might with a long report. I also have to change the way I think about presentation. I have to think about presentation!

At RK&A, we like to look at our report template to see what we can do to improve it – new ways to highlight key findings or call out visitor quotations. Not all of our ideas work out in the long run, but it is good to think about different ways to present information. At the end of the day, though, what our report looks like for any given project comes from a client’s needs—and not from professional standards. And I learned that when I wrote those 17 PowerPoint slides!

Reflection 16: Building Evaluation Capacity—when is it the right choice?

As evalua25th Anniversary Butterflytors, we are often asked to help our clients build evaluation capacity among staff in their organization. The motivation for these requests varies. Sometimes the primary motivator is professional development; other times it is perceived cost savings (since conducting professional evaluations can require resources that not all organizations have at their disposal). We welcome when an organization values evaluation enough to inquire about how to integrate it more fully into their staff’s daily work. If an organization has a true interest in using evaluation as a tool to learn about the relationship between its work and the public, building an organization’s evaluation capacity may be quite successful. On the other hand, if the primary motivator is to save costs associated with evaluation, often the outcome is much less successful, mostly because evaluation takes considerable time and invariably there is a trade-off; when the evaluation is being done, something else is being ignored.

Evaluation capacity building can take a variety of forms. It can range from building staff’s capacity to think like an evaluator, perhaps by helping staff learn how to articulate a project’s desired outcomes (I think this is the most valuable evaluation planning skill one can learn), to training staff to conduct an evaluation from beginning to end (identifying outcomes, creating an evaluation plan, designing instruments, collecting data, conducting analyses, and reporting findings). Even among the most interested parties, it is rare to find museum practitioners who are genuinely interested in all aspects of evaluation. As an evaluator, even I find certain aspects of evaluation more enjoyable than others. I’ve noticed that while practitioners may be initially intimidated by the data collection process, they often find talking with visitors rewarding and informative. On the other hand, they have much less enthusiasm for data analysis and reporting; I’ve only encountered a handful of museum practitioners who enjoy pouring over pages and pages of interview transcripts. We lovingly refer to these individuals as “data nerds” and proudly count ourselves among them.

There is yet another challenge, and it has to do with the fact that most museum practitioners are required to wear many mountain-data-mining33hats. Conducting evaluations is my one and only job; it is what I am trained to do and have intentionally chosen for my vocation. While a practitioner may be intrigued by what evaluation can offer, often it is not the job he or she was trained or hired to do, which means that evaluation can become a burden—just one more hat for a practitioner to wear. Some organizations have addressed an organization’s evaluation needs by creating a position for an in-house evaluator and the individual who might fill that position is usually someone who is schooled in evaluation and research methodologies, much like all of us here at RK&A. I would caution organizations to be very realistic when considering building their organization’s evaluation capacity. Does your staff have the time and skills to conduct a thoughtful study and follow through with analysis and reporting? What responsibilities is your organization willing to put aside to make time for the evaluation? And, do you want your staff to think like evaluators or become evaluators?—an important distinction, indeed. Otherwise, even those with the best of intentions may find themselves buried in mountains of data. Worse yet is that what was once an exciting proposition may be perceived as an annoyance in the end.

Reflection 14: Is It Real?

25th Anniversary ButterflyAs we move further and further into the digital age, museums hold something that is becoming a rare commodity—real objects and artifacts. It may be hard to believe, but one day, many tangible objects may be obsolete, the way that printed photographs and airplane tickets are becoming scarce items. Instead of going on “digs,” future archaeologists may primarily use computer-driven devices to search for clues of our ancestors. In the distant future, I can imagine that museums will be magical places where people can see “the real thing.” …But wait, maybe they already are?

 

This is my third reflection—informed by what I have learned about museum visitors in all my years studying them. I have found that there are many reasons people visit museums, but I believe the primary reason, one that we may take for granted, is that they want to see “the real thing.” A common question heard in museums is, “is it real?” especially in regard to bones and historical objects. I have heard it in our research, but you have probably heard it too, or said it yourself while walking through a museum. Why do people ask this question? What underlies the need to know if something is “real.” As museum goers, can’t observing a replica of a dinosaur skeleton or a 17th century Dutch ice skate tell us just as much as the real thing? Maybe so. But there is just something about being in the presence of authentic artifacts and objects that is thrilling; maybe it has to do with feeling connected to other people, to the past, or to other parts of the world. Professors David Thelen and Roy Rosenzweig say it best in their landmark 1998 national study, The Presence of the Past: “approaching artifacts and sites on their own terms, visitors could cut through all the intervening stories, step around all the agendas that have been advanced in the meantime, and feel that they were experiencing a moment from the past almost as it had originally been experienced—and with none of the overwhelming distortions that they associated with movies and television, the other purveyors of immediacy.”

 

Whenever studying museum visitors, I come face to face with their sense of wonder about and desire to get clDinosaur Sueose to (even touch) real objects. Whether evaluating text panels, interactive exhibits, touch tables, or ideas and concepts, visitors will usually keep coming back to the objects. As museum professionals, it is sometimes too easy to forget the centrality of the object when you are knee-deep in trying to interpret and contextualize something. We can get lost in these various mediums of interpretation, but visitors will usually remind us what they are really there for. For example, I was doing a study for a museum and historic site last year in which we were testing ideas for high-tech touch tables intended to convey information about the historic building the museum is housed in. I had gotten so wrapped up in testing all the information, that I had mentally pushed aside the museum’s biggest asset, the historic building it resides in. But visitors brought me back when they practically skipped over my questions about the touch tables and rather, kept circling back to the building itself—its authentic and tangible sense of history. Of course that is what they wanted to talk about and why they were there. This isn’t to say that interpretation of any kind is futile. But I believe it is important to keep reminding ourselves, as museum professionals, that interpretation should be used primarily to help visitors make sense of the objects and artifacts they are there to see—it’s really that simple.

 

The very reason I work with museums is because of my own sense of wonder and astonishment when it comes to objects and artifacts. Yes, I love studying people and how they learn and make sense of experiences, but I could do that in many different settings. I chose to do it in museums because of my own belief that we can learn so much from studying “the real thing.”

Reflection 13: Why I am Passionate about Intentional Practice

25th Anniversary ButterflyFor me, intentionality, a concept I view as essential to museum planning, emerged from two core experiences: results from hundreds of exhibition and program evaluations; and observing museum staff wanting to put too many concepts into an exhibition. Intuitively I knew there was a connection between exhibitions that didn’t fare too well (at least according to the evaluations) and staff not letting go of ideas that are near and dear to their hearts—regardless of whether those ideas supported the thesis of the exhibition.

When I have the good fortune to attend planning meetings, I always find myself thinking critically about what should be included in the exhibition under discussion and what could be saved for another time. My consideration always includes the big idea of the exhibition, what the museum would like to achieve with the exhibition vis-à-vis the public, humans’ capacity to process new ideas when in unfamiliar environments (like that of an exhibition hall), evaluation results from other projects that show what leads to quality visitor experiences and what might move visitors away from having quality experiences, and my utmost respect for scholars’ knowledge and passions. While passionate individuals love their subject matter (and really, I love their subject matter, too), one’s willingness to recognize that not all good ideas (or even great ones) belong in an exhibition and then exercising follow through are traits of intentional practice.

Embedded in intentional practice is the concept of alignment—ensuring that project concepts, components, and elements are present because they support the impact the team wants to achieve. If there are concepts, components, elements that do not contribute to the Intentionalitycore idea of the exhibition and its potential impact on audiences, they need to be omitted. I certainly don’t mean to sound ruthless, but I am acutely aware of how easy it is to keep putting more and more into an exhibition plan and how painfully difficult it is to take anything away. I am also aware of how challenging it is to stay focused on the exhibition’s big idea and have the discipline to say no to ideas because they do not support the intended impact of the exhibition. Learning to say “no” is a necessary survival skill and saying “no” is deeply connected to intentional practice. When practitioners are intentional, they are focused on the impact they want to achieve; they exercise discipline and restraint when determining how to best move forward; and their decision making is egoless and for the sake of achieving the results the team envisions.

Intentional practice represents the culmination of my experiences to date, and my passion for it is directly is tied to my evaluation experience. Over the years I started to realize that when exhibitions tried to do too much, visitors’ experiences didn’t amount to much—from their perspective; their heads were full but descriptions of their experiences were nebulous. Sense-making seemed futile. Nudging me was my memory of exhibition development discussions and tensions about what to put in (everything) and what to take out (nothing). Clarifying the intent of the exhibition and then staying focused on the intent of the exhibition is hard work—not likely to end soon, which is okay. Seeking clarity—whether in thought or action—is a never-ending pursuit.

Reflection 12: On puzzles and people

25th Anniversary ButterflyI’ve always loved solving puzzles. And to me, people are the most fascinating puzzle of all. Perhaps that’s why I studied two people-focused topics as an undergraduate- biological anthropology and history. Not only was I curious to learn about how our evolutionary past has shaped human behavior, but I also wanted to understand how our cultural experiences affect those behaviors.

To this point, one of the great things about museums is that they bring people together to share ideas and learn from one another. I’ve always thought of museums as some of the most innovative and fun places to spend my time. They’re also fascinating places to explore people. Where else can you encounter a blend of different age groups, cultural histories, educational backgrounds, learning styles, hands-on experiences, fascinating artifacts and ideas?

I like to think of evaluation as having a continuous set of “people puzzles” to solve. How do people behave in and use museums? What motivates their behaviors? What does learning look like in a museum? How can a museum craft meaningful experiences for the people that walk through its doors? What do those experiences look like?

The solutions to these puzzles are plentiful and ever-changing and that’s what I love about them. As someone who’s relatively new to the field, I’m excited to have the chance to observe and interpret peoples’ often mysterious behaviors to help museums continually create experiences that are truly responsive to their visitors’ needs.

But perhaps most importantly, as a new evaluator, I’m excited by the opportunity to help those that work in museums learn to continually ask this ever-evolving set of questions themselves. Because underlying our field of people watching and data analyzing is the idea that it’s perfectly alright to not have the solutions, as long as you’re willing to ask questions in order to uncover them and learn from mistakes along the way.

So, although as an evaluator I may never be able to definitively solve these people puzzles, I can help museums and their visitors understand and relate to each other in new and unanticipated ways.

Reflection 11: Order and Process

25th Anniversary ButterflyOne of the things I love about my job is the part where I need to find order in chaos, such as the trends in a large sample of individual in-depth interviews, and refining our processes for collecting and analyzing data in order to find that order more efficiently. I’d say that I have always had this tendency towards order and process. As a little girl, I could spend hours lining up figurines in a single line and moving them one by one like they are marching in a parade. I have learned immensely from and with my colleagues at RK&A particularly in regard to process. And while I can think of several examples about how we have refined process, my favorite example is Johanna Jones’s work with timing and tracking. Johanna was with RK&A for 13 years and over that span of time she honed RK&A’s timing and tracking processes to a truly distinct approach, learning a great deal from others like Beverly Serrell, who Randi considers the queen of timing and tracking.

photo

Timing and tracking was something that hooked me into the museum evaluation field. I remember being completely fascinated by Arthur Melton’s timing and tracking studies when I learned about them in grad school. As I was formulating my thesis research topic, I knew that I wanted to incorporate timing and tracking just to have the opportunity to try it. For my thesis research, I timed and tracked art museum visitors to determine whether security guards affected visitors’ movement in the gallery. I had meticulously drawn maps full of dots and lines to track a visitor’s and security guard’s behaviors, plus I wrote open-ended notes about what the security guard was doing in the space. The data was wholly exhausting to collect and time consuming to analyze, but I felt like it gave me a handle for what timing and tracking was. . .

 

. . . Then, Johanna shared with me a timing and tracking form for one of my earliest projects at RK&A. I was completely blown away by it. It was neat, clean, concise, and focused. Every component in the exhibition was clearly identified with columns to indicate when a visitor started and stopped using it. There was also a short list of behaviors that we looked for at each exhibit. It was perfect! Elegant even. I still marvel at how refined the timing and tracking process became and strive to apply that same clarity and creativity to thinking about the various other processes that make up the work we do.