I recently had the pleasure of participating in an online forum called Interactive Café, sponsored by the National Art Education Association (NAEA) Research Commission.  For a week, I exchanged ideas virtually with my co-hosts, Olga Hubard of Teachers College Columbia University, Michelle Grohe of the Isabella Stewart Gardner Museum, and Benjamin Tellie of the Charles E. Smith Jewish Day School in Rockville, Maryland, about assessing students’ responses to works of art.  Olga began the forum by posing the provocative question, “What is worth assessing in students’ responses to works of art?”  For me, the answer lies in another question: “For what purpose are you assessing students?” As a professional evaluator, the purpose is usually to help a museum understand the impact it has on the students it serves.  As Olga noted, there are many possible outcomes or benefits for students when they look at and respond to works of art, and it is my job to help a museum articulate its unique intentions for students.  Is the program designed to increase students’ critical thinking skills, curiosity or creativity, personal connections, or something else?  Once I truly understand a museum’s intent, the work of developing the assessment can begin.   In this post, I describe my work with one museum to illustrate intentionality in the process of developing a student assessment.

For the last eight months I have been working with the Katonah Museum of Art in Westchester County, New York, to assess the impact that the program, ArteJuntos /ArtTogether, has on the bilingual preschool students it serves.  The program is a partnership with a nearby local preschool that serves immigrant families.  Staff from the Museum visit the children (and their parents) at their school once a week to look at, talk about (through inquiry), and make art; the program also includes two visits to the Museum and parent training (an important part of the program that I have to leave out here for the sake of brevity).  I feel honored to be working with such a unique program and with people who understand that quality assessment takes time.  Fortunately, a full year of assessment (and other program activities) was generously funded by the National Endowment for the Arts.  As mentioned previously, I began by asking the very basic question, “How is your program designed to affect students?” and we continued from there.  To illustrate the intentional approach we took to developing the assessment, below I outline and explain the steps we have taken thus far.

  • The Museum described its intent for students primarily around literacy, especially emergent literacy. Remember these children are very young (on average 3 years old).  Museum staff believed (and had witnessed) that through facilitated, inquiry-based discussions about works of art, students have the unique opportunity to use and develop rich, descriptive language. Furthermore, they had heard from Pre-K teachers of students who had participated in ArteJuntos in previous years that these students seemed more verbal and better prepared for Pre-K.  The staff was eager to find out what was happening.
  • To hone and better understand the idea of literacy as it manifests in the context of ArteJuntos, we assembled a team of experts including Museum staff, teachers from the preschool, school administrators, and representatives from another community organization to talk about what literacy looks like for these young bilingual children and how the program affects their literacy. By the end of the day, I had a list of key indicators that would serve as evidence of students’ literacy in relation to looking at and talking about art.
  • I refined and honed the list of student outcomes and indicators and drafted a rubric that would be used to assess students’ literacy. The draft rubric included relatively simple indicators such as “The child names shapes (triangles, circles) to describe the work of art” and “The child names colors to describe the work of art” as well as more complex indicators like, “The child names two objects that are similar and/or two that are different and accurately describes similarities or differences.”  Museum staff, preschool teachers, and the principal reviewed the rubric and provided feedback.
  • I developed a protocol for the assessment.  In this protocol, each student sits one-on-one with a bilingual educator and a reproduction of a work of art (see the work of art by Carmen Lomas Garza below) and is asked a series of open-ended and close-ended questions that closely mirror the kinds of questions they are asked in the program.  For examples, questions include: “What do you see in this picture?” “What can you tell me about that?” and “What colors can you find in this picture?”
"Oranges" by Carmen Lomas Garza

“Oranges” by Carmen Lomas Garza

  • We tested the protocol by trying it out with some of the Museum staff’s children. As a result, we identified problem areas in the line of questioning and revised it as necessary.
  • In early fall, before ArteJuntos began, we did the first set of one-on-one student assessments with 12 children who would participate in the program that year.  These assessments serve as the pre-program assessment.  We videotaped the assessments, as shown here in this still clip from one of the videos:

Video still clip

  • Museum staff, preschool teachers, and I watched the videos together, discussing the emerging literacy we saw from the children. As a result of that meeting I revised the rubric. It remains focused on literacy, but now more closely aligns with what we saw happening among students.

We are about three-quarters of the way finished with the project.  This spring, once ArteJuntos ends, we will do a second round of assessments with the same children.  These will represent the post-program assessments.  At that point, we will score all of the videotaped children using the rubric, compare and contrast the pre- and the post-program assessments, and draw some preliminary conclusions about the way ArteJuntos impacts students’ literacy.  Our sample is small and we realize there are problems inherent in comparing pre-school children from pre- to post given how rapidly they develop; nevertheless, given our intentional process of developing the assessment tool, we feel confident that we will capture an accurate measure of students’ responses to works of art in the context of this unique program, and we hope that the assessment can continue to be utilized in years to come.

Foreword:

Our winning entry for our student blog competition reminded me that so many of us find our way into the museum field through other avenues, led by our passion for connecting people with art, science, history, you name it. For example, I started out studying non-human primate behavior which led to educating the public about non-human primates and now I study human primate behavior in museums and other informal learning environments! Intentionally following our passion for learning in and experiencing museums is often what unites us. Our student blogger, Kwasi, also reminded me of the passion that emerging museum professionals have for ensuring museums are accessible to many publics. Kwasi explains how he applied intentional practice—uniting his actions around the single goal of museum accessibility—to develop an app (www.thetravelsee.com) that helps people align their interests with cultural offerings in Cooperstown, NY. Read below for more about his journey.

Through your intentional practice, how do you help museums enrich the lives of others?IMG_3347

A few years ago, I ran a tour guide business in D.C. that focused on introducing visitors to the wonders of D.C.’s local museums. Every time I walked a tour group up to the entrance of a historic house or a national park site, someone in the group would always ask, “Can we really go inside?” I began to realize there was a disconnect between the way potential visitors viewed museums and how museums perceived themselves within communities.

I wanted to do something to help solve this issue and decided to close my tour guide business to pursue my masters in Museum Studies. A graduate course titled “Digital Technologies,” pushed me to think about how technology can be used to attract, engage and diversify museum audiences. The class spurred my intentional practice of creating a web application that redefines the way people engage with museums. I developed a prototype app called Travelsee which gives users an aggregate list of available cultural activities such as guided tours, seminars, or exhibitions in a given area based on the user’s keywords and GPS location.

Travelsee was formed by a need to show the general public that museums are fun and fascinating spaces. I decided to take a step outside of the museum advocate world, and I devised a simple way to use technology to strengthen the public’s awareness of museums and increase visitor engagement. I figured if I could come up with a way to gather all the local museum activities in one place, visitors would be able choose an activity based on their interests without any prior experience of visiting that specific museum.

I believe that museum activities should not be limited to a specific race, gender, class or any other social construct; unfortunately, most potential museum visitors look at museums as spaces for the elite. That is why redefining how museums engage potential visitors is important. As a museum advocate, I am using my web application Travelsee to engage new audiences. What methods are you using to engage new audiences?

For many years now, I have attended the National Art Education Association annual conference, and for the same many years, I have attended the Museum Division pre-conference—a day-long event for art educators who teach from original works of art.  These days, I am usually one of the more senior people attending—but not only because I am getting older.  The conference is often about teaching in the galleries, where many art museum educators begin their career, and as they advance to department leaders and even museum directors, they attend other conferences that help them manage their new challenges.  This year’s focus was on leadership, so a good number of other seasoned folks were in attendance.  While still outnumbered, there was a decent mix of people spanning as many as 50 years, providing a rich exchange of conversation, and for me, reflection.

Seated harp player, ca. 2800–2700 B.C.; Early Cycladic I–II The Metropolitan Museum of Art

Seated harp player, ca. 2800–2700 B.C.; Early Cycladic I–II
The Metropolitan Museum of Art

Attending such a focused conference provides a great opportunity to reflect on past years and changes that I have witnessed in the niche field of art museum education.  I have always felt a kinship with art museum educators; they are so passionate about their work (I am, too), they truly love what original works of art can do for people (the Cycladic sculpture at right is responsible for my very first deeply significant experience with a work of art when I visited the Metropolitan Museum of Art when I was 16), their stamina for intensely exploring ideas is commendable (and I am sometimes responsible for forcing the issue), and they seem to value intentionality (and I do, too).  In fact, during the pre-conference I heard the word “intentional” a lot—it seemed like everyone was talking about being intentional, which of course, delighted me.

In the preconference alone, I feel like I heard “intentional” or “intentionality” at least twice an hour—a whopping 16 times on that day alone, and because I attended museum division sessions, I heard it many times more throughout the larger conference.  So why were so many people talking about intentionality?   I’d like to think that my conference presentations over the last decade are starting to sink in (I have been discussing intentionality in every which way I can), but I suspect the somewhat recent issue of the Journal of Museum Education is mostly responsible—as it was titled, “Intentionality and the Twenty-First-Century Museum.”  I absolutely sense a shift taking place.  I think educators are starting to realize that often they try to do too much.  I realize sometime they are required to do too much—by their supervisors—but at least now, they are replying with a voice of reason as to why they may need to stop and take stock of what they are doing and why they are doing it.  They want their work to be purposeful and they want each and every action to support that purpose.

Simultaneously, they are also realizing that they just can’t continue doing more and more.  In order to manage their workloads (and we all know so many educators’ workloads are over the top), they are rethinking what they do and why they are doing it and this is where intentionality gains respect and momentum.  As with so many endeavors, the first step is recognizing that something needs to change.  I am so grateful to have experienced so many consecutive museum division pre-conferences; otherwise, I might not have witnessed this sea change.  Intentionality is hard, as our last post so noted.  I take a little bit of comfort in thinking that maybe RK&A can help by continuing to work with museums that want to become more intentional in their practice.  Maybe our blogging will help, or this article that I wrote in 2007.  I hope so—the future is looking brighter, thanks to art museum educators’ passion for wanting their work to make a difference!

Let’s not sugarcoat it—intentional practice is hard. It is not something you conquer, or something you do once and then pat yourself on the back for a job well done. It is a process not an end result and, here’s the honest truth, it never ends. So, why do it at all? Well, at RK&A we believe that informal learning organizations, like museums, want to make a difference in the quality of people’s lives, the way we define impact (as Randi wrote in her blog introducing our monthly series of writing about intentional practice). To make a difference in the quality of people’s lives, we believe organizations will need to be intentional in their practice—including consulting firms! Once people start down this path, they embark on a sort of journey, at least from our perspective. I’d like to describe a journey we had (and continue to have) with one of our clients.

We received an RFP from a botanic garden to conduct a summative evaluation of a permanent exhibition. As is typical, RFPs don’t always capture the essence of what our clients need. Instead, their needs become more apparent and transparent in the initial meeting for a project. During the initial meeting for this project, it quickly became clear that the organization would benefit from thinking about its permanent exhibition in the context of the whole visitor experience rather than as an isolated experience. It is common for organizations to think about each experience they offer as distinct but we know from years of conducting audience research and evaluation that visitors do not necessarily see such distinctions. Further, because achieving impact is so difficult, an organization really does need to align all its efforts around impact; and this is where intentional practice comes in.

Our suggested deviations from the RFP were fairly significant, but to this organization’s credit, it was open to a new approach—openness to new ideas or a new way of working is a key factor in embracing something like intentional practice. Not only were staff at various levels open to us significantly restructuring the scope of work, the director was on board as well, which is hugely important if intentional practice is to be a successful planning strategy. The new scope included intentional planning workshops to define the organization’s intended impact paired with an audience research study to assess the visitor experience in the context of this clarified definition of impact. The study maintained a focus on the permanent exhibition space and did so by exploring the space within the broader visitor experience.

After the first planning workshop, staff were invigorated by the idea of intentional practice but the journey had only really just begun—for us and for them. I want to make two main points about this type of journey: (1) it can be a little messy, even under the best of circumstances, and (2) it is ongoing (and somebody needs to be driving the bus or it doesn’t really happen). To illustrate the first point, we tried something during our second planning workshop that fell flat. We asked staff to do something that we, as evaluators, are quite comfortable doing on the spot—synthesizing (or detecting patterns among) ideas that emerged from a brainstorming session—but, really, such work is hard for others to do on the spot. We’ve since removed this exercise from subsequent iterations of this workshop and added others, as part of what makes this continuous process so effective (albeit considerably messy), is customizing and experimenting with ideas to help our clients practice intentionality.

To illustrate the second point, since the completion of the first project, we have worked with this client on several other occasions. We are not always afforded this luxury, but this organization has allowed us to support its intentional practice; we are now driving the bus together with each subsequent project. And, each time we work together, we incorporate the original intentional practice work into the current aspect of the visitor experience we are exploring; and, each time we do this, staff’s ideas for impact evolve and change and so does their practice. For instance, staff is currently focused on how one particular aspect of the impact statement—demonstrating its living collections’ connection to people—can be actualized in the garden’s interpretation. This is what I mean when I say that intentional practice is an ongoing process or pursuit. It truly never ends, but worthy pursuits rarely do.

Whole Garden and West Gallery exhibition [2013]

(Read the full report)

The United States Botanic Garden (USBG) contracted RK&A to study visitors’ experiences in the current West Gallery exhibition. However, after an initial meeting, USBG recognized that any changes to the West Gallery should be intentional and done in the context of staff’s aspirations for the whole Garden experience; thus, the study evolved into a more holistic endeavor with two main goals: (1) collect data about visitors’ experiences in the West Gallery exhibition to inform redesign of the Gallery; and (2) study visitors’ experiences in the whole Garden in the context of the newly-articulated visitor impact statement: Inspired by the welcoming, sensory, and restorative experience, visitors appreciate the diversity of plants, value the essential connection between plants and people, and embrace plant stewardship.

How did we approach this study?

RK&A facilitated a series of planning workshops with USBG staff to help them articulate the impact they aspire to achieve with their audiences. An Impact Framework resulted from these workshops. The Framework articulates the impact statement above, as well as audience outcomes and indicators which make the impact statement concrete and measurable. Guided by the Impact Framework, RK&A conducted an audience research study, employing four methods to explore West Gallery experiences and the Garden’s intended impact: (1) a standardized questionnaire; (2) in-depth interviews; (3) focused observations and interviews in the West Gallery; and (4) focus groups with teachers. Following the audience research study, RK&A facilitated two Using Evaluation Results workshops to help staff reflect on findings and develop action steps moving forward.

What did we learn?

The audience research study revealed many rich findings related to the Whole Garden, its audiences, and the West Gallery exhibition specifically, including visitor types that the Garden can use to inform their decision making (see full report for details). Study findings revealed that visitors’ experiences are, in some ways, well aligned with the Garden’s desired impact and, in other ways, not as well aligned. Specifically, staff used study findings to brainstorm more cohesive interpretive themes for the Whole Garden and West Gallery exhibition. Looking forward, USBG staff has two great opportunities to leverage these themes for an upcoming Garden-wide interpretive planning project and Conservatory Room evaluation.

What are the implications of the findings?

This project highlights the all-important link between planning and evaluation. Too often, evaluation is conducted in a vacuum (one program or exhibition at a time) as opposed to considering the organization’s aspirations for impacting the visitor. USBG staff recognized the need to consider changes to the West Gallery exhibition in the context of their intentions for the Whole Garden experience. In doing so, they now have baseline information about their audiences in the context of the impact they hope to achieve. This information helps USBG staff understand the alignment between their aspirations and visitors’ experiences and how they might need to change their practice to achieve greater impact.

When I read Emily’s reflection on comfort zone at the end of December, my first feeling was that of understanding.   As a parent of a now almost 2-year-old, I could relate to her story. Parenting really did force me to explore my comfort zone, learning zone, and definitely that panic zone too—although I try to avoid that as much as possible. Ironically, it was while operating in my learning zone as a parent—trying to keep myself ahead of the curve and away from that panic zone—that I came to another level of understanding of these zones through association.

I had just finished s book called “The Whole-Brain Child: 12 Revolutionary Strategies to Nurture Your Child’s Developing Mind” by Daniel J. Siegel, M.D. and Tina Payne Bryson, Ph.D. (Aside: Strategically placing those parenting books that I may not have sought out on my own in the children’s section right where all the kids gather to play is smart design). On page 12 of the Whole-Brain Child is a very simple image that I found clever at first and have found extremely compelling the more I have thought about. The image depicts a woman happily floating down a river between the banks of chaos and rigidity. In the book, the authors are describing a state they call integration. They write:

SiegelRiverImagine a peaceful river running through the countryside. That’s your river of well-being. Whenever you’re in the water, peacefully floating in your canoe, you feel like you’re generally in a good relationship with the world around you. You have a clear understanding of yourself, other people, and your life. You can be flexible and adjust when situations change. You’re stable and at peace.

The authors go on to describe the troubles encountered as one approaches the two banks of the river, which represent opposites. Chaos is where there is lack control; imagine “tumultuous rapids.” By contrast, rigidity is where there is too much control, and as the authors describe, the “water smells stagnant.” The image, both literal and the one they conjured through their writing, was very powerful to me. I could vividly sense what integration means to them. It also made me think about how this clever image and analogy could relate to so many aspects of my work.

My first association with Emily’s post was this image because I could easily imagine the river as the learning zone. In this scenario, one bank of the river is the comfort zone—I see Renoir’s Luncheon of the Boating Party … good food, wine, and company. By contrast, over in the panic zone, I am seeing Gericault’s The Raft of the Medusa banked on shore—nothing I want to be anywhere near. But my mind quickly left my new art historical river of comfort / learning/ and panic zones, to jump onto a second association—Mihaly Csikszentmihalyi and Kim Hermanson’s piece on flow experience in The Educational Role of the Museum (see “Intrinsic Motivation in Museums: Why Does One Want to Learn?”). And before I could get too transfixed reminiscing about how much I adored this article in grad school, then I was off thinking about how Siegel and Bryson’s analogy could be used to talk about the state required for innovation or creativity to happen.

Associations and associative thinking are so powerful. Sometimes they make you feel like you are taking a trip down memory lane. Other times it is like your own personal muse has arrived to show you new ways of looking at something. Anytime I am fortunate to have these moments, I am reminded to keep my mind and eyes open to what the world can show me (and how much is out there to know).

Intentional Museum is happy to announce its second student blogging competition! We believe that tomorrow’s museum professionals will shape and change the field through their unique perspectives and new ideas, and, because of that; there is a lot we can learn from students. New voices keep us on our toes and encourage us to consider alternate viewpoints.

We think a lot about intentional practice and would like to hear how students think about intentional practice and the impact it can have on the visitor experience. To that end, we ask that you reflect on the following question: Through your intentional practice, how do you help museums enrich the lives of others?

Perhaps you find joy in drafting a collections care plan, ensuring that objects and artifacts are around for many generations. Maybe you spend your time thinking about how museums can better use digital opportunities or social media to expand their reach beyond the traditional walls. From museum education to exhibitions, visitor services to administration, regardless of your focus, we want to hear from you. We often reflect on our professional experiences on Intentional Museum, but we appreciate the personal connection. We want your blogs to tell a story, to speak about your experience, and to highlight your unique insight into the museum field.

Guidelines:

  • Bloggers must be currently enrolled in an undergraduate or graduate degree or certificate program and have an interest in museum practice.
  • Blogs should be no longer than 500 words and written in a conversational style. Avoid jargon and academic language to ensure clarity.
  • You are welcome to share how the work of others has influenced your practice, but this isn’t required. If you include quotes, be sure to include citations.
  • We have no idea what the winning blogs will look like – if you look through our past posts, you will see we tell stories, share academic insights, and sometimes we are funny. We want to hear your story, so let your passion show.
  • Check your work carefully for spelling and accuracy. While no one is perfect, winning blogs will be error free.
  • Email your entry to sigmond@randikorn.com by 5:00pm (EST), Friday, March 13, 2015.

RK&A staff will review all entries and publish the top one or two responses on the Intentional Museum blog. Winners will be notified and announced at the end of March. Winning blog posts will be shared with our readers in April and May 2015. Winners will also receive a copy of one of our favorite museum books, Stephen Weil’s Making Museums Matter, with a personalized note from Randi.

How to Enter:

  • One (1) entry per blogger, please.
  • Send your blog as a Word document attached to an email.
  • Include your name, school, degree program and expected graduation date in the body of the email, with the subject line “Intentional Museum Blog Competition.”
  • Please do not include your name/identifying information as a header to your blog entry. Each entry will be assigned a number to ensure unbiased review.
  • Email your entry to sigmond@randikorn.com by 5:00pm (EST), Friday, March 13, 2015.

 Other Important Information:

  • RK&A reserves the right to edit winning blog entries for content and length.
  • Winners will be notified via email and will have 48 hours to respond with their contact information for book delivery.
  • Books will only be mailed to those in the United States and will be sent via the US Postal Service no later than May 1, 2015.
  • If a winner does not respond in the allotted timeframe, an alternate winner will take his/her place.
  • Winners will be asked to submit a picture of themselves for publication with their blog.

Still have questions? Contact us at sigmond@randikorn.com, or ask in the comments!

Follow

Get every new post delivered to your Inbox.

Join 168 other followers