Measuring the "Touchy-Feely" Stuff: Takeaways from a Qualitative Methods Workshop
“How do we measure the touchy-feely stuff?”
This question was posed to me at a recent pre-conference workshop I led at the 2018 HANO Conference, titled “Making Voices Heard: Capturing and Amplifying Constituent Perspectives Through Qualitative Methods.” The workshop was itself the result of an observation I’d made during previous presentations on qualitative data collection—namely, that many nonprofits are hungry to capture the perspectives of their constituents but feel they lack the know-how to do so. As live poll results from the workshop demonstrate, many nonprofits make the effort to collect qualitative data, but then aren’t sure what to do with it, leaving the data to collect proverbial dust on the shelf:
What is more, nonprofits frequently feel an either-or proposition exists, that in order to generate quantitative data, qualitative data must be sacrificed along the way. However, I strongly believe that qualitative data is the yin to quantitative data’s yang; together, they create a more balanced, more complete understanding of an organization, target population, or community. And as I’ve described elsewhere, the sum of each type of data is greater than its parts—yielding what I think of as a “1+1=3” understanding and richer basis for learning.
So how can you get started incorporating qualitative data collection into your organization’s practice? Here are six key takeaways from the workshop:
Take time to refine your qualitative question. I often say that good thinking begins with great questions. While it’s natural to get excited about the “doing”—creating a new survey, planning a focus group, or scheduling interviews—it’s important to be intentional about the “thinking” first. What question are we posing that we feel qualitative data will help us answer? What subjective information (e.g., opinions, motivations, beliefs, or interpretations of experiences) are we hoping to learn about? Taking time to explore and refine our learning question is important, because it will drive the qualitative data collection process. Some sample questions that might be informed by qualitative data collection:
· How is our organization perceived within the local community?
· Why do our older clients stay more engaged with our programs than our younger clients?
· How can we maximize the experience volunteers have with our organization?
· What do our supporters feel are the most important issues in the upcoming election?
· What are the greatest challenges facing arts and culture organizations today?
Sometimes less is more in capturing your constituent’s (literal) voices. While we are visual creatures and social media inundates us with videos, photos, and gifs, sometimes experiencing stories in their simplest form—through audio only—can be truly powerful. Fans of StoryCorps, the national nonprofit organization that works to preserve and share stories of people from all walks of life, can no doubt attest to being moved to laughter or tears simply by listening to an audio exchange of people having a conversation. For hearing audiences, audio stories can strip away all but the most essential elements of conversation, allowing emotions and meaning expressed through our voices as well as our words to shine through. In the workshop, we listened to the audio of a Vietnamese-American mother talking with her grown daughter, reflecting on the challenges of raising her within two cultural worlds:
Imagine your organization’s constituents, telling their stories in their own words and voice—what might they say, and what might audio recordings capture in a way that visuals cannot?
Look for themes to get an initial handle on qualitative data. For organizations that have collected stacks of open-ended survey responses or pages of interview transcripts, it can be overwhelming to know where to begin tackling all that narrative information. One way to make the gathered information manageable is to conduct a simple thematic analysis. In a nutshell, thematic analysis refers to a systematic review and coding of narrative data to identify patterns and themes that emerge from the text. While high-level thematic analysis can be complex and involve sophisticated software, for many organizations undertaking a light-touch, manual version of such analysis—what I dubbed “baby thematic analysis” in the workshop—is often enough to get an organization started on understanding the broad brushstrokes of respondents’ perspectives. What might that look like in practice? This brief and informative video below, produced by ModU, provides an excellent introduction to coding and thematic analysis:
Find a qualitative data “study buddy.” Like many new skills, analyzing qualitative data can be daunting to tackle on your own! It can be helpful to identify a study buddy—someone who is either walking the same journey of exploring qualitative data, or someone who is more experienced and can serve as a mentor—with whom you can practice these new ideas and skills. Not quite sure if the themes you see emerging from focus group responses are a single concept, or two distinct ones? Your study buddy may be able to help. Wondering if the wording for one of the survey questions you are drafting might be biased? Your study buddy can serve as a double-check. Hopefully you can return the favor, and each of you benefits by growing your knowledge, experience, and know-how in the process. Finding a study buddy outside of your own organization may help you gain objective feedback from someone who isn’t already steeped in your organizational lingo and assumptions.
Qualitative data collection can be an avenue to constituent empowerment. Fundamentally, qualitative data collection is a form of story collection. Each person’s experiences, perspectives, and opinions are part of the tale of who they are and shape how they move through the world. Simply having one’s story heard can be transformative and life-affirming, particularly for members of marginalized communities.
One of the forms of qualitative data collection that most resonated with the workshop participants was PhotoVoice. PhotoVoice combines visual storytelling with community building, and provides participants with guidance for using photography as a means of self-expression, community dialogue, and advocacy. The video below, produced by Health Share of Oregon, provides a powerful example of PhotoVoice in action—in this case, illuminating the intersection between individuals’ racial, ethnic, gender, and sexual identities and their experiences with health care:
Following group discussion and community-building, photographs are chosen and captioned by the participants. Capturing views of the world through participants’ eyes and accompanied by their own words can grow participants’ sense of agency while also educating a larger audience. One of the most visible examples of PhotoVoice in action in Hawaii was a community-based participatory evaluation of a Housing First program, conducted by a team of researchers at the University of Hawaii-Manoa. Sample photos from the project demonstrate the impact of Housing First participants’ images and captions.
Close the loop and amplify constituent voices. Almost all of us enjoy having our contributions acknowledged. When people take the time to provide their personal perspective, we owe it to them to close the loop on the information we’ve collected. For example, sharing out survey results allows respondents to see how their input relates to overall findings. Interviewees and focus group participants appreciate seeing final reports, white papers, and presentations that their insights helped inform. This loop-closing can be part of larger efforts to amplify the voices of your organization’s constituents, across multiple avenues for dissemination. Your organization’s website provides valuable real estate for moving first-person accounts into public spaces. Honolulu-based Domestic Violence Action Center, for example, features on its landing page a simple yet powerful video of a client who has benefitted from the organization’s programs.
Similarly, annual reports provide a vehicle for featuring the voices of your organization’s clients, supporters, and constituents; e-versions of annual reports (such as this 2017 annual report from StoryCorps) allow for live links to audio files, videos, or other online sources, adding additional dimensions to the data they typically include. Reports or white papers that include narrative passages or pull quotes (such as this report on institutional touchpoints of homelessness in Hawaii) can provide a visceral connection with individuals’ unique experiences. And public presentations—for instance, this share-out event featured at Honolulu Hale in July 2016 for the aforementioned Housing First PhotoVoice participants—can expand awareness and visibility of people whose voices might otherwise be overlooked.
How is your organization embracing qualitative methods of data collection? What have you or your organization learned through qualitative data collection that you wouldn’t have learned otherwise?