Let's be pragmatic: one approach to qualitative data analysis

Today, Hanna, one of my MSc students, has been asking interesting questions about doing a qualitative data analysis. Not the theory (there's plenty about that), but the basic practicalities.

I often point people at the Braun & Clarke (2006) paper on thematic analysis: it’s certainly a very good place to start. The Charmaz book on Grounded Theory (GT) is also a great resource about coding and analysis, even if you’re not doing a full GT. And I've written about Semi-Structured Qualitative Studies. For smallish projects (e.g. up to 20 hours of transcripts), computer-based tools such as  Atlas ti, nVivo and Dodoose tend to force the analyst to focus on the tool and on details rather than on themes.

I personally like improvised tools such as coloured pens and lots of notebooks, and/or simple Word files where I can do a first pass of approximate coding (either using the annotation feature or simply in a multi-column table). At that stage, I don’t worry about consistency of codes: I’m just trying to see what’s in the data: what seem to be the common patterns and themes, what are the surprises that might be worth looking at in more detail.

I then do a second pass through all the data looking systematically for the themes that seem most interesting / promising for analysis. At this stage, I usually copy-and-paste relevant chunks of text into a separate document organised according to the themes, without worrying about connections between the themes (just annotating each chunk with which participant it came from so that I don’t completely lose the context for each quotation).

Step 3 is to build a narrative within each of the themes; at this point, I will often realise that there’s other data that also relates to the theme that I hadn’t noticed on the previous passes, so the themes and the narrative get adapted. This requires looking through the data repeatedly, to spot omissions. While doing this, it's really important to look for contradictory evidence, which is generally an indication that the story isn't right: that there are nuances that haven't been captured. Such contradictions force a review of the themes. They may also highlight a need to gather more data to resolve ambiguities.

The fourth step is to develop a meta-narrative that links the themes together into an overall story. At this point, some themes will get ditched; maybe I’ll realise that there’s another theme in the data that should be part of this bigger narrative, so I go back to stage 2, or even stage 1.  Repeat until done!

At some point, you relate the themes to the literature. In some cases, the literature review (or a theory) will have guided all the data gathering and analysis. In other cases, you get to stage 4, realise that someone has already written exactly that paper, utter a few expletives, and review what alternative narratives there might be in your data that are equally well founded but more novel. Usually, it’s somewhere between these extremes.

This sounds ad-hoc, but done properly it’s both exploratory and systematic, and doesn’t have to be constrained by the features of a particular tool.

Hidden in full view: the daft things you overlook when designing and conducting studies

Several years ago, when Anne Adams and I were studying how people engaged with health information, we came up with the notion of an "information journey", with three main stages: recognising an information need; gathering information and interpreting that information. The important point (to us) in that work was highlighting the important of interpretation: the dominant view of information seeking at that time was that if people could find information then that was job done. But we found that an important role for clinicians is in helping lay people to interpret clinical information in terms of what it means for that individual – hence our focus on interpretation.

In later studies of lawyers' information work, Simon Attfield  and I realised that there were two important elements missing from the information journey as we'd formulated it: information validation and information use. When we looked back at the health data, we didn't see a lot of evidence of validation (it might have been there, but it was largely implicit, and rolled up with interpretation) but – now sensitised to it – we found lots of evidence of information use. Doh! Of course people use the information – e.g. in subsequent health management – but we simply hadn't noticed it because people didn't talk explicitly about it as "using" the information. Extend the model.

Wind forwards to today, and I'm writing a chapter for InteractionDesign.org on semi-structured qualitative studies. Don't hold your breath on this appearing: it's taking longer than I'd expected.

I've (partly) structured it according to the PRETAR framework for planning and conducting studies:

• what's the Purpose of the study?
• what Resources are available?
• what Ethical considerations need to be taken into account?
• what Techniques for data gathering?
• how to Analyse data?
• how to Report results?

...and, having been working with that framework for several years now, I have just realised that there's an important element missing, somewhere between resources and techniques for data gathering. What's missing is the step of taking the resources (which define what is possible) and using them to shape the detailed design of the study – e.g., in terms of interventions.

I've tended to lump the details of participant recruitment in with Resources (even though it's really part of the detailed study design), and of informed consent in with Ethics. But what about interventions such as giving people specific tasks to do for a think-aloud study? Or giving people a new device to use? Or planning the details of a semi-structured interview script? Just because a resource is available, that doesn't mean it's automatically going to be used in the study, and all those decisions – which of course get made in designing a study – precede data gathering. I don't think this means a total re-write of the chapter, but a certain amount of cutting and pasting is about to happen ...

The right tool for the job? Qualitative methods in HCI

It's sad to admit it, but my holiday reading has included Carla Willig's (2008) text on qualitative research methods in psychology and Jonathan Smith's (2007) edited collection on the same topic. I particularly enjoyed the chapters by Smith on Interpretive Phenomenological Analysis and by Kathy Charmaz on Grounded Theory in the edited collection. One striking feature of both books is that they have a narrative structure of "here's a method; here are its foundations; this is what it's good for; this is how to apply it". In other words, both seem to take the view that one becomes an expert in using a particular method, then builds a career by defining problems that are amenable to that method.

One of the features of Human–Computer Interaction (HCI) as a discipline is that it is not (with a few notable exceptions) fixated on what methods to apply. It is much more concerned with choosing the right tools for the job at hand, namely some aspect of the design or evaluation of interactive systems that enhance the user experience, productivity, safety or similar. So does it matter whether the method applied is "clean" Grounded Theory (in any of its variants) or "clean" IPA? I would argue not. The problem, though, is that we need better ways of planning qualitative studies in HCI, and then of describing how data was really gathered and what analysis was performed, so that we can better assess the quality, validity and scope of the reported findings.

There's a trade-off to be made between doing studies that can be done well because the method is clear and well-understood and doing studies that are important (e.g. making systems safer) but for which the method is unavoidably messy and improvisational. An important challenge for HCI (which has always adopted and adapted methods from other disciplines that have stronger methodological foundations) is to develop a better set of methods that address the important research challenges of interaction design. These aren't limited to qualitative research methods, but that is certainly one area where it's important to have a better repertoire of techniques that can be applied intelligently and accountably to address exciting problems.

When is a qualitative study a Grounded Theory study?

I recently came across Beki Grinter's blog posts on Grounded Theory. These make great reading.

The term has been used a lot in HCI as a "bumper sticker" for any and every qualitative analysis regardless of whether or not it follows any of the GT recipes closely, and whether or not it results in theory-building. I exaggerate slightly, but not much. As Beki says, GT is about developing theory, not just about doing a "bottom up" qualitative analysis, possibly without even having any particular questions or aims in mind.

Sometimes, the questions do change, as you discover that your initial questions or assumptions about what you might find are wrong. This has happened to us more than once. For example, we conducted a study of London Underground control rooms where the initial aim was to understand the commonalities and contrasts across different control rooms, and what effects these differences had on the work of controllers, and the ways they used the various artefacts in the environment. In practice, we found that the commonalities were much more interesting than the contrasts, and that there were several themes that emerged across all the contexts we studied. The most intriguing was discovering how much the controllers seemed to be playing with a big train set! This links in to the literature on "serious games", a literature that we hadn't even considered when we started the study (so we had to learn about it fast!).

In our experience, there's an interdependent cycle between qualitative data gathering and analysis and pre-existing theory. You start with questions, gather and analyse some data, realise your questions weren't quite right, so modify them (usually to be more interesting!), gather more data, analyse it much more deeply, realise that Theory X almost accounts for your data, see what insights relating your data to Theory X provides, gather yet more data, analyse it further... end up with either some radically new theory or a minor adaptation of Theory X. Or (as in our study of digital libraries deployment) end up using Theory X (in this case, Communities of Practice) to make sense of the situations you've studied.

Many would say that a "clean" GT doesn't draw explicitly on any existing theories, but builds theory from data. In practice, in my experience, you get a richer analysis if you do draw on other theory, but that's not an essential part of GT. The important thing is to be reflective and critical: to use theory to test and shine light on your data, but not to succumb to confirmation bias, where you only notice the data that fits the theory and ignore the rest. Theory is always there to be overturned!