Hidden in full view: the daft things you overlook when designing and conducting studies

Several years ago, when Anne Adams and I were studying how people engaged with health information, we came up with the notion of an "information journey", with three main stages: recognising an information need; gathering information and interpreting that information. The important point (to us) in that work was highlighting the important of interpretation: the dominant view of information seeking at that time was that if people could find information then that was job done. But we found that an important role for clinicians is in helping lay people to interpret clinical information in terms of what it means for that individual – hence our focus on interpretation.

In later studies of lawyers' information work, Simon Attfield  and I realised that there were two important elements missing from the information journey as we'd formulated it: information validation and information use. When we looked back at the health data, we didn't see a lot of evidence of validation (it might have been there, but it was largely implicit, and rolled up with interpretation) but – now sensitised to it – we found lots of evidence of information use. Doh! Of course people use the information – e.g. in subsequent health management – but we simply hadn't noticed it because people didn't talk explicitly about it as "using" the information. Extend the model.

Wind forwards to today, and I'm writing a chapter for InteractionDesign.org on semi-structured qualitative studies. Don't hold your breath on this appearing: it's taking longer than I'd expected.

I've (partly) structured it according to the PRETAR framework for planning and conducting studies:

• what's the Purpose of the study?
• what Resources are available?
• what Ethical considerations need to be taken into account?
• what Techniques for data gathering?
• how to Analyse data?
• how to Report results?

...and, having been working with that framework for several years now, I have just realised that there's an important element missing, somewhere between resources and techniques for data gathering. What's missing is the step of taking the resources (which define what is possible) and using them to shape the detailed design of the study – e.g., in terms of interventions.

I've tended to lump the details of participant recruitment in with Resources (even though it's really part of the detailed study design), and of informed consent in with Ethics. But what about interventions such as giving people specific tasks to do for a think-aloud study? Or giving people a new device to use? Or planning the details of a semi-structured interview script? Just because a resource is available, that doesn't mean it's automatically going to be used in the study, and all those decisions – which of course get made in designing a study – precede data gathering. I don't think this means a total re-write of the chapter, but a certain amount of cutting and pasting is about to happen ...

Wizard of Oz: the medium and the message

Last week, one of my colleagues asserted that it didn't matter how a message was communicated – that the medium and the message were independent. I raised a quizzical eyebrow. A few days previously, I'd been in Vancouver, and had visited the Museum of Anthropology. It's a delightful place: some amazing art and artefacts from many different cultures. Most of them relate to ceremony and celebration, rather than everyday life, but they give a flavour of people's cultures, beliefs and practices. And most of them are beautiful.

One object that caught my attention was a yakantakw, or "speaking through post". According to the accompanying description: "A carved figure such as this one, with its prominent, open mouth, was used during winter ceremonies. A person who held the privilege of speaking on behalf of the hosts would conceal himself behind the figure, projecting his voice forward. It was as though the ancestor himself was calling to the assembled guests." This particular speaking through post dates from 1860, predating the Wizard of Oz by about 40 years.

In HCI, we talk about "Wizard of Oz experiments" in which participants are intended to believe that they are interacting with a computer system when in fact they are interacting with a human being who is hiding behind that system. It matters that people think that they are interacting with a computer rather than another human being. The analogy with the Wizard of Oz is quite obvious. But is looks like the native people in that region beat L. Frank Baum to the idea, and we should really be calling them "Yakantakw experiments". Just as soon as soon as we Western people learn to pronounce that word.

The ethics of ethics

Is ethics about doing good or about avoiding doing harm?

Last week, I was chatting with a Danish colleague, who is running user studies in several European countries as part of the MONARCA project. They are developing and testing novel technologies for the detection and management of bipolar disorder. Apparently, it took 15 months to obtain ethical clearance to conduct studies in one country. 15 months!

Processes were faster in other countries, but still measured in months rather than weeks. In the UK, our experience is that the time taken to get approval to conduct user studies, even of established technologies, is highly variable, and unpredictable. But always measured in months rather than days or weeks. So it is impossible to plan a research project in detail before ethical clearance and R&D approval have been obtained. But this is a high-effort process, so you don't want to embark on it until you're sure the study will be going ahead.

The challenge of getting ethical clearance can be a real disincentive to proposing research projects on healthcare technologies. Why embark on projects in healthcare when you can do equally interesting projects in other areas that don't put such barriers in the way?

There have been some very welcome improvements over the past couple of years, with more streamlined processes for audit studies and proportionate review. But the focus is still on avoiding harm regardless of potential benefits. “VIP” is a useful mnemonic for the main concerns:

• Vulnerability: particular care needs to be taken when recruiting participants from groups that might be regarded as vulnerable, such as children, the elderly, or people with a particular condition (illness, addiction, etc.)
• Informed consent: participants should be informed of the purpose of the study, and of their right to withdraw at any time.
• Privacy and confidentiality should be respected.

However, our work with clinicians and patients has really brought home to me that ethics goes beyond VIP. It should be about doing good, not just avoiding doing harm. But "doing good" might be in the long term: understanding current design and user experiences to guide the design of future technologies. And that doesn't directly address the need to engage positively with research participants. What motivates an individual clinician, patient or carer to engage with research on the design and use of medical technologies? There should be some positive benefit to participants.

For some, it will be about the "common good": about being prepared to invest time and expertise for long-term benefits. For others, there's an indirect pay-back in terms of having expertise and experience recognised and valued, or of being listened to, or having a chance to reflect on their condition, or their use of technology. There are probably many other complex motivations for participating in research. As researchers, we need to better understand those motivations, and respect them and work with them.

Why do clinician and patients engage in research on healthcare technologies? Because the perceive value – whether personal or societal – outweighs the perceived costs. Why do researchers engage in research on healthcare technologies? Ditto. The costs to all parties need to be proportionate to the benefits. So the ethics processes need to be proportionate, to encourage essential research. And as researchers we need to be mindful of the benefits, as well as the costs, to participants in research.

The right tool for the job? Qualitative methods in HCI

It's sad to admit it, but my holiday reading has included Carla Willig's (2008) text on qualitative research methods in psychology and Jonathan Smith's (2007) edited collection on the same topic. I particularly enjoyed the chapters by Smith on Interpretive Phenomenological Analysis and by Kathy Charmaz on Grounded Theory in the edited collection. One striking feature of both books is that they have a narrative structure of "here's a method; here are its foundations; this is what it's good for; this is how to apply it". In other words, both seem to take the view that one becomes an expert in using a particular method, then builds a career by defining problems that are amenable to that method.

One of the features of Human–Computer Interaction (HCI) as a discipline is that it is not (with a few notable exceptions) fixated on what methods to apply. It is much more concerned with choosing the right tools for the job at hand, namely some aspect of the design or evaluation of interactive systems that enhance the user experience, productivity, safety or similar. So does it matter whether the method applied is "clean" Grounded Theory (in any of its variants) or "clean" IPA? I would argue not. The problem, though, is that we need better ways of planning qualitative studies in HCI, and then of describing how data was really gathered and what analysis was performed, so that we can better assess the quality, validity and scope of the reported findings.

There's a trade-off to be made between doing studies that can be done well because the method is clear and well-understood and doing studies that are important (e.g. making systems safer) but for which the method is unavoidably messy and improvisational. An important challenge for HCI (which has always adopted and adapted methods from other disciplines that have stronger methodological foundations) is to develop a better set of methods that address the important research challenges of interaction design. These aren't limited to qualitative research methods, but that is certainly one area where it's important to have a better repertoire of techniques that can be applied intelligently and accountably to address exciting problems.

The pushmepullyou of conceptual design

I've just been reading Jeff Johnson's and Austin Henderson's new book on 'conceptual models'. They say (p.18) that "A conceptual model describes how designers want users to think about the application." At first this worried me: surely the designers should be starting by understanding how users think about their activity and how the application can best support users?

Reading on, it's obvious that putting the user at the centre is important, and they include some compelling examples of this. But the question of how to develop a good conceptual model that is grounded in users' expectations and experiences is not the focus of the text: the focus is on how to go from that to an implementation. This is a very complementary approach to ours on CASSM, where we've been concerned with how to elicit and describe users' conceptual models, and then how to support them through design.

It seems to be impossible to simultaneously put both the user(s) and the technology at the centre of the discourse. In focusing on the users, CASSM is guilty of downplaying the challenges of implementation. Conversely, in focusing on implementation, Johnson and Henderson de-emphasise the challenges of eliciting users' conceptual models. These can seem, like the pushmepullyou from Dr Doolittle, to be pulling in opposite directions. But this text is a welcome reminder that conceptual models still matter in design.