Strategies for doing fieldwork for health technology design

The cartoons in this blog post are  from Fieldwork for Healthcare: Guidance for Investigating Human Factors in Computing Systems.  © 2015 Morgan and Claypool Publishers, www.morganclaypool.com. Used with permission.

One of the themes within CHI+MED has been better understanding how interactive medical devices are used in practice, recognising that there are often important differences between work as imagined and work as done.  This has meant working with many people directly involved in healthcare (clinicians, patients, relatives) to understand their work when interacting with medical devices: observing their interactions and interviewing them about their experiences. But doing fieldwork in hospitals and in people’s homes is challenging:

• You need to get formal ethical clearance to conduct any study involving clinicians or patients. As I’ve noted previously, this can be time-consuming and frustrating. It also means that it can be difficult to change the study design once you discover that things aren’t quite the way you’d imagined, however much preparatory work you’d tried to do. 
• Hospitals are populated by people from all walks of life, old and young, from many cultures and often in very vulnerable situations. They, their privacy and their confidentiality need to be respected at all times.
• Staff are working under high pressure. Their work is part-planned, part-reactive, and the environment is complex: organisationally, physically, and professionally. The work is safety-critical, and there is a widespread culture of accountability and blame that can make people wary of being observed by outsiders.
• Health is a caring profession and, for the vast majority of staff, technology use is a means to an end; the design of that technology is not of interest (beyond being a source of frustration in their work).
• You’re always an ‘outsider’: not staff, not patient, not visitor, and that’s a role that it can be difficult to make sense of (both for yourself and for the people you’re working with).
• Given the safety-critical nature of most technologies in healthcare, you can’t just prototype and test ‘in the wild’, so it can be difficult to work out how to improve practices through design.

When CHI+MED started, we couldn’t find many useful resources to guide us in designing and conducting studies, so we found ourselves ‘learning on the job’. And through discussions with others we realised that we were not alone: that other researchers had very similar experiences to ours, and that we could learn a lot from each other.

So we pooled expertise to develop resources to give future researchers a ‘leg up’ for planning and conducting studies. And we hope that the results are useful resources for future researchers:

• We’ve recently published a journal paper that focuses on themes of gaining access; developing good relations with clinicians and patients; being outsiders in healthcare settings; and managing the cultural divide between technology human factors and clinical practice.
• We’ve published two books on doing fieldwork in healthcare. The first volume reported the experiences of researchers through 12 case studies, covering experiences in hospitals and in people’s homes, in both developed and developing countries. The second volume presents guidance and advice on doing fieldwork in healthcare. The chapters cover ethical issues, preparing for the context and networking, developing a data collection plan, implementing a technology or practice, and thinking about impact.
• Most of our work is neither pure ethnography nor pure Grounded Theory, but somewhere between the two in terms of both data gathering and analysis techniques: semi-structured, interpretivist, pragmatic. There isn’t an agreed name for this, but we’re calling them semi-structuredqualitative studies, and have written about them in these terms.

If you know of other useful resources, do please let us know!

What's the value of human factors?

I was recently in a meeting where a technologist was reporting on a situation where a team had developed a new technology to support work in a clinical setting, but when they tried to deploy it they discovered that people didn't work in the way they had expected. From a user-centred design perspective, this is Rookie Error Number 1: so basic that it is amazing that it still happens so frequently.

It was the next statement that caught me slightly off-guard: that they should have consulted the clinicians ahead of time. Yes, of course they should. But actually, that's probably not enough either. Because clinicians (like all of us) have limited understanding of, or ability to articulate, their own work. The speaker's implicit assumption was clearly that we are all fully capable of describing our work to others in the detail that they would need to be able to design to support that work. That leaves me, as a Human Factors specialist without a job!

Which of course led me to reflect on what my role in such a situation is. What is it that I, or any other HCI / Human Factors specialist brings to the situation that the clinician or the technologist doesn't? In the case of the technologist (or at least, of the technologists who feature in this story), it's a limited understanding of what we're able to articulate, or even know, about our own activities: an assumption that if you ask someone a direct question they will give you a full, frank and accurate answer. In the case of the clinician, like anyone, it's a partial understanding of their own work practices and of how to describe them to a third party.

Many moons ago, Gordon Rugg and I wrote about this in the context of eliciting people's implicit knowledge of their own work, and I'm going to paraphrase some of our own writing:

In terms of understanding what people can articulate about their work, there are three main kinds of knowledge:

• explicit knowledge (knowledge which is readily available to introspection, and that people can readily articulate);
• semi-tacit knowledge (knowledge that can be accessed by some techniques but not by others); and
• tacit knowledge (knowledge which is not accessible to introspection via any elicitation technique). 

There are various types of semi-tacit knowledge, including taken-for-granted knowledge (that is so familiar to the speaker that they will not think to mention it); front and back versions (the official version of what should happen and a more realistic account of what actually happens); and knowledge that depends on recognition rather than recall

Tacit knowledge is subdivided into compiled skills (which have become so habitualised as to be inaccessible to introspection) and implicit learning (knowledge that is tacit throughout, never having been articulated).

So there I have at least a partial answer to my existential angst: that as a Human Factors specialist, I have knowledge of how to elicit semi-tacit knowledge about people's work – and specifically about that work for the purposes of designing technology to support it. So both the technologist and the clinician have a reason to get me involved. Whew!

A mutual failure of discovery: DIB and DiCoT

Today, I have been doing literature searching for a paper on Distributed Cognition (DCog). By following a chain of references, I happened upon a paper on Determining Information Flow Breakdown (DIB). DIB is a method for applying the theory of DCog in a semi-structured way in complex settings. The example the authors use in the paper comes from healthcare.

The authors state that "distributed cognition is a theoretical approach without an accepted analytical method; there is no single 'correct way' of using it. [...] the DIB method is a practical application of the theory." At the time that work was published (2007), there were at least two other published methods for applying DCog: the Resources Model (2000) and DiCoT (Distributed Cognition for Teamwork; 2006). The developers of DIB were clearly unaware of this previous work. Conversely, it has taken me seven years from when the DIB paper was published to become aware of it and my team have been working on DCog in healthcare for most of that time. How could that happen?

I can think of several answers involving parallel universes, different literatures, too many different journals to keep track of, the fragility of search terms, needles in haystacks. You take your pick.

Whatever the answer actually is (and it's probably something to do with a needle in another universe), it's close to being anti-serendipity: a connection that is obvious and should have been expected. We clearly have some way to go in developing information discovery tools that work well.

Hidden in full view: the daft things you overlook when designing and conducting studies

Several years ago, when Anne Adams and I were studying how people engaged with health information, we came up with the notion of an "information journey", with three main stages: recognising an information need; gathering information and interpreting that information. The important point (to us) in that work was highlighting the important of interpretation: the dominant view of information seeking at that time was that if people could find information then that was job done. But we found that an important role for clinicians is in helping lay people to interpret clinical information in terms of what it means for that individual – hence our focus on interpretation.

In later studies of lawyers' information work, Simon Attfield  and I realised that there were two important elements missing from the information journey as we'd formulated it: information validation and information use. When we looked back at the health data, we didn't see a lot of evidence of validation (it might have been there, but it was largely implicit, and rolled up with interpretation) but – now sensitised to it – we found lots of evidence of information use. Doh! Of course people use the information – e.g. in subsequent health management – but we simply hadn't noticed it because people didn't talk explicitly about it as "using" the information. Extend the model.

Wind forwards to today, and I'm writing a chapter for InteractionDesign.org on semi-structured qualitative studies. Don't hold your breath on this appearing: it's taking longer than I'd expected.

I've (partly) structured it according to the PRETAR framework for planning and conducting studies:

• what's the Purpose of the study?
• what Resources are available?
• what Ethical considerations need to be taken into account?
• what Techniques for data gathering?
• how to Analyse data?
• how to Report results?

...and, having been working with that framework for several years now, I have just realised that there's an important element missing, somewhere between resources and techniques for data gathering. What's missing is the step of taking the resources (which define what is possible) and using them to shape the detailed design of the study – e.g., in terms of interventions.

I've tended to lump the details of participant recruitment in with Resources (even though it's really part of the detailed study design), and of informed consent in with Ethics. But what about interventions such as giving people specific tasks to do for a think-aloud study? Or giving people a new device to use? Or planning the details of a semi-structured interview script? Just because a resource is available, that doesn't mean it's automatically going to be used in the study, and all those decisions – which of course get made in designing a study – precede data gathering. I don't think this means a total re-write of the chapter, but a certain amount of cutting and pasting is about to happen ...

Wizard of Oz: the medium and the message

Last week, one of my colleagues asserted that it didn't matter how a message was communicated – that the medium and the message were independent. I raised a quizzical eyebrow. A few days previously, I'd been in Vancouver, and had visited the Museum of Anthropology. It's a delightful place: some amazing art and artefacts from many different cultures. Most of them relate to ceremony and celebration, rather than everyday life, but they give a flavour of people's cultures, beliefs and practices. And most of them are beautiful.

One object that caught my attention was a yakantakw, or "speaking through post". According to the accompanying description: "A carved figure such as this one, with its prominent, open mouth, was used during winter ceremonies. A person who held the privilege of speaking on behalf of the hosts would conceal himself behind the figure, projecting his voice forward. It was as though the ancestor himself was calling to the assembled guests." This particular speaking through post dates from 1860, predating the Wizard of Oz by about 40 years.

In HCI, we talk about "Wizard of Oz experiments" in which participants are intended to believe that they are interacting with a computer system when in fact they are interacting with a human being who is hiding behind that system. It matters that people think that they are interacting with a computer rather than another human being. The analogy with the Wizard of Oz is quite obvious. But is looks like the native people in that region beat L. Frank Baum to the idea, and we should really be calling them "Yakantakw experiments". Just as soon as soon as we Western people learn to pronounce that word.

When is a qualitative study a Grounded Theory study?

I recently came across Beki Grinter's blog posts on Grounded Theory. These make great reading.

The term has been used a lot in HCI as a "bumper sticker" for any and every qualitative analysis regardless of whether or not it follows any of the GT recipes closely, and whether or not it results in theory-building. I exaggerate slightly, but not much. As Beki says, GT is about developing theory, not just about doing a "bottom up" qualitative analysis, possibly without even having any particular questions or aims in mind.

Sometimes, the questions do change, as you discover that your initial questions or assumptions about what you might find are wrong. This has happened to us more than once. For example, we conducted a study of London Underground control rooms where the initial aim was to understand the commonalities and contrasts across different control rooms, and what effects these differences had on the work of controllers, and the ways they used the various artefacts in the environment. In practice, we found that the commonalities were much more interesting than the contrasts, and that there were several themes that emerged across all the contexts we studied. The most intriguing was discovering how much the controllers seemed to be playing with a big train set! This links in to the literature on "serious games", a literature that we hadn't even considered when we started the study (so we had to learn about it fast!).

In our experience, there's an interdependent cycle between qualitative data gathering and analysis and pre-existing theory. You start with questions, gather and analyse some data, realise your questions weren't quite right, so modify them (usually to be more interesting!), gather more data, analyse it much more deeply, realise that Theory X almost accounts for your data, see what insights relating your data to Theory X provides, gather yet more data, analyse it further... end up with either some radically new theory or a minor adaptation of Theory X. Or (as in our study of digital libraries deployment) end up using Theory X (in this case, Communities of Practice) to make sense of the situations you've studied.

Many would say that a "clean" GT doesn't draw explicitly on any existing theories, but builds theory from data. In practice, in my experience, you get a richer analysis if you do draw on other theory, but that's not an essential part of GT. The important thing is to be reflective and critical: to use theory to test and shine light on your data, but not to succumb to confirmation bias, where you only notice the data that fits the theory and ignore the rest. Theory is always there to be overturned!

Usable security and the total customer experience

Last week, I had a problem with my online Santander account. This isn't particularly about that company, but a reflection on a multi-channel interactive experience and the nature of evidence. When I phoned to sort out the problem, I was asked a series of security questions that were essentially "trivia" questions about the account that could only be answered accurately by being logged in at the time. I'd been expecting a different kind of security question (mother's maiden name and the like), so didn't have the required details to hand. Every question I couldn't answer made my security rating worse, and quite quickly I was being referred to the fraud department. Except that they would only ring me back within 6 hours, at their convenience, not mine. I never did receive that call because I couldn't stay in for that long. The account got blocked, so now I couldn't get the answers to the security trivia questions even though I knew that would be needed to establish my identity. Total impasse.

After a couple more chicken-and-egg phone calls, I gathered up all the evidence I could muster to prove my identity and went to a branch to resolve the problem face-to-face. I was assured all was fine, and that they had put a note on my account to confirm that I had established my credentials. But I got home and the account was still blocked. So yet another chicken-and-egg phone call, another failed trivia test. Someone would call me back about it. Again, they called when I was out. Their refusal to adapt to the customer's context and constraints was costing them time and money, just as it was costing me time and stress.

I have learned a lot from the experience; for example, enter these conversations with every possible factoid of information at your fingertips; expect to be treated like a fraudster rather than a customer... The telephone interaction with a human being is not necessarily any more flexible than the interaction with an online system; the customer still has to conform to an interaction style determined by the organisation.

Of course, the nature of evidence is different in the digital world from the physical one, where (in this particular instance) credible photo ID is still regarded as the Gold Standard, but being able to answer account trivia seems like a pretty poor way of establishing identity. As discussed last week, evidence has to answer the question (in this case: is the caller the legitimate customer?). A trivia quiz is not usable by the average customer until they have learned to think like security people. This difference in thinking styles has been recognised for many years now (see for example "Users are not the enemy"); we talk about interactive system design being "user centred", but it is helpful if organisations can be user centred too, and this doesn't have to compromise security, if done well. I wonder how long it will take large companies to learn?

Seeing is believing?

In a recent interview, Mary Beard recounted a Roman joke: "A guy meets another in the street and says: 'I thought you were dead.' The bloke says: 'Can't you see I'm alive?' The first replies: 'But the person who told me you were dead is more reliable than you.'" She used the joke (apparently considered hilarious all those centuries ago) to illustrate a point about changing cultures and the nature of evidence. But the question of evidence is just as important in our work today. When are verbal reports a reliable form of evidence, and when do you need more direct forms of evidence? What can you learn from web analytics or the device log of an infusion pump? What does observing people tell you, as against interviewing them? Etc.

In general, device logs of any kind should tell you what happened, over a large number of instances, but they can't tell you anything much about the circumstances or the causes (what people thought they were doing, or what context they were in). So they give you an idea of where problems might lie, but not really what those problems are; they give quantity, but not necessarily quality.

Conversely, interviews and observations can potentially give quality, but not quantity. They have greater explanatory power; interviews are good for finding out people's perceptions (e.g. of why they behave in certain ways), and observations will give insights into the contexts within which people do things and the circumstances surrounding actions. Interviews may overlook details that people consider unremarkable, while observations may catch those details but not explain them. And of course the questions that are asked or the way an observational study is conducted will determine what data is gathered.

As I type this, most of it seems very self-evident, and yet people often seem to choose inappropriate data gathering methods that don't reliably answer the questions posed. I'll use an example from a researcher I have great respect for, and who is undeniably a leader in the field: Ever since I first read it, I have been perplexed by Jim Reason's analysis of photocopier errors – not because it is inconsistent with other studies, but because it is based entirely on retrospective self-reports. But our memories of past events are highly selective. I make errors every day, as we all do (see errordiary for both mundane and bizarre examples), but the ones I can recall later are the ones that were most embarrassing, most costly. most amusing or otherwise memorable. So what confidence can we have in retrospective reports as a way of measuring error? I don't know. And I don't think that's an admission of failure on my part; it's a recognition that retrospective self-report is an unreliable way of gathering data about human error. And that remains a challenge: to match research questions and data gathering and analysis methods appropriately.

The pushmepullyou of conceptual design

I've just been reading Jeff Johnson's and Austin Henderson's new book on 'conceptual models'. They say (p.18) that "A conceptual model describes how designers want users to think about the application." At first this worried me: surely the designers should be starting by understanding how users think about their activity and how the application can best support users?

Reading on, it's obvious that putting the user at the centre is important, and they include some compelling examples of this. But the question of how to develop a good conceptual model that is grounded in users' expectations and experiences is not the focus of the text: the focus is on how to go from that to an implementation. This is a very complementary approach to ours on CASSM, where we've been concerned with how to elicit and describe users' conceptual models, and then how to support them through design.

It seems to be impossible to simultaneously put both the user(s) and the technology at the centre of the discourse. In focusing on the users, CASSM is guilty of downplaying the challenges of implementation. Conversely, in focusing on implementation, Johnson and Henderson de-emphasise the challenges of eliciting users' conceptual models. These can seem, like the pushmepullyou from Dr Doolittle, to be pulling in opposite directions. But this text is a welcome reminder that conceptual models still matter in design.