There has been a lot of speculation, and also pessimism to some extent, regarding the implementation of Human Factors Engineering (HFE) into the design of medical devices and combination products in the current COVID-19 pandemic. This is not surprising given the challenges the situation has posed—travel restrictions, social distancing, government and local regulations and guidelines, and above all, the risk of contracting and transmitting the disease for which there is currently no definitive treatment or vaccine.

The situation, however, is not all doom and gloom. The majority of HF activities for a medical product development can still be carried out as normal, and HF studies can be executed with a bit of careful planning and due consideration. This article discusses the ins and outs of the HFE process for medical products during the COVID-19 situation in relation to the design process, and it provides step-by-step guidance on how to implement a successful HFE program during the pandemic. 

The topic discussed in the article is based on prevalent public information regarding the pandemic, the author’s extensive experience in the field of HFE, and some of the key presentations and discussions held at the recent virtual International Symposium on Human Factors and Ergonomics in Health Care organized by the Human Factors and Ergonomics Society (HFES) in May 2020. This symposium was attended by hundreds of HFE professionals, medical product manufacturers, and regulators such as the U.S. FDA and Health Canada.

HFE is Much More than Just Usability Testing

One thing that springs to everyone’s mind in the industry is the uncertainty about conducting an HF study (aka, a usability study or user testing) in the current climate. These studies involve real or representative users and are often moderated by an HF expert in the same room. Given the challenges posed by the COVID-19 pandemic, and our political, scientific, and social response to the problem, having a face-to-face usability study, such as a formative or validation study, is definitely a challenge that needs careful consideration. However, HFE is often misconstrued, even by experienced people in the industry, as being merely a usability study or testing. Although usability testing is an important aspect of the HFE process, it is only one part of the bigger picture. HFE is much broader than just conducting usability testing. In the current COVID-19 situation it is therefore crucial to understand that there are many other important HF activities that do not require usability testing. These HF activities can be accomplished in the current situation without much difficulty. On the other hand, although is it currently challenging to conduct face-to-face usability studies, there are ways to mitigate challenges and conduct studies. This is discussed later in the article.  

Let us first understand the holistic view of the HFE process. It is an integral part of the medical product development process and includes both analytical activities that do not involve usability testing with users, and empirical activities such as user studies. Refer to Figure 1.

Figure 1: Human Factors Engineering (HFE) as part of the overall medical product development

Figure 1 depicts a simplified schematic of the design process. The bottom bar (in orange) shows the key design activities, while the middle bar (in blue) presents the integrated HF activities within the process. The top bar (in grey) shows some of the widely used HF methods that facilitate the HF activities. It also shows the feasibility of these methods in the current COVID-19 situation. The green dots represent methods that do not involve users, and hence are not affected by the situation. The amber dots represent methods that do involve users but the analysis or testing is done remotely via telephone or video links. These methods have some challenges, such as recruitment of representative users, shipment of study materials to participants, safety of participants given the absence of the moderator in the vicinity of participants, and challenges in remote observation. These methods can still be carried out, although the outcomes may not be as rich and insightful as conducting face-to-face usability testing. Finally, the red dots represent face-to-face usability testing, e.g., formative and validation testing, usually in a simulated use environment, which is the most challenging part of the HFE process in the current COVID-19 pandemic.

As evident from Figure 1, most of the HF activities (represented by the green dots) can still be carried out despite the COVID-19 situation. Of course, gaining insights from the real users—either in the form of their needs, wants, preferences, or abilities to understand the use process—right from the onset of the design process is key to the success of a medical product. However, there are alternative but tested methods of gaining such information to guide the design process. For example, during the early stage of User Insight Research, desk-based online research into the users, their characteristics (e.g., backgrounds, experience, physical abilities, and impairments), the use environment characteristics pertaining to product use, and the likely workflow, can be researched from the plethora of information available online. This information can be augmented, if required, by interviewing real users over the phone or video link. A face-to-face contextual inquiry¹ or ethnography study² may be beneficial but not necessary. 

Similarly, Preliminary Analyses (e.g., task analysis¹, investigation into known use problems³, and use-related risk assessment⁴) that aim to identify potential use related issues affecting product safety and effectiveness can be performed comprehensively without involving any real users in the process. In fact, regulatory bodies such as FDA and European Medicines Agency (EMA) require manufacturers to conduct these analyses before product design in order to address these use-related issues through design. For more information on these methods, refer to an online article.⁵

Early design concepts should be evaluated to give the right direction to the design process. During Concept Evaluation, methods such as expert review¹ and heuristic analysis¹—evaluation and ranking against pre-defined criteria—can be powerful tools to select the most feasible and usable concept or user interface features of the concepts. Remote testing of the concepts with real users is also a viable option at this stage as it is relatively easy to send images, storyboards, and models of the concepts to the participants via email or post compared to sending a physical working prototype. Although a face-to-face evaluation may be ideal, it is not wholly necessary at this stage given the COVID-19 situation.

During the early stages of the Formative Evaluations¹ of the chosen concept, some of the above methods, such as expert review and heuristic analysis, can be used. It becomes more challenging as the design matures and there is a need for testing the user interaction, particularly physical interaction, with product prototypes. Remote testing can still be used as an alternative to a face-to-face usability study. When the design is nearing its final stage, it may be beneficial, but not mandatory, to conduct a face-to-face formative evaluation. The current COVID-19 situation does make face-to-face usability testing challenging but it does not completely preclude it. This is discussed later in the article.

The final activity in the HFE process before a regulatory submission is validation testing¹, aka summative testing⁶. A validation study may be possible to conduct remotely for some products, such as a Mobile Medical Application (MMA). However, remote validation testing may not be possible for physical devices, especially complex and large devices with a number of critical tasks to assess. Moreover, remote testing may not be acceptable to regulators, especially if they are not convinced that test scenarios represent real-life scenarios. This is discussed in detail later in the article.

Remote Testing and Regulators’ Perspectives

The increased risks and challenges in conducting face-to-face usability studies caused by the COVID-19 pandemic have forced medical product manufacturers and HF experts to devise new ways of gathering usability/HF data from real users. As a result, remote testing has become a hot topic of discussion. This is evident by the fact that this topic was discussed by many presenters, including some from FDA, at the virtual International Symposium on Human Factors and Ergonomics in Health Care in May 2020.

So, what is remote testing? It is a way of conducting an HF study or testing where the study participants are located at one end of a web conferencing channel, such as Zoom Meeting and Adobe Connect, usually at home, and the moderator and observer are located at the other end of the channel. This method physically separates the study participants and the researchers. While this separation is the key driver for the remote studies in the COVID-19 situation, it poses many challenges.

Remote testing is not new to the HF and market research communities. It has been used as a tool to test MMAs and web-based applications for a long time. However, the use of remote testing for physical medical products, especially for regulatory submissions, has been limited, until now. There are many challenges in using the remote testing method. Examples include:

• Would representative users of the product being tested have enough knowledge of and confidence in using a web conferencing platform?
• How would you find and recruit representative users?
• Would it be viable to ship multiple cameras and device prototypes to every participant?
• Would every participant have reliable internet/wi-fi connection at home?
• Would they have the physical and cognitive capabilities to set up the web conferencing and cameras themselves?
• Could a moderator run through a discussion guide (aka interviewer’s script) over the web conference without compromising the natural flow of the use process and without being too intrusive and leading?
• Could participants take the ‘non-leading’ instructions from the moderator and simulate the use process in a realistic manner?
• Could the observer capture all the nuances of the use process and more importantly, the subtle use errors and difficulties?
• What would they do with the device prototypes after the study?

On the top of the logistic and procedural challenges, there are pertinent risks of data security, confidentiality, and participant safety (e.g. what if they were to commit a use error that poses them a safety risk?). These are some of the many crucial questions that one has to answer before using remote testing for medical products. This needs careful consideration.

Moreover, would data gathered through remote testing be acceptable to regulators, such as FDA? This question was posed to FDA’s personnel at the above symposium. The answer is not as straightforward as one would like it to be. The conference was attended by personnel from FDA’s Center for Devices and Radiological Health (CDRH) and Center for Drug Evaluation and Research (CDER). One thing that has not ‘yet’ changed because of the current COVID-19 situation is the HFE process and FDA’s review process and requirements unless the product qualifies for an Emergency Use Authorizations (EUA).⁷ “FDA recognizes that the COVID-19 public health emergency may impact the feasibility and appropriateness of conducting in-person human factors (HF) testing of medical products due to any public health measures implemented to control the virus,” the agency shared at the virtual HFES conference. However, FDA is skeptical regarding the appropriateness and acceptability of HF data collected via remote testing, especially for the final validation testing. This is because “the Agency is currently not aware of any data that supports the use of remote HF validation testing or of any consensus scientific guidelines or standards that can inform as acceptable remote testing approach.” Therefore, FDA personnel mentioned on multiple occasions that “the Agency is unable to provide a general statement at this time about whether remote HF testing could potentially be an acceptable approach.” 

A presentation from an HF reviewer at CDRH helped synthesize the agency’s thought process behind the scepticism in regard to the HF data collected via remote testing. There are broadly two concerns about the remote testing method: representativeness of the real world and completeness and accuracy of the data. The personnel questioned the remote testing with regard to its ability to represent the real world—test participants (e.g., tech savvy) vs intended users (e.g. lay users), test environment vs intended use environment, test device user interface (e.g., modified due to remote testing) vs final device user interface, and test scenario vs real use scenario. He opined that remote testing may not be representative, and hence it may be difficult to use the method for a validation study. Similarly, it may be challenging to elicit and capture the necessary HF data—performance data, knowledge task data, root cause for use errors and difficulties, and subjective data—accurately and completely through the eyes and ears of a camera, hence compromising the data quality.

With the limiting nature of the current technologies and facilities, it is likely that remote testing may not be appropriate for validation testing for many types of medical products. However, this method can be used for formative evaluations where the intention is to learn as much as possible to improve medical product design. The appropriateness of remote testing for formative evaluations was not discussed explicitly at the symposium. One reason for this is that formative evaluations are recommended but they are not mandatory. FDA’s reviews primarily focus on the results from validation studies. Nevertheless, it is plausible to think that this method can provide some benefit to the design as opposed to conducting no user study in the current COVID-19 situation. The data may not be as rich but still helpful. However, it is more important to avoid any misleading data resulting from the shortfalls of this kind of testing.

As discussed above, in general, regulators’ requirements for validation testing remain the same as before. Also, remote testing methods may not be acceptable for many medical products that require physical interaction with their user interface. However, as the regulators are learning and monitoring the situation closely, it cannot be ruled out that remote testing, or some other form of testing that does not require a face-to-face meeting, will become acceptable in one form or other if the COVID-19 situation continues. In the meantime, FDA strongly encourages medical product manufacturers (sponsors of the products) to engage with the agency as soon as possible and to “reach out to their project manager” for guidance on the HFE process, HF data requirements, especially the validation testing, for their specific product in the COVID-19 situation. If a sponsor has no other choice but to do remote testing as part of the data gathering for their regulatory submission, they are encouraged to discuss the study design and protocol, along with samples of the products, with FDA before embarking on the study.

Face-to-face Validation Testing in COVID-19 Situation

As discussed above, face-to-face validation testing may still be the only option for HF data gathering for a regulatory submission despite the COVID-19 situation. Although there are many challenges, conducting face-to-face validation testing is not impossible. There are two options for medical product developers—do all the preparation work and wait for the situation to improve to conduct the study, or, if time is of the essence, go forward with the study with necessary safety precautions in place. Refer to Figure 2.

Figure 2: A potential approach of validation testing preparation and execution in the COVID-19 situation

Figure 2 provides a logical approach to preparing and conducting validation testing in the COVID-19 situation. The diagram should be self explanatory, but its main message is that a thorough study risk assessment accounting for the foreseeable risks associated with the COVID-19 situation in addition to other product- and study-related risks is key to determining whether a validation study can be conducted in safe and representative manner. It is important to identify potential risks and appropriate control measures (mitigations) and implement the measures, while still keeping the design of the study representative of real-life scenarios. If a study cannot be designed to be safe, it would not be acceptable to the Institutional Review Board (IRB) or a local ethics committee, and it would be regarded as unethical and ignorant to conduct it. On the other hand, if the study design is safe but not representative of the real world, it would not be acceptable to the regulators, and the findings are likely to be unrealistic and misleading. Therefore, balancing the safety of participants, researchers, and everyone else involved in the conduct of the study with the study design’s representation of the real world is key to the success of a face-to-face validation study in the current COVID-19 pandemic.

If the current situation does not allow for a study to be conducted safely, or the safety measures compromise the representation of the real world, it may be wise to wait until the COVID-19 situation improves. However, medical product manufacturers can do some preparation work in the meantime, which would give them a head start when the situation improves.

If the timing is critical, and the study design is safe and representative, it may still be beneficial to discuss the study design with the regulators such as FDA. This will provide an assurance that the HF data gathered through the study will be acceptable to the regulators.

The government’s and local rules and guidelines on travel restriction, social distancing, and any other public health measures implemented to control the virus must also be taken into account.

Conclusions and Take Away Messages

The COVID-19 pandemic has posed some challenges to implementing Human Factors Engineering (HFE) into medical product development. However, much of the HF work, such as the front-end research, preliminary analyses, and use-related risk assessment, can still be carried out without much difficulty. This kind of HF work is not only required for a regulatory submission but also essential in guiding the design process and avoiding a vicious cycle of design iteration and evaluation via usability testing later.

The current COVID-19 situation has made face-to-face testing, such as formative and validation studies, a real challenge. Remote testing is being experimented by many in the industry, especially for formative evaluations, as an alternative to a face-to-face study. Remote testing may be appropriate for some products, especially at the front end of the design process and for some types of formative evaluations. However, regulators are sceptical about its use for validation testing. The key concerns are the method’s limitations in ensuring the representativeness of the real world and the completeness and accuracy of the data. If these concerns can be addressed, this method has some merits in the current COVID-19 situation, especially to augment the data generated via preliminary analyses.

In many circumstances, face-to-face validation testing may be the only option for the medical product manufacturer. The key to planning and executing a successful face-to-face validation study is to conduct a thorough study risk assessment, identify potential risks, develop appropriate control measures, and implement the measures, whilst still ensuring study design represents the real world. The challenge is in striking the balance between ensuring participant safety and designing a study that represents real-life scenarios. The data collection methods must also ensure accuracy and completeness of the HF data. 

In summary, despite the current COVID-19 situation, with some careful planning and considerations, Human Factors Engineering can be successfully implemented to medical product development.

From：MDDI