The Need for Unbiased Scripts in Pharma Human Factors Evaluations

Leading language can be a significant issue when it comes to human factors evaluations. Specifically, the need for unbiased language in the scripts used during summative evaluations is a major concern. This is because non well curated scripts can easily prone the participants in a study to perform actions that are not aligned with what they would do in real life, making the research faulty. Sadly, this is an issue frequently occurring in our field, especially when the researcher is unaware of their own bias and is not properly trained in the concepts of biasing and leading language. 

For those not familiar with the concept of script or moderator guide, I want to provide a brief introduction. The goal of the script is to introduce the participant to the different tasks in the human factor evaluation. In my opinion, the script is a really powerful tool, it can easily affect the results of the evaluation and can be used in different ways to serve your purposes. As a general note, when you try to conduct a summative, the script must be as neutral as possible, since the goal is to simulate real life in the simulated environment. No matter how much you try to avoid it but the script is always a source of artifacts, it is important to be conscious about it effects in the study and try to minimize it and control it as much as possible. 

In order to achieve the expected neutrality of your script, a key part is being aware of the potential leading language that we all normally use and try to control it so it is not transfer to the Human Factors evaluation. It is important to accept that this will occur, and that it is not bad faith from any of the researchers or members involved in the study. I think is worth mentioning that we are more prone to introduce leading language if we are involved in the design and development process, and we frequently fail to identify those biases when we are heavily involved in the research process.

Best practices reviewing scripts

Best practices to me are to develop a framework with the most frequent biases or leading language and sanity check the script for the Human Factors evaluation once it is created, I will try to provide an example of it in a future post.  Such a framework must consider the experimental context, appropriate and meaningful criteria for assessing when leading language can be maintained or must be removed, and examples and lists with the most common leading language that the researcher must look for. This must include a thorough review of evaluation methods, assessment of the assumptions, data collection techniques (including root cause analysis), and other factors that contribute to bias. The design of unbiased scripts must also be such that they are repeatable and can effectively account for other factors that can influence results, such as the environment, motivation, and equipment used.

At the same time, these scripts must ensure that the experiment is conducted in an ethical and safely manner. This is especially important in drug related experiments where safety is a major concern. The protocol must fit the complexity of the experiment, focus on objectively assessing the safe and effective use of the drug delivery device, and ensure that all data is recorded precisely.

The instructions provided in scripts must be written in such a way that they influence as low as possible the use of the medical devices, trying to mimic the use that will occur in real life setting, when the moderator is not present in the room and the user has to interact with the device independently.

Unbiased scripts are a crucial factor in ensuring scientific rigor in pharmaceutical drug delivery human factors evaluations. Human Factors experts play a decisive role in providing the structure and frameworks needed to assure that data collected is both accurate and impartial.

Understanding why Rigor matters in Human Factors

In pharmaceutical research, the significance of scientific rigor, especially in human factors evaluation, cannot be overstated. Often, the critical nature of thorough human factors studies is underestimated, which I believe is a significant oversight. Consider the extensive efforts invested by various professionals in a pharmaceutical company – clinicians, chemists, medical doctors, engineers, and others. They dedicate countless hours and years to developing drugs that can improve patient outcomes and devices designed for precise treatment delivery. The success of these innovations largely hinges on the end-user's ability to effectively utilize the delivery device. Here, human factors play a pivotal role in ensuring user competency. Thus, underestimating the value of these evaluations not only undermines the collective effort of the development team but also potentially compromises the efficacy of the pharmaceutical product.

Additionally, the rules and guidelines set by authorities really emphasize how important it is to do careful and detailed studies on human factors. A few years ago, the Center for Devices and Radiological Health (CDRH) group from the FDA highlighted a startling statistic at the Human Factors and Ergonomics Society (HFES) annual symposium. They indicated that only 21 out of the 184 submission under the 510(k) in 2018 made it through without additional requests related to human factors, also only 6 of the 157 Q-Subs made it primarily due to problems in their human factors submissions. This revelation is a compelling reason for manufacturers to seriously prioritize human factors, ensuring adherence to stringent experimental standards.

Let’s now delve into the potential biases and leading language in human factors evaluations and explore strategies to mitigate them.

Controlling for Bias in Drug Delivery Devices HF Evaluations

Ensuring unbiased experimentation in drug delivery device evaluations is essential for maintaining scientific accuracy. Here are some key steps to control bias in pharmaceutical Human Factors evaluations:

Separate Designers from Evaluators

It's vital to distinguish between those who create the device or interface and those who evaluate it. Often in human factors, especially in medical device and pharmaceutical companies, there's a tendency to forget this separation, possibly because these companies may not have dedicated human factors teams from the start. Allowing designers to also conduct experiments can unintentionally introduce bias, as they are naturally influenced by their involvement in the development process. While it's beneficial for creators to observe experiments for feedback, the actual evaluation should be in the hands of professionals without a stake in the design.

Educate Researchers on Cognitive Biases

Training researchers about cognitive biases and logical fallacies is crucial. A helpful resource I often use is a book called “Logically Fallacious,” a book that illustrates common fallacies in conversations. This book, along with other articles and literature, can aid in identifying and avoiding leading language in human factors scripts, and is generally useful in understanding how word choice influences situations.

Blind the Experiment

Blinding is an effective method to prevent preconceived notions about companies from influencing results. This involves removing any logos or brand names from the products used in the study, replacing them with neutral terms. This step helps ensure that the participants' and researchers' opinions about the company do not affect their interaction with and evaluation of the product.

These strategies help in creating a more objective and reliable study environment, ensuring that the results reflect true user interaction without the influence of external biases.

Key Mistakes to Avoid in Script Development for Human Factors Evaluations

When creating scripts for human factors evaluations in the pharmaceutical field, certain common errors can affect the study's outcomes. It's important to understand these mistakes to ensure the most realistic environment is provided during the test. Let's explore these errors:

Thinking Scripts Control Outcomes

A common mistake in developing scripts for human factors evaluations is thinking that their main goal is to produce positive experiment outcomes. This idea misinterprets the true objective, which is to neutrally observe how people interact with pharmaceutical products or devices in real-life situations. The script needs to be impartial, mirroring actual use without swaying the participant's actions. When team members, particularly those involved in designing the product, contribute to script writing, they might unintentionally include suggestions that lead users in a certain direction. These kinds of prompts, which wouldn't occur in normal use, should be carefully avoided in the summative evaluations. The aim is to authentically capture how users would naturally interact with the product.

Skipping Training and Script Explanation

Not dedicating adequate time to training and explaining the script's purpose to both the research team and product developers is another common error. It's important to remember that scripts are tools for impartial investigation, not for directing the outcome of the study. This concept must be communicated effectively to everyone, particularly those who are newer to human factors research. Without proper understanding, there's a risk that team members might view the script as a controlling tool, which can skew the neutrality of the study as mentioned before, but also it may lead to discussions within the team since the goal may not be clear for everyone.

Being Too Flexible with External Vendors

Working with external teams can lead to the mistake of creating too flexible scripts, especially when the team is only used to conduct formative evaluations. These summative evaluations are critical and require scientific accuracy. Even though there needs to be some adaptability to identify issues with the device, the overall framework of the study, such as how to handle debriefs and root cause analysis, should be clearly set out in advance. This structure is vital to ensure that the same types of errors are examined in a consistent manner, allowing any differences in the findings to be attributed to the genuine reactions and views of the participants, rather than the influence of the researchers.

Leading Participants the Wrong Way

As mentioned, a key challenge in Human Factors evaluations is avoiding unintentional influence on participants, especially in final evaluations where observing natural usage is crucial. While leading language should be avoided in Human Factors, some guidance is necessary in earlier stages or when examining specific features like instructions for use (IFU). Directing attention to certain aspects in these cases helps evaluate their clarity. However, it's important to limit this guidance to ensure genuine interaction with the product, tailoring the approach to the study's goals for accurate and meaningful results.

By keeping these points in mind, we can create better scripts for our human factors evaluations. This helps us get truthful and useful information about how people will use our pharmaceutical products or devices.

Benefits of Hiring Human Factors Specialists for Pharma Experiments

The pharmaceutical industry is increasingly recognizing the value of integrating human factors specialists into their research teams. These professionals play a crucial role in maintaining scientific rigor, particularly in mitigating biases in experimental designs and ensuring neutral, reliable outcomes. Furthermore, the size of the human factors team is a critical consideration, with implications for the research process. It should be carefully determined based on the specific pipeline and complexity of the device being developed, as discussed in our detailed internal post about Team Sizes. This section of the post will explore why hiring such specialists is essential for the integrity and success of pharmaceutical human factors evaluations.

Experience Beyond Basic Research Training

While traditional research training provides a solid foundation, human factors specialists bring a deeper, more nuanced understanding of how humans interact with pharmaceutical products and medical devices. Their expertise goes beyond textbook knowledge; it encompasses a comprehensive understanding of real-world user behaviors, preferences, and limitations. This expertise is particularly vital in pharmaceutical human factors evaluations, where understanding the end-user's experience is crucial for the successful adoption and effectiveness of a product.

Proactive Problem Identification

Specialists in human factors are keen on identifying potential issues early in the research process. Their involvement can prevent problems that might otherwise emerge later, saving time and resources. Early identification of potential user-related issues can lead to more efficient experiment design, reducing the need for extensive revisions or additional studies and can drastically reduce the general development costs.

Ensuring Compliance with Regulatory Standards

Human factors specialists are familiar with the regulatory requirements related to medical device development. Their knowledge ensures that experiments not only meet scientific standards but also comply with regulatory expectations, like IEC62366 and all the FDA guidances. This aspect is critical, as non-compliance can result in significant setbacks, including the rejection of research findings by regulatory agencies.

Enhancing Overall Research Quality

The inclusion of a human factors specialist elevates the quality of drug delivery devices. Their insights contribute to more user-centered designs, ensuring that the products developed are not only safe and effective but also user-friendly and accessible. This focus on the user experience is increasingly recognized as a key factor in the successful deployment of pharmaceutical products and medical devices.