When conducting Human Factors (HF) studies in the medical device industry, choosing the appropriate sample size is a complex decision with significant implications. Contrary to the common assumption that larger sample sizes always yield better results, the key lies in balancing regulatory requirements, study objectives, and resource efficiency. This article explores the dynamics of sample sizes in HF evaluations, emphasizing their importance and providing insights into optimizing study outcomes while aligning with industry standards.

Common Sample Sizes in Human Factors Evaluations

Typically, formative evaluations—conducted iteratively to refine design—involve small sample sizes of 5 to 8 participants. Summative evaluations, designed to validate the final design, generally require at least 15 participants per user group as recommended by regulatory authorities like the FDA. These numbers reflect a balance between capturing sufficient data to identify use-related risks and maintaining manageable study logistics. For more details about the differences between formative and summative evaluations, check our article Differences Between Formative and Summative Evaluations.

For example, when evaluating an auto-injector, a formative study with eight participants might reveal critical usability issues, such as confusion in cap removal steps. Conversely, a summative study with 15 participants per user group—patients, caregivers, and healthcare professionals—would provide robust data to confirm that identified risks have been mitigated. To learn more about how to describe the operating tasks for auto-injectors, refer to our case study “Creating a Task Analysis for an Auto Injector”.

When Are Smaller Sample Sizes Appropriate?

Smaller sample sizes are not inherently inferior and can be strategically advantageous in certain scenarios:

1. Early Design Stages: In formative evaluations, smaller sample sizes enable quick, iterative feedback cycles, allowing designers to identify and address usability challenges efficiently. For instance, a formative study for a surgical device might prioritize feedback from a handful of surgeons to refine key design elements.

2. Focus on Specific User Groups: Studies targeting niche populations, such as pediatric users or individuals with disabilities, may involve fewer participants due to limited availability. Here, smaller sizes can still yield meaningful insights if the study design ensures representation and diversity. Explore more about designing for unique populations in our post Including Kids and Adolescents in Human Factors Evaluations.

3. Risk-Based Decisions: For low-risk devices, regulatory bodies might accept smaller sample sizes if justified by the study’s scope and risk analysis. A thorough Use-Related Risk Assessment (URRA) can support this approach, focusing on critical tasks rather than exhaustive testing. Learn how to create an effective use related risk assessments in our review of the FDA URRA guidance. 

Balancing Diversity and Sample Size

The ultimate goal of diversity within the participant pool is to identify and represent the most vulnerable populations within the intended user groups—those with the greatest cognitive, motor, or sensory challenges—to ensure that tasks critical to device operation are rigorously tested. For example, when developing an inhaler, focusing on individuals with reduced dexterity will be appropriate to investigate if your activation force is appropriate.

Regulatory Requirements for Sample Sizes

Regulatory agencies provide specific guidance on sample sizes to ensure study rigor:

• FDA Guidance: Recommends a minimum of 15 participants per user group for summative evaluations to identify use-related risks effectively. For detailed insights, consult our article Effective Summative Evaluations in Pharmaceutical Industry.

• IEC 62366 Standard: Emphasizes the need for representative participant groups and sufficient sample sizes to validate usability under realistic conditions.

• The NMPA (National Medical Products Administration) recent guidance on human factors evaluations recommends including a minimum of 20 participants per user group to ensure comprehensive and reliable usability testing outcomes.

It is general knowledge in the industry that sample size decisions should align with the device’s risk profile and use scenarios. But it is always better to ensure your final decisions align with the FDA recommendation in their guidance "Applying Human Factors and Usability Engineering to Medical Devices”.

While these guidelines set minimum thresholds, the actual sample size should be informed by factors such as device complexity, user group variability, and risk profile.

Is Small Always Enough?

Critics of small sample sizes often argue that they might overlook rare use errors or fail to capture the full range of user interactions. In my opinion, when the formative work is properly conducted, the iterative nature of HF studies significantly mitigates this risk. Formative evaluations address design flaws early, and summative evaluations validate that these refinements meet safety and usability standards. Additionally, complementary methods like task analysis, expert reviews, or cognitive walkthroughs enrich the findings from smaller samples. As the article "Calculating Sample Sizes for Human Factors Studies: What's the Magic Number?" published by Med Device Online highlights, determining the right sample size involves balancing statistical reliability with resource constraints.

Practical Considerations for HF Specialists

To optimize sample sizes effectively:

1. Conduct a Risk Assessment: Use the URRA to identify critical tasks and prioritize them in your evaluations. Also leverage the Perception, Cognition and Action (PCA) assessment, from your task analysis, to identify which user groups may struggle most with those tasks. I recommend checking our detailed "Review of URRA Draft Guidance from the FDA" for additional insights and practical guidance on this topic.

2. Leverage Formative Knowledge: Insights from formative evaluations can guide recruitment strategies in summative study designs, focusing resources on the most vulnerable populations identified during evaluations.

3. Plan for Iteration: Adopt a phased approach, starting with smaller samples and scaling up if additional risks emerge.

Conclusion

In HF evaluations for medical devices, the "less is more" approach often holds true, provided the study design is robust and strategically aligned with objectives. By focusing on diversity, regulatory compliance, and iterative refinement, HF specialists can achieve reliable results without unnecessary resource expenditure. Small sample sizes, when used judiciously, not only streamline the development process but also ensure that medical devices are safe, effective, and user-friendly.