Insight into RQM+ Consultation: How the Most Appropriate Study Design is Determined

A clinical study design describes the methodology with which to pursue the aims and objectives of a clinically important research question in a target population.

When reaching out to us at RQM+, we bring on board a range of internal stakeholders to deliberate with you the descriptors of your target populations, endpoints, study duration and designs. These are all budget relevant variables. Regulatory and ethical frameworks are considered alongside the intended analyses of study data. Being a full-service clinical research organization (CRO), we aptly assist in delineating objectively the suitability of an experimental or observational strategy to address a given clinical need in any medical discipline, including dentistry.

The regulatory framework is provided by the Medical Devices Regulation (EU) 2017/745. According to MDR Article 2 (45), the motivation to conduct a clinical investigation is to assess the safety or performance of a device in human subjects. Data integrity and subject safety are operationalized in ISO 14155 (the standard for Clinical investigation of medical devices for human subjects — Good clinical practice) and - importantly - are within the principal investigator´s remit. The ethical caveat for clinical research is to safeguard patient safety and rights and minimize unintentional harm or burden whilst pursuing to the highest scientific standards a topical question of medical relevance (Declaration of Helsinki).

A robust study design as well as the validity of the study data hinge on the manufacturer´s statement of intended use. Clinical performance of the device entails use of the device as intended by the manufacturer and should be clearly specified in an investigation. Safety of using the medical device must be monitored. Together, the assessments of clinical performance and safety will capture the clinical benefit of a medical device, i.e., according to MDR Article 2 (53), “the positive impact of a device on the health of an individual (…) or a positive impact on patient management or public health”. The outcomes of the investigation should be meaningful, measurable, patient-relevant and related to the diagnosis, according to MDR.

The intention of this article is to present an insight into the reasoning behind proposing certain designs of study, drawing from relationships with clients that have led to peer reviewed publication of their research activity (e.g., Park et al., 2022; Amin et al., 2024) and wider consulting activity. Two common discussion scenarios are chosen to showcase RQM+´s input in your strategic decisions on the best choice of a study design for your intended investigation.

The designating term `participant´ for the individual contributing to clinical research through their voluntary participation shall be used were applicable, in replacement of the term 'subject' (Corpuz, 2023).

Glossary of technical terms

For a controlled investigation – Randomized controlled trial (RCT) or Cross-over design?

Interventional studies, i.e. studies in which exposures / treatments are assigned, have the strongest evidence of cause-effect relationships. In fact, they are the gold standard to test the efficacy of the treatment and are especially impactful for public health policy.

To unequivocally demonstrate a treatment effect, a comparison to a control is required. A control group is a group which does not receive intervention with the investigational device. In statistical terms, the effect is termed 'Outcome variable', the cause is the so-called 'Treatment variable'. A covariate is a 'Confounding variable', a variable that influences both the selection of treatment and the outcome variables. We help to define and advise on these, as they are relevant for the level of statistical analyses the investigation will require.

To eliminate selection bias and reduce confounding variables, a process termed randomization is used. In this process, participants who satisfy inclusion and exclusion criteria are randomly allocated to the intervention or control group. In this manner, hypotheses can be tested.

To describe interventionist trials (they can be either RCT or follow a cross-over design), we use the PICO scheme. The scheme, essentially a brief table, crystallizes for quick reference and comparison to benchmarks, which we assist in, study population, type of intervention, control and outcomes. The scheme can be expanded by the investigation’s setting and the timeframe (PICOST). Study outcomes need to be seen in the context of clinical relevance, participant adherence measures and patients´ own reported outcomes whilst considering likely confounders.

To define 'control' in the medical device field, we need to critically revisit the concept that placebos are inert:

A Cochrane criterion-based, seminal analysis of contextual effects in RCT finds that over half of the overall treatment effect could be due to the placebo response i.e., the contextual response to a placebo effect (Hafliðadóttir et al., 2021). This observation may explain the so-called efficacy paradox, i.e. the lack of reproducibility of RCT-described efficacy in clinical practice, where contextual effects contribute to the overall treatment effects (Zhang and Doherty, 2018). However, patient-provider interactions and patient expectations of treatment are contextual factors whose relative impact on study results may equalize when the study is multicentric.

The implications are that the treatment of the control arm needs to be accurately defined and evaluated against routine practice in order to more accurately gauge a treatment effect. This is because the beneficial effect of placebo interventions is measurable and varied (Hróbjartsson and Gøtzsche, 2010). A more recent school of thought tries to engage with the placebo effects by viewing them as a threshold which the treatment needs to surpass to be deemed efficacious. This is because “therapeutic alliance and the beliefs and expectations of patients” engender placebo responses and should therefore be harnessed (Friesen, 2019). RQM+ is equipped to discuss these aspects in relation to the number of sites you have in mind for your investigation.

In a randomized control trial, participants are randomly allocated to one of more study arms where one arm during the same study period does not receive the intervention of interest.

In a cross-over design, randomized participants furnish their own control, and all will receive the intervention by the end of the study. Figure 1 illustrates the difference.

A	B

Figure 1. Schematic representation of RCT (parallel design) (A) and randomized trial in crossover design (B). Since each patient is his or her own baseline in a cross-over design, fewer patients may be required compared to the RCT design, but the study duration is longer.

Because treatment and periodic effects are separated in a cross-over design, hangover effects are meant to be eliminated by a wash out phase, i.e. an interim phase between one treatment and another, when the randomized group is switched between arms. To discern a treatment effect, a special statistical method needs to be used (analysis by sequency group; Wellek and Blettner, 2012).

It is important in clinical trials to document the disposition of participants, i.e. the extent to which participants have reached the intended trial milestones. For RCT and cross-over design alike, participant attrition requires adequate statistical planning (Bell et al., 2013). RQM+'s statistical team is poised to explain. Clinical monitoring of inclusion, disposition, endpoint relevant data points and safety should be well defined in support of the integrity of conclusions reached (Hsieh et al., 2023). RQM+'s clinical monitoring team is well trained to deliver on these points.

Because of the cross-over design, participants dropping out of the investigation may signify an attrition that jeopardizes attaining the endpoints of the study. It is also debatable whether patients at the timepoint of crossing over are truly comparable to the study start, especially when the study entails a behavioral change, entertains an expectation, or exerts a training effect to where the patients become more attuned to the study requirements. In fact, a cross-over design is not the experimental method of choice where the operator of a medical device is herself or himself on a learning curve (Bernard et al., 2014) or if the disease that is investigated is rapidly progressive (Zhang et al., 2022). RQM+'s consulting team will critically evaluate with you factors that are relevant to your intended investigation.

Adaptive designs are a statistical means to prospectively plan a pre-specified study modification (Pallmann et al., 2018). They are applicable to RCT (Bernard et al., 2014) and cross-over trials (Cook and Willan, 1996) and are advisable when the numbers of patients to recruit to the study need to be reviewed based on the primary outcome measures. This can be useful in the strategic management of your intended investigation. RQM+'s statistical team will address the use of adaptive designs in client consultations.

For real world evidence – medical chart review or device registry study?

Observational studies pursue a topical research question that has arisen from within routine medical practice. They may, during data collection and analysis, generate novel hypotheses for future clinical research. Observational studies are by definition 'non-interventional'. Typically, the extent of inclusion and exclusion criteria is much curtailed compared to interventional trials. Fringe populations may find representation in observational studies while they are typically not included in randomized in trials – which categorically aim to minimize influence of confounding factors (see above). For this reason, the outcome of observational studies is more generalizable to all patients receiving care for the same diagnosis (Kennedy-Martin et al., 2015).

Observational studies collect adherence data and data relating to safety and performance that may impact on the risk benefit analysis of a medical device, providing real-world evidence. Possibly off-label use may be found and requires stringent assessment (Team-NB position paper, 2022). The strength of observational studies is to capture the extent of real-world adherent behavior as well as treatment decisions that determine the effectiveness of a medical device (Gliklich et al., 2014).

Real-world data are data relating to patient health status and/or the delivery of health care and can be collected from a variety of sources, including patients' medical files in so-called chart review studies. Another example of real-world data is data derived from device registries.  Both, device registries and chart reviews can provide clinical evidence on “real-world effectiveness and (…) start to address the complications of managing other real-world problems such as multimorbidity” (de Lusignan et al., 2015). Device registries and chart reviews typically differ in size, extent of multicentricity, duration of data collection and the extent to which additional market relevant information of interest to the manufacturer is collected.

Guidance on the conduct of a retrospective chart review has been published (Vassar and Holzmann, 2013; Sarkar and Seshadri, 2014) and informs RQM+'s consulting. Particular attention should be paid to best practice in avoiding extraction and analysis bias (Kaji et al., 2014). RQM+ can assist in providing adequate training at the sites. In the design of registries, the inclusion of long-term follow-up data is important. 
To analyze study data emanating from observational studies, specific statistical tools are required, such as stratification into clusters of patients with similar characteristics prior to analysis (Carragher et al., 2020) or so-called propensity score matching to reduce the effect of systematic confounding at baseline in nonrandomized observational studies (Austin, 2011). RQM+'s statistical team routinely conducts these.   
Consecutive enrolment of patients against wide inclusion and exclusion criteria allows long-term data collections from routine care. These studies are important in the assessment of medical device safety and may have prospective and retrospective components by design.

Why are these important? Prospective studies have the advantage of being able to include measurements of risk factors, though detection of long-term effects may be limited by the period of follow-up. Retrospective studies are ideal in the study of rare diseases (Talari and Goyal, 2020). Observational prospective studies require clinical monitoring and site oversight. Monitoring of retrospective observational studies, by contrast, concentrates on completeness and accuracy of data entries from hospital records. Missing data typically are due to loss to follow-up (prospective) or incomplete data sets (retrospective). RQM+'s statistical team routinely considers their potential impact on the study's outcome in a statistical analysis plan.

Concluding statement 

Randomized trials are more likely to discern cause and effect because the process of randomization reduces the bias through confounding variables. However, a disproportionate drop out of participants may still alter the validity of the outcome of the investigation. Therefore, ensuring participation (treatment adherence and protocol compliance) is an important factor RQM+ will consider when setting up your clinical investigation.

Observational studies work out a quantitative relationship of variables, termed statistical association. They pursue a clearly defined research question which RQM+ can help to refine and support in its novelty through a systematic literature search. An expert statistical team is required for the analyses.  
Key elements in the discussions of clinical study design have proven to be client´s and RQM+´s views of

• representativeness of the study population and generalizability of outcomes,
• operationalizing manufacturer’s claims on the medical device (definition of endpoints),
• a priori decisions on subgroup analyses in support of the claims,
• type and frequency of clinical monitoring activities (dependent on the study design, see above).

To realize scientific integrity in medical device research, RQM+ refers to published guidelines for the design of medical device studies across device lifecycles and regulatory need (Fleetcroft et al., 2021). RQM+'s consulting team also speaks to justification of protocols through scientific peer review: STaRT-RWE (Wang et al., 2021) and CONSORT (Schulz et al., 2010) are guidelines for the conduct of real-world evidence studies and parallel group randomized trials, respectively. Publication of protocols allows for greater transparency in the clinical research community which, for the same reason, is called to publish their studies in searchable clinical trial databases such as clinicaltrials.gov or national study registries. The process of publishing a study protocol ensures a quality standard with which clinical research should be conducted and which RQM+'s medical writers aptly support.

To summarize, the balancing of the most appropriate study design (randomized controlled vs observational) in the medical device field is a highly topical issue (Páez et al., 2022) that RQM+ will help to view from within our common ethical framework to yield efficient and cost-effective solutions. Importantly, both designs have their value in generating clinical evidence (Monti et al., 2018) and can be efficiently tailored to the stage of development and market presence of the medical device. At RQM+, the clinical trial business unit offers the entire wealth of experience and expertise to anticipate risks and realize potential of your intended investigation.

Partner with us to ensure your clinical investigations are designed with precision, executed flawlessly, and deliver the data you need to succeed.

Incase you missed it, take a look at our latest related technical brief, Determining the Primary Endpoint in Clinical Research.

Join us on the 12th of December, 2024 for RQM+ Live! Medical Device Cybersecurity: Proven Strategies for Connected Devices and SaMD and be sure to bring your questions for our expert panelists!

References

• Amin M, Abdrakhmanov A, Kropotkin E, Traykov V, Salló Z, Gellér L, Lorgat F, Sapelnikov O, Toman O, Al-Muti K, Aljaabari M, Bystriansky A, Környei L, Mujović N, Simons S, Szegedi N. Ablation of Supraventricular Arrhythmias With as Low as Reasonably Achievable X-Ray exposure (AALARA): Results of Prospective, Observational, Multicenter, Multinational, Open-Label Registry Study on Real World Data Using Routine Ensite 3D Mapping During SVT Ablation. Pacing Clin Electrophysiol. 2024 Sep 27. doi: 10.1111/pace.15075.
• Austin PC. An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behav Res. 2011; 46(3):399-424. doi: 10.1080/00273171.2011.568786.
• Bell ML, Kenward MG, Fairclough DL, Horton NJ. Differential dropout and bias in randomised controlled trials: when it matters and when it may not. BMJ. 2013; 346:e8668. doi: 10.1136/bmj.e8668.
• Bernard A, Vaneau M, Fournel I, Galmiche H, Nony P, Dubernard JM. Methodological choices for the clinical development of medical devices. Med Devices (Auckl). 2014; 7:325-34. doi: 10.2147/MDER.S63869.
• Carragher R, Mueller T, Bennie M, Robertson C. A Bayesian hierarchical approach for multiple outcomes in routinely collected healthcare data. Stat Med. 2020; 39(20):2639-2654. doi: 10.1002/sim.8563.
• Cook RJ, Willan AR. Design considerations in crossover trials with a single interim analysis and serial patient entry. Biometrics. 1996; 52(2):732-9.
• Corpuz JC. Reconsidering the Terminology: Study Participants as "Subjects" or Not? Subst Abuse. 2023; 17:11782218231217783. doi: 10.1177/11782218231217783.
• Declaration of Helsinki. Ethical Principles for Medical Research Involving Human Subjects. Accessed 11.10.2024, accessible at  https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/
• de Lusignan S, Crawford L, Munro N. Creating and using real-world evidence to answer questions about clinical effectiveness. J Innov Health Inform. 2015; 22(3):368-73. doi: 10.14236/jhi.v22i3.177.
• Fleetcroft C, McCulloch P, Campbell B. IDEAL as a guide to designing clinical device studies consistent with the new European Medical Device Regulation. BMJ Surg Interv Health Technol. 2021;3(1):e000066. doi: 10.1136/bmjsit-2020-000066.
• Friesen P.   Mesmer, the placebo effect, and the efficacy paradox: lessons for evidence based medicine and complementary and alternative medicine. Critical Public Health 2019, VOL. 29, NO. 4, 435–447 doi:10.1080/09581596.2019.1597967
• Gliklich RE, Dreyer NA, Leavy MB, editors. Registries for Evaluating Patient Outcomes: A User's Guide [Internet]. 3rd edition. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr. 3, Registry Design. Available from: https://www.ncbi.nlm.nih.gov/books/NBK208632/
• Hafliðadóttir SH, Juhl CB, Nielsen SM, Henriksen M, Harris IA, Bliddal H, Christensen R. Placebo response and effect in randomized clinical trials: meta-research with focus on contextual effects. Trials. 2021; 22(1):493. doi: 10.1186/s13063-021-05454-8.
• Hróbjartsson A, Gøtzsche PC. Placebo interventions for all clinical conditions. Cochrane Database Syst Rev. 2010; 2010(1):CD003974. doi: 10.1002/14651858.CD003974.pub3.
• Hsieh SF, Yorke-Edwards V, Murray ML, Diaz-Montana C, Love SB, Sydes MR. Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers. Clin Trials. 2023; 20(2):121-132. doi: 10.1177/17407745221143449.
• Kaji AH, Schriger D, Green S. Looking through the retrospectoscope: reducing bias in emergency medicine chart review studies. Ann Emerg Med. 2014; 64(3):292-8. doi: 10.1016/j.annemergmed.2014.03.025. Epub 2014 Apr 18. PMID: 24746846.
• Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015; 16:495. doi: 10.1186/s13063-015-1023-4.
• Monti S, Grosso V, Todoerti M, Caporali R. Randomized controlled trials and real-world data: differences and similarities to untangle literature data. Rheumatology (Oxford). 2018; 57(57 Suppl 7):vii54-vii58.
• Páez A, Rovers M, Hutchison K, Rogers W, Vasey B, McCulloch P; IDEAL Collaboration. Beyond the RCT: When are Randomized Trials Unnecessary for New Therapeutic Devices, and What Should We Do Instead? Ann Surg. 2022; 275(2):324-331. doi: 10.1097/SLA.0000000000005053.
• Pallmann P, Bedding AW, Choodari-Oskooei B, Dimairo M, Flight L, Hampson LV, Holmes J, Mander AP, Odondi L, Sydes MR, Villar SS, Wason JMS, Weir CJ, Wheeler GM, Yap C, Jaki T. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Med. 2018;16(1):29. doi: 10.1186/s12916-018-1017-7.
• Park KJ, Meißner T, Günther E, Schmalz G, Kottmann T, Krause F, Haak R, Ziebolz D. Arrest of root caries with an adjuvant chlorhexidine-fluoride varnish over a 12-months observation period: a QLF-analyzed, placebo-controlled, randomized, clinical trial (RCT). Odontology. 2022;110(1):193-202. doi: 10.1007/s10266-021-00637-w.
• Sarkar S, Seshadri D. Conducting record review studies in clinical practice. J Clin Diagn Res. 2014; 8(9):JG01-4. doi: 10.7860/JCDR/2014/8301.4806.
• Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340:c332. doi: 10.1136/bmj.c332.
• Talari K, Goyal M. Retrospective studies - utility and caveats. J R Coll Physicians Edinb. 2020;50(4):398-402. doi: 10.4997/JRCPE.2020.409.
• Team-NB position paper “Data generated from ‘Off-Label’ Use of a device under the EU Medical Device Regulation 2017/745.” accessed 14.10.2024, accessible at https://www.team-nb.org/wp-content/uploads/2022/10/Team-NB-PositionPaper-Off-LabelUse-V1-20221005.pdf
• Vassar M, Holzmann M. The retrospective chart review: important methodological considerations. J Educ Eval Health Prof. 2013; 10:12. doi: 10.3352/jeehp.2013.10.12.
• Wang SV, Pinheiro S, Hua W, Arlett P, Uyama Y, Berlin JA, Bartels DB, Kahler KH, Bessette LG, Schneeweiss S. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021; 372:m4856.
• Wellek S, Blettner M. On the proper use of the crossover design in clinical trials: part 18 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2012;109(15):276-81. doi: 10.3238/arztebl.2012.0276.
• Zhang W, Doherty M. Efficacy paradox and proportional contextual effect (PCE). Clin Immunol. 2018; 186:82-86. doi: 10.1016/j.clim.2017.07.018.
• Zhang B, Guo J, Zhang H. Design and analysis of crossover trials for investigating high-risk medical devices: A review. Contemp Clin Trials Commun. 2022; 30:101004. doi: 10.1016/j.conctc.2022.101004.