Q&A: PMCF User Feedback Surveys

RQM+ recently held two PMCF Survey webinars that are now available on-demand. First, our April 2021 webinar focused on PMCF User Feedback Surveys: Practical solutions for leveraging user feedback surveys as PMCF activities under the EU MDR. Access the webinar on-demand here.

Next, we partnered with RAPS for a more in-depth webcast in May 2021 titled PMCF User Feedback Surveys: Practical Solutions, Lessons Learned, and Processes for Success. Access the webcast on-demand here.

In this follow-up blog post, our knowledgeable former notified body leadership and PMCF subject matter experts sat down to answer audience questions from both webinars.

RQM+ has conducted many user feedback surveys for our clients at this point in MDR implementation. We've encountered combinations of devices and clinical data where this is the perfect solution for meeting PMCF requirements, and others where a lot of strategy is needed to make it work. Our on-demand webcast with RAPS uses device examples and case studies to provide practical solutions for leveraging user feedback surveys as PMCF activities under the EU MDR.

Special thanks to RQM+ team members Torrie DeGennaro CBA, Niki Caporali Spaniel RAC, Amie Smirthwaite BEng, Ph.D., and Jaishankar Kutty Ph.D., PMP for their contributions to this blog post.

Q: How does establishing a product as well-established technology (per 2020-6), where evaluation of state of the art, including evaluation of clinical data from similar devices is defined as acceptable clinical data, play into the need for device-specific PMCF?

A: For WET devices, one can rely on lower levels (defined in MDCG 2020-6) of evidence to satisfy both pre- and/or post-market requirements. For example, for a suture which is considered a WET device and is enlisted as one of the twelve devices exempted from clinical investigations per Article 52(4) and 61 6(b), one can rely on a SOTA-based strategy/case report forms for initial MDR submissions and user/physician surveys (not patient level)/case report forms to satisfy PMCF requirements.

However, if the suture is an absorbable one or if it has novel features (such as infection detection) or if it has no previous patient specific data and is used in high risk cardiovascular- or neurovascular-type applications, then there is an expectation to generate a certain degree of patient level data as part of the PMCF activities.

Q: If not literature reviews, if not surveys, then what? As a manufacturer of a low-risk device, conducting a randomized controlled trial seems overkill. Thanks for your advice.

A: Per the MDR, the clinical data requirements do not differ with device classification. This is because classification in the EU is not based solely on risk. It’s foreseeable that the clinical data burden for a non-WET, non-implantable class IIa device with multiple indications may be significantly greater than that for a class III device intended for transient use in a severely underserved patient population.

The choice of data gathering effort needs to be rationalized on the basis of the gaps (or what has been previously “rationalized” under the MDD) in the existing dataset, rate of change of SOTA, standard of care considerations in the EU, emerging hazards, and expected benefits for the intended patient population. Moreover, such an effort needs to be performed with due consideration given to statistical significance and clinical relevance (once again regardless of device classification).

Q: For Class A devices, how do manufacturers verify/confirm the proposed risk class A for the device with the NBs to avoid any possible disagreements in the future?

A: Currently, notified bodies are not permitted to conduct a strategy review of any kind. However, there are some that may entertain a phone call on such items. Importantly, for any notified body, if this is class A sterile IVD, which will need a notified body review under the IVDR, you can discuss the device classification as part of the initial paperwork that’s submitted during the initial quotation review process (BSI calls it a “Device Schedule”) to help define the scope of the QMS certificate.

At this time, if there is a classification dispute between the notified body and the manufacturer, the question should be raised by the notified body to the competent authority of the member state where the legal manufacturer is located or the member state where the legal manufacturer’s authorized rep is located.

Q: It’s interesting that Amie said some kind of pro-active PMCF activity is always required even for low risk legacy devices. Can you give an example of what kind of activity should be conducted? If not literature reviews, if not surveys, then what? As a manufacturer of a low-risk device, conducting a randomized controlled trial seems overkill. Thanks for your advice.

A: For a very low risk, standard of care device with “sufficient clinical evidence”, a user survey could be entirely appropriate. It depends what the device is intended to do, what the state of the art looks like, what the clinical (or other) evidence of safety and performance looks like, and what the outputs of the risk management look like.

A literature review is not considered proactive, because it will only uncover information that someone has decided to publish. It is unlikely to identify specific issues with a given device, unless the size of the failure is causing significant concern such that it is of scientific interest (in which case, identifying a problem by literature review would be unsatisfactorily late in the process).

A survey, however, is considered proactive – so if the design of the survey is appropriate in light of the residual risks and clinical unknowns associated with a device, this could be fine. It has to be the right kind of survey, and access the right kind of data; what is “right” for any given device will depend on the intended purpose, clinical evidence, and risk assessment.

Q: MDR cites the general methods and procedures of the PMCF to include screening of scientific literature - can you provide more insight into this statement in the view of literature review being passive?

A: The “general” methods identified in Annex XIV 6.2a are normally interpreted to mean passive data collection methods not directed to any specific questions of clinical safety, performance or benefit-risk. These are gathered routinely, as all available sources of evidence should be reviewed and evaluated to maintain the clinical evaluation.

As noted above, methods such as literature screening or complaints and vigilance evaluation are considered “passive” because manufacturers do not have any control over what is published in the literature or reported in adverse events databases (except when they are reporting on their own studies and products).

The “specific” methods described in Annex XIV 6.2b are normally interpreted to mean proactive data collection methods, where the manufacturer has some control over the design of the data collection method, and can adapt it to answer specific questions relating to safety, performance, or benefit-risk of a device.

“Suitable registers”, including national registries, may be included as a proactive mechanism, because (depending on the registry) it may provide a comprehensive data set collecting clinical evidence relevant to the specific safety, performance and clinical-benefit endpoints which would be indicated by the outputs of the manufacturer’s own clinical and risk assessments.

The data needs to be evaluated and justified, as an example, the national registries only collect data on survivorship, but the outputs of your clinical evaluation and risk management assessments indicate that specific PROMS are required, then the registry would not in isolation be a sufficient specific method.

Similarly, if your outputs indicated that further data on a specific device variant that is not well-represented in the registry, then a registry would not be a suitable method, but a survey that accesses patient-level data might be.

Q: Amie, what did you mean by appropriate stratification?

A: Stratified at a level that enables you to reach appropriate conclusions about the specific variant, patient population, treatment indication, etc. For example, if you have four different combinations of femoral head and acetabular liner within a product family, but the data is not stratified to show outcomes for each possible combination, one of the four combinations could be getting a much poorer outcome than the other three, but this would be missed by considering the data in aggregate rather than stratified by specific combination.

Q: With safety endpoints built into a survey, does that automatically require the need to pursue IRB approval, patient consent, etc. or is there some considerations on ways to collect AE information while by-passing these expensive and time-consuming hurdles, particularly when the user is the physician and not the patient directly.

A: No, having safety endpoints in a survey does not automatically mandate the need to pursue IRB or EC approval. There are really multiple aspects to consider in this question:

1. 1. Physicians responding to a survey need to provide anonymized patient data;
   2. Physicians need to be prompted to submit AEs as complaints to the manufacturer’s complaint department (manufacturer should have a plan for how to handle complaints in advance);
   3. Additional country rules and regulations need to be followed to ensure IRB/EC approval is not required (is the study considered medical research, is the manufacturer going to use the data outside of regulatory purposes, etc.)

Q: Can the survey be based on a one year recall period or is an objective review of medical records required? For example related to the last question -- If there is a question to the physician to estimate the number of samples obtained of diagnostic quality? Can they provide a number based on recall and then the estimates be averaged across all respondents? Is it valid?

A: It depends on the specific details about the device. In some cases, 12-month recall (recollection) is acceptable. Other survey styles can include both physician review and also retrospective patient chart collection so that patient-level data is not subjective to the respondent’s recall but based on actual cases. These chart reviews must be anonymized.

For the IVD product mentioned, subjective recall on diagnostic quality is likely not adequate for a PMPF survey. Perhaps for a very low risk class device with sufficient clinical performance data it would be ok. In our opinion, it would be better to track patient charts to diagnostic outcome more directly. Here, it is less about “is it valid?” and more about designing a clinically-meaningful survey.

Q: Can PMCF surveys be appropriate for IVDs where the user is typically a medical provider using the “product” (IVD report) for decision making purposes?

A: Yes, this is possible. This would be a PMPF survey targeting both a clinician and collection of retrospective patient chart data.

Q: Can you expand on what you mean by "patient chart review" and how you would capture that in your survey?

A: These types of surveys are typically completed in two parts. The first part is asking the clinician to respond to questions. The second part is asking the clinician to provide anonymized patient data from a specific use-case. The entire survey is completed on a survey platform (must be GDPR compliant for EU and HIPAA compliant in the U.S.).

Q: How long should be considered for designing and executing PMCF surveys? Planning phase, data collection phase?

A: This will depend on the resources available and size of the portfolio. For just one survey, we plan four to six weeks to draft the protocol with our clients, one month for deployment which includes programming, translations if applicable, response collection, back translations if applicable, one month for data analysis and survey report.

This schedule becomes very difficult to keep when doing many surveys at once with a small team, or when survey objectives and endpoints change mid-protocol which can occur for many reasons (new feedback from a notified body, CER updates to align with notified body feedback, new medical reviewers, new templates approved with the client, etc.). So even though we outlined our example timeline, building in enough cushion early on will be helpful to creating a successful survey program.

Q: For instruments or implants, would a repeated survey be required after the initial survey?

A: Evaluating the need for PCMF should be completed as part of the post-market surveillance process. Class III and implantable devices require an annual update to the PMCF evaluation report and all other devices need to be updated as necessary. As part of this evaluation process, it may be determined that a follow-up survey is required. Good news, you can leverage your original survey design in your new survey and use the first data set to build a more robust survey activity.

Q: For a set of instruments which allows a choice of instruments for surgeons, how can we effectively collect data on a large instrument set?

A: The first step is understanding the relatedness and usage of the devices. Can one instrument be leveraged to support another? Is one device rarely used? This information can help you build a plan and survey to better target your devices and respondents. Using conditional formatting in your survey can help target questions to be applicable to certain respondents reducing survey fatigue.

Q: Is there any specific way in which surveys need to be collected? That is hard copies which the end use fills in or an online survey? Do they have to be signed by the individual taking the survey? And do they require any information about the person who filled out the surgery; age, gender etc.

A: There is no specific way per the regulation that surveys need to be collected. Different logistics and cost factors will need to be considered for each approach. Currently, the regulation and the available guidance documents do not specify information needed from the respondent. RQM+ recommends asking various screening questions to verify that these are users of the subject device and collecting some information on the respondent (e.g., how often they use the device, how long they have been using the device, etc.). RQM+ recommends reviewing all survey questions with your legal team to ensure compliance to local laws and regulations.

Q: Collection of patient-level data via survey was mentioned. What data should you not be collecting so you avoid patient consent issues?

A: Any information when used together or individually that can identify the patient should not be collected. If certain information, for example age, is needed, consider asking a multiple-choice question with age ranges provided. RQM+ recommends reviewing all survey questions with your legal team to ensure compliance to local laws and regulations.

Q: How would you ask for an off-label use of device without encouraging off- label use to end users?

A: The EU MDR requires that PMCF is used to “[identify] possible systematic misuse or off-label use of the device, with a view to verifying that the intended purpose is correct.” We have seen success when asking device usage related questions, to provide an “other” option with a follow-up free response field after listing the approved uses of the device. This allows the respondent to provide the off-label use without you suggesting what the off-label use is.

Q: Can surveys be designed with Yes/No or Likert scales?

A: Surveys can be designed with any response scale deemed appropriate for the question. Some surveys may need to use both Yes/No and Likert Scales. In the event that mixed scales are used it is recommended that questions with similar scales be grouped together. For more information on survey design refer to our Top Ten Tips.

Q: For safety and adverse event questions how do you ensure you collect enough information to determine if the adverse event was device or procedure related? How do you know how many devices were involved?

A: Consider framing safety and adverse event questions in a way that will lead the respondent to provide you with the information you need to make a complete report of the adverse event. Asking a free-response follow-up question is one way to address this. You could also ask a multiple choice question asking the respondent to identify if the event was device or procedure related.

This is a limitation of survey activities since additional follow-up with the respondent cannot be completed. RQM+ recommends having a process in place on how to handle potential adverse events collected via survey activities.

Q: If you get adverse events, how can you determine whether the event is caused by the guide wire versus the cardiac device?

A: Consider building follow-up questions into your survey that ask the respondent to identify the cause of the event. This is a limitation of survey activities since additional follow-up cannot be completed. RQM+ recommends having a process in place on how to handle potential adverse events collected via survey activities.

Q: What are our options to handle and meet data confidentiality requirements in EU and in other countries worldwide?

A: RQM+ recommends reviewing all survey activities with your legal team to ensure compliance to local laws and regulations. An easy first step is to ensure that a data collection platformed being used is GDPR compliant.

Q: Any incentives to get responses?

A: Manufacturers can choose to offer incentives to survey respondents. These incentives should be at the fair market rate and any local laws and regulations regarding reporting of payment need to be followed. Potential bias caused by incentives should be addressed in the survey protocol/report.

Q: How should a clinical gap analysis look like? And how much of it do you have to disclose to your notified body?

A: We have found that a spreadsheet is typically the best format for assessing clinical evidence. It should capture the literature: sample size, population type, objectives, success criteria, outcomes, subject device/variant, etc. while also taking into account the quality of the data and what device claim is supported by the data. This is a good tool to present to your notified body as it demonstrates that your clinical evidence was assessed with a methodological and scientific approach.

RQM+ is here to help address any other questions that you may have regarding EU MDR and PMCF related topics. Contact us today to find out how we can help! If you were not able to attend our webcast with RAPS, here's a link where you can sign up to view the on-demand version (and download the slides).

Looking for more PMCF resources? Check out the RQM+ Knowledge Center for more expert content.