Competency Assessment vs Proficiency Testing

Competency Assessment vs Proficiency Testing. Do IVF Labs Need Both? Tl;DR – YES! We explain why.

“One thing worse than training employees and losing them… is not training them and keeping them.” -Ed Metcalf

Competency Assessment vs Proficiency Testing. Does my IVF lab need to perform both? The short answer is yes!

Learning Objectives

— What is the difference between proficiency testing and competency assessment?

— When do you need to perform PT?

— When do you need to perform CA?

— What are the 6 elements needed to perform a CAP level CA?

— When do you need to perform PT?

— Define the criteria for satisfactory proficiency testing (PT) performance.

— Describe the consequences of unsatisfactory performance of a PT event.

— Take the appropriate steps to investigate a PT failure.

— Discuss the advantages of standardization and digitization for documentation of training, performance, and corrective actions.

Until each staff member has documented training, followed by successful assessment of competency of that task, they should not be permitted to perform or result patient testing independently of their trainer. Embryologist training, competency acquisition and documentation of these activities for inspections hasn’t advanced in 40 years, until now. ART Compass is outsmarting the embryologist shortage, reducing non-conformance events, and helping you pass quality inspections, defeating CAPs number one most cited deficiency, staff-related checklist items- download our complete full length, peer reviewed study here, “Assuring quality in assisted reproduction laboratories: assessing the performance of ART Compass – a digital ART staff management platform.”

Whether you’re subject to CLIA, CAP or your local state DOH requirements, you will be required to provide proof (documentation) of both training AND competency for each employee, for each task that they perform. Documentation of all of these tasks can be organized through the use of advanced digital IVF lab management tools, like ART Compass. ART Compass helps savvy IVF lab directors keep track of what each staff members’ specific job junctions include, which they need to be trained in, and which tasks have been completed by each trainer.

Proficiency Testing

Before passage of the Clinical Laboratory Improvement Amendments of 1988 (CLIA), participation in proficiency testing (PT) was voluntary for many labs and the standards for these tests were not defined. With passage of CLIA ‘88, proficiency testing evolved from an educational self-assessment and improvement tool to the primary measure of continued reimbursement and licensure.

Proficiency testing determines the performance of individuals for specific tests or measurements. Performance on PT has been found to be an indicator of the quality of patient testing. By CLIA standards, clinical laboratories are required to enroll in a CMS-approved PT program for all regulated nonwaived testing, and their performance is compared to a target value using statutory criteria for acceptable performance. For nonregulated, nonwaived testing or for tests that have no formal PT available, CLIA requires alternative assessment, twice per year.

Failure of a single event (<80%, or <100% for immunohematology) requires an investigation to ensure that processes were followed correctly throughout all phases of testing: preanalytic, analytic, and postanalytic. This type of unsuccessful performance does not require that a response be returned to the PT program or to the facility’s accreditation agency. However, if two consecutive events or two out of three consecutive events for the same procedure receive an unsuccessful grading, consequences are more severe. The laboratory must take steps to discontinue the procedure. It is best to notify the accrediting agency of the voluntary discontinuance before the accrediting agency notifies the laboratory.

Next, the problem should be investigated, corrective action implemented, and two consecutive successful PT events must be performed. The events can be independently obtained through the ART Compass program. Finally, the accrediting agency must be notified of the corrective action and the intent to reinstate testing.

Embryology procedures fall under nonregulated, nonwaived testing. This requirement can be met by subscribing to a formal program, such as ART Compass, which offers testing for all embryology procedures. The target values in ARTC are determined two ways: by the mean of all peer participant responses after removal of outliers, and by IVF Lab Director consensus. Acceptable performance is considered to be 80%.

You may access a Proficiency Testing Policy and Procedure in ART Compass, by logging in online via a Chrome Browser and navigating to the green tab “Documents and Forms,” then click on the “Quality” folder. Use the “Show to” feature to select any individual who needs to read and review the policy.

Competency Assessment

Embryologist competency assessment is an ongoing, continuous process of monitoring individuals’ abilities to perform their specific job functions.

Once training has been completed and documented in ART Compass, you will assess each staff member’s clinical decision-making ability to successfully perform these tasks using our robust suite of subjective “surveys.” This is their competency, where you assess if the training was successful and staff are able to perform each assigned task correctly.

You may access a Competency Assessment Policy and Procedure in ART Compass, by logging in online via a Chrome Browser and navigating to the green tab “Documents and Forms,” then click on the “Quality” folder. Use the “Show to” feature to select any individual who needs to read and review the policy.

CAP Requirements for Competency Assessment:

ART Compass contains a robust suite of tools for staff management, including a Staff-Quality Audit and subjective competency assessments for each embryology procedure, as well as objective surveys for topics of interest such as Media and pH, FDA regulations, HIPAA Compliance, PPE and Bloodborne Pathogens and much more! Two ART Compass digital tools, a conventional checklist and our Competency Assessment 2.0 document capture module (use your phone camera to record evidence) will guide your competency evaluations and document evidence of completion for all required procedural steps.

To fully demonstrate successful competency all 6 of the following elements must be documented for each employee, for each task:

Direct observation of routine patient testing — Watch (and snap a picture!) as employee performs routine work processes and procedures, including; patient preparation, specimen handling, processing, and testing.

Monitoring the recording and reporting of test results — Review worksheets, computer printouts, or manual test logs for results or entries that are incorrect or incomplete. The worksheets, logs, and printouts should be compared to final test reports, evaluated turnaround times, and for compliance with policies and procedures.

Review of intermediate test results, records, and proficiency testing results — Review testing documentation, to include test results or worksheets, documentation forms, QC logs, proficiency testing documentation, and other applicable documentation for completion of proper policies and procedure. Documentation of review will include follow up of corrective action related to problems in the laboratory.

Direct observation of instrument maintenance and function checks — Watch employee perform instrument maintenance and function checks as compared to documentation for completeness and correctness of process and expected outcome.

Assessment of test performance through testing previously analyzed specimens, internal blind testing samples or external proficiency testing samples — Assessment of previously tested samples; internal blind samples or external proficiency testing samples.

Assessment of problem-solving skills — Include review of problem logs, incident reports, and QC failures; Review corrective actions employed to resolve the problem. Personnel interviews (staff narrative) may also be included for documentation.

PT Failure: Now What?

Unsatisfactory PT performance makes up approximately 3.2% of all CMS survey deficiencies. Inappropriate PT enrollment makes up 1.8%, and failure to perform alternative assessment of nonregulated testing makes up 6% of deficiencies (Yost, 2011). The suggested steps for investigating PT failure are:

1. Survey Materials: On receipt of the survey materials, verify that they were received in acceptable condition. Should any unacceptable conditions or criteria be noted, notify the PT program as soon as possible for shipment of replacement samples. Ensure that the samples are stored according to the manufacturers’ instructions until analysis.
2. Clerical Error: Errors of this type are the most commonly reported and may be of various types. They include typographical errors as well as errors related to interpretation, transcription, transposition, misidentification, answer-form coding, miscalculation, or acceptance of a nonlinear result. Manual entry offers more chance for transcription or transposition errors. Electronic entry still allows for incorrect keystrokes and/or options to be submitted.
3. Technical Problems: The most common assumption when assessing technical issues is that the reagent and/or instrumentation reported an inaccurate result. This may be exhibited by the quality control value falling out of the acceptable range for the analyte and the run being accepted. Quality control results that demonstrate excessive repeats, shifts, or trends may also affect PT results and should be carefully scrutinized in the investigation.
4. Method Problems: These types of errors include instrument problems, maintenance issues, or faulty reagents and/or standards. Verify that the instrument was within performance guidelines the day of PT. Were there repairs or component replacements recently done that could have affected the PT results?
5. Could the error have affected patient testing? If so, document the actions taken to ensure that the patient testing reported at the time of the unsatisfactory PT was valid or whether amendment is needed. It is necessary to notify providers if any inaccurate results were reported.
6. The final steps of the investigation are to determine the problem(s) and summarize the findings in your documentation. Include any supporting documents for the findings. These may include proficiency report records, instrument data, quality control records, calibration records, maintenance logs, and any other records found to be of assistance. Document the corrective action taken to remedy the problem and that steps that will be taken to prevent the problem from recurring.

Documenting PT performance in ARTC

A logical approach and use of a standardized investigation forms may be a reassurance that possible critical steps in the investigative process were not overlooked, and variables were examined for the root cause of unsatisfactory proficiency testing performance. You can find free resources in ART Compass to troubleshoot failed or unsatisfactory PT events. 1. In the surveys module of ARTC you will find The IVF Lab Digital Playbook -> Investigation of Proficiency Testing Results.  2. Access ARTC online, navigate to the green tab “Documents and Forms” and assign the “Investigation of Proficiency Testing Results”to a staff member.