January 2013 - The NCCCO Auditor Program requires annual training as part of NCCCO’s continuous improvement program. Over the past several months, more than two dozen NCCCO Practical Exam Auditors received refresher training to verify that NCCCO Practical Examiners administer CCO exams consistently and accurately.
Four Auditor training sessions were held last fall, one each at H&E Equipment (Salt Lake City, UT), at Crane Edge, LLC (Milton, WV), at the Wisconsin Operating Engineers Training Center (Coloma, WI), and at the East Texas Crane Academy (La Porte, TX). The training covers recent program changes for operators, signalpersons, and riggers, and includes Inter-Rater Reliability (IRR) exercises to ensure consistency among Auditors and Examiners. The annual multi-day sessions ensure that Auditors receive calibration training and are briefed on multiple precepts of test integrity and behavior recognition strategies.
NCCCO Auditors are responsible for conducting performance reviews of Practical Examiners as well as site inspections to ensure the fairness, integrity, and safety of practical exams leading to CCO certification. In doing so, they also provide the impartial witness critical for fulfilling the accreditation requirements of the American National Standards Institute (ANSI) and the National Commission for Certifying Agencies (NCCA). Mandatory annual Auditor updates are also required to maintain NCCCO programs’ accreditation to international standards as defined by ISO 17024 General Requirements for Bodies Operating Certification Systems of Persons.
Phillip Kinser, Program Manager for the NCCCO Audit Program, explained, “We require our Practical Examiners to undergo thorough training and auditing to understand the intent of psychometric principles such as validity, reliability, bias avoidance, and Behaviorally Anchored Rating Scales (BARS). The auditing process provides feedback to the governing program committees and accrediting bodies (ANSI and NCCA) that the issues of test integrity, fairness, and adherence to exam procedures and policies are being followed.” This process is aimed, Kinser added, at ensuring that candidates seeking certification through NCCCO's programs are provided psychometrically fair and reliable exams.
Substantial time is spent at Auditor training sessions reviewing issues typically noted during site audits as well as coaching the auditors on appropriate responses. Examples include covering how to tally scores correctly; ensuring that participants are wearing appropriate personal protective equipment; and setting up the practical exam course precisely in accordance with the provided specifications. “This continuing training,” noted Kinser, “is intended to ensure confidence in the inter-examiner reliability of our Examiners as well as the inter-auditor reliability of Auditors.”