SEMS Skills & Knowledge – Why Qualifying Evaluators Is So Important

When you read the offshore industry guidelines on managing skills and knowledge under SEMS, one of the first things that jumps out at you is the need to make sure the evaluators of offshore personnel are qualified.   

As background, SEMS requires that operators verify that their contractors have the skills and knowledge to do their jobs safely and effectively.  Earlier this year, the Center for Offshore Safety came out with guidelines to help companies develop a Skills and Knowledge Management System (SKMS) and some operators will make having an SKMS a require for their contractors.  Section 8.1.2 of the guidelines says companies need a way to determine assessor (evaluator) qualifications.

Why is that so important?   It is because experience has shown that in any assessment system evaluators, administrators, instructors and proctors are the weak link.   Sometimes the people who are being evaluated or tested do cheat, but the hardest fraud to control is caused by the people who control the evaluation process.   Here are some recent headlines highlighting fraud by the people who were supposed to be safeguarding the system:

Navy sidelines 30 nuclear instructors suspected of cheating – The  Navy says it caught 30 staff members at an elite school fudging the test that allows people to train students to work on nuclear submarines.

Facing fraud investigation, Palo Alto Utilities changes tests for gas workers – An office worker told authorities his boss told him to fill out 30 test forms for workers to help them pass their pipeline operator qualification evaluations.

School Proctor and Admissions Officer Plead Guilty in Student Financial Aid Fraud Scheme – The FBI says school officials falsified test results to help students who had not finished high school qualify for aid at a trade school.

The message is that the people who perform evaluations, observe testing or administer programs have a unique opportunity to commit fraud.   That is why the COS recommends that companies make sure their assessors are qualified and that the program is audited.    It is also why you can expect operators to pay close attention to whether contractors are meeting those requirements.

Those are features of our SEMSReady™ skills and knowledge classes and follow up program.  In our classes, we focus on teaching companies how to develop objective evaluations that measure and document quantifiable skills.  We teach administrators and evaluators the importance of accurate, measurable skills and knowledge assessments.   Finally, we are able to audit results to make sure the process is being followed.

4 Replies to “SEMS Skills & Knowledge – Why Qualifying Evaluators Is So Important”

  1. I would think that evaluators with operations backgrounds would be an asset to any team because they can capitalize on past experience. Is this a consideration?

    1. Dan,

      You are absolutely right. The person who evaluates a skill absolutely has to have the operational experience to be a subject matter expert.

      Sometimes, SMEs know how to do things but have a hard time explaining how they do what they do. Then the evaluation form needs to guide them through the process. The other problem, which is the point I was stressing in the blog, is that they may have an incentive to pass workers without making them meet the skills requirements. That’s why we want the evaluation to include enough documentation that someone reviewing it can get a feel for whether it was faked.

  2. The assessment dilemma isn’t particular to our industry, any discipline that requires demonstrated skill has the same concerns. While I agree that knowledgeable evaluators are a valuable asset, more often than not I note that the evaluation itself is grievously flawed. If the evaluation doesn’t contain a very specific rubric for marking, if the questions leave room for interpretation or if it doesn’t contain measures for assessor accountability it’s not an objective assessment, merely a subjective opinion, regardless of who’s doing the grading. As with safety by design, assessment validation can stop a lot of these problems before they become problems. How many people qualified to do that validation do you suppose are in our industry?

    1. Ross,

      I think you have hit the nail on the head. This is a little like hazard barriers, where each control needs additional barriers if the first one fails. In our class on evaluations under SEMS, we stress the need to write the evaluation criteria in a way that minimized subjectivity, the need for the evaluator to be trained on performing objective assessments that are as bias-free as possible and the need to include metrics that allow an auditor to check the results.

      As for who is qualified to do validation – Based on your email, I suspect you are!

Leave a Reply