Over the past three months, I have spoken with a number of auditors and attended presentations on SEMS audit results. In particular, I was impressed by excellent presentations from the Center for Offshore Safety, representing the largest oil and gas operators, and M&H, which audited the lion’s share of small to medium operators. I use the information from those conversations and presentations for the workshop class on SEMS for Contractors that I teach. Today I wanted to draw some year-end lessons:
- The stuff we knew would be hard turns out to be hard – When companies were developing SEMS, everyone knew evaluating training/competence of contractor crews, operating procedures and mechanical integrity were industry-wide weaknesses. Based on the audits, it looks like we still have a long way to go.
- Consistency of audit processes and results is a problem – The first auditors had to make up the rules as they went. Is this a compliance audit or a systems audit? When is something a corrective action vs. a notation? What gets reported to BSEE? So it is no surprise that results were all over the map and looking for general trends is like comparing apples to oranges to golf balls.
- What gets looked for gets found – The old saying is, to a guy with a hammer, everything looks like a nail. SEMS is such a broad area, covering everything from the hard science of engineering to the softer science of human factors. That is a lot of territory for an audit team to cover and, especially when it comes to physical visits to facilities, with all of their complexities. As a result, individual auditors tended to focus on what they understood. Anecdotally, I have heard that many auditors had Coast Guard inspection backgrounds and if there is anything a Coastie understands, it is emergency response. So that is an area that some auditors focused attention on, even though it may be the area of SEMS the industry has the most experience with.
- Audits may not be the best way to evaluate the quality of a SEMS plan – The three previous trends will improve as we do more audits, but this last one may be a problem we can’t overcome. It really looks like most of the audit deficiencies were based on compliance with specific SEMS requirements. For example, one potential discrepancy from an audit was whether the operating procedures included “lease or concession stipulations” not whether the SOP is accurate or a review of JSA’s focused on whether the forms were filled out correctly, not whether the JSA accurately described the task the crew had to perform. In other words, audits look at black and white proof of documentation, not the grey areas of whether a process is effective, especially involving highly technical processes.
This is not the auditors fault. It is just the reality of what an audit does best – review documentation in an objective, non-judgemental manner. The failing of this was pinpointed by my friend Ian Sutton when he asked at a conference whether an audit looks qualitatively at safety-by-design alternatives that are considered by the original engineering team for a facility. The answer, of course, is no.
So after the first round of audits, the industry may be faced with a rather unappealing reality. The audits are a tool for assessing the programs, but they are a flawed tool and they cannot be relied on as the only tool. They can tell an operator whether its SEMS plan looks good, but, in some critical ways, they may miss the sort of fundamental problems that can cause a catastrophe. A historical comparison might be the Maginot Line, a system of concrete fortifications meant to protect France from German invasion before World War II. An audit would have shown it to be well-planned, well-built, and well-provisioned. The only problem was it failed in the first week of the German attack. The challenge in 2014 will be to use audits to help win the fight for safety, not just the battle over documentation.