Can operators be expected to learn a new control system by simply watching other units running a different processes? The cost of saving money by eliminating training.
Prior to the 2008 incident, the methomyl unit at the Institute facility had been shut down for two major equipment upgrades: the DCS had been replaced and the old carbon steel residue treater vessel had been replaced with a new stainless steel pressure vessel. Production in the unit is seasonal as there are specific times of the year when Larvin, Bayer’s carbamate-based agricultural insecticide, is used. Consequently, the unit is typically idle for several months each year during which time major repairs and upgrades are performed.
In this particular case, a new DCS from a new vendor was installed to replace an aging system that was beginning to cause operational problems. This was not the first unit in the plant to receive such an upgrade as the Larvin unit in the facility had changed to the same platform in 2007. On that earlier project, the DCS vendor and system integrator that installed the system had worked with Bayer to train the Larvin operators, and by early 2008, they were operating comfortably with the new system. When it was time to upgrade the methomyl unit DCS, management decided such formal training was too costly and unnecessary, in the belief that operators had seen enough of how the new system worked from the Larvin upgrade.
Glossing over a major challenge
DCS upgrades need to be treated as very critical events for operators under the best of circumstances, and leaving something this important to self-directed OJT proved to be a major contributor to the incident. Synthesis of methomyl is more complex than Larvin and operators that were not intimately familiar with the process were struggling. Various functions had moved to different screens making it difficult for operators to find what they were looking for. The HMI changed many control elements the operators were used to, such as indicating the contents of a tank by specifying the content in total gallons or pounds rather than as a percent-full. Operators had to create cheat-sheets to help with different conversions as they learned the new system, and that effort was far from complete.
Procedures and incidents
Procedures, such as startups, shutdowns, product grade changes and the like are critical times for operators as well. In most situations, procedures are performed manually following a written SOP describing the steps involved. Frequently, these depend on experienced operators capable of compensating for instructions that are outdated or otherwise inadequate. When trying to perform the startup procedure using the new DCS, operators reported that the new system was difficult to navigate and showed slower response times. With the older system, operators were able to change multiple variables simultaneously, but could only change one at a time with the new platform. Revisions to the SOP instructions had not been fully updated to reflect the changes necessary with the new DCS.
In this particular situation, the new HMI showed the solvent ratio in the residue treater using a different format than the older system. During investigations after the incident, operators admitted that they weren’t sure what they were seeing and were guessing about what was happening in the reactor. The startup procedure was also very complex, and involved multiple lab evaluations of the reactor contents to verify methomyl content was within acceptable limits. It was not, as the operators would soon realize.