TiPSTalk

Alarm Management and the 5S approach

Steve Apple — Wed, 03 Dec 2008 20:05:00 GMT

In the Wall Street Journal's (10/27) Business column, Julie Jargon writes, "5S is a key concept of the lean manufacturing techniques that have made makers of everything from cars to candy bars more efficient. The S's stand for sort, straighten, shine, standardize, and sustain." Recently, 5S "has been moving from the plant floor to the cubicle at hundreds of offices around the country, adding desk cleaning to the growing list of demands on employees." With regard to implementing 5S within a workplace, "it's not the initiative that's important, it's how managers communicate it," Gary Hayes, managing partner at Hayes Brunswick & Partners LLC explains. By clearly explaining "why they're doing something...most people will understand the rationale," he adds. According to a 5S inspector at one company who conducts quarterly inspections of the offices, supervisors should "figure out how to balance being too picky with upholding the purpose of the program," by letting go of some things that are not in compliance with the program.

When I first read this article, I thought- "What the heck is this? I know 6Sigma, and the Sigma is a take on standard deviation, and quality. Well, in the same vernacular, s stands for error, so is this a new quality approach?" So, I looked it up, and its all about cleaning up and maintaining a "tight ship" as my uncle in the navy used to say. It's from Japan, and the five s's refer to Japanese words that start with an s. Here's what the background is:

The first “S” is Sort. It is the process of removing all unnecessary items from the workplace area.
The second “S” is Set in Place. This is the process of moving the necessary items into the correct position for use.
The third “S” is Shine. This is the method of deep cleaning a machine or area to put it back into the condition it was when it was purchased.
The fourth “S” is Standardize. This is the process of standardizing the entire system, which is often the most difficult.
The last “S” is Sustain. Sustaining the system is thought to be one of the most difficult, primarily because experience proved years of cleaning and organization were not maintained. However, if the system is standardized in the fourth S, then sustaining it is much easier.

The best method of sustaining the system is to conduct audits. Care must be exercised so the audit system is not punitive. The 5S system relies on employee involvement and commitment at all levels, and a punitive audit system can destroy the system.

A good 5S implementation has many benefits. The assets of the company are kept in top condition which keeps the value high. Quality is kept at the level when the asset or machine was first installed. Maintenance costs are reduced as deterioration is immediately apparent. Setup times go down from better organization and reduced movement.

The best benefit is the morale improvement from an improved environment and culture.

Some managers think employees will not sustain a perfectly clean manufacturing environment. Like most systems, management is the reason the system succeeds or fails. Given the chance, employees will implement and sustain the 5S system. Most employees will choose an organized and clean workplace with a continuous improvement culture over a dirty disorganized facility.

Now, as a person who thinks a clean desk is the sign of a sick mind, I find this all rather amusing. But I also want to tell you that this very method is exactly the key to a better alarm handling system, and to improve the morale of operators who have to use it. Furthermore, as pointed out, operators will maintain a clean alarm system if you give them the tools and allow them to use them to do so. I know this because I have seen it.

I wont go into the intricate details, but let's examine the steps and how they apply.

Sort: Get the data from you system. use analysis tools to figure out what is going on in your control room. This is the first step to resolving alarm issues. If you don't have a handle on what is happening, then no amount of fixing is going to make it better.

Set In Place: Use that data to decide what is working and what is not. Eliminate things that are not working, and keep things that you have to. Note that as in past posts on this blog, this may mean you have to keep some things simply because you don't have the technology to get rid of them. In any case, clean up the work environment, and get rid of all the trash. Most people find a lot.

Shine: Now consider how to make this like a brand new alarm system. Document it. That may mean that you need to do a complete rationalization, or it may not. Risk factors will determine the depth of this step. Read my post on the engineer who decided to design his alarm system correctly if you have any questions about this. It takes the same amount of work to do it right as to do it wrong- so why not do it right? A fresh alarm design can cure a lot of ills, and gain the confidence of operations.

Standardize: This is often the step when you make a "gold standard" database of the configuration that you can refer to when things get out of hand again. It is the reference point of what you decided the perfect alarm system should look like if it were designed correctly to start with. Set the rules in place that make this become the rule rather than the exception, and design processes that keep it from ever going bad again.

Sustain: Design automated monitoring of the system so that it will indicate to you when it gets out of line. As is indicated above, with a shiny new alarm system, and very few spurious alarms on a regular basis, it becomse obvious very quickly when it goes bad, and the fixes are not only easy, but happen quickly as opertors want to have a clean system once they 've learned how valuable it can be.

That's it. Follow this method, and you will have a clean and shiny new alarm system that you can keep that way. You won't have to hire some fancy janitorial service to come in and clean it on a regular basis, because the users will keep it spic and span (as mom used to say). Nobody likes a messy system, but when it gets that way, few want to try to clean it up, or get it in order. They just find ways of adding to the junk since junk doesn't look bad on top of junk.

Now excuse me while I go clean my desk. Perhaps it will increase my own productivity?

The importance of alarm mangement lifecycle

Steve Apple — Fri, 07 Nov 2008 21:25:00 GMT

I'd like to return to one of my favorite topics for this blog posting. The issues is design versus fix. And let me just state it in clear terms: The reason why so many people are concerned about rationalizing their alarm systems is that the systems were never designed properly to start with. They just went from a bad design to a barely functional system with lots of bandaids in place. The risk associated with these systems should make you nervous about the potential they offer for incidents.

So, let's examine how you design a system properly to start with, and keep it from heading south. In doing so, I will use the reference to a lifecycle approach to alarm management.

Note that this is the approach that is being undertaken by the ISA18 committee, and reflected in the standard document soon to come. My opinion on this standard runs contrary to the assertions of one very service-oriented vendor who would like to sell you a book, and hundreds of thousands of dollars of services. They maintain that the ISA standard will essentially be unusable by the average person, and you will need to call on them for help. Au contraire.

Upon its release, this standard will allow you to understand completely how to design, analyze, control, and yes- even fix a broken alarm system. Six sigma junkies will love it. It is the culmination of several years collaboration of the world's top authorities in alarm management. When released, it will be the penultimate resource for understanding how to manage an alarm system and prevent issues in the control room associated with alarms. You will not need to buy a book to translate it. In fact, a book soon to be released will serve to explain in illustrative detail the underlying explanations of how alarm systems work, not just how to fix them. Save your money for that book, as it will be the ultimate textbook on alarm management- written by a truly recognized independent expert in the field.

If you use the ISA18 standard to first familiarize yourself with the lifecycle, and then read the associated text to understand what each of the elements represents, you will have a suscinct guide. It will help you approach any system in whatever stage of the lifecycle you may find it. Alarm management is not rocket science. It is solid, logical engineering and watchful monitoring utilizing the data an alarm system yields. You cannot get anywhere on any alarm system without first examining the data it produces. And you cannot fix it permanently without a well thought-out plan.

What quickly becomes clear to the average person is that if you design an alarm system properly,and continue to monitor that system for any unusual variance, you can maintain the alarm system in a highly usable and optimized state. You will need to document this design, and track it just the same way you document and track any good design. But I would warn you against wasting time to document an existing bad design. Proper documentation allows you to maintain original design concepts and logic of why alarms are placed where they are. Furthermore, it allows future engineers to track and change that logic as the process undergoes changes, or as process understanding exceeds what was preceived during initial design.

Utilizing ths approach, alarms are rationalized the way we did them in the days of yore- one at a time. How does that happen you might ask? New alarms surface as a process problem that caused an unexpected alarm to exhibit itself. Alternatively they may come as a request for a new alarm or configuration. Recall that in the old days, we "rationalized" each alarm as it came into existence, because we were limited to panel real estate, and cost of wire runs, etc. If it did not meet the criteria for availability, it was not approved for service. We need to re-establish that same rule set.

In today's systems, if you have a good alarm system design in place, and you have good management of change procedures, then a request for a new alarm should be treated in the same way. Each must be rationalized by the requestor as being necessary to the operation of the unit. It must then be approved for service and recognized as being beneficial to operations for the potential load it may add. Belive me when I tell you that this process reduces the number of requests for new alarms.

If you do not start this with a good design, then a bad design will just need bad fixes. And you may become subject to a decision to call in subject matter experts (SME's) to fix the problem. This often leads to the further discovery that so-called experts are only experts in the subject- they are not necessarily experts in your process. Suddenly, you experience what we used to refer to as “Matters to which Experts are Severely Suspect” otherwise known as a M.E.S.S.

This is the reason that the lifecycle of alarm management takes precedent over rationalization. Rationalization as a project is only necessary if you have not properly DESIGNed you alarm system to start with, and then not properly MONITORed and AUDITed its ongoing lifecycle. Note that the emphasis on these words is to point out that each is a part of the lifecycle. If you properly retain the knowledge of your operators and engineers, and the reasons for why certain alarms were necessary (or not necessary), then the historical record assists any future efforts. Others can simply review the alarm configuration to arrive at conclusions based on that past knowledge.

At this point in time in the history of alarm management, many people are talking about rationalization because so many of the systems lacked any adherence to these elements. If and when all systems have been through a proper redesign exercise, then adherence to the lifecycle will take precedence, as you should never need to undertake a rationalization project. It is for that reason that you should be careful of purchasing software packages that are totally focused on a rationalization project workflow. Their value diminishes quickly once rationlaization exercises have been completed.

This is the reason that TiPS LogMate was designed to address the entire alarm management lifecycle. We want to help you get your alarm system in order and keep it that way. Whether you need to rationalize, and want to use the automated tools built into LogMate to help you do so, or you want to monitor and maintain the system, we've got your backside.

Think about it. Opt for a good design, and you will gain the benefit for years to come.
If you are curious about the ISA18 Standard, contact us.

Searching for the momentous- making it right

Steve Apple — Tue, 04 Nov 2008 21:10:00 GMT

I'm not sure if you had time to attend the recent ISA show in Houston. I want to highlight a couple of technical presentations that were given there.

These two papers were presented by Eli Lilly, and showed a vision of alarm management that has been absent from a lot of the papers and presentations you will see given on the topic. In case you've missed a lot of the rhetoric lately on this topic, you can find it quickly by doing some internet searches. In a nutshell- most of the papers written focus around reducing alarms. I will tell you that is not the answer. It is certainly desirable when there are too many alarms, but once reduced- you still have an alarm problem. So, let's look at it through the eyes of two who have a different viewpoint.

The first was by Ryan I.- a young engineer at Lilly. Ryan was working in a unit that was replacing an older control system with a new one. This offered him the unique opportunity to set up the alarm system. In most plants that usually means that you put the alarms back the way they were so you don't confuse the operators when it comes on line. Ryan's hallmark statement on alarm system design was "I figured that it cost the same amount of time and money to design it wrong as it did to design it right. So I thought- why not just design it right to start with?" Stepping off that platform, Ryan took the opportunity to review previous alarm data- after all they weren't changing the process- just the system- and use that information to decide where alarms were hindering rather than helping the operator, and to fix that design.

Throughout his presentation, Ryan pointed to findings as he investigated why the alarm system worked the way it did, and things that they did to resolve issues. He pointed to the fact that this alarm system now works so well that when there's an alarm- the operators want to know why. And the operators have become the keepers of the alarm system. In fact, any alarm is a rarity. He quoted a new normal alarm rate of ten per day!

A later paper by Alan P. of Lilly expanded the thought process further. Alan's premise was that if you are only documenting the alarm system, and trying to reduce the number of alarms, you are missing the opportunity to find process discoveries. He called this striving for the momentous. He further denounced "trivializing the momentous, and complicating the obvious." Alan's greatest observation was that in normal operation, (once the alarm system was in order) that each alarm was an opportunity for process discovery.

This is a phenomenal breakthrough in alarm management thinking. While most of the efforts today assume that every alarm system is broken and needs a savior to come to the rescue (on a white horse?), Alan's observation takes us to the next phase of engineering thought on the subject. That level is the one we arrive at when we realize that we've fixed all the alarms, and we still have an alarm problem. In fact- we realize that every alarm is essentially a problem. Thus, our real goal should be NO ALARMS! That sure knocks the legs out from EEMUA's one alarm every ten minutes theory.

So, let's drift into that thought pattern for a minute- because a walk through history reveals that we were getting close to being there at one time, and decided that to go any further required new technology. The problem we identified was that we would need one of those new digital control systems. Our reasoning (seemed sound at the time) was that digital information transfer would resolve all of our instrumentation and communication issues and allow us to take on difficult process issues without the inherent shortcomings of analog instruments.

From an alarming standpoint, that's why we put alarm logging printers on those systems. We wanted to log each alarm, and use that information to investigate just why it happened. Once we knew that, we could prevent it from ever happening again. Alarms would be VALUABLE to our system. The vision was that we would make adjustments to the instruments, the system, and the process so that such alarms would soon disappear. Alternatively, we might discover a process anomaly that required us to use real engineering to figure out how better to produce the product, or how to avoid allowing the process to get into that particular state of upset ever again. And we wouldn't have to tweak potentiometers to make that happen. We'd just write new equations, and setpoints to the digital system.

I'm sure that many of you old-timers can recall having similar thoughts in the past. So, perhaps it's not new- just able to be considered once again- twenty years after the fact of its having been a pertinent thought.

Here's the key- if you're going to do some work on your alarm system, and you decide that a rationalization is what is called for- don't just document the current system. Documentation of a broken alarm system is worthless. Take the opportunity to fix it right. OK- it does cost a little more, because you actually have to look at the design of the unit. But the pay off is immediate, and creates a better work environment.

Look for the momentous opportunity when you are given it. Make your activities produce value. Otherwise, you'll be like a lot of others who wondered what you received for the flurry of activity that took place when you spent all that money and time. Hemmingway said it best: Don't mistake movement for action.

The long-lost art of true Alarm Management-- Forensics

Steve Apple — Tue, 02 Sep 2008 22:19:00 GMT

Over the past few years, the initial reason for alarm management has been hijacked. Lately, its been all about rationalization, and fixing the poor design. Bad designs have creeped up on us so numerously, that all efforts these days seem to be centered around fixing these bad designs. Some find that they can simply find all the nuisance alarms and ferret them out, and they're to a stable point. Others find that they need to sit down with the configuration and rationalize the system completely.

What this all hides is the fact that alarm management systems- originally alarm logging printers- were first designed to help us figure out what's going wrong in our processes, and use that information to fix the things that are wrong WITH THE PROCESS. That is as opposed to fixing things that are wrong with the alarms system. Which would you prefer to do?

At TiPS, many of our best customers- even when they are rationalizing units with the automated features of the LogMate (saving thousands on manpower), are still using the softwre to figure out what happened when something goes wrong. That is because the software was designed with a rich set of operator-specific tools to help figure out what happened. In fact, we commonly hear that the operations managers first question is "what does LogMate say?" when there is an incident.

The reason why we designed control systems with an alarm printer was so that we could tear that sheet off the printer, and bring it to the console where we could compare the alarm activity to real-time results. That allowed us to decipher what went wrong, and how. These old printers, when replace with a modern real-time database, can yield instantaneous results. That's one of the reasons why, unlike our competitors, TiPS has maintained a real-time databasing schema. This yileds the ability to instantly examine any alarm scenario, and decide what was the sequence of events. With proper information, operations can decide how best to avoid that situation in the future- sometimes through alarm settings, but often through process changes. Our recent news letter talks about the notion of using your alarm management system to resolve operations problems.

So- the deep dark secret of alarm management is that some companies have (properly) identified that most alarm systems in use today are seriously flawed. There's been an awful lot of talk about how to fix them, and how to rectify the problems these flawed systems can lead to. Yes, you need to use LogMate to fix your system- as a full featured lifecycle based software, it will do that for you.

However, let's put ourselves five years into the future. You've gotten rid of all the nuisance alarms. You've documented and rationalized all alarms in the system, so that you have a full understanding of what each alarm does. You spent over 5 Million Dollars doing it with some company that wrote a book. But, YOU STILL HAVE ALARM PROBLEMS. Management is pushing- when will the bleeding end?

Well, that's what alarms are. Every alarm is a problem. And keeping alarms from happening AT ALL is the only solution. Each alarm says that I (the control engineer) have not been able to automate the system in such a way that human intervention was not required. Otherwise, the system would handle every upset automatically. I am reminded of my old friend Warren Thompson who used to talk of the control room of the future. He said it would be a man and a dog. The man was there to feed the dog, and the dog was there to bite the man if he touched the control system.

Now, maybe that's a bit much, because some types of automation are simply too expensive either initially, or from a maintenance standpoint that a human is still the cheapest solution. We will not be fully automated until we have truly commoditized control to a no-brainer cost. So, we have to accept that we will live in a world with alarms. And we have to accept as well that we need to have operators trained to know what to do when things don't work the way they should. This means we need to give them every bit of information that is possible to resolve issues and in a manner that they can use it. One of these bits of information is the old alarm logging printer. Yes, it is dead, and now replaced by a SQL file on a computer, but its value to the system still survives.

So- let me make a suggestion. If you're going to hire a company to do a rationalization project for you, and they also happen to have a package that does it, then let them use it, but don't keep it installed. If it is really good at what it does, you should be able to use it, and then throw it out and never need it again, right? Then fall back to the good old LogMate that everybody knows and loves for your day-to-day operations.

Now- if you're smart enough to start with LogMate- you're already ahead of the game. Here's an example. Suppose you've been using LogMate for years to monitor your alarm system, and perform exactly what I've discussed- forensic analysis. Now you have a leg up on alarm management activities. LogMate has a database of alarms that you can use to analyze the system. Simply send that database off to TiPS- they'll put it up on their server,and let you analyze the situation with your alarm system. Most of the alarm management efforts fall on their face if they do not have a good long-term database. This is the reason for a lot of failures we have seen. A vendor arrives on site, and downloads you alarm journal. It reflects on ly the activity you have seen over the past few weeks. And they then try to fix your alarm system for how it will run next year. You need a decent database, with lots of modes of operation to be fully representative. LogMate provides that.

Intall LogMate first. You'll be ahead of the game, and your operators will thank you for the forensic visibility it gives them into the process. After you've cleaned up all the alarms it will again revert to that usage, as well as keeping automated tabs on your alarm system.

Buying American

Steve Apple — Fri, 22 Aug 2008 20:09:00 GMT

I bought a Harley Davidson about a month ago. I've always wanted a Harley- since I was about 3 years old. My mom could tell you of the tape marks on my walls from motorcycle pictures. Now, don't blame me for it, but I always wanted a Husqvarna 450 as well, I just never felt I was up to spending that much money on something that I wasn't enough rider to make use of. And my motocrossing days are long since over. And I always wanted Corvettes. I've had a few of those, and they're great machines. The classic American, I guess. Wish I'd kept a few of them that I once owned.. that's a different story.

The thing is, I felt good about it. I felt good about buying an American product that is unquestionably better than anything out there (No e-mail please- this is entirely personal opinion). I felt good about giving money to a local business that employs local people, and helps out a lot in the community. I felt good about not sending my money to a company overseas.

Those of you who know me, know I'm a big buy-USA advocate. I own a few Chevy's- my Suburban now has 180,000 miles on it, and has only had the water pump and the alternator replaced since new, along with regular oil changes. When I worked at Beech Aircraft a long time ago, I remember asking a fellow worker why he bought a Japanese car. He told me it was because American workers were overpaid, and lazy, and didn't make good products, and didn't deserve his money. He said this while he was leaning back in his chair with his feet on his desk. I asked him what he was going to do when the Japanese used the investment money he gave them to make the products he made, and put him out of a job by hiring workers that worked harder than him for less money.

I remember in the 70's all of the auto magazines (whose pictures added more tape marks to my wall) started telling us that American products were not as good as their foreign competitors. Some people believed them and invested internationally. We lost jobs. Yet, despite all the carrying on, the majority of vehicles from that period still on the road today are American Iron, unless they're a piece of artwork like a Ferrari that never gets driven (some American iron is approaching this category lately as well).

And the press got in on the game. Now it seems there's a group out there who thinks that everything American is bad. According to them, our businesses are all wrong- we're all money grabbers who pollute the atmosphere at the public's expense. Having traveled all over the world, I can tell you that we're the cleanest nation I have seen- perhaps with the exception of Australia who doesn't have enough manufacturing density to be compared.

Throughout all of this, I have maintained my faith in American workers. I know wer'e better than they say, becasue we have some of the best and brightest that came here from all over the world. America truly is a melting pot, and we have both native, and foreign-born brilliant people who make it great.

So, I'm going to continue to buy American every chance I get. Even if it's more costly to do so in some instances. If I dont' keep my next door neighbor hired, then he will not be able to keep me hired.

Besides that fact, I think that good old American ingenuity makes some of the best products and the most reliable in the world. That goes for software as well. I trust in the American work ethic, and I trust in the inventiveness of Americans to make something right for me.

Buy something American today. If you need some good Alarm Management software, there's only one all-American company in the business. And I promise you will be happy with your purchase. Furthermore, I can promise that none of your investment money will be sent overseas to compete with you. I'll use it to buy the products you make, and the products your neighbors make. I'll keep the investment circle going to be sure that you are employed. I can't speak for everybody else, but that's my promise to you. I can't keep your companies afloat on my investment in you alone, but, if everybody else takes a note from my page, the US could kick its economic worries aside.

As the World Turns- Quality is alway eventually recognized

Steve Apple — Thu, 14 Aug 2008 16:47:00 GMT

No, the title of this Blog isn't intended to invoke a daily soap opera, though some of the actions that led up to its writing may play out that way.

Lately, TiPS has been receiving a lot of inquiries from clients who purchased another competitor's alarm management solution. Of course, in some cases it is blamed on a predecessor. When they purchased the solution, they were given the demo that showed all of the things they could do with it, and they were "wowed". Of course, they bought enough services with the product to ensure they'd get it to work anyway, so they weren't concerned. What is now happening is that as the products age, they're starting to sense the true Total Cost of Ownership. Note that I wrote a warning blog about these issues previously. http://tipstalk.tipsweb.com/2007/05/14/total-cost-of-ownership-tco.aspx That is why they are coming back to TiPS, who did not try to "wow" them, but told them simple truths and designed easy to use products that worked year after year, and upgraded easily.

They're finding several interesting things,:

Case 1, Vendor 1
- One vendor simply sold them modified Excel spreadsheets. The attraction was that spreadsheets are known by everybody, and are easy to modify for your own purposes. At least, that was the promise. Now, they're realizing that somebody else's spreadsheet is hard to modify, and spreadsheets are not commercially supportable, especially if you actually do try to modify them. Now every upgrade of new revisions becomes a nightmare. The interesting part about this vendor is that they will claim the product can do ANYTHING any competitors can do. OF COURSE IT CAN! IT"S A SPREADSHEET, for gosh sake. Heck, I can even write multivariable controllers in spreadsheets. But you wouldn't want me deploying them in your plant. I hear they get away with that stuff up in Canada. Perhaps that explains it.

- Next, you try to do more advanced things with your application. They sell you add-ins. You find that the modifications you've done to the spreadsheet don't work with their add-ins. They use something called pivot tables. For the weak of heart, I will avoid explaining what those are. You pay them to come out and try to fix it. What the heck, its a spreadsheet- it can do anything. At the same time, you also order their web browser window, and discover its an add-on that has to be loaded to each machine (not exactly thin client?). And it only allows access to specific things, and a subset of preset analysis options. And it certainly doesn't access the things you'e added yourself. Bummer. However, they've offered to come on site and do some custom programming... but they do offer 24 hour support lines out of India.

- You then decide to go full boat with their management of change, and find that the cost of it was more than expected. But you have to have it to do anything more with the product and get back on track. So, you sink the bucks and find --- you guessed it --- another Excel add-on. Why didn't you just save a lot of money and write your own Excel application to do this? Or hire somebody to do it on contract. You'd have been money ahead, and you'd have owned the design so you could modify it. Man, do you feel taken.

Case 2, Vendor 2
- Another vendor sold them on their approach to alarm management, and sold them the book, the software and the services- hook line and sinker as dad says. So, they sold the software, and came to their site to use it to make it solve their alarm management problems. After a lengthy install, it seemed to be working. Because they were using it for you, they used it to do only the things it was capable of, and they were able to hide any faults. If something screwed up while they were there, they shipped out a quick-fix revision. That impressed the customer actually. Little did he know that he now had a one-off untested version of a software release soon to cause an upgrade nightmare. They did their job- rationalized the alarms, and left the site. After departure, things didn't seem right. The rationalization was all wrong. They'd only rationalized a small percentage of the I/O. The software didn't do a lot of things expected. You didn't know what you received for the massive investment. But they were willing to charge you more to fix it. And it kept crashing- losing data. All of this was hidden while they were on site babysitting the stuff. Now you owned it. But your management felt good about it, so you didn't make noise about the problems, hoping that a new problem would arise, and this one would be ignored. Who wants to be accused of making the wrong decision?

- You decided to try to use their "still beta" application to do dynamic alarming. And you buy it as an add-on. You find that its actually a whole separate product, as their base application cannot deal with real-time alarm data, since it only batches alarm data on a scheduled basis. As you investigate further, you find that the application is just a dumb state analysis tool that changes configurations on the fly. You start realizing that you could have done that inside the DCS and saved a lot of trouble. And Money. The first time you use it, it corrupts your DCS configuration database. You pay them some more money, and they come out and fix it so that won't happen again. It then gets hung up in the middle of a state change, and alters some settings while leaving others untouched. Realizing the danger of that should you be in a critical condition, you side-line the product, and institute some simple state diagnostics in side the DCS- where they can be more easily tracked.

- After all you have spent with this company, you still wonder what you have got. You did everything they recommended. You still have a big problem. And your management is starting to ask things like "What did we get for all that money?"All you can any is that you got a book, and a method that now looks questionable, because it didn't deal with a full lifecycle approach to alarm management. In fact, now that you've blown your budget in that area, you may not have enough left to fix the true problems that all of their busy work was masking. You hear they're opening up a new office in the Middle East with your money, so they can find unknowing victims.

Vendor 3 (TiPS)
- Some bought a TiPS product. The installation was done in about one day, and it was gathering data on everything from the newest DCS to the older legacy systems. They even delivered updated filters, and extraction routines to match specific alterations you had made in your control system. Yes, there may have been little glitches here and there. But you found that rather than ship a quick fix, TiPS worked diligently on the problem and sent you a fixed solution in a new release once it was totally resolved, and integrated into the complete release. Meanwhile, the product was busily gathering data to be certain no information was lost, and still yielding valuable results. They didn't try for a one-off fix. And now, when you upgrade, it's just like any other customer. Your release works the same as everybody else's. You downloaded it from their FTP site, and it installed quickly and worked immediately. You're happy that you only have to upgrade one version at the server. The true thin-client saves you from having to upgrade all clients as well.

- You start to realize that everything you need to do for alarm management is integrated into the product. And its all available via a web browser- even remotely if you have proper securities in place. It can't calculate your monthly payments on your new motorcycle, because it doesn't have a spreadsheet built in, but it is purpose-designed with a work flow to resolve alarm management lifecycle issues as you face them.

- You invoke the Management of Change (TiPS Calls it their ALarm KB- for Knowledge Base, and it's built into their analysis tool- ACE). You realize that it downloads your configuration from your DCS with a few clicks, and a rather long wait. But its all there. It is linked to the dynamic database, so that configuration and KB information is available to operators in real-time, and it has a KB template that allows you to enter other knowledge about the tags and alarms. You can even allow useful documents to be linked to it, for operator viewing. You can allow the operators to add notes. Wow- its not a spreadsheet- but it does everything you need to do in alarm management and meet EEMUA guidelines and other specifications. And when you want, it exports anything wanted to a spreadsheet, or a specific web page report built to meet your needs. It can kick out HTML, Excel, Web pages, whatever you want. And it all works out of the box because it was all rigorously tested before you ever saw it.

- You decide to do more. They introduce you to UReason (www.ureason.com) for dynamic alarming. You realize their offering is truly unique, as it deals with alarms that are coming to the operators in real time. You begin a path of successfully handling dynamic alarming, and providing better advisory information to the operator. They introduce you to UCDS (www.mycontrolroom.com). UCDS helps you to get into a plan to resolve deeper seeded control room issues. WIth UCDS, you lay out a plan forward to resolve the issues that lead to poor alarms, and start you down the path of true operator Situation Awareness. You are beginning to realize that alarms are only part of the problem- its really an operations effectiveness issue, and alarms are just an indicator. Together with Mustang Engineering (www.mustangeng.com) you start to resolve control room issues. On issues where you have available manpower, you take care of it. When it causes a bump in resource availability, you call Mustang in to assist, because you know you can trust them to follow up to see the job is done correctly. Because now, you are educated by the best in the world, and you know what needs to be done.

- One year later you are promoted. You managers realize that you know how to find the right solution to their problems, and they want you to replicate this success all over the company. They realize you do not frivolously spend money on things that seem to have questionable value after the fact. You realize it all started with LogMate, and those guys at Tips who were willing to introduce you to people who were the leaders in the industry, rather than trying to send a so-called expert of their own making. You send Steve Maddox a thank you e-mail.

Note: As I wrote this, I noted that one of these companies is advertising to hire a person they hope to send to you as their self-made expert on these issues. Trust TiPS that we will stick to software, and bring the true experts to you when you need them.
---------------------------
Yes, there's no software that's perfect, but clients are finding out that true commercial software is easy to use, and purpose-built for the problem which it is to resolve. It is tested rigorously to internal specifications that meet quality standards even of pharmaceutical companies which perform regular quality audits.

GOOD COMMERCIAL QUALITY SOFTWARE differs greatly from modified Excel spreadsheets. It differs greatly from software built to support a service offering. It is something you should expect to be able to install, use, and maintain yourself, and it should cause very little downstream conflict after installing. If there are problems, the revisions should be tested and included in a regular release cycle, rather than just shipped to you willy nilly. And once the vendor has left site, you should be able to do everything you expected without having to pay extra for each fix.

That's the way its done at TiPS, and we are happy to welcome clients who are coming to us after having tried the "wow" stuff they were promised, but never delivered.

Just as a side note- we don't even recommend their books. We view them as just large, costly marketing brochures advocating a questionable approach to alarm management.

Alarm Management versus Procedure Enforcement

Steve Apple — Fri, 18 Jul 2008 21:55:00 GMT

Procedure enforcement is the holy grail of operations of the future. In today's market we are seeing expertise disappear on a daily basis. Fewer college grads choose the automation market. Regulations and competition push us to have less people in the plant. The EPA discourages us from making products that require carbon-based fossil energy consumption. The government makes onerous requirements for any employer silly enough to actually employ people. Training of personnel is an afterthought. Personal legal responsibility is being assigned to people who are only doing their jobs. And systems continue to become more complex.

Frankly, I'd be scared half to death to be an operator in a plant today. And management knows this. So, how do we continue to run ever more complex manufacturing systems given such a dilema? The real answer lies in procedure enforcement, and today's Process Automation vendors are in a postion where we need them to arise to meet this requirement. The unfortunate part of that statement is that few automation vendors would have the expertise to understand ALL of the processes that need to be procedurally enforced, and I'm not sure they are the proper source for that anyway. But just as they provided control engineers with the platform that we needed to institute process control, and then advanced process control, they now need to meet this need.

What it will mean is that they need to make a systems that STOP alarming operators unless they can also give information on the next step that needs to be taken. It is a major shortcoming that there is no control over the steps an operator can take after receiving an alarm. The same operator can handle the same alarm a different way each time. The problem is that the next step is not always something that can be written by prescription. Sometimes, it requires investigation, and further input from the operator. In this way, the DCS becomes a true guidance system for the operator. It must be accessible by a number of vendors who understand their equipment, systems, and procedures to insert the proper logic necessary to make a system work.

We know this can be done. We have seen it done. It was a promise made to us by Expert Systems, Neural Networks, and Fuzzy Logic systems in the 90's. The promise was not followed up on, mainly due to the high maintenance component of such systems. But that maintenance component was there simply because the DCS systems of the day did not lend themsleves readily to logic-based guidance systems. Thus each of these types of technologies was subject to Human-Machine interactions designed by people who had very little experience in the systems on which it would need to be deployed. It created a mis-match, and training requirement that was hard to undertake.

In my past I have seen sever such systems successfully deployed. The unfortunate part of their deployment was that since they were not done on mainstream software, for which there was a ready supply of trained individuals, there was little chance ithey would continue operating correctly once the original designers left the building. This can be remedied by incorporating some of the inventions of those systems into the basic control technology employed in the plant. Just like modern automobiles that employ systems that you could once only buy at an add-on shop. I believe we are seeing some of that today. Yesterday's third-party option is slowly becoming todays off-the-shelf offering.

This is where automation vendors need to be focusing their attention. Give us systems that allows us to put our process knowledge into the guidance the operator will receive. Let us automate when necessary, calculate when necessary, ask for input when necessary, and call on the human when all else fails. And lastly, let us tell that human how to go about remedying the situation when he is called on to do so. We'd like to reserve alarms for situations where's there's been a breach in the outer hull. Alarm management will still have its place- it will be the external monitor that audits with checks and balances on the alarm system to be certain it does not invoke human action unnecessarily.

Incomplete rationalization?

Steve Apple — Mon, 07 Jul 2008 17:55:00 GMT

Lately, we've been confused when we receive what we are told is a rationalized database. Those of you familiar with TiPS LogMate package (http://www.tipsweb.com/products/logmate/) know that we create a KB (that stands for knowledgebase) that contains all of the configuration information and design parameters surrounding alarms.

This KB is important, in that it captures all of the alarm and tag configuration, including information about cause, consequence, corrective action, trip points, time to resolve, and even links to documentation about standard procedures, etc. It becomes your alarm datamart repository. A datamart is a subset of a data warehouse. It is set aside and optimized specifically for the use of alarm management, and the various methodologies surrounding the professional practice in treatment of alarm management issues.

As such, this rationalized configuration should be rather complete once somebody has spent over $100k with a service company who writes books on alarm management. Unfortunately, that is far from the truth. Our experiences show that on many occasions, less than 10% of the configuration has been completely rationalized and documented. Furthermore, it is usually contained in a spreadsheet that is not current with the prevailing configuration, as it seems to be delivered months after the fact. How does a company stay in business delivering this level of quality workmanship?

The only answer we can come up with is that the general user community is not really aware of what they should expect as a deliverable when they hire a service provider to do this for them. Realizing this, our Chris Wilson undertook the task of writing a white paper a couple of months back on just what you should expect. This whitepaper is available through Control Global white paper releases. It is entitled the "Alarm Rationalization Quality Assurance Guide". You can find it at http://www.controlglobal.com/wp_downloads/TiPS.html.

In numerous conversations with people from the industry, we heard comments of "we've spent lots of money having this done, but we're not sure what we received for our money". When we saw the results, and tried to enter them into our KB, we understood why. Download this paper, and ask for a full quality deliverable when you want a rationalization done on your process units. There are several other companies out there who deliver a complete job. One is Mustang Engineering. They have some of the worlds leading experts on staff. We have also seen impressive work from Invensys Services group. They will supply this work as part of an ongoing service contract. Each of these companies gives a complete delivery of a fully rationalized configuration.

Why is a fully rationalized KB important? The first and most obvious reason is that the documentation is priceless. Not only does it help to have that information available to your operators online, but when regulations come (and they will) their requirement will be more for documentation than anything else. Get things documented now, and you will be ahead of that game.

The second major reason for a fully rationalized system is that not only do you learn a lot in the process, but you identify areas for opportunity. One of those opportunities is dynamic alarming, and operator advisory systems. To see where you can go with this, check out our partners at UReason (www.ureason.com). Their OASYS system adds directly on top of LogMate, and makes use of that rationalized information to dynamically control alarm information that is routed to the operator.

Note that this is more than just a dumb system that sets up a bunch of plant states. It actually advises the operator in times of trouble, and guides his efforts in resolving situations. It does this in three ways. Firstly, it identifies commonly occurring alarm groups, and reduces the overflow of information (thus reducing alarm floods). Secondly, it shows causal connections for events that occur, and reveals information which can help the operator to avoid situations before they ever occur. Thus, avoiding the types of incidents that cause or result from alarm floods. Many who use it see it as the flood avoidance package of the future. Flood avoidance is the holy grail of alarm management.

Lastly, the resultant datamining effort reveals areas of your process that can benefit from additional efforts. Once you are at that point, you are starting to address the full situation awareness issues in your control room. You will soon realize that alarms are only an indicator, not the problem. To learn more on this subject, see www.mycontrolroom.com Alarms are a great place to go to find out where your true automation issues lie. This data yields all of the cost/benefit analysis necessary to justify automation upgrades, should they be called for.

If you think you could benefit from such a system, call us, and we'll have a UReason discussion with you.

In the meantime, be careful to see that you identify what you want for a deliverable for your alarm management efforts. Download Chris's white paper from Control Global to get a good start.

Regulatory confusion, fear uncertainty and doubt (FUD)

Steve Apple — Thu, 26 Jun 2008 23:04:00 GMT

In my blog, I always try to relate to you things I am seeing and hearing in the marketplace that have an effect on how you go about resolving alarm management issues. I've lately been amused by the efforts of vendors to scare people with their interpretation of upcoming regulatory requirements.

There's a lot of FUD being distributed out there. Sales people use FUD as a short term for Fear, Uncertainty, and Doubt. FUD is a sales tactic popularized in the 70's to help non-technical sales people sell technical items. It is employed to try to frighten you into buying a product out of fear of cleverly described consequences.

In this case, it's a perceived need to meet a looming regulatory situation that may or may not exist. Lately, I have been bombarded with questions about these looming regulatory issues, and some competitors have begun to sow the seeds of doubt (D) about whether you will be able to meet these regulatory requirements. Canadian vendors tend to do that. They must have a lack of regulations up there in the tundra. It's always exciting to a vendor when they believe you might be required to buy their product by legal mandate. And at this point in time, there are several vendors lurking and hoping that you will be forced to buy something from them soon. So, without selling the issue short, I will try to allay those fears, and tell you what might be coming, and how you should prepare to deal with it.

Firstly, most government regulations follow the guidelines of "best practices" or "most available" technology. There is a term often overused called RAGAGEP. It stands for Recognized And Generally Accepted Good Engineering Practice. This is what the government expects of you- nothing more and nothing less. OK- sometimes they stretch a bit and get into practicing some "political" science that doesn't necessarily conform to thermodynamic principles, but that's it for the most part. So, assuming I'm talking to good engineers, I will continue with what that might mean in the area of alarm management, with a light on previous regulations.

I will start by highlighting some past regulations with which we should all be familiar. Let's look at drug testing. The government did not tell us HOW to do drug testing when they started requiring its presence in the workplace. Instead, they said essentially "make a plan" and "work that plan". And here are some guidelines about what we think works. So, at small companies that sell software, they adopted the policy that if somebody exhibited abnormal behavior, they would send them for drug testing. I used to have a sales manager who joked that if they did any more, they would chance losing half their staff, and perhaps would never be able to hire.

At the other end of the spectrum are companies who performed highly dangerous jobs which involved the public. They instituted a drug screen on hiring, and regular drug screenings throughout your work life. They had more at stake should an employee be at the root of a drug-related incident.

Similar to this has been environmental regulations. If you were running a plant in the middle of a highly populated area that could put out phosgene gas, or fuming nitric acid, or some other dangerous thing (like Carbon Dioxide?), then they wanted continuous monitoring of your sources to see that you would not cause a meltdown. If you were operating an 8-cylinder compression station in the middle of nowhere, perhaps you could be ignored (at least in the first round of regulations). And essentially, your need to measure was regulated by whether you were in an attainment, or non-attainment region. Yes, that had a bit of political science behind it, so we'll ignore that one.

From these, I think you can see a regulatory pattern. Alarm overload has proven to be responsible for contributing to major industrial incidents. Alarm overload needs to be watched more closely in situations where it might lead to an incident which costs human life, or health, or contributes to environmental waste. Do you operate a plant producing highly volatile chemical compounds in the middle of a municipal environment? If so, then you will need to keep a closer eye on your alarms. Do you operate a plant that may only cost you money if it shuts down? In this case, your observance of alarm management may be dependant on self-protection from capital loss. And there will be everything in between.

At a minimum, companies will be expected to quantify alarm issuance as a function of operator loading. They will be expected to show how they monitor those loads, and how they mitigate issues should they arise. There are a myriad of ways to slice that problem up. In some cases, this may mean simply counting the alarms, and issuing a report. In others, it may mean a full-blown alarm management effort with complete rationalization of processes for cause, consequence and corrective action on each alarm. It may mean all alarm priorities need to be properly set, and the alarm philosophy is available for review by inspectors should they come to your site.

I think you will be able to tell which one of these categories you operations will most likely fall into. Mostly, its just a matter of common sense. If you need a guide, examine your environmental category, and the relative amount of required regulation should be approximately on par with where you stand there compared to your peers.

In any case, as a blatant commercial statement, TIPS stands ready to serve you wherever you stand in this broad spectrum of regulatory issues. We adhere tightly to the ISA SP-18 concept that alarm management is a lifecycle issue, not just a project where you rationalize alarms and leave. We offer a full-range product suite that allows everything from simple alarm logging to full-blown alarm resolution and automated regulatory reporting systems. Call us and we'll help you get something in place that will address these issues when they begin to take effect.

I would sugest you get started with alarm logging at the very least and see where that leads you. It is not expensive, and the data it produces can prove extremely valuable for both alarm reduction efforts, and process upset forensic analysis. In any case, it will provide you the precursor data for any regulatory requirements that might be enacted down the line. It will inform you just how much work you might have ahead of you when regulations actually come into existence.

If you would like an accounting of the existing regulations, and what they might mean to you, contact us directly and we will give you our latest listing of where regulatory issues currently stand.

Making Alarm Management Myths by using DMAIC

Steve Apple — Mon, 02 Jun 2008 21:59:00 GMT

A friend sent me a blog posting recently where somebody was trying to allay so-called "myths" of alarm management. I once attended a sales seminar on these types of tactics they called land mines. Like the kid on the playground, the idea was to call you over to the side and mention all the bad things you might hear before somebody else pointed them out, and tell you why they were false. In this way, if another person mentioned them, it blew up in his face by the now pre-educated recipient. I don't encourage this type of practice from our own sales people. We usually give most of our clients the benefit of a doubt in recognizing such sophomoric attempts at self-aggrandizement.

The first "myth" this blogger attempts to expose is: Alarm management is all about counting alarms.

The misdirection is purposeful, and attempts to misinterpret for the reader's benefit the inherent value of logging alarms. It's an easy thing to do when you have a great shortcoming in that area. So I will attempt to clarify- What is being done by alarm logging software is collecting and measuring data-Nothing more. But it is a necessary step to a properly controlled alarm system, and only one part of the life cycle of proper management of an alarm system.

Six sigma concepts recognize that data collection is a precursory step to control of a process. That is why we at TiPS put such an emphasis on that step. If you don't get the data right, it is virtually assured the rest of the process will be more dysfunctional. DMAIC: Define, Measure, Analyze, Improve, Control. I refer you to recent issues of Control magazine where you will see a story by a Monsanto plant where they did exactly that process. See this two-part story at http://www.controlglobal.com/articles/2008/051.html and http://www.controlglobal.com/articles/2008/087.html . In this analysis, the engineer- Brent Thomas- follows the six sigma process to show how an alarm management project is undertaken in a logical and intelligent approach. Whether you're a fan of the six sigma process or not, all engineers who have any scientific understanding know that the six sigma process has its roots in what we in the US learned in grade school as the scientific method.

At Pavilion Technologies, we used to say "In God we trust- all others bring data". And Pavilion made a slogan of the saying "Turning your Data into Gold". What we understood was that in order to control anything you first have to measure it. Our above-mentioned blogger would have us believe that the proper approach is to just go in and re-design the alarm system without ever having first quantified the problem. That's the "M" in DMAIC. Collecting measurement data reveals much information:

- Firstly: Is the problem big enough to justify further efforts?
- If so, just how bad is it?
- Where and when does the problem most exhibit itself?
- Does the data reveal any other outstanding issues that were not initially considered?
- Where should I start my effort?
- What actions are the operators taking to resolve these issues on their own?
- Do I have all the data I need to analyze the problem?

What does the data yield? In Mr. Thomas's case, he shares that much of the problem actually was associated with bad data, and instrument messages. Had they tried to simply fix the alarm system, it would still be just as bad. That doesn't build respect from operators.

Once you have measured the problem, and resolved nuisance issues that stick out like a sore thumb- and these should be resolved just to clean the tables- you can set about making a plan for what you need to do to resolve the issues that made the alarm system get that bad to start with. What are some of those issues?

- Was the system poorly designed? If so- a rationalization may be in order so long as the cost of the rationalization effort does not outweigh the value it will deliver, or the risk it will alleviate.

- Do I have poor procedures for alarm management in place? If so, perhaps I need to update my alarm philosophy, and my procedure enforcement.

- Do things change constantly? If so, then I need to institute some sort of management of change procedures to stop the continuous alterations that affect performance, and eventually lead to incidents.

- Are alarms used to substitute for poor operator communication? These take the form of:

Poor graphics design
Poor inter-operator communications
Poor control room layout
Poor control panel layout
Operator training
etc.

Once you have analyzed the issues (the "A" in DMAIC) that led to a badly performing alarm system, you need to now institute the improvements (I) identified as necessary to resolve the current situation. These MAY come in the form of a rationalization exercise, if a system redesign is called for, or they may take other forms. Often, the information that this exercise yields allows you to make a prioritized list of improvements. If you need help in that area, we have several people who can help you get there.

In the long run, a rationalized system is extremely valuable. Many will tell you the effort itself is an education process and yields greatintrinsic value. However, such an effort runs the risk of dominating the scenery from a management viewpoint, and returning very little obvious value for its effort. In the future, regulatory issues may require a complete documentation of the system, in which case this may become a requirement, but other pressing issues should not go untreated at the expense of the effort.

In other words- use good common engineering judgment to preserve the capital to perform that task when it is absolutely required. Build respect from your management for wise use of the resources they put at your disposal. That will yield returns in the form of political capital that is necessary to succeed.

All roads lead to the logical last step of control (C). Put the proper controls in place to see that what you have fixed does not revert to its former broken state. The proper way to do this is to institute continuous monitoring (usually in the form of KPI's- Key Performance Indicators) that allow you to know when the system is getting out of control, and requires a revisit. Additional treatment is even easier, as all the methods, measurements, and tools are in place for quick updates. These ready measurements yield speedy- even automated- analysis and improvement.

The real secret here is that you don't want to look back at this problem in three years and find out you blew your entire capital on resolving only a part of the issue. Be aware that there are others who will try to convince you to take that path. Avoid that trap and make a few myths of your own.

2008 TiPS Expertune Technical User Conference

Steve Apple — Mon, 28 Apr 2008 21:23:00 GMT

For those of you who missed the 08 user conference, all we can say is that we are sorry. As usual, TiPS and Expertune took great measures to stay away from commercial issues (yes, there were parallel training tracks for those interested), and brought those who attended a wealth of information in the latest best practices in alarm management, process health monitoring, and loop tuning.

Entitled "Common Sense Automation" the conference kick off was from Leroy Chiao, a NASA Space shuttle pilot who spent six months on the space station, in addition to numerous shuttle missions. As a Chemical Engineer, he had a lot of experiences to tell about the practical automation that was necessary when you are sending a plant to outer space, where the choice to "shut it down" was not an option. An amateur photographer, it was worth attending the show if for nothing more than the photos he shared from his personal collection of shots from space.

Since I did not attend the Expertune sessions, I will limit my discussion to what was seen in alarm management. As usual, the theme from TiPS was that alarms are only part of the story in an overall Situation Awareness scenario. If you get stuck on solving alarm problems, you'll not have enough capital (both monetary and political) or energy left to resolve the real issues that alarm information can reveal to you.

I wish everybody could have been there to hear Ray Martinez describe his experiences at the control panel, and why they led him to resolve alarm issues in a very stern-yet understanding approach. You felt as if you were there with him when he described the conditions that led he and another operator to decide who had to open the control room door to decide if there was a chance to escape and save their lives. Today, Ray works in a plant that has NO alarms on the operator's screen most of the time. When an alarm happens, it is worked on immediately. The operators love this so much, they are enforcing it now. We're hoping Ray will contribute more information - perhaps a magazine article on this in the near future.

We heard Alan Phipps share the collective knowledge of his organization- a pharmaceutical company. They have been involved in alarm management issues for many years. Their conclusions showed that once you've cleaned up nuisance alarms, the number of alarms don't take on as much importance as the quality of alarms. Alan is digging deeper into the portions of the EEMUA 191 document that talk about the alarm quality index, and better metrics around that issue. Alan should be giving a flow-up presentation for the general public at this year's ISA conference in Houston, TX. You can register for that conference at www.isa.org.

The world's leading experts showed up again as well. Ian Nimmo gave a presentation on Situation Awareness, and the things that lead to incidents. People are starting to understand under Ian's tutelage that the problems indicated by alarms can be resolved, and there have been millions of dollars of research that have shown how to do so. For the average company that means you don't have to spend a lot of research money yourself. Hire Ian as a consultant, and he can quickly put you on the path of righteous resolution. I think we saw many converts at this conference. Gladly so, since it has been our continued mantra for the last four years. I encourage you to visit Ian's company's website for more papers and downloads at www.mycontrolroom.com.

Bridget Fitzpatrick (author of the ASM consortium guidelines for alarm management- precursor to EEMUA 191) gave a short workshop on alarm rationalization 101. She stressed the need for a good alarm management plan, and the steps to carry out that plan. Bridget is available via Mustang Engineering, where she is a consulting engineer. Those who have made use of her experience tell me that it is priceless. See more about Mustang Engineering at www.mustangeng.com.

Dave Strobhar- the godfather of alarm management (and he's not really very old) shared information on Human Performance factors, and how they influence an operator's ability to react to alarms. Dave started up a recent industry consortium entitled the Center for Operator Performance under the umbrella of Wright State University. Mark Nixon Of Emerson Process, who chairs that consortium gave us insight into its current projects, and efforts. See more about Dave's company (Beville Engineering) at www.beville.com. See more about the Center for Operator Performance at www.operatorperformance.org.

Doug Rothenberg gave us insights into the attitudes, and perceptions that prevent human use of technology, and how we can overcome these in our own daily efforts. Additionally, Doug showed us that the light at the end of the tunnel truly is delight,and there is a positive outcome to be expected if we continue to strive for the goal. See more about Doug's company, and his consulting services at www.d-roth.com.

These are highlights. There were many more. See our post-conference listing for copies of these presentations.

The last I will mention here is the discussion by Jules Oudmans of UReason. Jules gave the audience an appreciation of the value of dynamic alarming, process event data mining, and how those both contribute to higher level operator advisory systems. He showed some real-world cases, and indicated methods that can be used to resolve critical condition occurrences. Further, he gave the audience an appreciation of why this cannot be done with simple state-based tools. The UReason engine is an add-on to TiPS LogMate system. You can see more about it at www.ureason.com.

We hope you'll be able to attend in the future. Put a spot in your calendar and budget for sometime in April of next year. We can guarantee you won't regret the valuable information you receive from the world's leading experts in alarm management and critical condition management.

The Smell of Mendacity

Steve Apple — Mon, 28 Apr 2008 20:02:00 GMT

Faux pas- (Webster's): An embarrassing blunder that breaks a social convention, a gaffe, error of judgement, etc. Perhaps it should now be termed faux-PAS.

One of my favorite lines from a movie was in "Cat On a Hot Tin Roof". When his daughter-in-law Mae makes a positioning statement for his inheritance, the father Big Daddy (Burl Ives) asks the son (Paul Newman) a question. He says "Do you smell that, Brick? (Brick is his son's name.) That's the smell of mendacity." I had to look the word up the first time I heard it as a kid. I often recall this line when I witness some of the things vendors tell potential clients when attempting to position themselves for "inheritance". Most of you can guess just what odor he refers to. This turned out to be a faux-pas on the part of the daughter-in-law.

I'd like to share a recent faux pas we witnessed. It was like watching a train wreck. You just can't turn your head when its happening, though you probably should for your own future sanity.

Faux pas #1- A service vendor releases a 150 page hardbacked marketing brochure, and postions it as a best-seller non-fiction release with truths that have been until now kept under wraps. They even go so far as to claim that their book is "better than EEMUA". That's Mendacity. Further, the so-called book does not address alarm management as a lifecycle issue. Rather, it deals only with rationalization as if that were a means to the end of alarm managment issues. It is not.

I'm here to tell you right now: THERE IS NO DOCUMENT CURRENTLY AVAILABLE, EITHER PURCHASED, DOWNLOADED, OR STOLEN THAT ECLIPSES EEMUA 191 as an Alarm management reference. It is simply the pre-eminent authority on alarm management available today. All written alarm management works point to it.

Faux pas #2- Since that time, the same service vendor has tried to blur the lines of truth between that book, and another published with the ISA, and their adherance to and ownership of ISA "specifications". They even went so far as to announce the same at the ISA Expo in Houston, where it was almost immediately repudiated by the ISA, and any others involved in the establishment of a specification. Their goal is to spread what is referred to in the sales world as "FUD" or Fear, Uncertainty and Doubt. The hope is that they can make alarm management seem so complex as to require assistance from an outside consultant- perhaps one who wrote a book on part of it.

Faux pas #3- To befuddle this issue even more, they claim the existence of a so-called specification that only they can meet. Since no specification exists, they hope the FUD will convince you one is looming. Truth: The ISA is currently working on a specification, but to date none exists other than the SP-18.01 which deals with annunciator panels. I don't believe they can claim to address that specification since their book contains a specific chapter on the death of the annunciator panel.

Alarm management is neither a secret "void", proprietary to one company, nor is it rocket science. Most of the information needed to perform Alarm Management activities (the how-to) has been published in public documents. (see www.tipsweb.com/amlibrary.asp ) And, as stated, the EEMUA dcument gives a very concise view of the technical expectations.

So, buy a copy of the new EEMUA 191 Guide, and you will be 80% of the way there. It tells you how to measure, what to examine, and how to go about it. (Order one at www.eemua.co.uk )

If you're looking for a book on the subject, save your money until one comes out from a true expert on the subject- one who has no service sales axe to grind- one that addresses more than just the rationalization side of alarm management. Until that time, there is plenty available in the papers (www.tipsweb.com/amlibrary.asp ) and EEMUA (www.eemua.co.uk ) guidelines.

Mendacity stinks. Don't get it stuck to your shoes.

Rationalization Quality Assurance

Steve Apple — Wed, 30 May 2007 21:23:00 GMT

This month, TiPS is releasing for immediate consumption a white paper, and a spreadsheet which will be of enormous benefit to those who are thinking about rationalizing their alarm system. Why are we doing this instead of charging for it?

At the TiPS user group, one of the repeating themes was the fact that there seems to be very little consistency in the fulfillment element of alarm management- specifically in the area of rationalization. I've written on this issue before, so its not a new thing for me to address. And perhaps you already know my views on this subject, so I'll share those of others instead.

One attendee commented that they had a service company (who just happens to publish their own software and books) come in to perform the rationalization work for them. Though they didn't really know what to fully expect from the result of the rationalization exercise, they did know that what they received was NOT what they expected. This experience was shared as being common by several others. It seems some service vendors have been playing "hide the sausage" on alarm management.

Another comment was made that a major expectation was that the result would be an overall reduction in alarms. After all, this seems to be the commonly advertised thread, and therefore expectation. So, to test this, the so-called "rationalized settings" were played back against the data from the previous few months, and the resulting change in number of alarms was "negligible". The response from the service vendor when confronted with this fact was that the expectation of alarm rationalization should not necessarily be to provide a reduction in the number of alarms. In order to do that you needed to do a nuisance alarm reduction, and the client had not paid for one of those. The point is that for the price, it was expected. And, from all that is written, it seems it has been advertised as an expected result.

Additional comments indicated that only part of the database of alarms configured have actually been rationalized (sometimes less than 10%). The justification given for this by the vendor held that only high-priority alarms needed to be annunciated, and therefore rationalized for full design. Meanwhile, we've told most of our clients that its as important to document why something is NOT an alarm as why some things ARE.

Now add to that some of our own past experiences in importing rationalized data. We are often asked to include the results of a rationalization study into our alarm KB, so that the results will be tracked for auditing, enforcement, and Management of Change. The array of spreadsheets we have received show no continuity, and seem to be only partially completed. This, despite the fact that a healthy sum of money was paid for the services. We have seen exceptions- more on that later. These rationalization results were delivered by companies who should know better. They participate in all the standards boards, and write software, and lots of papers on what you ought to do with your money. Seems they've been collecting exorbitant fees, and delivering shoddy work/software. I guess your money spends a little more freely outside US borders...

Well, we found this just downright un-American, and decided to do something about it. Thus the result that is available to you now, and will continue to be updated and made available to you: the alarm rationalization quality assurance kit. It consists of a spreadsheet in Excel format, and instructions detailing how to use it. With these instructions, you should be able to hold anybody's feet to the flames for what you expect of an alarm rationalization exercise done at your site.

As strictly a software vendor, I think you are aware that TiPS does not perform such services. We've done lots of studies for our clients, but few projects. Our intent is to make our software easy enough, and to perform enough in an automated fashion that you should not need external services should you decide to do this yourself. Software these days should offer you that option. From a purely commercial standpoint, ours is the only one that is not offered by a company with a service desire built in as shortcomings. See more about it at www.tipsweb.com/products/logmate.

However, rationalization- the process of fully documenting cause, consequence, and corrective action for alarms and prioritizing those alarms properly can be a time-consuming effort and perhaps a little intimidating to one who has not done it. So TiPS partners with companies who are the state of the art in delivering services to that end. If you attended our user group, many of them were in attendance, and you were able to hear what they had to say for themselves. These vendors all offer the complete service one might expect when performing a rationalization exercise. In other words, they:

1. Help you to review or create an acceptable alarm philosophy.
2. FULLY configure and document your alarm system.
3. Perform a nuisance analysis so that those pesky alarms are reduced.
4. Complete the documentation that shows the expected result you pay for.
5. See that your expectation was achieved.

So, this means they adhere to the quality assurance process offered by the kit. Some even offer services in excess of what we have stated as the minimum expectation.

We're hoping that our alarm rationalization quality assurance kit will help to shed light on the minimum of what you should be expecting in return for your efforts- whether internally, or contracted. Additionally, our software is designed to help you accomplish that end.

So, please enjoy what we have taken pains to put into print for you. Use it, test it, and send us back your comments. We intend to maintain this kit as an evergreen item that will always be available in the latest best practices rendition. It will adhere to the principles of EEMUA, ISA's SP-18 Standard, and the experience of persons key to those efforts, including past members of the ASM consortium (ASM Is a trademark of Honeywell). Our intent is to be certain that alarm management as a practice does not get a bad name from the habits of others. The more educated our user base is about what to expect, the less chance they will be surprised with the results.

Total Cost of Ownership (TCO)

Steve Apple — Mon, 14 May 2007 23:01:00 GMT

TCO is a term slowly gaining acceptance in the process market. It has long been a term of endearment for many other industries- such as Risk Management, Banking/Finance, and others who have been very software- dependent. These were the industries who first gave us computing a la COBOL, and other historical giants. In fact, the first spreadsheets (anybody remember Visicalc, Lotus 123, etc.?) were designed for those industries.

Its not surprising therefore that the processing industry remains a step behind these other markets in acceptance of both technologies, and norms that seem everyday in those markets.

So, just what is TCO? Total Cost of Ownership is a measurement of the cradle-to-grave costs of owning a specific piece of software. It gives recognition to the fact that when you buy a specific piece of software, the price you pay is not your total cost of owning it. There are other factors to consider. As a result of this, the aforementioned industries generally have professional IT software guys who grill you about the underlying architecture of your product. They could care less about the features if the architecture is one that they know to be problematic. These days, most of them point to MS .NET ASP as the preferred platform, with thin-client web browser access to the components. They've learned that no matter how attractive the features of a software may be, that if it takes a human pretzel to make it work, it just isn't worth the investment.

So, TCO should be top on your list when considering software, and many factors add up to influence it. What are some of those factors?

Ongoing license costs- Is this package all that you need? Are there additional licenses from this vendor, or other vendors that will be necessary to realize the desired benefit from this package? Some vendors try to cloud this picture by doing a very good job of listening to exactly what you think you expect, and giving you a price for only those pieces, failing to tell you that you will not resolve your problem until you expend further investment. Good software vendors listen closely to your problem, and help you to understand the additional software license costs involved and potentially other technologies invovled in handling that problem. We know of many clients who went with a competitor (who shall remain nameless) only to find out that the pieces they really needed were not included in what was delivered, because it was not specifically asked for.The old saying is be careful of what you ask for...

Ease of Use- What will be the learning curve associated with this package? Can I get in quickly and get many of the things I want, or will I need specific training? One good example of this is Microsoft Word. It does a lot of things I will never use, but for the most part, I can get into it, and type up a report, and print it out with very little help. One area to be very wary of in the technical arena is products that are built on Microsoft Excel. On the surface it seems like these would be the easiest to use. On the contrary, remember back to the first time somebody gave you their Microsoft Excel spreadsheet that they had designed. Each of us has learned how to use Excel in our own way, and we like to use it that way. Add to that the fact that when you use it as a product basis, you lock down certain things- so you may even be blocked from using it the way you like. And if the product is trying to do serious statistical calculations, you enter the realm of things like pivot tables, etc. Have you ever used them? If not, you won't like a tool that does. If you are attracted to Excel spreadsheets to resolve a technical issue, I would encourage you to build your own rather than buying somebody else's. It's much less frustrating, and the learning curve of what you CANNOT do is much less. Oh- and be sure that results are exportable to Excel- That part IS valuable whether for further processing, or for writing reports, and sharing.

Of course, most vendors who offer high-difficulty packages make up for it with a deep services offering. Their hope is that you will purchase the (hook) software having seen the demo. You’ll then try to fly into it like the experienced tech guy who demo’ed it to you and get lost. You then immediately phone for help, and experience what the guys at Fluor used to refer to as a change-order-driven income stream. It's not what you expected when you purchased, but you are now trapped. An initial small investment in software can lead to a very large investment in ongoing services to obtain the expected value. I wrote another blog referring to stone-soup software that approximates this problem. One way of checking this is to examine a vendor's historical and expected income. A true Software Vendor expects less than 20% of their income to come from services. And that means on any given project (so examine the quote for this balance) and for any given product. Note that some companies sell certain products that almost install themselves, but have higher services on others. If the latter is a smaller percentage of their business, they may skew the numbers when talking with a prospect to invite them into the trap fro the services-oriented piece.

A side note in the Alarm Management field here is that we’ve learned over the last few months that many people went to certain service vendors to get a rationalization performed. They didn’t really know what to expect when it was performed, but they did know that what they received for their money did not meet their expectations. (there’s some deep irony in that statement.) In fact, they’re still not sure just what they got for their money. TiPS is soon to release some standards that will help to shed some light on this issue, and increase both the expectation and understanding of the recipient of such services.

Speed of Processing the job- This often has to do with work flow built into the product, and the basic product design elements. It also gets into issues like ease of installation. Has the vendor made the software so it is easy to install, configure, and process the information necessary to accomplish the job? Or, will they be on site inventing a new way to tie their so-called OPC driver into your system, while you teach them how to do it? The sooner you start receiving results, the sooner you can recoup your investment. Conversely, the longer you spend making it work, the more its base installation is costing you, and the more man-hours are adding to TCO- whether they are apparent or not.

The next piece of that is of course, how much they have automated their product to perform the task at hand. Did they build the product with the mindset that they would augment it with services, so it was not necessary to automate its actions? Or did they build it with the hope of making it so powerful they would never have to visit your installation site? Have they strived to make most functions have two-click access to the underlying information? One easy way of judging this is to find out how many of their clients are using the product to accomplish the end-use goals themselves. Is it greater than 50%? Or do they try to make excuses for how busy their clients are, and how this operation is one that just shouldn’t be tried by those not familiar with its intricacies.

One other test for this is to notice how many times they talk about what they have done for their users versus how many things their users have done. A very experienced technical person may often be the sign of a very weak software package. The more they try to impress you with their own knowledge, and the less they try to show their software architecture, the greater chance their software is not designed with automation in mind.

Note that in some particular instances, software simply cannot do everything, and services will be necessary. In those instances, you’re looking for the package that requires the least amount of services, having built in as much automation as possible. The package which requires the least on-site time usually wins this category.

Costs to Upgrade- What will be the upgrade path? If it is a client server application, do I have to install a client package on every platform that wants to use it, or does it use pure thin-client architecture, requiring only server installation? The cost for this difference can be HUGE, and is the least examined. Many try to cloud the issue by having certain parts of their application be thin-client oriented, while the upper level application layers require additional client installations. The other dodge is to offer a "web-based" wrapper that accesses the legacy applications via a "look-in" window. This is very dangerous, and maintenance-prone. Think through the process of what will happen when Microsoft releases a new Service Pack, and you require an upgrade installed. The average software company has one major release a year, and almost quarterly releases for patches to accommodate Microsoft issues, etc. What is your corporate upgrade policy? Does it guard against periodic upgrades? What is the danger of being caught a few releases behind? What is your potential cost to stay up to date with their releases? Get a true IT professional involved in the negotiations to help examine these costs.

Training Costs- These costs go hand-in-hand with those mentioned above. Sometimes you may find that the company either requires extensive training, or doesn’t even offer training. The second Is acceptable if the product has high ease-of-use. Ask which different layers of personnel need training? Do you need specialized administrative training for DBA’s or such? How far does the invovlement string continue into your personnel? Does every department have to attend for the software to work? Or is it controllable at the source by a singular person? THese are two extremes, but it illustrates teh difference.

Periodic Maintenance- What is the maintenance cycle on the product? Doe it require constant attention? How often must an administrator go to the server that contains the application to keep it refreshed, and running? How does it fail when things go wrong? Does it fail “gracefully”, and restore itself without a hitch? Or does it require human intervention every time it has a glitch? The cumulative costs of this aspect can be staggering. Not to mention that operations personnel will eventually just turn it off, and you will find it is not running at the time you want its results the most.

Other intangibles- Has the vendor included any specialty features that add to the value, and lower the cost to operate? Some examples might be automated report generation, remote service access, error report generation, and even e-mail of glitches and system issues. Some even provide pager access to prevent extended down-time. Others provide remote service support of the application, so long as the client’s security procedures will allow it. For example, we maintain remote access keys to some systems, and we can actually log into the client systems from our offices. There may be small things which can greatly reduce TCO.

Examine TCO. This is just a primer. You'll find more to it than I have listed. A good look at these costs on the front end can help to avoid a massive unplanned outlay.

Alarmaholics Anonymous

Steve Apple — Fri, 09 Mar 2007 21:24:00 GMT

I was discussing this post with a friend and he noted that Alcoholism is a serious problem in the US, not to be taken lightly. I know this- my brother has a coin that he is very proud of for his years away from the bottle. Fate and genetics can strike any of us. I know that in ways I don't care to reveal. However, I also know that sometimes we have to look at the world in a lighter context. So I hope you appreciate the humorous side of this posting.

This same friend was noting that one service company out there claims that they have a 7-step approach to alarm management. Another claims a 6-step approach. I'm trying to figure out what it is supposed to cure you of, since everybody that has tried them is still looking for answers. It's like the Britney Spears of rehab programs. They want you to have to return for rehab often and expensively.

This friend advocated a 12-step approach which stole from the famous AA aproach. A few conversations later, and shazaam- the concept was born of - Alarmaholics Anonymous. - We urge you to join if you share the symoptoms, and are ready to solve the problem. Help us end the senseless addiction to more alarms. What are the symptoms? Here are some:

Have you been sneaking in at lunch when nobody was watching and putting an alarm on the system?

Do you need an alarm when you get up in the morning, and perhaps do you prepare one before you go to bed at night?

Do you have dreams about alarms and you can't reach the acknowledge button? (this one came from a client)

Do you feel you can handle a larger number of alarms than others?

Do you keep a small alarming system in your pocket in case you need to be alarmed in a place where you might otherwise have not had access to them? (Crackberry users apply here)

When you arrive at work, do you just not feel right until you've had your first alarm?

If you go to a party, is it important that the house have alarms for you to feel comfortable?

Do you prefer to have your alarms alone? Do you find yourself not wanting to be in a crowd when an alarm goes off?

Do you feel that others don't notice that you are dealing with alarm issues every day?

Do others tell lies for you to cover up your alarm problem?

All of these can be symptoms. Please e-mail me if you have more. So here's the 12-step approach:

1. You have to admit that you have an alarm problem, and you want to solve it.

2. You must believe that you can be helped.

3. You're willing to ask for that help no matter what it means.

4. You are ready to inventory how big the problem is.

5. You begin by admitting to others that you have an alarm problem.

6. You are prepared to resolve whatever defects are found.

7, Now- make the call to ask somebody to help you. (TiPS has operators waiting...)

8. Make a list of all the problems this has caused besides just the alarms themselves. (We call this the situation awareness approach)

9. You fix those problems wherever possible except when the cost is greater than the return.

10. Continue to inventory and track the problem symptoms. (KPI's come in here.)

11. You now enter a greater path of situation awareness understanding, and study all of its nuances. Continuing to keep a monitor on potentials for back-sliding.

12. Have a spiritual awakening as a result of these steps, and want to spread the results in the form of assistance to others who are trapped in the same issues. (Attend TiPS User group meeting.)

This was written with a tongue-in-cheek approach, but as I examine the steps, I realize how close they are to what actually has to be done to resolve an alarm management problem. And I can guarantee you that every alarm management problem is so different that there is no tried-and-true 6- or 7-step approach that will cure them all. Each alarm management situation needs to be considered in its own standing, and an approach crafted to fit the particular situation.

As I write that last paragraph, I also note that there is another service company who advocates that they have utilized four drastically different approaches depending on the circumstances. I believe that is getting a little closer to the reality of the situation. But they negate this idea by then proposing the aforementioned 6-step approach.

Do you have an alarm problem? We can help. We've been helping people SUCCESSFULLY resolve alarm issues since 1990. Come to see us at the TiPS/ Expertune user conference. Held in Beautiful Austin, Texas- we promise to have at your availability the world's leading experts in alarm management. Sure cures to alarmaholism. You can schedule some private time with these experts, and hear their presentations. Product training is available.

Yep- LogMate (http://www.tipsweb.com/products/logmate/) is the sure cure for Alarmaholics. It won't just help you inventory the size of the problem, and get it behind you- it will keep you informed should it ever rear its ugly head again.

Rations on Libations?

Steve Apple — Thu, 22 Feb 2007 21:18:00 GMT

If you're about my age, you probably grew up on a diet of Saturday Night Live, with John Belushi, Steve Martin, Gilda Radner and the rest of that wacky bunch. I loved the one when Gilda Radner took off (as Rosanne Rosanna Danna) on the righteous question of what was wrong with Saxophones and Violins on TV? When they stopped her and explained that it was actually sex and violence, she responded in the traditional "Oh-well- Never Mind" for which her character was famous. Recall she also did pieces on the "Youth in Asia" and other memorable fractured phrases.

So-to borrow from Gilda, a lot of you are probably wondering "What's all this talk about rations on libations"? Well, I'm here to enlighten you. And like the "Oh-well- Never mind." I also realized they're talking about Rationalization. Was that ever a shock. I was wondering why so many people were inviting a service company to come in for a project that never seemed to make anybody much better off. Especially if it meant they were cut off on libations. Here's what my research has found.

To start with, I looked up the word RATIONALIZE in Websters. ra·tio·nal·ize: to attribute (one's actions) to rational and creditable motives without analysis of true and especially unconscious motives (rationalized his dislike of his brother) broadly : to create an EXCUSE or more attractive explanation for (rationalize the problem)

So, do I need for people to come to my plant and make up excuses for my alarms? Rationalization as a word and a practice is at the least overused, and is nominally something done to make an excuse for the fact that we put in a bad design to start with. So, let's re-look at alarm systems as a design problem rather than a fix-it problem. If you get the design right, it's fixed for good.

Perhaps we should look at the problem in the way that is supported by the ISA's SP-18 committee. As already stated, the fact that you have a bad alarm system is due to the fact that it was not designed properly to start with. Or, perhaps it was designed properly, but it was not engineered properly with the protections and safeguards in place to ensure that it would not grow to an unmanageable state. That is because there was no attention paid to the alarm system design lifecycle. The upcoming ISA SP-18 specification will pay particular attention to alarm system design lifecycle. The point is that if you put the proper design in place, and install proper safeguards and practices, your alarm system should NEVER become a problem. So, let's stop paying so much lip service to alarm rationalization, and instead pay particular attention to alarm system design and the tools to protect its integrity.

How do I do that?

Alarm management is not rocket science. There's no special algorithms, and no special proprietary secrets necessary to make it work. The EEMUA 191 document is written just for those who need to know how to get their arms around it, and it's not really that bad to read. You can buy it with your credit card at www.eemua.co.uk. If you have designed your system such that it is now out of hand, an alarm system lifecycle review will tell you that you need to rectify that to get it back within a state of control.

In the late 80's and early 90's there was a lot of talk about process optimization. There were people who would come to your site, and optimize your process. When they left, things ran well- all the loop controllers were tuned, all the targets were properly set, and things seemed to run fantastic for a few weeks. A few weeks later- back to square one. They did not install the tools to protect your unit from falling out of an optimized state. In fact, they did not want to install such tools, as that was how they made money. If they did install a tool, it was so rudimentary as to make you need their help anyway.

Such is the state of alarm management that is being sold by a handful of service companies. Yes, they offer software packages, but it is not in their best interests to make that package address your alarm system lifecycle. If it did, their income would drop drastically.

What is the pattern? A team comes on site- does a rationalization, and all is well for awhile, and then we're right back where we started. Why is that? It's because to be successful, we need to institutionalize and internalize the processes associated with good alarm management. And we need to correct the bad practices that allowed the alarm system to get into the shape it is in to start with. These need to be incorporated as a part of our internal processes and practices. Just as we did with process optimization (if we did). And as Mr. Ian Nimmo says (www.mycontrolroom.com) you need to consider all the factors associated with good SITUATION AWARENESS (his term- read his papers).

What is meant by that? It's simple- other than absolute nuisance alarms- all alarms exist for a reason. Unless you discover and resolve the cause for an alarm, it makes no sense to get rid of it just for the sake of reducing alarms. It will simply reappear, or- lead to an incident. Too few alarms can be more damaging than too many alarms. This means that some alarms just cannot be reduced without additional work around the control automation subsystems. Some have learned this the hard way on some of their initial rationalization attempts.

So- just what am I getting at?

Some service vendors out there want to sell you a rationalization project. They would also like to sell you a product which supports rationalization the way they see it. And at least a few of them have a very narrow view of the subject. They do not recognize the lifecycle aspects of an alarm system, just their desire to rationalize. So what do you do with their product once you've rationalized? Well- its only valuable if you need to rationalize again, and it is their goal to ascertain that need will exist. That's rather disappointing if you've designed a complete bid process around finding a product that will help you rationalize your systems. Quite frankly if that product did what it was desired to do, you could use it once and throw it away.

Much like the Webster's definition of the word rationalize, their motives are to create a rationale to hide the true motives behind their actions. i.e they've rationalized rationalizatrion, and made it sound as if it's a final answer when it is not. It's ignoring the causal drivers for bad alarms Essentially, it is akin to fixing a flat tire on your car everyday,rather than finding why it goes flat.

Ignore the noise you're hearing in this area long enough to consider a lifecycle approach to alarm management. Think of your alarm system as a system that needs to be properly designed to start with. If it is not, or was not, then rectify that situation, but be certain that you consider the need to get the system under control. There's a lot that's been written on this issue, but its not attractive because it says you have to alter practices and procedures and perhaps even take on additional automation changes rather than just doing a project and checking it off the list of to-do's.

So, if you have a new alarm system, put the tools on to start with to be certain it is properly tracked and kept within reasonable limits. You'll save massively in the years to come. If you have an old one, consider what is needed to get it back to a state of good design.

Yes, excuses may thus have to be made for your existing alarms, but your mindset will be one of getting it within a realm of control and tracking to be certain it stays there rather than just having to re-do it every six months. And the tools you choose will support this approach rather than one of continuous rationalization.

Place rations on their libations and they might come to the same conclusion...

I'll be addressing this issue more in the coming months.

�

Do you have alarm floods?

Steve Apple — Thu, 21 Dec 2006 18:35:00 GMT

Recently there has been a lot of discussion about avoidance of alarm floods- the holy grail of alarm management. Despite the efforts of hundreds of smart engineers and scientists in eliminating alarm floods, they still plague operators. In my opinion, it is because we keep trying to eliminate it, rather than just trying to help the operator deal with it.

I recently attended a presentation where the discussion was mainly focused on alarm rationalization. One of the reasons given for alarm rationalization was to reduce the number of floods. There was disagreement about what that meant, and several opinions offered. Prior to that I had been at another meeting where half of the group seemed to think they were in a state of constant flood, while the other half seemed to think that they had no alarm issues at all. At this point, I realized that perhaps everybody does not measure a flood using the same metrics. So, that gets us to the root of today's subject - just what is a "flood"?

According to EEMUA guide, a flood is anything exceeding 10 alarms in 10 minutes - or one per minute. I have looked at enough data to tell you many units are in constant flood by that measurement. However, the operating team does not perceive themselves as being in a flood condition. It is simply their normal alarm load. That team considers a flood as a group of alarms that they simply don't have time to deal with before something bad might happen. In other words, alarms associated with an upset condition, an equipment malfunction, or a process wobble, and perhaps a potential incident in the making. Often, a flood consists of unfamiliar alarms since they are outside of normal operating conditions and a range of normal training/familiarity.

That leads us to the unusual conclusion that "flood" as specified by data is not necessarily a flood to the operator. The operators know why the alarms are there, and they are essentially ignoring them because he knows they will not lead to anything significant. Some of them may even be there because they elect to leave them active for any number of reasons. In fact, once innured to the sound of the alarm annunciator, he is so comfortable with these alarms that he'd probably feel uncomfortable if they were not there. It's kind of like the friend with the overly talkative spouse. Did you ever notice how they were able to function through the incessant prattle despite the fact that it was starting to drive you up the wall? The human mind is capable of ignoring things in this way. This is why in the case of many incidents, ignorance of alarms has been noted as a contributing factor. Ignorance is a strong tool, and is in fact can be a good quality in the right circumstances. Soldiers learn to focus on their goals with bullets flying over their heads. Air traffic controllers learn to analyze situations under circumstances that are increasingly pressurized. Beekeepers learn how to ignore even more.

So, how does this relate to EEMUA guidelines? Simply stated, EEMUA guidelines are not applicable to a unit until that unit has observed, and put into practice ALL of the EEMUA recommendations for alarm management. Paradoxically, if you do not clean up your nuisance alarms, you will not pass the EEMUA spec of normal alarm state, yet your operator may not feel that he is in any state of disrepair. The EEMUA benchmark becomes vital only at the moment you accept the first tenet of the EEMUA documant:: Every alarm has to have an associated operator action. Subject to those conditions, every alarm that occurs beyond the allowable level contributes to a flood. On the flip side, every alarm that does not require action may not be contributory to a flood. From a purist viewpoint, it is, but not necessarily to the operator who is ignoring it. So why all the rush to rationalize alarm systems to reduce flooding? Does it really work?

Our data shows rationalization costs a lot of money, but doesn't necessarily solve the problem. Thus the reason for all of the papers purporting new cost-effective methods. In fact, our experience shows that the most bang for the buck comes when you get rid of all nuisance alarms, and using the information around the others to point you to the situations for which they exist. In other words, it points to Situation Awareness as the cure. This is contrary to all the written expert opinions currently in print.

Much study has been done surrounding this problem by the ASM Consortium (ASM is a registered trademark of Honeywell). Their members were the first to explore alarm rationalization some 15+ years ago. Shouldn't that mean they are now satisfying the EEMUA constraints of the EEMUA 191 report which they co-produced? Their evidence says no.

See the paper they published on this issue.

Yet, their members don't seem to think they have drastic alarm problems, and many feel they have the alarm problem pretty well in hand. Not totally resolved, but in hand. And that's fifteen years after having started. Again that points to the fact that perhaps the problem solution does not lie in all of the things they've tried. Again- I point to a study in Situation Awareness as the required answer. That is the real jewel they have uncovered.

As a short diversion, another path has been taken lately that deserves some attention- tools that will resolve the issue once you have fixed the basics. These tools are based on principles for smart alarming, or state-based alarming. I have met lately with DCS providers and learned that tools are coming which will address this situation in most new DCS systems. I think you'll like them once you see them. Unfortunately, most require system upgrades to recent releases to make use of them. Where does that leave us with respect to legacy systems which will be with us for many years to come?

It has not been TiPS policy to recommend tools that attempt to handle alarm dynamics POST-DCS. There are many reasons for that. Read my post on dynamic alarming to see a few. However, we have seen some tool sets that deserve consideration.

The first is from a company called UReason. See their website at http://www.ureason.com/ . These tools allow for a super-imposed alarm handling screen that "subsumes" alarms, creating a display that makes more sense. For example, it will recognize patterns, and reduce the alarm count automatically by knowing such things as when a pump shuts off, or when a start-up or shut down condition is occurring. Their OASIS system will use pre-designed filters to deliver only the information pertinent to the situation. In other words, those seventeen alarms you once received will be subsumed, and replaced by a single alarm that tells you the pump is down. Or, if the pump is automatically replaced, it may not bother the operator with the information at all - only sending its data to maintenance for service follow-up.

Note that the level of attention and maintenance for such a super-imposed system is increased to levels beyond even the maintenance you have NOT been providing to your alarm system. However, for those who have gone to the trouble of cleaning up their alarms, this could be a next logical step. WARNING!! DON'T TRY IT WITHOUT HAVING CLEANED UP THE BASIC ALARM SYSTEM FIRST. As a voice of experience from one who has tried it both ways, I can tell you that it has no chance of success if you don't do it in the right order. Also, don't try it in conjunction with complex state-estimation models. They have never worked, for a variety of reasons. Handle only the simplest configurations first, increasing your complexity to a level as you see the operations team can support it. The rule here is that if it must be maintained by engineers or mathemeticians, then operators will not receive sustained benefits from it.

To be fair, I should state that I have seen cases where complex model-based systems did work, and were maintained past their initial installation for some period. All of these successful examples share two common traits. The first was that the problem could not have been solved any other way. The second was that the value of solving the problem was so great that it justified the PhD mathemetician who had to be kept on the problem. Note that one other issue was that the PhD mathemetician's interest was also maintained because there were enough twists and turns to the solution that it required his ongoing inventiveness to keep resolving new issues. Perhaps more of these would be in existence if somebody were to offer an inexpensive and efficient expert system package...

We have also seen products from a company in Louisiana called Prosys. http://www.prosys.com/. Having not seen their tools first-hand, I can only guess what they have from descriptions of their products. The approach is similar - a replacement of the DCS alarm screen, but giving a more powerful view to the operator. My understanding is that they have a few examples of these having been installed and maintained for an extended period.

With proper implementation of these tools - not trying to take on the world, but simply to give the operator better information, it is possible that flood management - the holy grail of alarm management - may be within reach. We're certainly heading that way.

The ROI on Alarm Management

Steve Apple — Wed, 06 Dec 2006 22:16:00 GMT

I was at a mixed-company training a while back. There were companies there who used software from various vendors, and had solution projects performed by the same, and a greater mixture of "alarm management newbies"- i.e. those only recently addicted/afflicted. The question of ROI for alarm management came up. The Newbies wanted to know how the experienced ones evaluated ROI for the effort.

The comment from a battle-scarred veteran was "Yes- our management is asking that same question. They want to know what they have received back for the $x Million (it was a shocking number) they've invested in all of these rationalizations, etc. They noticed that the control room seemed a bit quieter, but they still saw the same shut-downs and plant upsets as before."

Their answer to management was that they wanted to work with the same vendor who had not yet solved their problem to invent plant state estimators, and develop a smart alarming system. WARNING Will Robinson! Aliens approaching. Be careful of what these aliens promise you. You'll be stuck with the result, and they'll exigrate back to their home land. Management's response back was "You mean we haven't already got that for what we've paid?" I repeat my mantra. You ain't seen an alarm flood until you've seen a smart alarm flood. As a veteran with battle scars in that area, I've seen too many brilliant people wreck their ships on those shoals to bet that a service provider is going to resolve this problem.

Don't take this wrong- the vendor they hired did just as promised, and did a good job of rationalizing their alarm system. Its just that at the end of that alarm rationalization rainbow, the leprechaun didn't have the pot of gold they'd hoped for. This is not necessarily the service vendor's fault (though promises were PROBABLY made to get the project started). It should have been obvious that a simple rationalization could not solve the problems that made the system get bad to start with- after all , it didn't change any processes, just an underlying configuration.

Again- a warning. This alarm management stuff is NOT rocket science, and reducing alarms is not the end resolution of your problems. Alarms are the automation backstop of the plant, and thus the best indicator of your problems. But, other than nuisance alarms which are easily dispatched, the real bottlenecks lie elsewhere. So, perhaps you can save all that rationalization money and invest it instead in the bottlenecks your alarm system is pointing to. Upgrade your control room. Redo your graphics. Increase your maintenance interval. Install better communications. These (and their cousins) are usually the root cause of alarm problems.

So- lets take this smart alarming thought process a step further- because I've been there and done that. Suppose you and I- with our vast intelligence- sat in a room and came up with every possible scenario the plant might face, and designed an expert system that would automate the plant through and past each problem. Well, guess what- we'd miss. And the ones that caught us would be the no-brainers that we were too smart to think of. We used to say that the solutions we offered didn't solve the problem, but only confused you more than before. However, we felt you were confused on a higher level and about more important things.

Alright- where is the ROI on alarm management? Actually, it's pretty easy, and it has to come from operations. How many times have you lost a pump, and how much did it cost? If you can avoid a lost pump due to a clearer message, what does it save? The answer to this question lies in benchmarking pre- and post-alarm management effort. Problem here is- if I do prevent a lost pump, how can I claim that savings if I don't know whether it would have happened otherwise? I cannot run parallel timelines and see what happens in another universe with different decisions...

So, let me help you get true ROI out of your alarms management investment. Don't bite off too much too quickly. Put the measurement tools and benchmarks in place up front. Let the data tell you where your problems lie. If you have a problem that can be solved by alarm rationalization, analysis of the data will indicate that. If your problems lie elsewhere, analysis of the data will indicate that. But don't put the cart before the horse. Don't spend $5Million on rationalization exercises that will produce no measurable effect. If you've got $5Million to spend, let the data point you to where your problems lie.

I suspect you'll learn that your alarm data will point you to several areas that seem obvious once you've seen the data. Oh- and stop looking at alarm management as an ROI exercise- it's NOT. It's really a RISK MANAGEMENT exercise. You should be looking at the dollars you are putting at risk by NOT optimizing your alarm system. This is always the scarier side of this equation, but infinitely more realistic. We'll have more articles dealing with this issue in the future.

The Value of Process Data to Alarm Management

Steve Apple — Fri, 10 Nov 2006 19:48:00 GMT

For those of you who haven't seen it yet, TiPS has an add-on feature to our LogMate product called TRAC. http://www.tipsweb.com/products/logmate/forensics_trends.asp This product was something that came out of our relationships with pharmaceutical clients. Those clients have a great need to have alarm data and process data together in the same spot to assist with variance reviews. In a straightforward implementation of the best of Microsoft, we are able to extract the process data, and align it with alarm data for any given period. There's even a relative time capability so you can view it batch-by-batch.

Cool- huh? But, that's not the data I want to discuss. The item I'm interested in today is that of the process data's importance to alarm management during the rationalization exercise.

Rationalization is something which TiPS advocates should be internalized as much as possible, and in most cases, should be under the realm of operations management. I expand on this philosophy in a two-part technical paper series. YOu can see the first installment of that series at http://www.tipsweb.com/downloads/A_Practical_Approach_to_Alarm_Management_Part_1.pdf

The process of rationalization often is one of the few opportunities for the DCS/ instrument engineers, control engineers, process engineers, maintenance and operators to sit together in a room and talk about how operators use the control environment. The tools involved are what each of these other factions work to have available for what the process engineer hopes is making what is planned. As such, there is a lot revealed about just which pieces are used, and how they are used, maintained, and intended. This process is thus very valuable. In fact, it is so valuable that it should not be shopped out to a vendor, except as an enabler, or as an internal partner to the process. Much of the time spent with a vendor is in the process of educating them to the level of what your internal people already know. The exception here is that alarm management specialists DO have a knowledge of the process necessary to accomplish a good project, and a partner in this process has a stake in making things work. Bottom line- avoid strictly project work except to fill in the blanks.

I used to visit an operations manager (who shall remain nameless) in Canada who told me a story which I'll share.

"I've looked this process over from a historical standpoint, and discovered we're operating at about 150% efficiency. I base that discovery on the fact of all of the justifications that have been put through for all of the projects we've performed over the past 10 years. Each small project was justified to get, and post-audited as receiving x% improvement. Well, all those improvements added up equals 150% or thereabouts. Now, we both know this can't be true, but it's taught me one thing. That is the importance of having continual projects in the plant. It seems that projects cause people to be interested in what we're doing, and they continue to fight against the natural tendencies for things to return to chaos. (remember the second law of thermodynamics- all things tend to greater randomness- it's still true). So, I like to keep projects going that keep people communicating and pushing each other. Otherwise, they get bored, and things fall into disrepair."

That's an interesting concept. I get from that the need to communicate to one another what is going on, and what is working, and what is not.

Years ago, I was doing some modeling on a unit. I was using one of those fancy Neural Net packages. (Not the nets with scribed steel handles- those would be knurled nets- totally different) Again- no names. I was told by the engineer who gave me the data that two specific variables were the most important variables to the control of the unit. Guess what? They were flat-lined. The engineer thought this was possibly correct, as they needed to be relatively flat to accomplish tight control. No- I mean they were flat-lined. NO data change at all. Just one value. In talking with operations, those sensors had been dead for over two years. They had developed other methods to accomplish control (essentially things which mirrored the action of these variables). Without that discussion, and a review of the data, the engineers would have never understood how operations was controlling the unit. This is just one story among many where similar discoveries were made. Neural nets were cool for the fact that they renderd the sensitivity, and data range of the variables involved in the model. I could go on, as that's fun stuff to talk about, but let's get back to how this relates to alarm management.

The point here is that the rationalization process allows a unique opportunity for a thorough operations review. TREAT IT AS SUCH, and you will benefit from more than a rationalized alarm list out of the exercise. One of the most overlooked items in this process- and you will not see this in any book, guidelines, or paper yet published- is examining the data activity around the alarms you assign. What does that mean?

Within the rationalization process (assuming you rationalize the complete operating unit), you will end up with a prioritized list of alarms. For at least the top priority alarms, and safety critical alarms, examine the data activity under these alarms. If you have time, you should do it for ALL configured alarms. To do this, you will need access to your process data historian, and the statistical toolkit it contains. Examine the data distribution, range, and standard deviation of each of the sensors you have rationalized. What do you see? Is it what you would expect? Why or why not? Have you assigned a high priority to alarms on sensors that are flat-lined? Have you assigned a low priority to an alarm on a sensor that is all over the map, and reflects something important to the process? This is just the tip of the iceberg of the discoveries you will make with a process data review of the important variables.

With this data in hand, and the process specifics it reveals, intercommunication between these factions of your plant is priceless. As my Canadian friend would point out- it forces communication for the purpose of a project that might not otherwise have happened. It gives people who might rather avoid each other a reason to have to talk. And that's valuable to your operation.

Lastly, let me take this opportunity to also issue a warning. There is a practice being proposed by a service company which they call "focused rationalization" or something like that. RUN AWAY from this methodology. I point to my earlier logic to help you understand the danger of this approach. The method of this madness is to focus rationalization efforts only on the alarms which have activated recently, or ones you designate as important. On the surface, this may sound rational.

Where does it fall apart? If you have alarms which do not activate because they are flat-lined, they may not get rationalized. If you have a sensor which is being used by operations to take the place of that sensor, you may not even place a properly prioritized alarm on that sensor. And even if one is identified as important- is it valuable to have an alarm on a flat-lined sensor?? A review of the historical process data would reveal this. Additionally, we all know that by Murphy's law, it will be the alarm which hasn't activated- and the operators therefore had little experience with, that will be the one which will cause the next incident- the one we ignore because it never activates. This method must have been hatched by somebody with very little statistical process data experience. It is statistically unsound, and could be very dangerous to your process. Unless you like talking to regulatory bodies and lawyers.

TiPS advocates that if you are looking to reduce your rationalization effort, it must be with a consideration of overall process unit risk. And any unit which is slated for rationalization must be FULLY rationalized. You can read more about this in our recent two-part series on Practical Alarm Management. The first part is available on our web site now, and the second part will be released next month. I hope you will read it, and comment.