Tales of troubleshooting: Don't panic

Pressure leads to panic. Having to troubleshoot and solve a difficult engineering problem under pressure can be paralyzing, particularly to an inexperienced engineer.

Six months into my second job and just 2 years out of college our production started having failures I had to investigate. Keep in mind, I was one of 2 electrical engineers at this very small company, and we served as design, test, and manufacturing support. After a few days of trying to find the root cause of the PCB failures, the president (an EE by degree) comes down to production and he sees the 100 feet of workbenches still covered with product that cannot ship because of this issue. He then begins yelling at me in a tirade laced with F-bombs and pointing out that $500K of inventory is sitting here at the end of the month and wanting to know why this was not solved yet.

I should probably mention that I was just learning high voltage motor controls since that was not my background. Also, the failure was that the power FET would end up 100% in the ON state which resulted in a violent extraction/evaporation of all the motor drive parts from the PCB and many of the traces as well. Not only did the pressure focus the mind, but also the danger of working on mains connected high voltage circuits.

A few days of study showed that nothing in the design was violating the published specs of the parts on the PCB. Even the application engineer at the semiconductor company verified that what we were doing should work.

I should mention that I was fortunate to have a talented mentor at the time from whom I was learning motor control. He is still one of the most talented engineers I have ever worked with and possibly the greatest storyteller (more on that in another post). He didn’t come out to the production floor and solve this for me or even offer. He let me learn from the school of hard knocks and I am thankful for it. We had several panic situations like this and whenever this came up, he always reminded me “There is always a reason…the laws of physics still apply. You may not always be allowed the time or resources to find the reason and may have to pursue an alternate route to solve the immediate problem and ship product, but you just keep digging…the answer is always there.”

This advice has stuck with me for over 30 years now. You can’t stop that feeling in your stomach and your mind will let you start to believe that something almost magical is happening…a problem that can’t be fixed or understood. I have found that you must calm yourself and step back from the problem repeatedly. I have solved more of these issues by lying on my back on the couch mentally envisioning every detail of what could be wrong than I have standing at the problem with a host of equipment at hand. (Usually with a bunch of other panicked people shouting advice at you.)

So, I am sure you fellow geeks want to know the root cause. It turned out that the PWM motor control chip had an issue where some production parts would draw a large inrush current. The off-line voltage regulator (and highly specialized) chip that could produce a few milliamps of current from rectified line voltage could not produce enough for this initial surge and the output voltage got stuck just below required operating level for the PWM motor drive chip. This caused the FET drive to go full ON. The issue was thankfully solved by just kludging on a Zener that would not allow power to the PWM chip until the regulator had charged the cap to almost the 15V needed to drive the PWM chip.

So, why do I tell you this story? I simply want to prepare you for this inevitable situation. Don’t panic and don’t spiral into worrying about your job or what might happen. I felt like I was way over my head in this situation and I am sure many of you have had or will have this feeling as well. Once you successfully navigate a situation like this once, you need to tuck that confidence away for the next time it happens because this is what a career in engineering will bring you.

Don’t be afraid to step back and don’t stay at your office until all hours of the night…that is not helpful. Focus, clarity, and being mentally sharp wins the day. Tune out the loads of advice (usually bad) and the applied pressure. There is always a reason…you just grind it out using basic scientific method, changing one painful variable at a time, and relying on your training. These things always reveal themselves eventually.

Love to hear feedback or questions