Lessons Learned in Diagnosing Pulsed Power Systems

While APELC does everything possible to make our systems low-maintenance and reliable, the honest truth is that any high-voltage system has unique failure modes that require some specialized knowledge to diagnose and repair. We pride ourselves on good customer service and regularly visit our customers to provide support and training in the maintenance of our systems. In this blog entry, APELC would like to share some lessons learned from decades of pulsed power troubleshooting. Before getting into this, a few important warnings:

• Personnel who have not been trained in high-voltage safety should never attempt to operate or diagnose a pulsed power system.
• ALWAYS disconnect prime power and ensure all elements capable of holding charge have been grounded and verified as discharged before handling any element of a high-voltage system.
• Always consult the authority having jurisdiction (AHJ) before working on any pulsed power or high-voltage system.
• If you are having an issue with a system you bought from APELC- call us before attempting any of the recommendations below.
• Failure to follow safety guidelines for a high-voltage system can result in extreme harm or death.

All that being said, here are our top 6 recommendations for diagnosing pulsed power systems:

 

1. Diagnostics, diagnostics, diagnostics.

Operating a pulsed power system is often like flying an airplane in low-visibility: If you don’t have your instruments to guide you and monitor the operating conditions of the aircraft, bad things are going to happen. Given the incredibly high-EMI (electromagnetic interference) environment created by high-voltage/current transients, diagnostics can be incredibly misleading. Many times, we have seen customers (ok, and ourselves too) start to diagnose what they think is a broken Marx generator simply because they are confusing noise for signal on the oscilloscope. The pulse-widths generated by many of our systems are in the tens of nanoseconds- this also means that diagnostic cabling and ground lines start to look electrically long in comparison to the transient environment, and suddenly a diagnostic signal that may be in the millivolt to volt range is swamped by common-mode noise that is an order of magnitude higher. Here are some basic troubleshooting steps we take well before cracking open the Marx:

• • Are the attenuators still good? Because some of the signals generated by pulsed-power diagnostics can reach amplitudes in the hundreds of volts to kilovolts, it is not unusual for attenuators to fail. A quick check of the attenuator’s S-parameters on a Vector Network Analyzer (VNA), or even swapping it out for a fresh one can quickly tell if a failed attenuator is leading to an erroneous reading. If this is the case, consider adding a smaller attenuator before the larger one to decrease the voltage drop. E.g. If you are using a single 40 dB attenuator, consider swapping it for a 30dB attenuator with a 10dB attenuator in front of it.
  • Are the connectors good? Often times, someone either over-torques a connector causing the center pin receptacle to become damaged, or the connection is left too loose to make good connection. Also, it is not unusual for a connector or cable to get stepped on or rolled over by equipment to the point that it stops working.
  • How good is the shielding effectiveness of your cable? Folks, I’m only going to say this once: If you have RG-58 in your lab, throw it away! Poor cable shielding can result in high-levels of common-mode noise. APELC prefers using RG-400, RG-223, or RG-214 with double-layered shielding to ensure high-levels of shielding effectiveness.
  • Are you seeing signal or noise? In many situations, this can be the first step taken in checking your diagnostics. Place a 50-ohm terminator or short-circuit at the end of your diagnostic cable in place of the probe. Take special care to leave the cable in the same orientation it was in, as this can be a major factor in how noise is coupling onto the cable. Fire the system and monitor the oscilloscope. Are you seeing a similar waveform without a probe compared to what you were seeing with a probe? If so, you are looking at noise and not signal.
  • If you are seeing a high-level of noise on the channel, and you know your cables and attenuators are good, there are a few steps you can take to start cutting down on noise:
    • Make sure your oscilloscope is in a shielded enclosure
    • Ensure the power coming into your oscilloscope is filtered and not carrying noise into the shielded enclosure.
    • Place ferrites onto the diagnostic cable either just before or after the bulkhead feedthrough of your shielded enclosure. This well help to eliminate the common mode noise that may be present on the cable’s shield.
    • Move diagnostic cabling away from high-voltage cabling, and minimize coupling by keeping the diagnostic cable orthogonal to any radiated fields.
    • Consider replacing long runs of coaxial cable with an analog fiber-optic link, such as the p2p series of links available here: https://gmw.com/product/analog-signals-via-optical-cable/
  • Is your probe still working? Whether you are using a d-dot probe, b-dot probe, current-viewing resistor (CVR), or current transformer/Rogowski coil, check the mechanical/electrical integrity of the probe and measure its impedance to see if it matches an expected value (e.g. a d-dot should measure open, a b-dot should measure short-circuit). If you are using a current-transformer or CVR, it is always good to keep an unused spare on-hand to quickly swap out and see if the existing one is bad.
  • If you have gone through every step to eliminate noise and are still having trouble seeing your signal above the noise floor (which should never exceed a few millivolts worst-case), decrease attenuation or utilize a probe with a higher sensitivity to make sure your signal level is well above an order of magnitude higher than your noise.
  • Lastly, if you have tried everything else, and are fairly sure the system is producing some kind of measurable output, swap oscilloscope channels (or even oscilloscopes). It is not unusual for a channel to fail because someone before you drove too high of a signal into it.

2. What are you expecting?

Having a prior understanding of the expected performance/output of the system is critical. APELC provides our customers with the data from our factory acceptance testing as a baseline for both operating parameters (e.g. pressure vs. voltage) as well as output waveforms. If you are reading this and did not purchase an APELC system (drop us a line!), and you don’t have an example output waveform, simulate the system performance using a simple lumped-element LRC model to get an idea of the expected performance.

3. Know the source impedance and load impedance.

This point cannot be stressed enough. You cannot accurately determine the performance of a pulsed power system without knowing the exact characteristics of the load it is firing into as well as the impedance of the source. APELC typically determines the source impedance of our generators by firing them at their lowest-possible charge voltage (to ensure capacitors are not damaged by voltage reversal) into a short-circuit. We use the period of the resulting “ring-down” (i.e. damped-sinusoid) waveform to determine the equivalent series inductance (ESL) of the entire source. Because the series capacitance of the source can be determined from the manufacturer specifications and/or verified with an LCR meter, these two values can be used to determine the source impedance.

A common problem with getting the measured signal to match up with a simulated/expected waveform is in not fully considering all factors contributing to the load impedance. One major benefit of the APELC quick-disconnect connectors on the output of our Marx generators is that the customer load can be quickly and easily swapped out for one of our coaxial dummy loads (see below). This ensures that the entire source and load combination maintains a coaxial topology, which not only helps to control/predict load impedance, but also minimizes radiated emissions that could couple onto the diagnostics and flood the measurement with noise. It is not unusual to see a low-inductance load resistor (e.g. carbon or ceramic) being connected by a large flying-lead from the output of the source. This has the dual effect of adding a large inductive loop to the load (significantly raising the impedance), and creating a powerful transmitting antenna for the energy delivered from the generator. The coaxial dummy load mentioned above not only maintains a very controlled load impedance, but also provides a NIST-traceable CVR on the low-side of the load resistance to create an accurate voltage divider to determine both amplitude and temporal characteristics of the output pulse. This can be used to both diagnose the Marx generator and to calibrate other probes in the output chain.

 

Figure 1 APELC Coaxial Dummy Load

4. What does the waveform tell you?

With trustworthy diagnostics and a good knowledge of expected performance in-hand, you can begin using the output waveform to tell you a great deal about the performance of the system well before turning a wrench. The APELC-provided factory-acceptance test waveform or expected waveform from simulation should always be generated using a well-matched load. From a practical perspective, this protects the generator from damage due to voltage reversal or internal flashover. From a diagnostic perspective, this provides a means of determining the generator’s health by going back to your knowledge of a simple LRC circuit. For example, if you are expecting a critically damped waveform and start seeing a waveform that is ringing like crazy, your source has either started looking much higher in impedance, or your load has started looking much lower in impedance. Similarly, when an APELC Marx generator erects correctly (i.e. triggered vs. misfire), it is typically characterized by a fast rising-edge and double-exponential decay. A slow rising edge (on the order of the falling edge) is a typical sign of a misfire and can mean the pressure is either too high or too low by a few psi, or that there is an issue with the trigger circuit causing the Marx to only fire due to a “self-break” (gaps breakdown prematurely due to low pressure). Some example waveforms are shown below:

Another culprit that can be diagnosed by looking at the output waveform is internal breakdown either within the generator or the cable between the source and the load. If you see a pulse that starts coming up to full amplitude and then suddenly falls to zero (often followed by significant ringing), you can be fairly certain that there is a low-impedance breakdown occurring at the output of the Marx. If you have an output cable that is suspect, this is a good time to swap it out for a new one or high-pot test the existing one to determine if it is where the breakdown is occurring. If it is happening inside of the Marx generator, you are likely to also hear a loud “clunk” that sounds like there is someone in there striking the wall of the generator with a ball-peen hammer. At this point, it is definitely time to crack it open and take a look inside.

Sometimes the breakdown that is occurring within the source is what we would characterize as a “long-path” breakdown or surface flashover. “Triple points” formed by high-potential conductors impinging upon an insulator surface within a gaseous insulation medium (hence, the three points) emit electrons that can cascade along the insulator surface on their way to ground (in the case of a negative-polarity output). Over time, this can begin to deposit carbon along the insulator that forms an electrical path to ground. Because this breakdown phenomenon takes time to develop, the resulting change in the waveform (often referred to as a “crowbar”, because it abruptly takes the output to ground) may not occur until later in the pulse. With each additional shot, the falling edge created by the crowbar may drift closer to the rising edge of the pulse as carbon is deposited on the surface of the insulator. Additionally, partial discharge can begin to occur well before this, which can look like a relatively high-impedance (versus an arc or bulk-breakdown) and will result in a reduction in output amplitude even before enough of a discharge forms to create the sudden drop to zero. Some fun examples of various insulator failures are shown below:

The other waveform to keep an eye on is the charge waveform. If you are using a capacitor-charging constant-current high-voltage power supply, there will likely be a voltage monitor port that can be viewed from a high-impedance oscilloscope input. Typically, these monitors operate either with a 1000:1 or 0-10V full-scale reading ratio and can be used to determine whether or not the generator is reaching full charge and/or triggering at the correct point in time. In the case of pulsed power systems using a gaseous insulation medium (APELC systems typically use compressed dry air), the pressure within the switch volume determines the hold-off of the switches/spark-gaps within the source. While APELC provides our customers with factory-determined pressure-voltage characteristics for our generators, changes in atmospheric pressure at the customer site, or even damage during shipping can cause these values to change and the switches to either break-down prematurely (“self-break”), or not fire at all. Whereas a correct charging waveform should show the voltage coming up to peak, holding for some amount of time, and then falling as soon as the trigger signal is sent, a self-break waveform would show the charge voltage going to zero before the charging cycle has completed. This either means a spark gap has fallen out of tolerance or the pressure is too high by a few psi. Conversely, if the charge waveform falls off slowly and does not suddenly go to zero when the trigger signal is sent, there is either an issue with the trigger, or the pressure in the generator is set too high. Lastly, if the power supply is current-limiting and only getting up to a few kV throughout the charge cycle- STOP! This could mean a spark gap is locking on, an insulator has punched through, or a capacitor has failed. An example of this occurring in a charge waveform is shown below:

 

If you are using a constant-voltage supply, such as a Glassman or Spellman, watching the current and voltage meters can tell you some of the information described above. We were sad to see power-supply manufacturers stop using analog panel meters, because oftentimes, watching a needle “bounce” could tell you if some small internal discharge was occurring within the Marx that might be the result of a broken lead or failing capacitor. Sadly, digital displays don’t provide this same level of feedback.

5. Take a look

Once you are quite certain that you have thoroughly vetted the diagnostics and attempted all possible settings- or if you are clearly hearing/seeing an indication of an internal failure, it is time to take a look. For an APELC system, we always recommend calling us first- especially if the system is still under warranty! Also, if you are not trained in proper high-voltage safety, you need to leave the lab immediately and not attempt to open, diagnose, or repair any high-voltage system. You are always responsible for your own safety. This is by no means a comprehensive training document, but hopefully provides some lessons learned for trained personnel to use who have not yet experienced some of the unique quirks related to pulsed power phenomena. Before working on any high-voltage system, ensure prime power is off and disconnected and that all elements capable of holding charge are connected to ground and positively discharged.

Here are some of the things we typically look (or sometimes smell) for when we open up a pulsed power system:

• • Is there the smell of burnt plastic? If so, there has either been a spark gap that is locking on or a bulk breakdown of the high-voltage insulation.
  • If there is solid insulation in between the Marx and ground-plane, can you shine a flashlight into the pressure vessel and see evidence of breakdown, flashover, or partial discharge near the output? Sometimes partial discharge requires getting at just the right angle with a flashlight to see the tell-tell “tree” pattern associated with the avalanche of charge finding it’s way to ground.
  • Is there any evidence of flashover/breakdown on the stage insulators?
  • Is there any debris in there? If a capacitor has failed, it will often blow a chunk out of its insulation.
  • Are all of the charge elements fully intact? A quick continuity measurement along the high and low-side charge elements can quickly diagnose this.
  • Are all of the spark gaps within the expected tolerance? Using a feeler gauge, measure each spark gap. It should be within several thousandths of the expected value.
  • Is there continuity all the way from the charge and trigger lines to their destinations within the Marx? Sometimes these cables can fail and a quick hi-pot test can determine if there is a breakdown within the cable insulation.
  • If the Marx is in an oil insulation, is there excessive carbon in the oil? This can happen over time in any oil-insulated system due to partial discharge, and a lack of regular filtering can lead to carbon deposits along insulator surfaces that may result in breakdown.

6. Don’t turn it up to 11!

Lastly, don’t be the person who turns the system all the way up just to make sure it is good and dead. Many of us (myself included) have been guilty of this when we reach a point of frustration in troubleshooting, but the result is often a repair that could cost $100 dollars becoming a complete replacement or rebuild of the system costing tens of thousands of dollars because a catastrophic failure resulted from continuing to push the system well past failure. Also, if you purchased the system from APELC, call us as soon as something is not operating as expected! We can save you a massive amount of time and cost by catching things well before they get worse.