[SANS ISC] When Lightning Strikes. What works and doesn’t work., (Thu, Aug 19th)

Living in Florida, afternoon thunderstorms are a regular occurrence with Florida having the highest lightning density of any state in the US [1]. In my time in Florida, I had close or direct strikes damage equipment twice. The most recent incident was about a month ago. So I am sharing here some of the things that work and don’t work.

This most recent strike didn’t hit the house directly as far as I can tell. But the house was engulfed in a large flash lasting several seconds. I am including a video clip of the strike as it was captured on a security camera. Lightning can cause damage in different ways. First of all, a direct strike will of course inject significant voltage and current, causing equipment to melt and even fire. A few neighbors were affected by such a strike hitting a cable TV line, melting several cable modems, and causing one small fire. The more likely damage is however not from a direct strike. Even a close strike, like the one I experienced, can cause damage as the strong electric field will induce currents that will in particular affect low voltage equipment like networks. Networks again are particularly sensitive. Longer cables may pick up more of the electric field.

First a list of the equipment the strike damaged (none of the equipment was visibly damaged):

Cable modem.
Firewall (only the port connected to the cable modem)
One PoE switch lost its PoE function but still passed traffic.
A second PoE switch lost a couple of ports.
a small PoE powered switch was completely dead
Apple TV wired network port was dead but works otherwise
The projector powered on but displayed no picture
The receiver connected to the projector would no longer output an HDMI signal
The subwoofer connected to the receiver was dead.

I may have missed a couple of things, but needless to say, the damage was substantial.

Surge Protectors / UPS

All equipment but the project was connected to a UPS. But I don’t think the UPS played a role here. It may have prevented worse damage. So far, it looks like all the damage was caused via network ports (the receiver was connected to the dead PoE switch, but there is also a long HDMI cable from receiver to project that may have played a role).

I do have a surge protector that is well-grounded as the cable enters the house. Note that cable companies usually only ground the cable, and do not install a surge protected. Coax surge protectors are a bit tricky. “Gas Tube” surge protectors that are sometimes used can wear out over as little as 5 years or even built up a charge that itself can cause damage. The surge protector in my case had no visible damage and still appears to work fine, For damage caused by current induced by a close-by lightning discharge, surge protectors are not of too much use. 

PoE Equipment

My Power-over-Ethernet (PoE) equipment did a lot worse than other equipment. There are various anecdotes that can be found that may support that PoE is most sensitive to lightning. The Ethernet standard does include some requirements for over-voltage protection [2] but of course, there are limits, PoE in particular adds additional components like transformers that are vulnerable to excessive voltages. 

Fiber

My network uses some fiber runs in part to electrically isolate network segments. Part of the network is in a separate building with its own power feed and ground point. Back in my physics days, I dealt a lot with sensitive electronics close to high voltage systems and ground loops were an ongoing issue, so I decided early on to use fiber for some of the longer connections (still well within the 100m copper ethernet limit). This strategy worked very well and likely helped contain the damage. Most of the damage appears to have happened either by currents induced by the high potentials of the lightning, or by voltage spikes traveling via network cables.

Recovery

Initial debugging showed that the firewall and the cable modem were out. I do have an LTE modem connected to the firewall. Only the port connected to the cable modem was damaged, and the LTE modem worked well, but due to the cheap data plan I am using, the LTE modem ran out of data within about an hour. Comcast sent a technician next day to replace the modem (I am using static IP addresses which requires a leased modem). 

For the firewall, I did have a spare that was not powered on and I replaced the damaged firewall. The damaged switches worked well enough initially.For the most part, only PoE devices (couple of security cameras and wireless access points) didn’t work. I had an old spare switch around, but it was a different type and would have required significant configuration so I decided to wait the two days until the replacement switch arrived. Luckily a replacement switch was readily available.

Lessons Learned

Fiber works! It probably protected my main workstation and with that the most valuable asset that would have been expensive to replace. It is hard to tell if UPSs played a role. Another important lesson is to have some powered-off “cold standby” equipment. Automatic failover and such is nice to have, but in this case, the failover switch/firewall would likely have been damaged as well. As for backup internet connectivity, I will be trying the unlimited 5G home internet which just became available. The speed of the LTE modem was barely usable and having limited data plans was a pain.

Final Note

A couple of days after I replaced the cable modem, I had another odd network issue: All of a sudden, only IPv6 connections worked, and IPv4 failed. At the time, I was just reconfiguring IPv6 on the firewall, as my IPv6 allocation changed with the modem swap. So I suspected the firewall, undid my last change, but still no luck. It took me a couple of hours until I realized that IPv4 still went over the LTE modem, while IPv6 used the cable modem, and the LTE modem had just run out of data again. But due to multiple equipment changes along the way, this was the last thing I checked “retracing” my configuration path.

Video of the lightning strike

[1]https://www.weather.gov/media/safety/09-18Flashes_Flash_Density_State.pdf
[2]https://incompliancemag.com/article/designing-ethernet-cable-ports-to-withstand-lightning-surges/

Johannes B. Ullrich, Ph.D. , Dean of Research

 

 

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Source: Read More (SANS Internet Storm Center, InfoCON: green)

You might be interested in …

[ZDNet] Human Rights Commission calls for a freeze on ‘high-risk’ facial recognition

All posts, ZDNet

Until protections around the use of such technologies are in place, the Australian Human Rights Commission has asked for a moratorium on the use of biometrics, including facial recognition, in ‘high-risk’ areas. It has also recommended the creation of an AI Safety Commissioner. Source: Read More (Latest topics for ZDNet in Security)

Read More

[TheRecord] Google removes support for FTP and old-gen U2F security keys in Chrome 95

Google has released today Chrome v95, the latest version of its popular web browser, a version that contains several changes that will likely cause problems for a considerable part of its users. The problematic changes include: removing support for File Transfer Protocol (FTP) URLs — ftp://removing support for the Universal 2nd Factor (U2F) standard, used […]

Read More

[BleepingComputer] The Week in Ransomware – July 23rd 2021 – Kaseya decrypted

This week has quite a bit of news ranging from the USA formally accusing China of the recent ProxyLogon vulnerability and Kaseya mysteriously obtaining the universal decryption key. […] Source: Read More (BleepingComputer)

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.