Verizon’s throttling of data rates used by a fire department that subscribed to one of the company’s “unlimited” plans hampered the firefighters’ command and control at the fire.
While battling the Mendocino Complex, which has become the largest wildfire in the recorded history of California, the Santa Clara Fire Department deployed OES Incident Support Unit 5262, a command and control resource. Its primary function is to track, organize, and prioritize routing of resources from around the state and country to the sites where they are most needed. OES 5262 relies heavily on the internet to do near-real-time resource tracking.
This unit and other resources in Santa Clara County use web-based applications that rely on high-bandwidth, latency-sensitive exchanges of information with the public and to provide crucial public safety services.
While fighting the fire the County discovered the Verizon data connection for OES 5262 was being throttled. Data rates had been reduced to 1/200th, or less, than the previous speeds. Fire Chief Anthony Bowden wrote in a court filing that the “reduced speeds severely interfered with the OES 5262’s ability to function effectively”. The County has signed on to a legal effort to overturn the Federal Communication Commission’s repeal of net neutrality rules.
Below is an excerpt from an article in the San Francisco Chronicle:
Despite having paid for what it thought was an unlimited data plan, the Santa Clara County Central Fire Protection District saw its data flow “throttled” down to 1/200th of its usual speed as it fought the complex — now the biggest wildfire in state history — because Verizon officials said it had exceeded its plan limit, district Fire Chief Anthony Bowden wrote. This primarily hampered a specialized vehicle the department depends on to coordinate its machinery and staff in such emergencies, and Bowden said that put his battalions at risk.
Without full-speed service for the high-tech command and communications rig, which goes by the arcane name of OES 5262, Bowden wrote, “resources could be deployed to the wrong fire, the wrong part of a fire, or fail to be deployed at all. Even small delays in response translate into devastating effect, including loss of property, and, in some cases, loss of life.
One of the fire captains complained to Verizon that the command and control unit had been so hobbled that “it has no meaningful functionality”.
The battle with the fire morphed into a battle with Verizon as fire department personnel fought with the company about restoring their “unlimited” data rate. Eventually after getting various sections in Verizon and the Fire District involved, the cell phone plan in OES 5262 was upgraded to a more expensive plan that had more capability.
In the last couple of years all four major cell phone providers have advertised “unlimited” data plans. All of them ARE LIMITED in various ways, so it is inconceivable how the Federal Trade Commission lets them get away with false and misleading advertising.
An article published by C|NET on August 9 does a good job of comparing “unlimited” plans offered by Verizon, Sprint, T-Mobile, and AT&T. Of the 10 plans described, all except one have data limits, while the one that does not, limits speed used on hotspots to only 3G. Everyone is now used to 4G speeds or the even faster LTE. 5G, with much higher data rates, is just around the corner. The companies disguise how speeds will be greatly reduced after a data limit is obtained, by using words like “prioritize your data”, “deprioritized”, or just blatantly saying “customer may temporarily experience reduced speeds on these line(s) during times of network congestion”. It likely that during an emergency that affects a large number of citizens, “network congestion” will occur.
We have written many times about the “Holy Grail of Wildland Firefighting Safety”, knowing the real time location of the fire and firefighters. Depending on how these systems are configured they could rely on data delivered through the internet. If that data stream is throttled to 1/200th, is cut off, or becomes unreliable, the safety of firefighters and the public could be threatened.
The intentionally misleading use of the term “unlimited” by the four cell phone carriers is part of the problem here. The FCC and the Federal Trade Commission should do their job and stop this practice.
It’s interesting that the Wildfire Today readers aren’t familiar with FirstNet. I really enjoy reading everything here, and rely heavily on this site as a reference source for the R&D work I am doing. In fact, I came to this site to see how firefighters responded to this incredible Verizon foul-up, so maybe I can provide some useful information in return.
Long story short, prioritized public safety (PS) broadband cellular access is what FirstNet is all about. It has been a long time coming, but AT&T, as the network provider, has begun implementing the plan for a nationwide, interoperable PS communications network. It promises to revolutionize PS comms. Here’s a link:
https://firstnet.gov/
Verizon, who didn’t even bid on the opportunity to host, is engaging AT&T in a what I suspect will be a fiercely competitive battle to retain their PS business, if it isn’t already so. Here’s a couple more links:
https://www.linkedin.com/pulse/being-best-network-matters-public-safety-mike-maiorana/
https://www.critcomminsights.com/analysis/2017/9/29/verizon
Looking at search results related to this news item, I would think they received a huge black eye from this, and I would think Mr. Buss might have gotten a talking-to.
https://calfire.blogspot.com/2018/08/verizon-throttled-santa-clara-county.html
In addition to what has already been said, which is spot-on, personally, I wonder why the Officer Farrelly re-engaged with Verizon by e-mail after they hit their data limit in the field? I see it took Verizon another 1/2 day to get back to him, at a minimum. My other personal comment is that I don’t think this has anything to do with Net Neutrality.
More generally, hardening of FirstNet infrastructure, capacity and coverage, and AT&T’s ideas for using the PS Band 14 as part of commercial offerings are among the questions that all PS practitioners will need to track and see addressed to fulfill the promise of FirstNet. I hope the readers of WFT can educate themselves about FirstNet and join the conversation. From what I’ve seen fireground communications need updating, and there are significant technological opportunities that would make your engagement worthwhile, including potentially achieving the Holy Grail. I’d be happy to help in developing this understanding in any way if I can. Thanks for the opportunity to learn and comment here on WFT.
Respectfully,
Ed
Having lived through numerous fires, the last thing I count on is cell service. Towers get jammed with callers, lose power, and burn. This is not a reliable service for disaster operations.
Fire opps needs a robust, stand alone, redundant system that can be counted on.
UCSD’s HPWREN has just that in San Diego.
Having to call customer service located in some unknown county/country during a major indecent is a sure sign of poor planning on your part.
You are right this proves that you can not count on for profit cell phone companies to provide reliable service in a disaster. I am in a remote region of Ontario we have a number of coop cell companies that work wit the fire and ambulance depts.
So who’s job is it to read the fine print in contracts; the provider, or the customer?
I know an old IMT salt who has an I-phone that’s configured to get data priority over other traffic on the network. He got it so he could still make calls and the like during hurricane response when connectivity is severely restricted. Apparently it came with lots of restrictions on the security side like not being able to download any third party apps and so forth and even the feds are very tightfisted with it. I wish I knew more about it but the technology does exist to give first responders priority access to cellular networks.
As a federal employee I used to have a card with a special phone number and access code that would give me access to priority land lines if needed during an emergency when regular phone lines were overwhelmed.
Verizon has been doing that for years. I used to work for a Verizon construction contractor. My work phone was Verizon, and in places where there was limites cell bandwidth I could dial a number and see another person immediately lose service.
I think OES needs to not rely on hotspots, but this shows the value of net neutrality.
As I understand things, the repeal of net neutrality creates the potential for a carrier to prioritize traffic.
That would seem to be the other side of the debate where emergency services could legally have unfettered priority in certain circumstances which, presumably, would have been quite welcome to fire authorities in this case.
Reinstating neutrality would seem to throw away the priority potential. I sure don’t know what’s best but, as always, there are trade-offs.
I understand the frustration with having slow internet service, I have Verizon and an unlimited plan and I was aware that after a certain data level I get throttled down. I understand the need for OES 5262 to be connected but just relying on Verizon “hot spots” for the internet is interesting. I feel old saying this, but I remember a time where we fought fires without cell phones and all of that. I feel like someone should have had the foresight to look into the overall system and potential issues with so many resources in an area all using bandwidth, may it be through a LAN cable or Broadband.
Picture it this way…when there is a small town with a fire near it, what happens to the local gas station?…Everyone comes in a buys all the chew and energy drinks, then they don’t have them for a few days or even the duration of the incident. Its like when we get mad that we cant use the fairgrounds for fire camp because there is currently a County fair going on, some people don’t seem to grasp that the world doesn’t stop because we want it to. It is very easy for us as an emergency response community to overwhelm the infrastructure of any given area, be it the roads, gas stations, internet, ect.
Good points in your post. And Verizon should have done the right thing in this case and put a priority on life and safety and resolved the inadequate plan issue later.
Perhaps fire suppression could be throttled back for Verizon CEOs if they’re ever in a fire event, wildland or structural?
The public those directly and indirectly affected with home and business losses, insurance companies, municipalities, counties, energy grid providers should be involved in any class action litigation going forward. Thank the current administration and Congress for allowing and encouraging the FCC and FTC for allowing this practice to continue unabated.
Wow. I hope Verizon gets their a**es handed to them in a lawsuit.