InciWeb is failing

InciWeb fail

(Originally published at 5:31 p.m. MDT, June 20, 2013; updated June 21, 2013)

I have tried to be patient. I have given InciWeb the benefit of the doubt. I WANT to like InciWeb. The user interface looks fine, it strives to contain useful information, and if it worked well, it could provide a very valuable service.

Before U.S. Forest Service employee Jon Holden created it in California 10 years ago, it was difficult for the public to ferret out current information about fires that may be threatening their property. Some Incident Management Teams created web sites for fires, but they were all different, and if another IMTeam took over a fire they would sometimes start over again with a different web site, keeping the affected citizen guessing …  and hunting.

But now in its tenth year, I have to admit, InciWeb is a failure. To be precise, too often it fails to perform in a manner that meets the needs of its customers. After 10 years, it is reasonable to expect that a web service could have worked out the bugs and would be extremely reliable. ESPECIALLY, if the purpose of the web site is to provide critical information to users who in many cases at that moment are being adversely affected by an emergency. For some it may be one of the worst days of their lives. If a site with this purpose can’t have at least 99.99 percent uptime, and load a page reliably in less than three to four seconds, it needs to be put out of its misery.

How embarrassing is it that a web site existing primarily to provide information during emergencies has THIS as a banner across the top of the page:

Inciweb unresponsive

That banner has been on the site for months, not just for a few hours while a bug is being fixed.

In recent weeks the performance of the site has gotten even worse. Sometimes the home page loads in less than a second. Other times it will not load at all and an error message appears instead. Frequently if a page does load correctly, it can take 10 to 20 seconds for it to appear. Several times in recent days when the page loaded the formatting was completely screwed up and was unusable. This problem, I have found, is intermittent. At times the information is available very quickly, but too often, the site is virtually worthless.

And this is June — we are not yet into the busy part of the western wildfire season.

A person with knowledge of the system explained to me that the U.S. Forest Service insists that InciWeb reside on U.S. Forest Service government computer servers running on same network that hosts a crapload of routine USFS data and applications, and this results in conflicts and compromises that degrade the functionality of InciWeb.

Another person, Jennifer Jones, a spokesperson for the U.S. Forest Service which is the agency responsible for running the site, said another federal government site which has some similarities, in that many agencies use it, is Recreation.gov, a site used for planning vacations at federal sites and making campground reservations. But of course that site does not experience web traffic that surges when fires suddenly threaten or take out hundreds of homes. There are approximately ten U.S. Forest Service employees working full time on Recreation.gov. The site has a budget of around $2 million, while InciWeb does not have one person working full time to keep it running smoothly and has around $200,000 to work with each year under a contract with IBM. Another difference is that a portion of the fees that campers pay to make reservations on Recreation.gov is set aside to run the web site.

Ms. Jones said the USFS recognizes the problems with InciWeb and the agency is trying hard to mitigate them, using the limited resources available.

The software, servers, and the network hosting the site need to be robust enough that the site will continue to work when it is most needed — that is, when there are several large fires burning at the same time that suddenly affect many thousands of people. If we can’t depend on it to work when it is most needed, then it is a failure.

Web site designers and Information Technology specialists in the U.S. Forest Service and other land management agencies are not oriented to think about configuring a web site to handle a surging emergency affecting large numbers of citizens. For many of them their mind set is providing pages like “Camping with kids“, not providing information about ongoing emergencies that are affecting tens of thousands of citizens at the same time.

On April 22, the last time I ranted about InciWeb failures, I quoted a USFS spokesperson who said:

It appears that the current problem isn’t with the inciweb application itself but with the new Ness Application Portal through which you now log in to inciweb to enter and update incident information. The portal was instituted recently to enhance the security of fire and aviation management IT applications. The fire and aviation management IT staff is working to correct the issues as we speak.

Whatever the excuse, it is not relevant. This is a site primarily for dealing with emergencies. It needs to work, all the time.

The following agencies are listed as being responsible for InciWeb: National Wildfire Coordinating Group, U.S. Forest Service, Bureau of Land Management, Bureau of Indian Affairs, Fish and Wildlife Service, National Park Service, National Association of State Foresters, and FEMA.

These agencies should be ashamed to be listed as responsible for a service that has failed to get it right after 10 years.

They need to admit the emperor is wearing no clothes, and they must take InciWeb off the U.S. Forest Service computer system and put it on a non-government hosting service with a 99.999 percent uptime that can handle a surging load of tens (or hundreds) of thousands of site visitors at the same time. And the software, if necessary, should be upgraded to handle massive visitor traffic. And to do this and make it reliable, it may require additional funds from the federal and state agencies that have an interest and responsibility to make it work.

(UPDATE at 10:15 MDT, June 21, 2013:)

After publishing the article, we heard from some people who are charged with the task of inputting information about fires into InciWeb, and it turns out that their experience is similar or even worse than what the public endures when they try to access the site.

Here is an example we received in an email from someone on an Incident Management Team:

Thank you for your truthful words on Inciweb. We were told this “new and improved” version was going to actually be “new and improved.” It was frustrating before it was “fixed” but I spent one morning while on [a recent fire] trying to get the site to work for me. People knock those of us working for the government who have devoted our lives to public service don’t realize how many computer programs we are give that aren’t ready for prime time and never will be. I appreciate you saying what those of us in the government can’t. Keep up the good work.

R. Singh left a comment here on June 20, 2013:

You think that was bad, try being a PIO on a IMT and trying to update InciWeb. Your halfway done typing stuff in and the system kicks you off or won’t save the information and you lose it and have to redo it. And most Line Officers require the IMT PIO update the website 3-4 times a day. Sometimes 3-4 times a day requires a full time PIO spending 2-3 hours during each update. Very frustrating!!!

Today we talked with Richard Del Hierro, the U.S. Forest Service Branch Chief for Fire and Aviation Management Information Technology. Mr. Del Hierro told us that the federal land management agencies are moving toward combining 87 major fire-related programs and applications into a new program called Wildland Fire Information and Technology, affectionately called WFIT. They are even considering hosting it on a non-government computer system, such as Amazon Web Services or another cloud-type service. But WFIT is just getting off the ground so it could be a while before it has an impact.

WFIT sounds like a huge undertaking. We asked Mr. Del Hierro if it could turn into a computer software money pit we hear about too often, like the $170 million “upgrade” the FBI had to scrap. He said, no, that’s not going to happen.

In the meantime, Mr. Del Hierro, who was recently hired into a position that had been vacant for an extended period of time, said they are aware of the issues and complaints about InciWeb and within the next six to eight months most of the problems will be rectified. He said many of them are caused by the layers of security that are necessary when allowing the public to access data on a government computer system.

****

These articles with more information about InciWeb do not document every time we noticed a problem, but are links to where we wrote about it:

Author: Bill Gabbert

After working full time in wildland fire for 33 years, he continues to learn, and strives to be a Student of Fire. Google+

23 thoughts on “InciWeb is failing”

  1. A hundred simultaneous users, or even a thousand, should not take down a web site, not even a hobby site.

    Btw, I stumbled across this site while looking for info on the Doce fire. Good info here …

  2. Holy cow! You posting about INCIWEB is spot on and must have made a difference. I just went to INCIWEB and the sucker is LIGHTNING fast now. Someone important must have cruised by WT and flipped the power switch to low from ridiculously low!

  3. Yup. Could not agree more. Well said! I visit this site much more frequently than inci web anymore. Just got tired of the wait to connect and the information is just too slow. Thanks for what you do here.

  4. I’ve encountered all the issues Bill reported, plus some. Inciweb only started listing the Black Forest fire after the majority of damage was done. And the last time I looked, the East Peak fire wasn’t even listed.

    Here in Colorado Springs, the police are pleading with people to not call 911 to report smoke. If Inciweb worked as it should, this wouldn’t happen. People would use Inciweb to answer their own questions.

    The problem is obviously funding. Inciweb is a worthwhile tool and deserves proper support so it can work well.

  5. You think that was bad, try being a PIO on a IMT and trying to update InciWeb. Your halfway done typing stuff in and the system kicks you off or won’t save the information and you lose it and have to redo it. And most Line Officers require the IMT PIO update the website 3-4 times a day. Sometimes 3-4 times a day requires a full time PIO spending 2-3 hours during each update. Very frustrating!!!

  6. i find this website incredibly useful…even with the above…thank you for your time in providing this great resource…hope all the above comes around…your words are wise…take it to the cloud…they need to contact and learn from sites that are used to the high loads…go see how the big ticket broker sites handles it…and tell them to stop charging such exorbitant “convenience fees” 😉

  7. Regarding the Black Forest Fire and in defense of Inciweb (?!): Black Forest was a county fire until it transitioned to a federal management team. Initially, Black Forest was not a national incident and that is what Inciweb is intended for.

    1. InciWeb is available to by used by local and state organizations, as well as federal. But it is extremely user-UNfriendly and full of bugs for the unfortunate person charged with inputting the information — that is one reason it is not used as often as it could be by cities, counties, and states.

      1. It often seems that no one updates it on the weekends, even though fires burn on Saturday and Sunday. Or at least that is my perception.

  8. While I agree with many of the comments, I would also like to point out the obvious which is that the USFS does not get sufficient funding from Congress to support what people think Inciweb should be. Due to the new world of the internet, the public has expectations that Inciweb will provide real-time updates of fire locations, maps, evacuations, and burning activity, information that is not now available to fire fighters let alone the public. Add in that many incident commanders feel that Inciweb is a lower priority job than others, plus remote communication connections are tenuous, it is surprising that it works at all some days.

    1. Re. tenuous communications…Anecdotally, it took ~3 days for the SWCC Type 2 IMT on Tres Lagunas incident to get any internet at all at the ICP – eventually it was DSL. Those who’s job depended on it (PIO, IMET, FBAN, etc) had to shuttle back and forth to the Ranger Stn in Pecos, NM to get online. That was only a few minutes away…imagine if it was a half-hour.

      1. BGAN terminals work anywhere in the world. Emergency services were using hundreds of them during the Haiti Earthquake. IMET’s carry them. They worked great duing the Deep Water Horizon oil spill.

  9. I dont have problems any of the other Federal wildland fire sites I read daily, some multiple times a day….

  10. Most of what has been said here is correct, and Bill pointing out that Inciweb is available for any municipality, whether it be local or federal is correct. As a PIO of a large national IMT I can state for sure that many of the type 1 and type 2 national IMTs do not activate Inciweb for their incidents as quickly as they should. With my team either someone I delegate or myself creates the incident in Inciweb with current information remotely in order to get it out as quickly as possible – BECAUSE PEOPLE WANT TO KNOW! I also make every effort to update my incidents EVERY HOUR in Inciweb which, aside from keeping our incident near the top of the list (which I consider extremely important when my fire is in the early and often most volatile stages), it lets the public know that we care and are making a serious effort to provide the information experience tells us they want to know. Do other type 1 teams perform at this level? I must admit that the answer is no. And unfortunately, even when my team and the PIOs I supervise update Inciweb hourly, the system we call Inciweb is often bogged down.

    Solutions? I won’t get into this here as they hinge on how it is managed, or rather in my opinion mismanaged. However I will say that sites like this one and wildlandfire.com do an incredible service and fill a huge void that probably won’t be going away soon!

    PIO for the public

  11. Wooowee

    After reading this combined with the airtanker program………

    Would I entrust this bunch with a UAS ( Unmanned Aerial Systems) program?

    I mean, if you can not manage, troubleshoot, and get something such as servers and other IT systems up to speed…

    Would I want be flying around in GA or civilian aircraft around the National Airspace System (NAS) while a USFS desired UAS / UAV is up roaming around in the future looking for hotspots or bug kill (if and when the USFS enters the UAS program)

    I mean if they can not master the airtanker program and now Inciweb……I would be truly fearful that with diminishing budgets…..and if I was a Congressman….I would DEMAND the USFS try to master the programs they are already charged with (AKA land management) and try not to screw up any more systems or at least try to SUSTAIN the programs they already have

    Like grant programs that come with strings attached such sustainability after the grant program life……..how are these programs going to run without the dinero required? The land management agencies sure ought to be required to be better stewards of the land…..and DEMONSTRATE it before they get anymore bright ideas of more technology, especially anything in the flying arena…manned or unmanned

    Sheeeeeeshh!!

  12. I believe that Bill’s influence had made a difference! I have never seen INCIWEB so fast as it is this morning!

  13. In the last few weeks since Colorado fire season started, InciWeb has been *consistently* impossibly slow to load. (:-( Bewildered that *anyone* is *ever* experiencing fast load times; for me, it *always sucks*. (:-( JFTHOI, I found a website that checks site performance; it says InciWeb is slower than 97% of other sites checked – couldn’t get much worse. (:-(

    What’s especially embarrassing and frustrating to this site user is that it’s branded as a collaboration of all these various government entities, suggesting they could each kick in some funds for infrastructrure, hardware, software and personnel (design, engineering tech support), and produce a rich and proud exemplar of tech innovation. Instead, it’s consistently hands down the slowest website I (try to) use and, sadly, just makes the government look bad. (:-(

    Where do concerened citizens like us apply political pressure to fix this mess so the firefighters can quickly and efficiently communicate important information to an interested – and expecting – public in a timely fashion?

Comments are closed.