InciWeb is failing

Bill Gabbert
June 20, 2013

(Originally published at 5:31 p.m. MDT, June 20, 2013; updated June 21, 2013)

I have tried to be patient. I have given InciWeb the benefit of the doubt. I WANT to like InciWeb. The user interface looks fine, it strives to contain useful information, and if it worked well, it could provide a very valuable service.

Before U.S. Forest Service employee Jon Holden created it in California 10 years ago, it was difficult for the public to ferret out current information about fires that may be threatening their property. Some Incident Management Teams created web sites for fires, but they were all different, and if another IMTeam took over a fire they would sometimes start over again with a different web site, keeping the affected citizen guessing … and hunting.

But now in its tenth year, I have to admit, InciWeb is a failure. To be precise, too often it fails to perform in a manner that meets the needs of its customers. After 10 years, it is reasonable to expect that a web service could have worked out the bugs and would be extremely reliable. ESPECIALLY, if the purpose of the web site is to provide critical information to users who in many cases at that moment are being adversely affected by an emergency. For some it may be one of the worst days of their lives. If a site with this purpose can’t have at least 99.99 percent uptime, and load a page reliably in less than three to four seconds, it needs to be put out of its misery.

How embarrassing is it that a web site existing primarily to provide information during emergencies has THIS as a banner across the top of the page:

That banner has been on the site for months, not just for a few hours while a bug is being fixed.

In recent weeks the performance of the site has gotten even worse. Sometimes the home page loads in less than a second. Other times it will not load at all and an error message appears instead. Frequently if a page does load correctly, it can take 10 to 20 seconds for it to appear. Several times in recent days when the page loaded the formatting was completely screwed up and was unusable. This problem, I have found, is intermittent. At times the information is available very quickly, but too often, the site is virtually worthless.

And this is June — we are not yet into the busy part of the western wildfire season.

A person with knowledge of the system explained to me that the U.S. Forest Service insists that InciWeb reside on U.S. Forest Service government computer servers running on same network that hosts a crapload of routine USFS data and applications, and this results in conflicts and compromises that degrade the functionality of InciWeb.

Another person, Jennifer Jones, a spokesperson for the U.S. Forest Service which is the agency responsible for running the site, said another federal government site which has some similarities, in that many agencies use it, is Recreation.gov, a site used for planning vacations at federal sites and making campground reservations. But of course that site does not experience web traffic that surges when fires suddenly threaten or take out hundreds of homes. There are approximately ten U.S. Forest Service employees working full time on Recreation.gov. The site has a budget of around $2 million, while InciWeb does not have one person working full time to keep it running smoothly and has around $200,000 to work with each year under a contract with IBM. Another difference is that a portion of the fees that campers pay to make reservations on Recreation.gov is set aside to run the web site.

Ms. Jones said the USFS recognizes the problems with InciWeb and the agency is trying hard to mitigate them, using the limited resources available.

The software, servers, and the network hosting the site need to be robust enough that the site will continue to work when it is most needed — that is, when there are several large fires burning at the same time that suddenly affect many thousands of people. If we can’t depend on it to work when it is most needed, then it is a failure.

Web site designers and Information Technology specialists in the U.S. Forest Service and other land management agencies are not oriented to think about configuring a web site to handle a surging emergency affecting large numbers of citizens. For many of them their mind set is providing pages like “Camping with kids“, not providing information about ongoing emergencies that are affecting tens of thousands of citizens at the same time.

On April 22, the last time I ranted about InciWeb failures, I quoted a USFS spokesperson who said:

It appears that the current problem isn’t with the inciweb application itself but with the new Ness Application Portal through which you now log in to inciweb to enter and update incident information. The portal was instituted recently to enhance the security of fire and aviation management IT applications. The fire and aviation management IT staff is working to correct the issues as we speak.

Whatever the excuse, it is not relevant. This is a site primarily for dealing with emergencies. It needs to work, all the time.

The following agencies are listed as being responsible for InciWeb: National Wildfire Coordinating Group, U.S. Forest Service, Bureau of Land Management, Bureau of Indian Affairs, Fish and Wildlife Service, National Park Service, National Association of State Foresters, and FEMA.

These agencies should be ashamed to be listed as responsible for a service that has failed to get it right after 10 years.

They need to admit the emperor is wearing no clothes, and they must take InciWeb off the U.S. Forest Service computer system and put it on a non-government hosting service with a 99.999 percent uptime that can handle a surging load of tens (or hundreds) of thousands of site visitors at the same time. And the software, if necessary, should be upgraded to handle massive visitor traffic. And to do this and make it reliable, it may require additional funds from the federal and state agencies that have an interest and responsibility to make it work.

(UPDATE at 10:15 MDT, June 21, 2013:)

After publishing the article, we heard from some people who are charged with the task of inputting information about fires into InciWeb, and it turns out that their experience is similar or even worse than what the public endures when they try to access the site.

Here is an example we received in an email from someone on an Incident Management Team:

Thank you for your truthful words on Inciweb. We were told this “new and improved” version was going to actually be “new and improved.” It was frustrating before it was “fixed” but I spent one morning while on [a recent fire] trying to get the site to work for me. People knock those of us working for the government who have devoted our lives to public service don’t realize how many computer programs we are give that aren’t ready for prime time and never will be. I appreciate you saying what those of us in the government can’t. Keep up the good work.

R. Singh left a comment here on June 20, 2013:

You think that was bad, try being a PIO on a IMT and trying to update InciWeb. Your halfway done typing stuff in and the system kicks you off or won’t save the information and you lose it and have to redo it. And most Line Officers require the IMT PIO update the website 3-4 times a day. Sometimes 3-4 times a day requires a full time PIO spending 2-3 hours during each update. Very frustrating!!!

Today we talked with Richard Del Hierro, the U.S. Forest Service Branch Chief for Fire and Aviation Management Information Technology. Mr. Del Hierro told us that the federal land management agencies are moving toward combining 87 major fire-related programs and applications into a new program called Wildland Fire Information and Technology, affectionately called WFIT. They are even considering hosting it on a non-government computer system, such as Amazon Web Services or another cloud-type service. But WFIT is just getting off the ground so it could be a while before it has an impact.

WFIT sounds like a huge undertaking. We asked Mr. Del Hierro if it could turn into a computer software money pit we hear about too often, like the $170 million “upgrade” the FBI had to scrap. He said, no, that’s not going to happen.

In the meantime, Mr. Del Hierro, who was recently hired into a position that had been vacant for an extended period of time, said they are aware of the issues and complaints about InciWeb and within the next six to eight months most of the problems will be rectified. He said many of them are caused by the layers of security that are necessary when allowing the public to access data on a government computer system.

****

These articles with more information about InciWeb do not document every time we noticed a problem, but are links to where we wrote about it: