A 9-year USFS aerial firefighting study left many questions unanswered

After 9 years and more than $11 million

Air Tanker 02 Drops on the Creek Fire
Air Tanker 02 Drops on the Creek Fire on Camp Pendelton Marine Base, December 24, 2020. CAL FIRE image.

-This article was first published on Fire Aviation-

In fiscal year 2018 the U.S. Forest Service spent more than half a billion dollars, $507,000,000, on air tankers, helicopters and other firefighting aircraft.

The agency’s spending on aircraft contracts, support, and fire suppression operations has gone on for decades with little meaningful oversight. The Forest Service has been repeatedly asked to justify the expense by the Government Accountability Office, the Department of Agriculture’s Inspector General, and Senators and Representatives in committee hearings — “How do you know air tankers are effective?”

A report by the GAO in August, 2013 said, “None of the agencies’ studies and strategy documents contained information on aircraft performance and effectiveness in supporting firefighting operations, which limits the agencies’ understanding of the strengths and limitations of each type of firefighting aircraft and their abilities to identify the number and type of aircraft they need,”

The Inspector General’s investigation concluded, “[The Forest Service] has not used aviation firefighting performance measures that directly demonstrate cost-impact…”

In 2012 the Forest Service began the Aerial Firefighting Use and Effectiveness (AFUE) study to address those concerns. After nine years and an annual cost of $1.3 million plus overtime for the field data collectors, a report about the study was quietly released August 20, 2020 during the peak of an exceptionally busy wildland fire season.

The AFUE had very ambitious goals initially when Tom Harbour was the Director of Fire and Aviation for the U.S. Forest Service.

“AFUE was initially intended to eventually help answer questions about the size and composition of aviation assets needed by the USFS,” Mr. Harbour told Fire Aviation recently.

From the agency’s AFUE website:

The desired outcome is to support training, mission selection and execution, and overall aerial fleet planning to enhance effectiveness and cost-effectiveness, potentially reducing aviation and fire suppression costs by answering a general, but complex question: “What are the best mixes of aircraft to do any fire suppression job?”

The data in the study was collected by four crews, or modules, of three to four single resource qualified firefighters, each with 10 to 25 years of firefighting experience. The modules mapped aerial drop activity and recorded incident objectives, outcomes, and conditions for aerial suppression actions that supported tactical and strategic incident objectives. The module coordinator coordinated crew movements.

AFUE personnel applied analysis protocols to data after observing 27,611 drops from 2015 to 2018 at incident locations throughout the USA in 18 States and across all nine Forest Service regions.

Other studies

This was not the first time that a study took on the task of determining the aircraft mix needed to assist wildland firefighters in the United States or to evaluate aerially applied fire retardant. The Inspector General’s report listed seven, most of which are on the Wildfire Today Documents page.

Additional studies not mentioned in the Inspector General’s report:

Size of USFS Large Air Tanker Fleet
Number of USFS Large Air Tankers on Exclusive Use contracts at the beginning of each year.

Which fires were analyzed in the AFUE study?

The fires at which data was collected were primarily large that escaped initial attack, since it takes time to mobilize the modules. Smaller fires that were stopped by ground and air resources are likely underrepresented; that is, fires on which aircraft were most effective may not show up in the data. Fires burning during high or extreme fire danger that grew large because of the burning conditions may be overrepresented. As conditions become extreme, firefighting aircraft are less effective.

From the study:

[T]he sample may be biased towards incidents with substantial aircraft activity and especially those with any airtanker activity. Because AFUE was launched primarily to evaluate large and very large airtankers, choices were consistently made to observe fires with airtanker activity. Recognizing that many fires that receive any airtanker drops typically only receive a few drops, the sample could be underrepresenting fires with limited airtanker activity. Further, many aerial firefighting drops occur on remote fires that make direct observation challenging.

What were the findings of the AFUE?

Much of the AFUE report is based on two performance measures that the study used to determine the effectiveness of an aircraft, Interaction Percentage (IP) and Probability of Success (POS). IP, a term apparently invented, is defined as the proportion of drops that interacted with fire. POS is the number of effective drops divided by the total number of drops with known and interacting outcomes.

Interaction Percentages firefighting aircraft AFUE
Interaction Percentages, from AFUE

The interaction percentage data compares apples and oranges. Helicopters and scoopers primarily drop water, while fixed wing tankers that are not scoopers almost always drop long term fire retardant. Since water is a very short term fire retarding agent, it is usually dropped directly on the flaming front. If it were dropped out ahead of the fire, much of it would run off the fuel, soak into the ground, or evaporate before the fire reached that location.

Long term fire retardant dropped by air tankers is usually placed ahead of the fire. It might be dozens of feet away, or when pretreating a ridgeline, protecting a point, or securing a planned indirect fireline it could be thousands of feet away from the flaming front. Retardant, much more viscous than water, adheres to the vegetation more so than water, retains moisture for a while, and can even interfere with the process of combustion after it dries.

Therefore, comparing the interactions of water dropping and retardant dropping aircraft is not a reasonable exercise. Water droppers should always be very close to 100 percent on the interaction scale, while retardant droppers will have lower numbers, in part because some of the drops are done to support indirect firelines or ignition operations that did not interact with the main fire.

Helicopter 3PA, an AS350B (N833PA)
Helicopter 3PA, an AS350B (N833PA) on the Elephant Butte Fire southwest of Denver, July 13, 2020. Photo by skippyscage.com.

The chart which shows small Type 3 helicopters having 100 percent interaction does not mean that dropping 100 gallons of water is going to have a larger overall fire-slowing result than a 75 percent interaction DC-10 very large air tanker dropping 94 times as much liquid.

The interaction rates of single engine, large, and very large air tankers all range from about 74 percent to 80 percent. And in the helicopter category, it is about 87 percent to 100, with the small 100-gallon Type 3 having the highest number. The largest Type 1 helicopters carry 2,500 to 3,000 gallons; their interaction percentage is about 10 points higher than the average retardant dropping air tanker.

Drop Outcomes, AFUE
Drop Outcomes, AFUE

The study also rates the aircraft on the probability of success, only taking into account drops that actually interacted with the fire. When used on a large fire the helicopters averaged about 0.73 and the retardant dropping air tankers, about 0.72. If excluding the small Type 3 helicopters which are not often used to drop water on large fires, the helicopter average increases to about 0.84

What did the AFUE study recommend?

Continue reading “A 9-year USFS aerial firefighting study left many questions unanswered”

Congress to hold hearing on “Wildfire Management in the Midst of COVID-19”

Save the date: June 9, 2020 at 10 am EDT, 7 am PDT

senate committee hearing fire four-person panel
On June 13, 2019 a four-person panel provided testimony before the Senate Energy and Natural Resources Committee. L-R: Shawna Legarza, Director, Fire Aviation and Management, Forest Service; Jeff Rupert, Office of Wildland Fire, Department of the Interior; Chris Maisch, Alaska State Forester & National Association of State Foresters; and Wade Crowfoot, Secretary, California Natural Resources Agency. Screenshot from the Committee video.

The Senate Energy and Natural Resources Committee will hold a hearing at 10 a.m. EDT June 9, 2020 on “Wildfire Management in the Midst of COVID-19”. It will be webcast live on the committee’s website, and an archived video will be available shortly after the conclusion. Written witness testimony will be available on the website at the start of the hearing.

This could be very interesting, depending on what questions the Senators ask and if they follow up when the witnesses give vague or evasive answers. In 2019 Shawna Legarza, the Forest Service Fire and Aviation Director, was one of four on a panel. If she appears this year it will be with only 21 days remaining before her announced retirement date.

Shawna Legarza, Fire and Aviation Director, Forest Service
Shawna Legarza, Fire and Aviation Director, Forest Service, June 13, 2019 at a hearing before the Senate Energy and Natural Resources Committee. Screenshot from the Committee video.

This committee regularly holds hearings about the activities of the land management agencies, but also has hearings specifically about wildland fire topics. Senators sometimes press the Forest Service and the Department of the Interior agencies on topics such as the number of air tankers on contract, using technology to track fires and resources, transferring management of the Job Corps Civilian Conservation Centers from the FS to the Department of Labor, is the FS asking for enough funding to accomplish their goals, sexual harassment, and the outlook for the coming fire season.

Obviously this year the issue of fighting fire during the pandemic will come up. Another possible topic is accountability and lack of transparency for how decisions are made about contracting for firefighting aircraft and how taxpayers’ dollars are being used. Are they being spent wisely? When will they release the Aerial Firefighting Use and Effectiveness Study that has been going on for eight years? Launched in 2012 at a cost of about $1.3 million annually, the study is supposed to quantify the effectiveness of the various types of fixed and rotor wing aircraft used on wildfires. In FY 2017 for example, the most recent year with exact numbers available, the agency spent over half a billion dollars on fire aviation; $507,000,000. If ever completed the AFUE study could make it possible to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet — to use the best, most efficient, and effective tools for the job.

In hearings before the Senate Energy and Natural Resources Committee in 2018 and 2019 the Forest Service told the Senators the results of the study would be released “soon”. In another hearing in February, 2020 Forest Service Chief Vicki Christiansen again said it would be released “soon”. When pressed by Colorado Senator Sen. Cory Gardner, who last year made his opinion about the delay very clear, she said it would be released “this Spring”. Senator Gardner said, “Before June?”. She said, “Yes”.  A clip from that exchange is below.

Link to the entire hearing

(Members of the Energy and Natural Resources Committee)

(The list of members of the committee was replaced with the more current link above)

Senate committee discusses Forest Service budget issues with the Chief

5% reduction in USFS budget, fuel reduction funds, air tanker study, CFLRP, and LWCF

CL 415 on Colby Fire
CL-415 on the Colby Fire near Glendora, California, January, 2014. Photo by Jeff Zimmerman.

In addition to grilling the Chief of the Forest Service about hostile workplaces, several other issues were covered in a hearing Tuesday before the Senate Energy and Natural Resources Committee.

A video recording of the hearing is available at the Committee’s website. It begins at 19:48.

At 56:30 in the video Washington Senator Maria Cantwell asked Forest Service Chief Vicki Christiansen about the $545 million that was appropriated for fuel management in the recent omnibus legislation but was not mentioned in the administration’s proposed budget for FY 2020 which begins October 1. The Senator asked for assurances that the funds would still be available and would be used for that purpose. The Chief would not commit to the funds still being available, saying, “We will use whatever resources are given to the agency”.

The Chief reminded the Senator that the White House directed the Forest Service to cut its overall budget by five percent.

Senator Cantwell also mentioned very briefly at 59:00 in the video the availability of CL-415 water scooping air tankers but the issue was not discussed. The Forest Service, even though funds are available and a vendor offered the aircraft at a greatly reduced rate this year in a meeting with Chief Christiansen and Fire Director Shawna Legarza (according to our sources), the agency does not plan to have any scoopers on exclusive use contracts for the second year in a row. Historically the FS does not hold scoopers in high esteem even though they are used extensively in Canada and Europe. The 2012 Rand Study, which the agency attempted to keep secret (and did so successfully for two years), recommended a heavy emphasis on water-scooping air tankers and fewer conventional air tankers, which would have been a monumental shift in the paradigm.

Senator Murkowski said (at 1:39:30 in the video) that during a hearing a year ago the committee was told that results from the Aerial Firefighting Use and Effectiveness (AFUE) study would be released “soon”. The study, launched in 2012, is supposed to quantify the effectiveness of the various types of fixed and rotor wing aircraft when they are used on wildfires, in order to better justify the hundreds of millions of dollars spent by the Forest Service on firefighting aircraft. In FY 2017 for example, the most recent year with exact numbers available, the agency spent over half a billion dollars on fire aviation; $507,000,000. If ever completed and the results implemented, the study could make it possible to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet — to use the best, most efficient tools for the job.

However, to date no detailed reports have been released from the AFUE.

The Senator asked about the results of the study, now entering its eighth year. The data is being collected by four “observation modules,” each comprised of three qualified firefighters and a dedicated aircraft, to collect ground and aerial data at wildfires throughout the nation during fire season. In addition to the 12 firefighters, 3 analysts/managers evaluate the data. Christine Schuldheisz, a spokesperson for the USFS, has said the annual cost of the project is approximately $1,300,000.

Chief Christiansen, referring to the lack of any detailed results being released, said, “I absolutely share your concern and your question….. I am low on patience as well, Senator. This is a complex and labor intensive endeavor.”

Senator Murkowski: “But should it really require seven years to get a report like this?”

Chief Christiansen: “To have enough, when you have to take these assessment teams and have to be on the fire scene and to get enough data to get what the trend line is, it does take some time.”

The Chief then referred to a very small amount of preliminary data that was released in a two-page document in March which in a vague manner referred to the probability of success of direct vs. indirect attack by aircraft. This was was reported by Fire Aviation April 8, 2019.

Senator Murkowski asked the Chief to have more details from the AFUE study when the Committee holds their annual fire outlook hearing in about a month.

Since after seven years the Forest Service has not released any significant data about the study, a person has to wonder what have they found that is so embarrassing, controversial, or perhaps critical of specific models of aircraft, retardant products, or vendors?

Some people think the Forest Service will never release the full results of the AFUE study.

The Committee might have to subpoena the data.

Later in the hearing (at 1:43:30) Colorado Senator Sen. Cory Gardner referred to the study, saying in his rapid-fire speaking style: “There is a technical term I want to use to describe the length of time it is taking to get that study done, and it is Bunk! I’m sorry, it’s just a bunch of Bunk that it has taken seven years to get this done. We fought a world war in four years, we built the Pentagon in 16 months, we can’t do a study in 2 years, 1 year, 3 years, 4 years, maybe 5 years? It has taken seven years to do this? In the meantime we have western states that have had significant and catastrophic fires. I understand it’s important to get the information right. But doggonnit, someone needs to get a fire lit underneath them to get something done on this study.”

New Mexico Senator Martin Heinrich expressed concern that the Administration intends for both the Collaborative Forest Landscape Restoration Program (part of the Forest Service) and the Land and Water Conservation Fund (under the Department of the Interior) to be unfunded beginning in October. Again, the Chief mentioned that the White House directed the Forest Service to cut its overall budget by five percent.

Referring to the fact that the “fire fix” has reduced the necessity for the Forest Service to borrow funds from unrelated accounts to pay for fire suppression, Senator Heinrich said, “We’re giving you the tools, you’re not using the tools we are giving you.”