Professional ethics and the fire dynamics simulator
By Richard Schulte
Schulte & Associates, Evanston, Ill.
The series of four articles titled "Standard engineering practice or junk science?" which recently appeared in this column discussed the use of the Fire Dynamics Simulator (FDS) to predict the activation times of multiple sprinklers in conjunction with research on the concept of the "ganged" operation of smoke/heat vents in storage buildings protected by standard spray sprinklers. (This research is presented in a report written by Hughes Associates Inc., titled "Analysis of the Performance of Ganged Operation of Smoke and Heat Vents with Sprinklers and Draft Curtains," dated February 18, 2008.) This article will continue the discussion and address the appropriate use of the FDS.
Background
The Life Safety Code® (LSC) is considered by most in the fire protection field to be "accepted engineering practice." Section 4.4 in the 2006 edition of the LSC indicates that building designers are permitted to utilize the prescriptive provisions contained in the LSC or, as an alternative to the prescriptive provisions, the "performance-based" provisions contained in the Code.
The "performance-based" provisions included in the LSC are contained in Chapters 1 through 5, and explanatory material on "performance-based" design is included in Annex A. The following are excerpts from the "performance-based" provisions and the explanatory material included in the LSC:
"Fire and Similar Emergency. The goal of this Code is to provide an environment for the occupants that is reasonably safe from fire and similar emergencies by the following means: (1) Protection of occupants not intimate with the initial fire development, (2) Improvement of the survivability of occupants intimate with the initial fire development." (section 4.1.1)
"Crowd Movement. An additional goal is to provide for reasonably safe emergency crowd movement and, where required, reasonably safe non-emergency crowd movement." (section 4.1.2)
"Occupant Protection. A structure shall be designed, constructed and maintained to protect occupants who are not intimate with the initial fire development for the time needed to evacuate, relocate or defend in place." (section 4.2.1)
"Structural Integrity. Structural integrity shall be maintained for the time needed to evacuate, relocate, or defend in place occupants who are not intimate with the initial fire development." (section 4.2.2)
"Systems Effectiveness. Systems utilized to achieve the goals of Section 4.1 shall be effective in mitigating the hazard or condition for which they are being used, shall be reliable, shall be maintained to the level at which they were designed to operate, and shall remain operational." (section 4.2.3)
"Goals and Objectives. The performance-based design shall meet the goals and objectives of this Code in accordance with Section 4.1 and Section 4.2." (section 5.1.2)
"Sources of Data. Data sources shall be identified and documented for each input data requirement that must be met using a source other than a design fire scenario, an assumption, or a building design specification. The degree of conservatism reflected in such data shall be specified, and a justification for the source shall be provided." (section 5.1.5)
"Any assumption and design specifications that the design analyses do not explicitly address or incorporate and that are, therefore, omitted from input data specifications shall be identified, and a sensitivity analysis of the consequences of that omission shall be performed." (section 5.4.2.2)
"Data Requirements. A complete listing of input data requirements for all models, engineering methods, and other calculation or verification methods required or proposed as part of the performance-based design shall be provided." (section 5.6.3.2)
"Validity. Evidence shall be provided to confirm that the assessment methods are valid and appropriate for the proposed building, use, and conditions." (section 5.6.5, LSC)
"Safety Factors. Approved safety factors shall be included in the design methods and calculations to reflect uncertainty in the assumptions, data, and other factors associated with the performance-based design." (section 5.7)
"Technical References and Resources. The authority having jurisdiction shall be provided with sufficient documentation to support the validity, accuracy, relevance, and precision of the proposed methods. The engineering standards, calculation methods, and other forms of scientific information provided shall be appropriate for the particular application and methodologies used." (section 5.8.2)
"Assumptions made by the model user, and descriptions of models and methods used, including known limitations, shall be documented." (section 5.8.11.1)
"Documentation shall be provided to verify that the assessment methods have been used validly and appropriately to address the design specifications, assumptions, and scenarios." (section 5.8.11.2)
"Modeling. Models can be used to predict the performance criteria for a given scenario. Because of the limitations on using only tests for this purpose, models are expected to be used in most, if not all, performance-based design assessments." (section A.5.6)
"CFD models can provide more accurate predictions than other deterministic models, because they divide a given space into many smaller volumes. However, since they are still models, they are not absolute in their depiction of reality." (section A.5.6)
"Validation. Models undergo limited validation. Most can be considered demonstrated only for the experimental results they were based on or the limited set of scenarios to which the model developers compared the model's output, or a combination of both." (section A.5.6)
"Models have limitations. ... The intent is not to discourage the use of models, only to indicate that they should be used with caution by those who are well versed in their nuances." (section A.5.6)
These last excerpts from Annex A of the LSC address the issue of model validation. In early August 2008, the National Institute of Standards and Technology (NIST) released Volume 3 of the Fire Dynamics Simulator (Version 5) Technical Reference Guide. Volume 3 addresses the issue of the validation of the FDS. The following are excerpts from the volume on FDS validation:
"The model evaluation process consists of two main components: verification and validation. Verification is a process to check the correctness of the solution of the governing equations. Verification does not imply that the governing equations are appropriate; only that the equations are being solved correctly. Validation is a process to determine the appropriateness of the governing equations as a mathematical model of the physical phenomena of interest. Typically, validation involves comparing model results with experimental measurement." (page i)
"Evaluation is critical to establishing both the acceptable uses and limitations of a model. Throughout its development, FDS has undergone various forms of evaluation, both at NIST and beyond. This volume provides a survey of validation work conducted to date to evaluate FDS." (page i)
"Although there are various definitions of model validation, for example those contained in ASTM E 1355 [2], most define it as the process of determining how well the mathematical model predicts the actual physical phenomena of interest. Validation typically involves (1) comparing model predictions with experimental measurements, (2) quantifying the differences in light of uncertainties in both the measurements and the model inputs, and (3) deciding if the model is appropriate for the given application." (page 1)
"A common question asked of any mathematical model is whether it is validated. To say that FDS is 'validated' means that the model has been shown to be of a given level of accuracy for a given range of parameters for a given type of fire scenario." (page 1)
"Keep in mind that FDS can be used to model most any fire scenario and predict almost any quantity of interest, but the prediction may not be accurate because of limitations in the description of the fire physics, and also because of limited information about the fuels, geometry, and so on." (page 1)
"Keep in mind that for any fire experiment, FDS might predict a particular quantity accurately (within the experimental uncertainty bounds, for example), but another quantity less accurately. For example, in the a series of 15 full-scale fire experiments conducted at NIST in 2003, sponsored by the U.S. Nuclear Regulatory Commission, the average hot gas layer (HGL) temperature predictions were within the accuracy of the experiments themselves, yet the smoke concentration predictions differed from the measurements by as much as a factor of 3." (page 2)
"This Guide is merely a repository of calculation results. As FDS develops, it will expand to include new experimental measurements of newly modeled physical phenomena. With each minor release of FDS (version 5.2 to 5.3, for example), the plots and graphs will all be redone to ensure that changes to the model have not decreased the accuracy of a previous version." (page 3)
"In this chapter, a survey of FDS validation work will be presented. Some of the work has been performed at NIST, some by its grantees and some by engineering firms using the model. Because each organization has its own reasons for validating the model, the referenced papers and reports do not follow any particular guidelines. Some of the works only provide a qualitative assessment of the model, concluding that the model agreement with a particular experiment is 'good' or 'reasonable.' Sometimes, the conclusion is that the model works well in certain cases, not as well in others." (page 5 )
"FDS was officially released in 2000. However, for two decades various CFD codes using the basic FDS hydrodynamic framework were developed at NIST for different applications and for research. In the mid 1990s, many of these different codes were consolidated into what eventually became FDS. Before FDS, the various models were referred to as LES, NIST-LES, LES3D, IFS (Industrial Fire Simulator), and ALOFT (A Large Outdoor Fire Plume Trajectory)." (page 5)
"There is an on-going effort at NIST and elsewhere to validate FDS as new capabilities are added. To date, most of the validation work has evaluated the model's ability to predict the transport of heat and exhaust products from a fire through an enclosure. In these studies, the heat release rate is usually prescribed, along with the production rates of various products of combustion. More recently, validation efforts have moved beyond just transport issues to consider fire growth, flame spread, suppression, sprinkler/detector activation, and other fire-specific phenomena." (page 6)
"Formal, rigorous validation exercises are time-consuming and expensive. Most validation exercises are done simply to assess if the model can be used for a very specific purpose. While not comprehensive on their own, these studies collectively constitute a valuable assessment of the model." (page 6)
"As part of the NIST investigation of the World Trade Center fires and collapse, a series of large scale fire experiments were performed specifically to validate FDS [23]. The tests were performed in a rectangular compartment 7.2 m long by 3.6 m wide by 3.8 m tall. The fires were fueled by heptane for some tests and a heptane/toluene mixture for the others." (page 6)
"A second set of experiments to validate FDS for use in the World Trade Center investigation is documented in Ref. [24]. The experiments are not described as part of this Guide. The intent of these tests was to evaluate the ability of the model to simulate the growth of a fire burning three office workstations within a compartment of dimensions 11 m by 7 m by 4 m, open at one end to mimic the ventilation of windows similar to those in the WTC towers. Six tests were performed with various initial conditions exploring the effect of jet fuel spray and ceiling tiles covering the surface of the desks and carpet. Measurements were made of the heat release rate and compartment gas temperatures at four locations using vertical thermocouple arrays. Six different material samples were tested in the NIST cone calorimeter: desk, chair, paper, computer case, privacy panel, and carpet. Data for the carpet, desk and privacy panel were input directly into FDS, with the other three materials lumped together to form an idealized fuel type. Open burns of single workstations were used to calibrate the simplified fuel package. Details of the modeling are contained in Ref. [25]." (page 7)
"Experiments conducted solely for model validation are somewhat rare. More common are validation studies that use data from past experiments." (page 7)
"Although FDS simulations have been compared to actual and experimental large-scale fires, it is difficult to quantify the accuracy because of the uncertainty associated with material properties. Most quantified validation work associated with flame spread has been for small, laminar flames with length scales ranging from millimeters to a few centimeters." (page 10)
"For real wood products, it is unlikely that all of the necessary properties can be obtained easily. Thus, grid sensitivity and uncertain material properties make blind predictions of fire growth on real materials beyond the reach of the current version of the model." (page 10)
"A significant validation effort for sprinkler activation and suppression was a project entitled the International Fire Sprinkler, Smoke and Heat Vent, Draft Curtain Fire Test Project organized by the National Fire Protection Research Foundation [60]. Thirty-nine large scale fire tests were conducted at Underwriters Laboratories in Northbrook, Ill. ... At the time, FDS had not been publicly released and was referred to as the Industrial Fire Simulator (IFS), but it was essentially the same as FDS version 1. ... However, five experiments were performed with 6 m high racks containing the Factory Mutual Standard Plastic Commodity, or Group A Plastic. To model these fires, bench scale experiments were performed to characterize the burning behavior of the commodity, and larger test fires provided validation data with which to test the model predictions of the burning rate and flame spread behavior [61, 62]. Two to four tier configurations were evaluated. For the period of time prior to application of water, the simulated heat release rate was within 20 % of the experimental heat release rates." (page 10)
"The scope of the VTT [Technical Research Centre of Finland] work is considerable. Assessing the accuracy of the model must be done on a case by case basis. In some cases, predictions of the burning rate of the material were based solely on its fundamental properties, as in the heptane pool fire simulations. In other cases, some properties of the material are unknown, as in the spruce timber simulations. Thus, some of the simulations are true predictions, some are calibrations. The intent of the authors was to provide guidance to engineers using the model as to appropriate grid sizes and material properties. In many cases, the numerical grid was made fairly coarse to account for the fact that in practice, FDS is used to model large spaces of which the fuel may only comprise a small fraction." (page 13)
"In January 1997, a series of 22 heptane spray burner experiments was conducted at the Large Scale Fire Test Facility at Underwriters Laboratories (UL) in Northbrook, Illinois [96]. The objective of the experiments was to characterize the temperature and flow field for fire scenarios with a controlled heat release rate in the presence of sprinklers, draft curtains and a single smoke & heat vent." (page 18)
"The documentation of the experiments described in this chapter have varying descriptions of the uncertainty in the reported measurements. However, in order to assess the accuracy of FDS, there must be some estimate of the combined effect of the uncertainty in the reported input parameters, like the heat release rate of the fire, and the reported measurement of the quantity of interest, like the hot gas layer (HGL) temperature." (page 28)
"Figure 6.2 displays graphically the difference between predicted and measured sprinkler activation times as a function of burner position. Note that there are no experimental uncertainty bounds on the plot because it is difficult to estimate the combined uncertainty related to the various parameters that are input into the model. For example, changing the median volumetric droplet size from 1000 μm to 750 μm led to a reduction of approximately 50 % in the number of predicted sprinkler activations due to the increased cooling of the smaller droplets. Consequently, three replicate experiments are compared to show how much variation one can expect in sprinkler activation times in repeat experiments." (page 63)
Sections 2.0 and 3.1 in the HAI report on research on the "ganged" operation of smoke/ heat vent address the issue of the validation of the Fire Dynamics Simulator. The following are excerpts addressing validation from the report:
"The Fire Dynamics Simulator, version 4 (FDS4), was used to perform the field calculations. FDS4 is a three-dimensional large eddy simulation CFD program developed at the National Institute of Standards and Technology's (NIST's) Building and Fire Research Laboratory (BFRL) [McGrattan & Forney, 2004; McGrattan, 2005]. FDS4 is a multidimensional, multiphysics simulation that solves the low Mach number equations of expandable flow [Rehm & Baum, 1978]. FDS was specifically written to address fire scenarios. It has over a twenty year development history. Some of its antecedents include the Industrial Fire Simulator (IFS) and LES3D. The current version of FDS (4.0.7) contains updated source code from these previous projects. FDS can handle isothermal or thermally variable flows. It can directly simulate the effects of turbulence or it can perform large eddy simulations of turbulence. It can also handle axisymmetric cylindrical, two-dimensional Cartesian, and three-dimensional Cartesian coordinates. FDS4 uses Lagrangian droplet transport to simulate the delivery of water from suppression systems. The droplets and the fluid mechanics are coupled. The flow of air and gas components affect the drag on the droplets. The force that the droplets exert on the surrounding gas shows up as a body force in the Eulerian momentum equations. This coupling allows the model to simulate sprinkler-smoke layer interaction. Some validation studies for FDS4 and its predecessors are given in [Baum, McGrattan, & Rehm, R.G., 1994; Baum, McGrattan, & Rehm, 1996; Baum, McGrattan, & Rehm, 1997; Floyd & Lattimer, 2004; McGrattan, 2005; McGrattan, Baum, & Rehm, 1996; McGrattan, Baum, Walton, & Trelles, 1997; McGrattan, Hamins, & Stroup, 1998; Najafi, Salley, Joglar, and Dreisbach, 2006; Trelles, Mawhinney, & DiNenno, 2004]." (pages 14 and 15)
"As was indicated above, the reason for choosing the t2 growth rate of 1.78 kW/s2 (0.157 BTU/s/ft2) was to facilitate comparisons with the results of the tests reported in [McGrattan, Hamins, & Stroup, 1998]. For Runs 1 - 12, the center of the burning racks coincided with the center of the four nearest sprinklers. However, only Runs 4 - 12 went up to 10 MW (9,480 BTU/s). The average of the first sprinkler activation times reported in Table 5 for these runs is 70.0 s. The sprinkler performance results for the heptane tests from [McGrattan, Hamins, & Stroup, 1998] are reported in Table 6. Only burner positions E and F correspond to the burner being centered on four sprinklers. The average of the first sprinkler activation times reported in Table 5 for these two cases is 70.5 s. The difference between these tests and the current study is 0.7%. For Runs 4 - 12, an average of 19 sprinklers operated. From Table 6, an average of 21 sprinklers operated with the heptane burner tests II-9 and II-10. The difference is 9.5%. (pages 30 and 31)
"The remaining positions in Table 6 are for the spray burner centered on two sprinklers. Although this is not exactly the case with Runs 13 - 16, a comparison will be made just the same. From Table 5, the averages are 65.9 s for the first activation and 21 sprinklers overall. From Table 6, the averages are 67 s for the first activation and 22 sprinklers overall. The differences are 1.6% and 5%, respectively." (page 31)
"For a variety of reasons, comparisons with the commodity tests in [McGrattan, Hamins, & Stroup, 1998] are not appropriate. For example, in the current set of simulations, the heat release rate is well defined. It was not measured for the commodity tests in [McGrattan, Hamins, & Stroup, 1998]. Without this information, it is difficult to determine which experiments best correspond with the sixteen simulations available in this report. Some of the NFPARF tests, such as P-3, are not applicable at all because fuel was placed directly underneath the draft curtain. Nonetheless, the averages of the first sprinkler activations will be compared. For tests P-1 - P-5 of [McGrattan, Hamins, & Stroup, 1998], the average is 70.6 s. For Runs 1 - 9, the average is 71 s. The difference is 14%. The comparisons presented in the section are of suitably low percent differences to conclude that the validation exercise is a success." (pages 31 and 32)
Discussion
The Life Safety Code provisions that address the use of "performance-based" design specifically discuss the issue of validation of fire models used in a performance design. Section 5.6.5 in the LSC indicates that evidence is required to be provided that models used in a performance design are "valid and appropriate" for the proposed "use and conditions." Section 5.8.11.1 indicates that "known limitations" of fire models used in the design are required to be clearly indicated.
Explanatory material on "performance-based" design included in Annex A of the Life Safety Code states that models "are not absolute in their depiction of reality" and "that models undergo limited validation." The explanatory material further indicates that "models have limitations" and that "models should be used with caution." Further, the explanatory material in Annex A states that "most [models] can be considered demonstrated only for the experimental results they were based on or the limited set of scenarios to which the model developers compared the model's output, or a combination of both."
Given the provisions for use of fire models in "performance-based" design and explanatory material contained in the Life Safety Code, it seems reasonable to ask: Is the model validation documentation included in the Hughes Associates Inc.'s report on the "ganged" roof vent concept sufficient to consider the FDS validated for the purposes of predicting the activation times of multiple sprinklers under all fire and sprinkler system design conditions that could potentially occur in a building (where the combination of standard spray sprinklers and smoke/heat vents would be provided)?
Without giving the response to this question much thought, it appears, at least to me, that the answer is plainly obvious. The test data used by HAI to determine that the FDS is validated is limited to tests using heptane spray burners using the same sprinkler model (Model ELO-231 manufactured by Central Sprinkler Corporation) with only one sprinkler spacing (10 feet by 10 feet) and only one operating pressure (19 psi). Based upon the limited scope of the experimental data provided in the HAI report, the validation of the FDS for the purposes of predicting the activation times of multiple sprinklers appears to be extremely limited. Hence, in my opinion, the documentation provided by HAI does not validate the use of the model for the purpose for which it is being utilized by HAI.
Conclusion
As was noted in Volume 3 of the FDS Technical Reference Guide, the use of Fire Dynamics Simulator in building design is relatively new (mid-1990s), as is the fire protection field itself. (The Society of Fire Protection Engineers was formed in 1950.) In order to govern the practice of fire protection engineering, the Society of Fire Protection Engineers (SFPE) has adopted a Canon of Ethics. The following are excerpts from the SFPE Canon of Ethics:
"Preamble. Fire protection engineering is an important learned profession. The members of the profession recognize that their work has a direct and vital impact on the quality of life for all people. Accordingly, the services provided by fire protection engineers require honesty, impartiality, fairness and equity, and must be dedicated to the protection and enhancement of the public safety, health and welfare; and the environment. In the practice of their profession, fire protection engineers must maintain and constantly improve their competence and perform under a standard of professional behavior which requires adherence to the highest principles of ethical conduct with balanced regard for the interests of the public, clients, employers, colleagues, and the profession. Fire protection engineers are expected to act in accordance with this Code and all applicable laws and actively encourage others to do so."
"Fundamental Principles. Fire protection engineers uphold and advance the honor and integrity of their profession by: ... Being honest and impartial, and serving with fidelity the public, their employers, and clients. ... "
"Canon 6. Fire protection engineers shall be honest and truthful in presenting data and estimates, professional opinions and conclusions, and in their public statements dealing with professional matters ... "
"Canon 12. Fire protection engineers having knowledge of any alleged violation of this Code shall cooperate with the proper authorities in furnishing such information or assistance as may be required."
"Canon 15. Fire protection engineers shall perform professional services using only those engineering methods and tools which are appropriate for the specific application."
Pages 14 and 15 in the Hughes Associates Inc. study of the concept of the "ganged" operation of smoke/heat vents includes the following statement:
"Some validation studies for FDS4 and its predecessors are given in [Baum, McGrattan, & Rehm, R.G., 1994; Baum, McGrattan, & Rehm, 1996; Baum, McGrattan, & Rehm, 1997; Floyd & Lattimer, 2004; McGrattan, 2005; McGrattan, Baum, & Rehm, 1996; McGrattan, Baum, Walton, & Trelles, 1997; McGrattan, Hamins, & Stroup, 1998; Najafi, Salley, Joglar, and Dreisbach, 2006; Trelles, Mawhinney, & DiNenno, 2004]."
Curiously, one validation study that is not included in the list above is a study titled "Verification and Validation of Selected Fire Models for Nuclear Power Plant Applications" (NUREG-1824) dated May 2007. NUREG-1824 is a study of fire models conducted by NIST for the Nuclear Regulatory Commission (NRC) and consists of 7 volumes.
Volume 7 of NUREG-1824 addresses the "verification and validation" of Version 4.06 of the Fire Dynamics Simulator. (Version 4 of the FDS is the version utilized by HAI in their research on the "ganged" operation of smoke/heat vents.) Table 3-1 in Volume 7 (pages 3-1 and 3-2) indicates that "estimating sprinkler activation" and the "suppression by water spray" routines included in the FDS have not been "verified and validated" by the NIST study. Presumably, the reason for this is the lack of experimental data to validate the algorithm/methodology included in the FDS, which perform these tasks.
Of interest is the fact that Dr. Craig Beyler and Phil DiNenno of Hughes Associates Inc. are both listed as peer reviewers of Volumes 1, 2, 3 and 7 of NUREG-1824. Given that both Beyler and DiNenno are listed as peer reviewers of these volumes, it would seem probable that both were aware that the validation of the FDS for the purpose of predicting the activation times of multiple sprinklers is, at the very least, questionable.
Since section 5.8.11.1 of the Life Safety Code and Canon 6 of the SFPE Canon of Ethics both require that "known limitations" of fire models be explicitly stated in a "performance-based" design, and that nowhere in their report does HAI mention the conclusions contained in Volume 7 of NUREG-1824 regarding the "estimating sprinkler activation" and "suppression by water spray" routines, a case can be made that the HAI report violates the SFPE Canon of Ethics.
Prior to formulating an opinion regarding the ethical conduct of Hughes Associates Inc. in regard to their research on the concept of "ganged" smoke/heat vent operation, it is suggested that readers of this column review all three volumes of the Fire Dynamics Simulator Technical Reference Guide and NUREG 1824 in detail. Besides being an excellent introduction into the use of fire models, these references will help define the current limitations on the use of fire models and, hence, the ethical use of these models.
Editor's Note: The report on the research conducted by Hughes Associates Inc. on the concept of the "ganged" operation of smoke/heat vents can be found at the following Web address: www.iccsafe.org/cs/ctc/balanced/Thornberry_AAMA_Modeling_Study.pdf. Volumes 1, 2 and 3 of the FDS Technical Reference Guide and also the FDS Users Guide can be found at http://fire.nist.gov/fds/documentation.html. Volumes 1 through 7 of NUREG 1824 can be found at www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1824/.
Richard Schulte is a 1976 graduate of the fire protection engineering program at the Illinois Institute of Technology. After working in various positions within the fire protection field, he formed Schulte & Associates in 1988. His consulting experience includes work on the Sears Tower and numerous other notable structures. He has also acted as an expert witness in the litigation involving the fire at the New Orleans Distribution Center. He can be contacted by sending e-mail to rschulte@plumbingengineer.com.