Author's Commentary on "Crashing into Law"

Two issues arise in this case. The first is how models should be used. Sven argues that in this case, his model is not being used properly and that it is poor science for Oleson to use the model's output to prove the lawyer's points. The second issue is how an expert witness should present scientific information.

There has been a lot of discussion on what makes a good experiment, including a good experimental design, appropriate use of statistical tools, and honesty in the manner in which data are included or excluded. However, there has not been much discussion of how models should be used. Models are used more and more frequently in engineering and science to examine problems that cannot be examined by direct experimentation. For instance, in this case, it would be impractical to examine crash damage in aircraft by crashing planes of every type in each kind of terrain. This strategy would be prohibitively expensive and dangerous. However, it is important to know how an airplane might come apart in a crash in order to design safer aircraft.

What rules should govern the use of models? How might one go about protecting the interests of science and the public through judicious use of models? Here are few guidelines that I believe most modelers would agree on.

The first guideline is that a model makes appropriate use of underlying scientific principles and works within the limitations of the tools it uses. In mathematical modeling, models that are made up of lots of variables instead of a few underlying principles are sometimes described as able to "fit an elephant." That means that such models can fit every data point but might also be able to fit any other data, such as the shape of an elephant. In modeling, it isn't enough for a model to fit all available data; it should also make sense scientifically. Without scientific validity, one cannot extend a model beyond where it has been tested.

The second guideline is that a model should be validated by experiment. It is not enough that a model has a basis in some scientific principle. It should also be validated by comparing the model's predictions with experimentally measured data. These data should not be the data used as input in designing the model. This guideline can be problematic, since models are generally created to deal with problems that cannot easily be investigated experimentally. However, the more one can prove a model works, the more confidence can be placed in it.

The final guideline is the admission of the limitations of the model. Everyone who creates or uses a model should be aware of its limitations. They should have an understanding of the scientific principles and tools on which the model is based. Without such an awareness, even a good, well-validated model can be used inappropriately.

This case study calls the final guideline into question. Sven believes the limitations of his model make it inappropriate for Oleson's purposes. Sven believes that the model is being used outside its range of applicability. However, in Part 2, Oleson believes in his use of the model. This conflict makes the ethics of the case less obvious. We want to believe that modeling is science, but an element of faith is also involved. Each of the proposed guidelines enhances confidence in a model, but models inevitably contain a degree of uncertainty. In Part 2, the issue is who has a better understanding of the model. It becomes a question of whom we believe -- Sven or Oleson? Did Oleson try to convince himself the model was valid because he wanted it to be valid? Because he had already promised the lawyers that he could deliver, Sven might not have been able to persuade him to reconsider. On the other hand, Sven has less experience in the field and may not understand the limitations of the model as well as his professor.

The second issue in this case is that of the expert witness. In Engineering Ethics, Harris, Pritchard and Rabin present five guidelines for expert testimony:

1. There should be adequate time for a thorough investigation.
2. A witness should not accept a case if testimony cannot be presented in good conscience. (He or she should be able to testify honestly, without withholding information.)
3. The witness should consult extensively with the lawyer to ensure the lawyer is familiar with the technical aspects of the case.
4. The witness should maintain an objective and unbiased demeanor on the witness stand.
5. The witness should always be open to new information, even during the course of the trial. (Harris, Pritchard and Rabins, 1995, pp. 203-204)

Guidelines 2 and 5 are called into question in this case. In Part 1, Oleson evidently plans to withhold information on the limitations of the model. Along with the ethical issue of dishonesty, he may be acting foolishly. If the lawyers learn of the inadequacy of the model and question him in court, Oleson may be embarrassed and his professional reputation may be damaged.

In Part 2, Oleson may encounter a problem with Guideline 5. He does not seem to be open to Sven's criticism of the model. Oleson has a possible bias because he wants to keep his commitment to the lawyer that the model could find the source of the damage.

The case also raises the question of Sven's options for expressing his concerns. One option would be to include his assessment of the model's limitations, along with an analysis of its uncertainty, in any writeup. Although a writeup would still be filtered by Oleson, such a report generally goes to lawyers on both sides of a suit and would allow communication of the model's limitations to the lawyers.

This case study raises two separate issues. First is the issue of how a model should be used. Second is the issue of an expert witness's obligation to report the whole truth.

This case can provide an opener into discussing the broader question of scientists' or engineers' obligations in reporting their work, particularly modeling work, to the general public. The discussion can be expanded to include not only the role of expert witnesses, but also the roles of public policy consultants to government agencies or sources in a newspaper article or a TV news program.

Back to Top

References

• Harris, C. E., Jr.; Pritchard, M.S.; and Rabins, M.J. Engineering Ethics. Belmont, Calif.: Wadsworth Publishing Co., 1995.