What is the error margin of your signature analysis?

Highlights

• Assigning a likelihood ratio in a discipline where there are no tabulated data.

• Practical way of addressing the question of error margin in a given case.

• Example of how one can present the Bayesian approach at a Court hearing.

Abstract

In our experience, it seems to become more and more common for mandating authorities or parties to ask forensic signature examiners to quantify the degree of certainty of their conclusion regarding a signature analysis. This paper reports the likelihood ratio approach followed by examiners to answer such a question, in a case where the Court asked whether a questioned signature was written, or not, by Mr Jones. The Court also required an assessment of the error margin of the signature analysis. This question was answered using Bayes’ theorem (i.e., a full Bayesian approach) and this paper seeks to show that such an approach can be used despite the popular belief that Bayes’ theorem is beyond what courts may accept.

Using a practical example, we present advantages of the approach we have chosen to assess our results and show that a logical approach for evidence evaluation can be followed even in a forensic discipline where no tabulated data are available. This example also illustrates a practical way of addressing the error margin question, which helps the Court understand what can be the risk of being wrong in this particular case (and not in cases in general). We further present the way these results were communicated to the fact finders in the case at hand and provide guidance as how forensic observations can logically be combined with the other elements of the case.

Introduction

Recently, due to changes in the Swiss Criminal Procedure Code, forensic signature examiners have been facing new requests from lawyers and the judiciary about the weight of evidence and the degree of certainty of their analysis. This paper addresses the issue with the intent of satisfying both the expert and the Court. To do so, we will discuss a recent case, where authors of this paper were asked a fairly typical question in the field of disputed signatures, followed by a less common one. These two questions were as follows:

• ‘Was the questioned signature written by Mr Jones?’

• ‘Please state, preferably in %, what is the degree of certainty of your conclusions. If, even under the best case scenario there remains an unavoidable error margin for this analysis according to the state of scientific and technical knowledge, please state what that error margin is (preferably in %).’

How were we to answer these two questions? A few decades ago, we would have gladly provided the following direct answer to the first question: ‘In my opinion, Mr Jones probably did not write the questioned signature’. Such a conclusion would have been at that time in agreement with the recommendations of the ASTM (E 1658-08) [1]. But, with the knowledge accumulated over the years in the field of evidence interpretation, we could not make such a conclusion anymore, for reasons that will be detailed later in this paper.

At first sight, one could wonder why one would stop a practice that had never been challenged, in this case replying that Jones probably did not write the signature. The Court never complained, on the contrary, this type of answer was exactly why it appointed forensic experts. If we stopped answering the Court’s questions, when we had done it for such a long time, the Court may not understand. How were we to explain this change to the Court and to our colleagues? Would it not be less demanding to continue with the method that had always worked and perhaps add a caveat stating that document examination was based on expert knowledge and was after all only opinion evidence? It would be easier than explaining why forensic scientists should not state their opinion on whom has signed (which would go beyond the role of the examiners, who should give their opinion on the value of their observations). Moreover, that would be less complicated than assigning a numerical value to the observations without tabulated data. This is not saying that no data exist in the field of signature examination but to the best of our knowledge, the published data do not really help the examiners in the evaluation of their results in casework.

To prevent any misunderstanding of our conclusions by the Court, temptation was also high to answer the second question as follows: “I have taken proficiency tests for more than 10 years and have never been wrong. If I had not been certain of my opinion I would not have reported my conclusions.” We could also have informed the Court about the error rate of other practitioners, based on studies designed to determine the number of wrong answers reported by examiners in proficiency tests or collaborative exercises [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]. However, we are of the opinion that the error rate of practitioners is not useful for the Court who must take a decision on a specific case. If we use such error rates, the value of the results would always be the same, whatever the signature, whatever the references and whatever the laboratory. From discussions with the Court, we understood that the mission was to inform about the chance of being wrong in the case at hand.

We decided it was time to abandon incoherent statements such as “Mr Jones probably did not write the questioned signature.” We also decided that reporting: “the findings are more probable if Mr Jones did not write the questioned signature than if he wrote it” was insufficient. A statement about probability must necessarily refer to numbers, since a probability is defined by a value comprised between 0 and 1. Therefore, an examiner should be able to express the strength of the evidence with numbers. Finally, we decided to adopt a practical position to handle the error rate issue, which provides an attractive guide to support the decision maker.

Solving both of these questions in a rigorous and transparent manner was not an easy task, facing the risk of being misunderstood by the Court. However we knew from the early discussions with the mandating authority that the appointed examiners would attend the judgement hearing. So, there was the challenging prospect and opportunity of explaining our approach to evaluation, both in the written statement and during the Court hearing. The presentation of the approach would have the advantage of helping the Court understand the reasons for the formulation of our conclusions. While the likelihood ratio approach is well appreciated in many fields (e.g., economics, medicine, risk analysis), and in spite of the numerous publications on this approach [15], [16], forensic scientists are still not comfortable to deal with it in the absence of data, and surprisingly, all the more in disciplines where one traditionally individualises.¹ This is especially the case in the field of signature examination, where occurrences of particular features in given populations are largely unstudied, and where experts rely mainly on their experience to assess the significance of their observations.

Needless to say, lawyers and magistrates are not used either to deal with conclusions formulated in agreement with the likelihood ratio approach. Even when they have been exposed to such statements for many years, conclusions can be misunderstood² [17], [18], [19], [20], [21]. Moreover, they have often been taught little on how to use this information and they generally do not know how to combine this evidence with the other information available to them.

In the case at hand, we therefore took the decision to face the dominant ideology (the one shared by a majority of experts and lawyers, who assume that the expert can give a straightforward answer to the question: ‘Was the questioned signature written by Mr Jones?’) and to adopt a likelihood ratio approach. This was explained during the Court hearing and will be stepwise described thereafter.

This paper is structured as follows:

• Section 2 focuses on the expert’s role and answers the first question of the Court: “Was the questioned signature written by Mr Jones?”. It presents a logical methodology for the assessment of the observations made by document examiners. We will apply a model based on a likelihood ratio approach, which represents the contribution of the expert, to a typical case of disputed signature.

• Section 3 will address the “error margin” question of the Court. It will hopefully help lawyers understand the conclusion of the expert, which fulfils the requirements of logic, but remains counter-intuitive. The likelihood ratio will be discussed within a full Bayesian framework (i.e. using the Bayes’ theorem).³ This section will enable lawyers and the judiciary to know how one can combine the results of forensic document examination with other information, such as case circumstances and/or other evidence.

• Section 4 summarises the discussions held during the court hearing. It relates the major issues and questions raised concerning the approach followed for the interpretation of the forensic observations. It will hopefully help forensic examiners who will have to testify using the Bayesian approach. It will also show that unaccustomed lawyers, with stepwise explanations, can understand this approach.