Lies, irony, and contradiction — an annotation of semantic conflict in the movie "Forrest Gump"

Introduction

Detection of semantic conflict is an important cognitive skill for human social interaction. It is required to identify lies (false statements made with the intention to deceive) but also to correctly interpret stylistic devices — such as sarcasm and irony (statements with direct meaning that is the opposite¹ or contrary² to the implied semantic content). As the interpretation of such events is highly context dependent, it is difficult to study how the brain processes these in the context of real-life like interactions in complex natural environments.

In this study we explored occurrences of semantic conflict in the core stimulus of the studyforrest project (http://studyforrest.org) — the motion picture “Forrest Gump” — in order to evaluate whether the available brain imaging data^3,4 can be readily used to study this aspect of cognition. We annotated the presence of contradictory statements, including lies and ironic statements, as well as the portrayal of cues, such as exaggeration or raised eyebrows, that are often associated with making ironic statements. Additionally, we recorded the context that allowed observers to classify an event as contradictory.

Depending on the exact criterion used for identifying events across observers, we found only between 64 and 36 occurrences of semantic conflict or portrayal of irony cues in the entire movie stimulus. These are likely insufficient numbers for an investigation based on these data alone. However, these new annotations nevertheless contribute to a more comprehensive description of this complex movie stimulus^5,6 and may be useful as confound variables in subsequent studies.

Materials and methods

Data legend

For each annotated event, a total of 10 properties were recorded, each of which are described in the following sections.

Start and end The duration of each event is recorded in start and end as the number of seconds from movie start (no sub-second precision, due to limitations of the video player time display). The time-points correspond to the onset and offset of the respective evidence. Both times can be identical in the case of events with less than one second duration. For contradictory statements, the duration covers the time from the onset of evidence of a contradiction until the end of the statement.

Sender and receiver The identity of a character making a contradictory statement or portraying an irony cue is encoded in sender using character labels listed in 5. In the case that the respective statement is directed to another present movie character, its identity is encoded in receiver.

Evidence of a contradiction The contradiction flag indicates the presence of a contradiction in an event (1: present, 0: absent). The variable proof qualifies if the current or previous events provide the viewer with information to allow the detection of this contradiction (see Table 1). If proof is empty, the movie itself does not contain such information (e.g. a common sense contradiction).

Irony cues The variable cues contains a space-separated list of labels for all irony cues present in a particular events. See Table 1 for a description of all possible labels.

Event category The category variable classifies events into lies, ironic statements, and other events (value empty).

Intention Two more variables encode whether a contradiction was used deliberately and whether this was noticed by the receiver. The variable intended encodes the presence of evidence for deliberate use (1: yes, 0: no). The variable is empty if there is no evidence for either case. The second variable intention_decoded encodes, in the same way, whether a potential receiver noticed a deliberate ironic statement or lie.

Dataset validation

We used an automated procedure to check the annotation records of individual observers for errors or potential problems. Observers submitted their annotations in tabular form to a script that generated a list of error and warning messages. Using this feedback, observers double-checked their annotations as often as necessary until no objective errors were found and all warning messages were confirmed to be false positives. The tests included, for example, plausibility of timing information (no end time before the respective start time) or the presence of unknown condition labels.

In order to assess inter-observer agreement of annotations, we used a two-step approach. First, the temporal location of events depicting any relevant property were determined by comparing annotation timing across observers. The columns in Table 2 report agreement statistics for events defined by at least one, two, or all three observers recording an annotation for the same sender at the same time. In the case that individual observers reported events of different length, or with only partially overlapping duration, only the time-windows with the minimum number of observers reporting an event were considered.

In the second step, we computed Fleiss’ Kappa⁷ for each individual property of an annotation separately with respect to being consistently assigned or non-assigned to the identified events (Table 2). We observe increasing inter-observer agreement of all annotated properties with increasing agreement of annotation timing, approaching “substantial” or “almost perfect” agreement — according to the conventions put forth by 8.

Data and software availability

F1000Research: Dataset 1. Events of semantic conflict and occurrences of irony cues in the motion picture "Forrest Gump", 10.5256/f1000research.9635.d136203⁹

F1000Research: Dataset 2. Python script, 10.5256/f1000research.9635.d136204¹⁰

In addition, released data, code, and manuscript sources are also available on Github (https://github.com/psychoinformatics-studyforrest-paper-ironyannotation).