Measuring diagram quality through semiotic morphisms

2.1 What is a diagram? Peircian perspectives

In Peircian semiotics, diagrams can be found within the world of iconic sinsigns, as “a representamen which is predominantly an icon of relations” (CP 4.418). For Peirce, iconic sinsigns then include portraits, models, diagrams and maps. Recall also Peirce’s definitions used in his triadic model of signification:

A representation is that character of a thing by virtue of which, for the production of a certain mental effect, it may stand in place of another thing. The thing having this character I term a representamen, the mental effect, or thought, its interpretant, the thing for which it stands, its object. (CP 1.564)

Building on Peirce’s definitions to allow greater specificity, a diagrammatic representamen is the substance of an icon of relation (often the substance is “lines on paper” and referred to as a “diagram”). Given the anchoring of this paper in Peircian semiotics, it seems prudent to examine Peirce’s thoughts on diagrams. As a mathematician, these naturally focused on geometry, logic and Euler diagrams. In his thoughts on mental diagrams, Peirce asserted that all mathematical thought is diagrammatic:

For mathematical reasoning consists in constructing a diagram according to a general precept, in observing certain relations between parts of that diagram not explicitly required by the precept, showing that these relations will hold for all such diagrams, and in formulating this conclusion in general terms. All valid necessary reasoning is in fact thus diagrammatic. This, however, is far from being obviously true. There was nothing to draw the attention of the early reasoners to the need of a diagram in such reasoning. (CP 1.54)

This seems plausible, if we use the definition of a diagram as an icon of relations, since mathematics can be seen as primarily relational in its abstraction. Further, for Peirce mental diagrams have claims on deduction itself:

Deduction is that mode of reasoning which examines the state of things asserted in the premisses, forms a diagram of that state of things, perceives in the parts of that diagram relations not explicitly mentioned in the premisses, satisfies itself by mental experiments upon the diagram that these relations would always subsist, or at least would do so in a certain proportion of cases, and concludes their necessary, or probable, truth. (CP 1.66)

In terms of diagrams in their specific instantiations, describing a particular situation outside mathematics, Peirce is relatively quiet. In terms of physical manifestations, regarding inference over existential graphs, he says:

What we have to do, therefore, is to form a perfectly consistent method of expressing any assertion diagrammatically. The diagram must then evidently be something that we can see and contemplate. Now what we see appears spread out as upon a sheet. Consequently our diagram must be drawn upon a sheet. We must appropriate a sheet to the purpose, and the diagram drawn or written on the sheet is to express an assertion. We can, then, approximately call this sheet our sheet of assertion. The entire graph, or all that is drawn on the sheet, is to express a proposition, which the act of writing is to assert. (CP 4.430)

In this context, he also suggests that “A diagram ought to be as iconic as possible; that is, it should represent relations by visible relations analogous to them.” (CP 4.433) Beyond Peirce, the emphasis in definitions of a diagram are various:

1. “An illustrative figure which, without representing the exact appearance of an object, gives an outline or general scheme of it, so as to exhibit the shape and relations of its various parts.” Oxford English Dictionary (Oxford University Press 2018)

2. “The diagrams, then, are maps of thought, which may be used “to stick pins into” in order to mark anticipated changes.” (Kiryushchenko 2015: 117)

3. “Diagrams are usually simplified figures, caricatures in a way, intended to convey essential meanings. Diagrams by their very nature do not pretend to be naturalistic; they seek to represent whatever their author regards as the salient features of the subject.” (Hall 1996: 9)

4. “Diagrammatic form itself provides the facility to describe whole processes and structures, often at levels of great complexity in one representation.” (Scaife and Rogers 1996: 1)

It will not escape the reader that, while compatible, even within Peirce’s own work these definitions are not identical. The line to tread in defining a diagram is perhaps not as obvious as it may seem. Indeed, “landscape art should be considered a branch of topographical illustration, and thus may be classified as scientific illustration.” (Topper 1996) By restricting our study to diagrammatic and schematic representations, we are spared this conflict and are able to research with depth. As such, we move forward with the Oxford English Dictionary definition, a restriction of scope from Peirce’s thinking.

2.2 Why diagrams matter

Diagrams can be useful aids for describing, interpreting and reasoning about systems. Given the above comments by Peirce, it seems clear he would not be surprised by the quantity of praise for diagrams since his death. Tylén et al. (2014) concisely summarize that diagrams:

1. Are external representational support to cognitive processes (Clark and Chalmers 1998).

2. Make abstract properties and relations accessible (Hutchins 1995).

3. Can be in a public space, therefore enabling collective and temporally distributed forms of thinking (Peirce 1931–1966).

4. Are manipulated in order to profile known information in an optimal fashion.

Further commentators have praised diagrams for controlling search space (Sloman 1984), their explicit spatial relation advantage (Karaca 2012) and explanatory value (Burnston 2016). Stenning and Oberlander (1995) state that the power of visual representations is in the omission of information, limiting abstraction to aid “processibility.” In a similar vein, Levesque (1989) argues for simplicity, that inferential and computational tractability is maintained by minimising the number of cases that must be computed over. Shimojima’s (2015) work on “free rides” includes a number of examples, the core concept of which is that by establishing one relationship in a diagram there is also established a relationship to all objects within the diagram. This leads to well-grounded claims for diagrams aiding inference and consistency checking, at the expense of potential over-specificity. To summarize these crucial works, some of the key properties of a diagram are:

1. Being an external support to cognitive process, allowing for reduced utilisation of memory

2. Making abstract properties and relations accessible

3. Making topics simpler, leading to reduced search space and fewer cases to be computed over, by including minimal salient information

4. Being public, allowing for articulation of concepts and distributed thinking

5. Facilitating perceptual “free rides” in inference

2.3 A brief history of schematic diagrams

Diagrams in the modern sense are quite new. While some of da Vinci’s illustrations perhaps have the abstraction to be considered diagrams, there is a technological piece missing. This is described in Lawrence’s wonderful “History of Descriptive Geometry in England” (Lawrence 2003), relevant aspects of which are drawing upon and summarized throughout this section. Orthographic projection, the technical key to mechanical diagrams, was invented by mathematician and draughtsman Gaspar Monge in Napoleon’s French military, eventually published in his “Géométrie descriptive” (Monge 1798). Orthographic projection allows three dimensions to be reduced to two, leading to the ability to describe those dimensions with simplicity and accuracy. This travelled to the US via West Point, but was not propagated in Britain, according to Lawrence perhaps due to the war with France, and the idea originating from a Republican educationalist.

In Britain, Nicholson’s Parallel Oblique Projection achieved similar objectives, in appropriately representing a 3D object in a 2D medium. Quoting Lawrence’s (2003: 1274) account of Nicholson: “In 1794 I first attempted the Orthographical Projection of objects in any given position to the plane of projection; and, by means of a profile, succeeded in describing the iconography and elevation of a rectangular parallelpipedon: this was published in vol. ii of the Principles of Architecture (1822).” Farish, a professor at Cambridge, had educational mechanical models which he made of smaller components, and he devised a way of graphically describing these to communicate how his mechanical models should be assembled in advance of lectures. He called these graphical communications “isometrical perspective.” This was disseminated through the Mechanics Institute as the “British System of Projection” (Lawrence 2003).

To illustrate this historical development timeline, in 1824 in order to articulate his heat-engine concept Carnot included only a single diagram, a high level diagram of a piston without connections (Boon and Knuuttila 2009). In 1840, Babbage’s Analytical Engine theoretical designs (Science Museum Group 2019) include plans that could be described as diagrammatic in that they outline key attributes without excessive redundant flourishes. The theoretical nature of Babbage’s work makes this abstraction understandable and natural, and indeed in this case the nature of the diagram is not so much iconic, but rather indexical, in that a change to the diagrammatic representamen directly changes the nature of the represented hypothetical object.

“It seems fairly certain that Marc [Isambard Brunel]’s drawings of his block-making machinery [in 1799] made a contribution to British engineering technique much greater than the machines they represented. For it is safe to assume that he had mastered the art of presenting three-dimensional objects in a two-dimensional plane which we now call mechanical drawing.” (Rolt 1957: 29–30) These diagrams included projections, but still included what could be seen as unnecessary flourishes such as detailing the lever handles and ironwork decorations. Isambard Kingdom Brunel’s 1840–1847 plans for the SS Great Britain (Weale 1847) still utilized illustrations without projections, alongside schematics that were more abstract than his father Marc’s, and comparable to modern engineering diagrams. After this time, more modern schematics become ubiquitous, if not yet standard. As such, the 1840s appear to be a key turning point for the popular dissemination of this combined projection-and-abstraction technology, coinciding with the expansion of the Mechanics Institute but predating Public Libraries. We argue what would be considered today as schematic systems diagrams (distinct from illustrations) have perhaps 200 years of heritage.

2.4 Diagrammatic lament

Throughout history, diagrams have been a neglected area of academic research. Much attention has been given in semiotics to language, images and icons.

For the semiotics community, perhaps diagrams lack overtly the aesthetic dimensions that makes visual semiotics interesting. It is unfortunate that the Prague Linguistic Circle (1929–1952), with their definition of language as “a system of goal-oriented means of expression” (Steiner 1982: 5) and their visual work did not explore with depth diagrammatic representations. Within the semiotics community, being neither art nor text, diagrams were overlooked by the Tartu-Moscow Circle (1964–present). In modern times, we would argue semiotics is having success down a number of paths including linguistics, education, unification, anthropology and marketing. In considering diagrams as a communication medium, very little has been done. Relating to internal representation and reasoning, related work includes the examination of space on computational speed (Bryant and Tversky 1999), tests on the impact of different visual arrangements on tasks (Kosslyn 1980), and diagrams to support mathematical reasoning and abduction (Hoffmann 2004 and others).

By their nature, scientific illustrations can often be considered as iconic sinsigns, and the set of types of scientific illustrations includes diagrams. This content is nicely summarized by Topper, which we utilize for this paragraph. He cites the crucial nature of scientific illustrations as famously described by Gould (1992: 171), “Scientific illustrations are not frills or summaries; they are foci for modes of thought.” From this academic scientific perspective, “Historians of science have shown a little more curiosity, but they too tend to treat scientific pictures only as afterimages of verbal ideas.” (Edgerton 1985: 168), and Historian of Science Eugene S. Ferguson (1977: 835) states that visual imagery has been ignored by historians of technology “because its origins lie in art and not in science.” To further illustrate the point, his seminal work “The Mind’s Eye: Nonverbal Thought in Technology” (1977) there is not a single mention of diagrams or schemata in his historical review. Despite this omission, in modern times diagrams are heavily utilized, particularly in science and engineering. Examples include the periodic table of elements, engineering schematics, and biological life-cycles. Perhaps, where funding exists in these spaces, it is invested in direct technical advancements rather than improving the effectiveness of communication, such as communication through diagrammatic representations.

In more recent times, diagrams are becoming more popular as a research topic. Since 2000 there has been a conference for a diagrammatic interest group, the biannual “Diagrams: International Conference on Theory and Application of Diagrams” which publishes proceedings on philosophy and psychology of diagrams in “Diagrammatic Representation and Inference.” The absence of a diagram focused conference until this recent time is indicative of the attention this crucial form of visual communication has received. Shimojima (2015) signposts three as yet un-investigated cognitive potentials of diagrams in envisaging, aspect shifting, and transfer of spatial analysis. These topics are at the core of diagrams as an aid to reasoning. Work such as this, together with the adoption of visual languages for computing, may inspire an increasing rise in popularity of the discipline. We continue to examine the semiotic foundations of diagrams.

3.1 Diagrammatic structuralism

Engelhardt’s thesis examined graphics, including diagrams, and provided a framework for the description (but not creation or utilisation) of these. This work is abbreviated and modernized in Engelhardt (2006) (using trees of objects and sub-objects) and later Engelhardt and Richards (2018) (with a comprehensive listing). Engelhardt proposes the following meta-taxonomy:

1. Signs (the components of a diagram): 1 Basic graphic vocabulary, 2 Conventional elements, 3 Pictorial abstraction

2. Graphic structure of a diagram: 4 Graphic structure

3. Meaning: 5 Mode of correspondence, 6 The represented information

4. Context-related aspects: 7 Task and interaction, 8 Cognitive processes, 9 Social context

This taxonomy captures a variety of object-attribute properties, which can be considered semantic, syntactic and pragmatic in nature.

3.2 Diagram complexity

The number of semantic objects includes both the objects themselves and the relationship between them (though often in diagrammatic representamen, this can be considered a syntactic encoding of a semantic property of the underlying overall object). A commonplace example is a diagram (or map) of a rail network, which prioritizes encoding relationships. The important part of this example is that a rail map includes minimal semantic objects, excluding for example time between stations or physical distance, both of which might be helpful to the diagram user. We do not say we make a diagram of a set of stations and their relationships, but rather we make a diagram (or map) of the rail network. The stations are a subset of the rail network, though we would argue that it is often the relationship between these sub-signs which is the key to utility (for example, to navigate from station A to station B).

Perhaps the simplest way to quantify diagram complexity is to count features (semantic and syntactic, excluding pragmatic as they assist cognitive process). We can list and enumerate each Sign and Graphic Structure for a particular instance of a diagram.

4.1 A morphic perspective

This section contains the substance of the paper, by facilitating quantification of the signification process. In both Peircian and Saussurian (De Saussure 2011) semiotic thinking, the model of thought and representation is oriented around the object and its representation. In a mathematical formalism called “category theory,” the objects are present, but more important are the morphisms between objects (Eilenberg and MacLane 1945). Category theory is a mathematical foundation for describing general abstract structures. It includes categories, objects, morphisms, functors, and natural transformations. A category is a collection of morphisms. Morphisms between categories are called functors, and morphisms between functors are called natural transformations.

This morphic paradigm allows us to consider the Peircian triadic relation in a new way. Category theory has previously been applied to Visual Semiotics. It is a suitable framework, since it is abstract, and requires simply the existence of objects and relations between them. For the Peircian triadic relation, Vickers et al. (2013) identified some assumptions necessary for the mathematical morphism properties of commutativity, identity and associativity for this visualisation to be considered a category, and engagingly applied this to “chart junk.”

Caterina and Gangle (2016) formalize mathematically the use of category theory within the Peircian triad, with additional commentary and mathematical framework for icons, allowing for additional insight and precision into iconicity. In the abstraction of our paper, their semiotic morphism would be a functor from the Triadic category to itself.

In the concrete instance of an information system, Vickers et al. (2013) captured its objects and morphisms. The existence, even if not explicitly defined, of an Ontology for each category-theoretic object allows us to represent the objects of each morphism as “sets” in the mathematical sense.

Note that the properties of these morphisms determine operations such as generalization, abstraction, amplification etc. (“operations” noted by Peirce; see CP 4.505–4.509; Wason and Johnson-Laird 1972). It is in these properties of the morphisms that we hope to analyze the quality properties of the diagrammatic representation.

Figure 1 shows the familiar Peircian triadic model, with a reorientation to the implicit representational morphisms that occur in the Peircian semiotic process. Note that we have included directional arrows for clarity of definitions, and in an attempt to understand one aspect of the semiotic process. The encoding morphism is about modelling the real world, in doing so “parsing” or instantiating the real world into a set of visual semantic and syntactic actualities. We consider the encoding morphism as a function of the sender, rather than the receiver.

Figure 1:

Peirce’s triadic model with representational morphisms.

In order for the diagram to be considered a representation (or more strongly as a sign), there should be a relationship between “real world” and “diagrammatic representation.” Each step of this mapping sequence can lead to error or inefficiency. Unlike an Information System, where the preservation and completeness of information is desirable, in a diagrammatic representation it may be desirable to lose information in the encoding morphism order to make the pragmatic morphism more cognitively computationally efficient. Stenning and Oberlander (1995) assert that graphical representations limit abstraction and in doing so support better “processibility”. As such, this morphism of diagram creation is not automatically invertible and the category diagram in Figure 1 does not automatically commute. Vickers et al. (2013) explored the restrictions required for different properties of these morphisms. In keeping with our understanding of semiotic theory, the perceptual morphism is not a naïve view on the real world without mediation, but is an extremely complex morphism, in itself a function of the receiver’s context. The pragmatic morphism is concerned with the usage of the diagram in a particular task or set of tasks. It is dependent on the receiver.

4.2 Encoding morphism quality

To discuss the encoding morphism, it is helpful to have a specific application in mind. In computer science, diagrams are often used to signify system designs. While some aspects of these diagrams may be prevalent and recurrent (such as the use of arrows, and the spatial encoding of temporal dimension), common practice does not employ any explicit formalism. Even for the case of Unified Modelling Language (UML), a formally defined and ontologically grounded diagrammatic language for modelling, there is no convention around use of color, size and location, and these properties may or may not convey meaning (Moody 2009). Recently, it has been shown that the diagrammatic representations employed within a sub-field of computer science are extremely heterogeneous (Marshall and Freitas 2018). This example indicates that we may be unable to usefully consider “types” of diagrams at a pure representational level, except in cases where more fully defined frameworks are both available and utilized.

Wand and Wang (1996) ground “data quality” in an ontological framework between Real-World states and the Information System states representing this. Adapting this, in Table 1 we propose intrinsic quality dimensions for the transformation from a Real-world system to a Diagrammatic representamen (i.e., the Encoding Morphism). These remain relevant in this new context, as they are derived directly from the morphism properties (for example if there are objects in the domain for which there is no mapping that would lead to incompleteness). These are at the morphism level, and for specific instances this is instructive. For a given morphism, unlike with data quality, we would not wish for the representation to be complete due to the unnecessary semantic content. Unambiguity, Meaningfulness and Correctness are still usefully considered at this individual level.

The apprehension principle (Tversky et al. 2002) suggests that external representations should be readily and accurately perceived and conceived. As stated, this principle would be a property of the pragmatic morphism. In practice, this would involve omitting (in terms of usage, unnecessary) details from the encoding morphism. Through a taxonomical representation of the diagram, we can explicitly define and reduce these. For example, it is clear that color, outline, and position can be used for grouping objects, so if multiple groupings are required it could be advisable to use these in a coordinated fashion, for example aligning objects which are colored similarly. Conversely if multiple groupings are not required to transmit the required semiotic information from the Interpretant, then perhaps entire diagrammatic properties (such as color) should also be omitted. However at the structural level, we agree with Boon and Knuuttila (2009) that a representation should be structurally isomorphic with that which it represents. For our purposes, the encoding morphism should result in a diagrammatic representamen structurally isomorphic to the object. This is naturally in keeping with the Peircian definition of a diagram.

In this work we are concerned with the quality of this representation, which involves understanding of the diagram’s context, including characteristics of those who will use it, and how it will be used.

4.3 Pragmatic morphism quality

4.4 Summarized diagram quality measures

In total, we propose the following diagram quality measures:

1. Diagram complexity = Count of diagrammatic elements

2. Diagram pragmatic utility quality = correct response ratemean response time (for a specific task)

3. Diagram pragmatic aesthetic quality = Nešetřil’s Combinatorial Entropy

4. Diagram semiotic quality = Correct, Meaningful, and Unambiguous mapping expert assessment

5. Diagram feedback = qualitative desirability feedback from users