The prominence of spoken language elements in a sign language

1 Introduction

Bilingual speakers’ language use has been characterized in terms of a variety of possible language contact phenomena, sometimes showing a dense integration of elements from different languages (see Muysken [2013] for an overview). Deaf and hearing users of signed languages form a special case in the study of bilingualism and language contact, as there are no physical limitations for both languages to be produced simultaneously: spoken language by the oral articulators (with voice or without) and sign language by other visible articulators. Although lexical content is mainly conveyed by the hands in all sign languages, other articulators play an essential role as well, such as eye gaze, position of head and torso, and mouth actions (see Crasborn [2006] for an overview). These mouth actions may be divided into mouth gestures and mouthings (e. g., Boyes Braem and Sutton-Spence 2001; Johnston et al. 2015). An example of each is provided in (1). ^[1]

The mouth action transcribed as “pt” (two subsequent plosive-like gestures) in (1a) does not have a relation to any Dutch word or phrase related to the meaning of the sign, and is therefore classified as a mouth gesture. The mouth action transcribed in (1b) as afhankelijk ‘dependent’, by contrast, is a full Dutch word that corresponds to the lexical meaning of the manual sign, and it is therefore classified as a mouthing.

Mouth gestures are sign language inherent, linguistically relevant mouth movements that co-occur with signs. They have various forms and functions, such as puffed cheeks to indicate large size of the referent of the co-occurring sign; or the pursed lips with the sign for be-present, adding to the phonological well-formedness of the sign (see Crasborn et al. [2008a] for an overview). Mouthings, on the other hand, originate from the spoken language of the hearing community in which a deaf community is embedded (see the various contributions to Boyes Braem and Sutton-Spence [2001]). They are (usually silently) mouthed words from the surrounding spoken language, or parts thereof. Typically, the semantics of the spoken word overlaps with that of the manual sign (Bank et al. 2011). In some cases, mouthings can be used to distinguish different meanings of a sign, such as in Example (2).

In Sign Language of the Netherlands (Nederlandse Gebarentaal, NGT), all signers appear to use spoken language mouthings with their signing at some point (Bank et al. 2011; Schermer 1990). This is in fact the case in many sign languages studied to date (e. g., Boyes Braem [2001] for Swiss-German Sign Language; Crasborn et al. [2008a] for British, Dutch, and Swedish sign languages; Ebbinghaus and Heßmann [1994] for German Sign Language; Mohr [2012] for Irish Sign Language; Nadolske and Rosenstock [2007] for American Sign Language; Nyst [2007] for Adamorobe Sign Language; Schuit [2012] for Inuktitut Sign Language; Sutton-Spence [2007] for British Sign Language; Sze et al. [2013] for Jakarta Sign language; Vogt-Svendsen [2001] for Norwegian Sign Language). This long list makes clear that signers around the world use a combination of the language most accessible to them (i. e., a signed language) and the language used by most people in their surroundings (i. e., a spoken language). However, the effects of language contact in the case of signed and spoken languages remains largely unexplored. For most sign languages mentioned above, however, studies were conducted with only a few signers, in laboratory settings, or were focused on a general description of the language instead of studying language contact. Moreover, cross-language activation studies often focus on perception instead of production (e. g., Morford et al. 2011) or only describe hearing bimodal bilinguals (Emmorey et al. 2008, 2012).

Schermer (1985) suggested earlier that “the existence of a pure sign language, without the occurrence of any speech, among deaf adults, is more or less a theoretical construct” (Schermer 1985: 288). Since there is no Deaf country or other place only inhabited with monolingual deaf signers, there is no way we could witness how a signed language would develop without any influences from outside spoken languages. One may hypothesize that the occurrence of mouthings is related to amount of education in the spoken language. The only case of a sign language without mouthings that we are aware of is Kata Kolok in Indonesia, a rural sign language in which “there are virtually no mouthings” (de Vos and Zeshan 2012: 17). Indeed deaf education is a very recent phenomenon for the signers in this village, but at the same time the amount of schooling for deaf users of the languages listed above is highly variable and still they appear to incorporate mouthings on a substantial scale. Kata Kolok remains an exception, it appears. Obviously, more empirical research is needed to chart the use of the mouth in more sign languages than the ones mentioned above.

One of the questions sign language researchers are pursuing concerns the linguistic status of mouthings in sign language. Some consider mouthings to be an inextricable, inherent part of the sign language lexicon (e. g., Boyes Braem 2001; Sutton-Spence and Day 2001), yet others argue that, while relevant for communication, mouthings should not be regarded as part of the lexicon (Ebbinghaus and Heßmann 2001). In support of the latter view, Vinson et al. (2010) showed with picture-naming tasks and word-translation tasks that signs and mouthings may be represented and accessed largely independently from each other. Thus it seems that the manifestation of two languages simultaneously is a special case of code-mixing (Muysken 2000). However, code-mixing by unimodal bilinguals is a sequential phenomenon by nature of the modality. To describe the bimodal code-mixing of hearing bimodal bilinguals (e. g., CODAs, children of deaf adults), Emmorey et al. (2005; also Emmorey et al. 2008) proposed the term code-blending. Van den Bogaerde and Baker (2005; Baker and van den Bogaerde 2008) adopted this term to also cover the bimodal bilingual input to children from their deaf mothers, and the output from both deaf and hearing children to their deaf mothers. Since virtually all deaf signers in developed countries are functionally bilingual – as a result of growing up in a hearing world and receiving education in (at least) spoken language (cf. Ann 2001; Lucas and Valli 1992) – we follow van den Bogaerde and Baker (2005; Baker and van den Bogaerde 2008) and use the term code-blending for deaf signers using mouthings.

An open question in the research field concerns to what extent the use of mouthings is determined by individual-level or group-level variables: to what extent do elements of spoken Dutch play a role in conversations between deaf people whose primary language is sign language? Browsing the video recordings of the Corpus NGT (Crasborn and Zwitserlood 2008; Crasborn et al. 2008b), one gets the impression that there is a lot of variation between individual signers, both in their style of articulation of manual signs and in their style of using non-manual features such as mouthings.

Variation in signing between individual learners may be related to social distinctions. The body of research into the sociolinguistics of signed languages is steadily growing, most notably through the work of Lucas (e. g., Lucas 1995, 2001; Lucas and Valli 1992). In Lucas et al. (2001), the authors report that the social constraints that influence variation in American Sign Language (ASL) are the same as the ones that influence variation in spoken language: age, gender, ethnicity, social class and region. In the domain of manual phonology, for instance, they find regional differences for handshape, and age differences and social class differences in the use of a specific handshape variant of a sign. They further report on gender variation in the use of overt pronouns. Similar sociolinguistic studies have been conducted for other sign languages, for example, British Sign Language (BSL; Fenlon et al. 2013; Schembri 2008).

When the language of an age group is contrasted with that of an older or younger adult group, differences are expected to occur, reflecting diachronic language change (Kerswill 1996). In the case of sign languages, educational reforms are expected to have a major impact on change due to the fact that for many deaf children with hearing parents, most sign language input is available in deaf schools. In the course of the last century, the language policies of deaf schools in Western Europe have changed considerably, showing an overall development from oralist education (i. e., teaching deaf children to speak and to lip-read) to the inclusion of sign language in the curriculum. At the same time, language policies have varied from school to school. Both in the literature and in informal accounts of deaf people, even in the most oralist schools, there was plenty of sign language use outside the classrooms in the breaks and after school. The impact of education policies on a sign language is well documented for Ireland, where strict separation of boys and girls has led to many gender differences in the language (LeMaster 1990, 2000; LeMaster and Dwyer 1991).

Studies on variation in the use of mouthings, however, are scarce. This is hardly surprising, since the sign language that is most studied is ASL, a language that has a longstanding reputation of hardly featuring any mouthings –but see Nadolske and Rosenstock (2007), who show that “contrary to what has been claimed in the literature, mouthings contribute significantly to the formal and semantic aspects of ASL” (2007: 35). Sutton-Spence and Day (2001) did not find differences between groups of different backgrounds in their exploratory study on BSL. Regarding differences between age groups, Hoyer (2004) mentions in passing that mouthings are more frequently used by elderly signers in Finnish Sign Language. However, she does not refer to empirical evidence for this claim. Mohr (2012) finds both gender differences and age differences in a study on mouthings in Irish Sign Language (ISL), which may not be surprising in the light of the overall gender differences in ISL lexicon referred to above. In Mohr’s study, women on average use considerably more mouthings than men. When split into age groups, the number of mouthings by women turns out to be quite constant, whereas the use of mouthings by men decreases over age groups. Younger male signers use more mouthings than older male signers. In the youngest age group, males and females are comparable in their use of mouthings. Mohr ascribes the Irish situation to the separate institutions for deaf boys and girls in Ireland: oralism was introduced at a later stage to the boys’ school (1957) than to the girls’ school (1946), thus accounting for the lower numbers of mouthings by elderly men. Regarding recent changes in language policy away from oralism, Mohr mentions that “today, the schools’ current language policy states that teachers should communicate with the children in whatever language is most suitable to their needs, be it Irish Sign Language, Signed English or spoken English” (Mohr 2012: 51), a change in policy that started to take shape during the 1980s (Leeson and Saeed 2012).

In a study on NGT, van de Sande and Crasborn (2009) looked at the proportions of mouthings and mouth gestures between registers (narrative vs. interactive) and between early and late learners, where they classify late learners as those who start sign language acquisition at around 3 years of age, with an average of 4.5 years in their sample. They found a significant difference between registers, with more use of mouthing in interactive registers, and a non-significant tendency towards more frequent use of mouthings by late learners. It should be noted that the division of their sample into early and late learners largely coincides with the age groups that we will distinguish in our study reported below. In a study on ASL, Nadolske and Rosenstock (2007) found the same register effect as well. Stadthaus (2010) used the same sample as van de Sande and Crasborn (2009), consisting of signers from the Amsterdam and Groningen regions, as well as a few signers labeled “other region”. She looked at jaw drop in mouthings in NGT, operationalizing the hypothesis that it may not be the number of mouthings in relation to manual signs that varies but rather the way they are articulated that results in the impression that older signers use more mouthings than younger signers: speech and/or mouthings of older signers may be less sloppy because of stern speech therapy in oralist times. However, she did not find any differences between the age groups. Finally, Bank et al. (2015) looking at spreading of mouthings over adjacent signs in NGT (where one mouthing encompasses multiple manual signs), found no differences in spreading behavior between older and younger signers.

Mouth actions can vary between individuals or groups in several possible ways. First, there can be variation in the number of mouthings relative to the number of signs. Although van de Sande and Crasborn (2009) found no differences between age groups, it may be that their sample was just too small. In the present study we therefore investigate frequency differences based on a much larger sample.

Second, there may be variation in articulation. Stadthaus (2010) investigated this by manually scoring jaw drop, or how far the mouth opens during mouth actions. While it seems like a good idea to annotate the entire corpus for properties like amplitude or precision of articulation, at present we have no automated ways of doing this for a large data set, and we discarded this as a possible option for the present study.

Third, it may be the case that older signers indeed do not use more or more pronounced mouthings, but that they simply point at a location in signing space while mouthing the intended spoken word when they do not know a sign or when they want to be efficient in terms of manual articulation. This would explain why earlier research did not find differences in manual sign to mouth action ratios for different age groups, but would still add to the impression of older people using more mouthings. This effect could not show up in the Bank et al. (2011) study on variation in highly frequent signs, because they left pointing signs out of their study.

Finally, there may be variation between groups in their use of mouthings as compared to mouth gestures. Bank et al. (2011) found variation in the choice between mouthings and mouth gestures accompanying a sign, but they did not look into variation of that choice over groups. That is, older signers may be more consistent in their mouthings, while younger signers may show more variation in their choices.

In this paper, we will seek answers to the following research questions. To begin with, how prominent is the use of mouthings – the code-blends of simultaneously articulated manual signs and spoken language words – exactly in NGT in terms of frequency? Second, are mouthing frequencies related to other signer characteristics, such as age, gender, region, school or having deaf or hearing parents? Our third research question relates to pointing signs (pt) that may compensate for not knowing a sign or may be a shortcut relying on the information conveyed by the mouthing. Do particular groups of signers more often combine mouthings with pointing signs than with content signs?

2 Methodology

We analyzed parts of the Corpus NGT (Crasborn and Zwitserlood 2008; Crasborn et al. 2008b). The Corpus NGT is one of the largest sign language corpora in the world, and available online under an open access license. It contains video data of 92 prelingually deaf signers, recorded in pairs, who retell video clips and picture stories, and discuss issues related to deafness, education and sign language. The signers in the Corpus NGT were selected on the base of signing skill, not their knowledge of the Dutch spoken language. Still, all participants have followed secondary education where they successfully completed exams in Dutch.

Annotations in the corpus are made on a variety of tiers for the different articulators. Figure 1 gives an idea of what the annotations look like, in a fragment where both signers in the video (not depicted) are signing at the same time.

Figure 1:

Part of the annotations in the Corpus NGT on a selection of tiers. The affixes S1 and S2 in the tier titles (first column) denote signers one and two (each occurring in a separate video, not depicted); GlossL and GlossR are the tiers where glosses for the signs on the left and right hands, respectively, are annotated; Mouth is the tier for mouth action annotations, and the annotations on the MouthType tiers denote the types of mouth action.

For manual components, there are separate, independent tiers for the left and right hands, ensuring an accurate annotation of what each hand does. This benefits, for instance, research on handedness, or on spreading of the non-dominant hand (e. g., Sáfár and Crasborn 2013), but in an automated analysis it is not immediately clear whether two co-occurring manual glosses actually comprise one two-handed sign or two unrelated but (partly) overlapping one-handed signs. Since the Corpus NGT reflects everyday language use, signers may articulate more sloppily or deviate from citation forms (by articulating one-handed signs as two-handed and vice versa), use their non-dominant hands as a buoy (i. e., holding the end location of a sign while the preferred hand continues signing), or articulate two one-handed signs simultaneously. This currently makes determining the number of signs in the corpus a non-trivial task.

Annotations for mouth actions are made on a tier independent from the gloss tiers, and these have dependent tiers attached to denote the types of mouth actions. In the example in Figure 1, four different types can be observed on the MouthType tiers for the two signers (S1 and S2): M denotes a mouthing (i. e., the item described on the Mouth tier is a lexical item from spoken language); M-add denotes a mouthing without an accompanying sign; A denotes an adverbial or adjectival mouth gesture (modifying the co-occurring sign to indicate size or manner, for instance); W denotes a whole face action (such as a surprised look). In the remainder of this paper, we will only be concerned with the M category of mouth actions (including the subtype of added mouthings, M-add).

In the past, annotation work for manual components has concentrated on signers from the Amsterdam and Groningen regions, and therefore mouth annotations are concentrated in this part of the corpus as well. Further, for most signers only clips in the conversational register (discussions) are currently annotated. There are just a few signers for whom data are available in both conversational and narrative registers (retelling stories). Van de Sande and Crasborn (2009) showed that there is a significant effect for register in the use of mouthings, with more mouthings being used in a conversational register. With not much narrative data at our disposal we decided to concentrate on the conversational register only, reflecting everyday language use.

For some previous studies, mouth annotation work was only done relating to specific signs throughout the corpus, while other studies resulted in clips fully annotated for the mouth. For the current study, we annotated mouth actions for many clips that had been partially annotated before, thus resulting in a more coherently annotated corpus. There are 257 clips that contain conversational data and that are at least partly annotated for mouth actions, containing data from 75 signers from all five regions in the Netherlands where educational institutes for the deaf are established (i. e., Groningen, Amsterdam, Rotterdam, Voorburg, St Michielsgestel). Five signers with less than ten mouth annotations each were excluded from our analyses. All results for this sample (Sample 1) are thus based on 70 signers from 252 clips. To see if there would be any language change, for instance influenced by educational reforms, we divided the sample into three age groups: 17–26, 27–40 and 41–84. Furthermore, to see if there would be an effect in highest form of education that participants had followed, we distributed the signers over five groups: primary and secondary education, and three types of vocational education common in the Netherlands: lower, middle and higher vocational education (LBO, MBO and HBO, respectively).

We will be looking at several sociolinguistic variables to see if variation in mouthings occurs: region, gender, age, highest level of education and whether a signer has deaf parents or not (which we will be calling nativeness). Sign languages are a special case in that respect because the L1 of a child’s parents will not necessarily become the L1 of the child (as most deaf children have hearing parents). Moreover, it is often hard to tell whether the signed language is the L1 or the L2, since they are often learned at the same time as the spoken language. There are very few native signers if you consider a native signer to be someone who learns sign language right from birth from fluently signing parents. Only a small minority of deaf born children have deaf parents (usually estimated between 5 and 10 %; Mitchell and Karchmer 2004), of whom, in turn, only a small minority would have been real native signers themselves.

To investigate the question of mouthings co-occurring with pointing signs, we could only use clips that were fully annotated for mouth actions, because for every pointing sign we wanted to see what kind of mouthing co-occurred with it, including “no mouth action on pointing signs”. If we would have been using partly annotated clips, we wouldn’t have been able to distinguish between “no mouth action” or “not annotated yet” in an automated analysis. This subset of Sample 1 (that we will dub Sample 2) consists of 38 clips fully annotated for hands and mouth, spanning 86 minutes. There are, coincidentally, 38 signers in this sample, all from the Amsterdam and Groningen areas.

To get an idea of how often mouth actions actually happen, we calculated the ratios between signs and mouth actions and between signs and mouthings, using Sample 2 (because, again, we had to be sure that all mouth actions in a clip would be annotated). Since the gloss annotations in the Corpus NGT do not easily allow for the count of signed lexical items (see above on the separate annotation of the left and right hand), we assumed that, with one-handed signs, all right-handed signers are strictly right-handed, and all left-handed signers are strictly left-handed (thus disregarding changes in preferred hand, or dominance reversal). Thus, for right-handed signers, the number of glosses for the right hand then illustrates both the number of one-handed and two-handed signs, and for left-handed signers vice versa. Ambidextrous signers were left aside (see Sáfár [forthcoming] on how to determine handedness in an automated way). We realize this is an oversimplification, but it will give us a general idea about the number of one-and two-signs in the context of mouth actions.

3 Results

3.2 Results on mouthing frequencies for various sociolinguistic variables

Our main Sample 1 (70 signers, 252 clips at least partly annotated for mouth actions) contains 10,814 mouth actions. Of those, 9,120 (84.3 %) are mouthings and 1,463 (13.5 %) are mouth gestures. The remaining 231 were either not visible or undecided. Figure 2 shows the outcomes for the mean percentages of mouthings when the data are split up for the five sociolinguistic variables.

Figure 2:

Percentages of mouthings, for gender, age, nativeness and region. Error bars have the value of two standard errors.

One of the most striking results that can be seen in Figure 2 is the large percentage of mouthings: there are no groups where the average number of mouthings per 100 mouth actions is lower than 80. Further, it is evident that the differences between the subgroups are small, the biggest difference being between 80 and 100 % for region.

It should be noted, however, that the few signers from the Voorburg, Rotterdam and St Michielsgestel regions – i. e., the three highest bars in the graph – are represented with only a small amount of annotated data: these 11 signers account for only 274 mouth actions, a mere 2.5 % of the total amount.

Figure 2 effectively shows how small the differences are between groups, but does not reflect the differences within the groups. The scatter plot in Figure 3 visualizes how the number of mouthings to mouth actions is spread across age, gender and native signers (in percentages), including the contribution of the individual signers in terms of numbers of annotations.

Figure 3:

Scatter plot for percentage of mouthings in relation to mouth gestures as a function of age. Black triangles represent male signers, blue circles represent female signers; solid marks represent native signers, outlined marks represent non-native signers. The size of a mark represents the number of tokens for that signer, in five categories (10–100, 101–200, 201–300, 301–400 or more than 400 tokens).

Again, one of the most striking results is the high percentage of mouthings: the lowest percentage of mouthings is 64.6 %. There is a high score for mouthings, but the variation between participants is considerable. Further, it can be seen that no distinctive groups can be formed within gender, nativeness or age. There are a number of signers with a mouthing percentage of 100 %, most of them contributing less than 100 mouth actions to the sample. There are also signers who contribute a high number of mouth actions and still score almost a 100 % for mouthings (such as 98.4 % for a 61-year old non-native male, contributing 369 mouth actions to the sample).

We investigated the effect of the set of sociolinguistic variables by applying analysis of variance. No effects were found for gender (F(1,68)=0.678, p=0.413), age (F(2,67)=2.304, p=0.108), and nativeness (F(1,68)=1.201, p=0.277). Significant effects were found for region (F(5,64)=6.525, p=0.000) in that signers from Voorburg use more mouthings, and for education (F(4,65)=3.931, p=0.008) in that the use of mouthings decreases with better education (post-hoc analysis, excluding Rotterdam, St. Michielsgestel and Mixed regions). ^[2]

3.3 Results for combinations of mouth actions and pointing signs (pts)

In the continuously annotated Sample 2 (38 signers, 38 clips, 86 minutes), there are 1,758 annotations for pointing signs. These are distributed over five groups, as summarized in Table 1. The first group are pt annotations that co-occur with one mouth action, not spreading over or from adjacent signs. There are 458 (26.1 %) such annotations; 140 (30.6 %) of those mouth actions are mouth gestures, 318 (69.4 %) are mouthings. The largest word class present in this segment is that of pronouns (72 tokens, 22.6 %), mainly consisting of ik ‘I’ (57 tokens, such as in Example [3] where ik occurs together with a point to self [annotated as pt:1]).

(3)

Manual:	pt:1
Mouth:	ik
Meaning:	‘I’

Other large groups are those of verbs (46 tokens [14.5 %], with 20 tokens from the zijn ‘to be’ paradigm), adverbs (13.8 %), prepositions (12.6 %) and possessives (10.4 %). We thus find a much lower percentage of mouthings for this category than the 84.3 % average for all signs as displayed in Figure 2.

The second group are pt annotations that co-occur with multiple mouth actions. This is a rather small group, containing 18 (1.0 %) pt annotations; most of the mouth actions in these constructions are mouthings. This small group consists of the following constructions:

ben jij ‘are you’	tuinman bleek A ‘gardener turned out
bij mij ‘at my place’	<adverbial mouth gesture>’
bij mij thuis ‘at my home’	van mij ‘mine’
hij ook waterpokken ‘he chicken pox too’	vind ik ‘I think’
hoeft niet ‘not necessary’	volgens mij ‘I think’
ik ben ‘I am’	voor mij ‘for me’; three tokens
in Hilversum ‘in Hilversum’	voor mij, voor mij ‘for me, for me’
of niet ‘or not’; two tokens	wijs daar ‘point there’

With very few exceptions (hoeft niet; of niet; tuinman bleek <mouth gesture>), these multi-word mouthings contain a pronoun (ik, mij, jij) or a locative expression (thuis, daar).

The third group are pt annotations that co-occur with a mouth action that is also present with one of the adjacent signs (thus, a spreading mouth action). There are 548 (31.2 %) such annotations; 433 of those are mouthings, 115 are mouth gestures. An example is given in (4), where the extended line under the mouthing ‘school’ indicates that the mouthing spreads over the second pt annotation. ^[3]

(4)

Manual:	pt	past	school	pt
Mouth:			school
Meaning:	‘Where did you go to school?’

In this case, both pt annotations are indicating the conversation partner (so-called pronoun copy, see Bos [1995]). The first pt annotation in Example (4) is not accompanied by a mouth action, see below (group five). Note that questions in NGT are marked by non-manuals (raised eyebrows) that are not yet annotated our dataset.

The fourth group are pt annotations that co-occur with a mouth action that accompanies a sign on the other (dominant) hand. There are 96 (5.5 %) such mouth actions; the two hands articulate different signs simultaneously, and the mouth action can be traced back to accompanying the sign on the hand not articulating the pt. An example is given in (5), where the mouthing ‘hour’ accompanies the sign hour-long on the right hand, while pt is articulated on the left hand. ^[4] The first pt annotation (pt:1) co-occurs with a palm-up sign on the other hand, but there is no accompanying mouth action.

(5)

Manual left:	palm-up	stay	pt
Manual right:	pt:1	stay	hour-long
Mouth:		Blijf	uur
		‘stay’	‘hour’
Meaning:	‘I (will) stay one hour’

Finally, the fifth group are pt annotations that do not co-occur with any mouth action, the mouth being in a neutral state. There are 638 (36.3 %) such annotations. Table 1 summarizes the results.

Table 1:

Combinations of co-occurrence of pointing signs (pt) and mouth actions.

All pt annotations	1758 (100 %)
pt co-occurs with only one mouth action	458 (26.1 %)		(100 %)
		Mouthing	318 (69.4 %)
		Mouth gesture	140 (30.6 %)
pt co-occurs with multiple mouth actions	18 (1.0 %)	Mouthing	18 (100 %)
pt co-occurs with mouth action that spreads over multiple signs	548 (31.2 %)		(100 %)
		Spread from mouthing	433 (79.0 %)
		Spread with mouth gesture	115 (21.0 %)
pt is on nondominant hand	96 (5.5 %)
No mouth action	638 (36.3 %)

We found no significant differences in ratios of pt annotations with or without various types of mouth actions for age, region, gender, or having deaf parents or not. There was a significant effect for education level, but only for the combination of a pt co-occurring with one mouth action (F(4,20)=3.503, p=0.018). When we split this category into its subcategories mouthings and mouth gestures, the effect lies with mouthings (F(4,30)=4.048, p=0.010). Figure 4 visualizes the findings summarized in Table 1.

Figure 4:

Distribution of types of mouth action (MA) occurring with pointing signs (pt).

4 Discussion

5 Conclusion

We showed that the majority of manual signs is accompanied by a mouth action in NGT dialogues, and that the majority of these mouth actions consist of (fragments of) words from spoken Dutch. We further showed that there are no age, gender, region or nativeness-related differences in the use of these mouthings in NGT. A small effect was found for level of education, higher educated signers using fewer mouthings. Future research is needed to explore this effect further, for example by looking at the variation in sign/mouthing combinations or in sign/mouth gesture combinations related to educational level. Are higher educated signers perhaps more skilled in combining different semantic elements from the two languages? Further, perception and recognition studies of sign language need to take into account the overwhelming presence of mouth actions in signed interaction, something that is largely ignored in studies that do not focus exclusively on the mouth. What is the relative role of the two information streams in different stages of the comprehension process?

Our findings also impact our understanding of the cognitive representation of signs and the organization of mental lexicon. Although Vinson et al. (2010) showed that signs and mouthings are processed largely in separate channels, our findings show that the two are tightly linked in terms of frequency and co-occurrence. Since there is no linguistic necessity for inhibition of mouthings in the production of manual signs, the mental representations of the two languages may be even more linked for bimodal bilinguals than for unimodal bilinguals (Dijkstra and van Heuven 2012; Giezen and Emmorey 2015).

We can only speculate as to why we did not find any differences in the use of code-blends in different groups of signers. The explicit attention to spoken Dutch in deaf education may have been such a dominant part of primary education for all signers that any variations in educational approach by different schools or at different times was only seen in the amount of (positive or negative) attention for NGT. An alternative possibility is that we looked at the wrong dependent variables, namely the frequency of mouthing. Traditional sociolinguistic and more recently sociophonetic studies of spoken (and signed) languages look at details of the pronunciation, which is something we did not do. We know from personal experience that deaf speakers adopt regional accents in their speech even if they only had visual access to spoken language all their life. Thus, when deaf people produce audible Dutch speech, one can clearly distinguish Flemish versus northern deaf speakers after hearing only a few syllables. While the attention for visible speech is increasing (e. g., Jesse and Massaro 2010; van der Zande 2013), we know of no studies looking at dialectal variation in visible oral articulations. Perhaps investigating the fine articulatory detail of mouthings could bring to light group differences that did not appear using the present methodology. At the same time, in so doing one might well be investigating speech characteristics that differ in corresponding groups in the hearing language community, while not establishing differences in relative role of spoken language in the sign language for these groups. By looking at frequency of spoken language elements in relation to manual elements in the way we did, we obtain a more direct measure of the relative importance of mouthings.

We conclude on the basis of corpus evidence that spoken Dutch provides an almost continuous stream of linguistic information in parallel with the manual sign stream, and that the code-blends identified by Emmorey for bilingual hearing native signers of ASL are a core feature in the communication of all deaf NGT users. It remains an open question which of the two information channels is more important in activating lexical items during language perception. Deaf people are known to be proficient lip readers (Bernstein et al. 2000; Mohammed et al. 2006), and given the prominence of (parts of) Dutch words in deaf interaction as demonstrated in our study it would appear plausible that they use lip-reading as a main strategy in communicating with other deaf signers. Posing this question may appear to bring back the once sensitive question of sign languages being full-fledged linguistic systems independent from spoken languages, but the present study suggests that there is no sign language without Dutch mouthings for deaf people in the Netherlands. In other words, there are no monolingual users of NGT, even if their level of proficiency in spoken Dutch may be variable. Neither have we found monolingual use of NGT in our dataset, not even for short sequences of a few sentences. While this does not contradict the overwhelming evidence that NGT like other sign languages has a lexicon and a grammar that is markedly different from the related spoken language, it is unlikely that processing of sign language input is fully independent from processing spoken language, opening up a new domain of research in bilingualism studies.