Introduction

Serotonin (5-hydroxy-tryptamin, 5-HT) is a phylogenetically ancient molecule being present in almost every living organism1. In mammals, 5-HT affects various behavioural traits, among them mood, emotion and cognition as well as social behaviour2. Especially 5-HT’s role in the regulation of aggression has been aim of intensive research3,4,5,6. First results in mice7 and humans8 indicated an inverse relation between 5-HT concentrations and aggressiveness, i.e. the lower the 5-HT level, the higher the aggression. This so-called “serotonin deficiency hypothesis”3,9 has been supported by further experimental research in rodents: reduced 5-HT concentrations caused by diet, pharmacological interventions or genetic variation increase aggressive behaviour10,11,12, while activation of certain 5-HT receptors as well as genetically increased 5-HT concentrations result in decreased levels of aggressive behaviour6,13,14,15,16.

While so far most studies in animals have investigated effects of higher vs. lower extracellular 5-HT levels, a tryptophan hydroxylase 2 (Tph2) knockout mouse model offers a new approach. TPH2 is the rate-limiting enzyme in the synthesis of brain 5-HT. Thus, homozygous knockout of the corresponding gene allow to investigate effects of complete brain 5-HT deficiency17. Despite the lack of this important neurotransmitter that was confirmed by immunohistochemical analysis, these mice are viable and even develop “serotonergic”-like neurons in the brainstem raphe region17. While conflicting effects on anxiety-like and depression-related behaviours have been reported17,18,19,20, the most prominent result was indeed increased aggressiveness in both sexes19,21,22,23. In addition, compromised social memory and decreased social interest were described23.

Yet, so far social behaviour of Tph2 knockout mice has only been assessed in common short tests used for phenotyping. Aggressiveness, for example, has been measured in the context of a resident-intruder paradigm: mice were confronted with a same-sex wildtype conspecific that was introduced into their cage for a duration of 5 or 10 minutes and attacks of the resident towards the intruder were recorded19,21.

In contrast to these first investigations, the present study applied a different approach not relying on short test paradigms. We aimed to assess spontaneous social behaviour of wildtype and knockout females in a more natural setting and for a longer duration. Furthermore, we intended to investigate aggressive as well as non-aggressive behaviours towards wildtype as well as knockout individuals. For this purpose, female Tph2 wildtype and knockout mice (focal animals) were housed with an unfamiliar same-sex mouse (partner animal) of either the same or the other genotype. To allow the display of a wide range of behaviours and offer the possibility to avoid each other, the animals were housed in large, enriched terraria24,25. Various social behaviours of the focal towards the partner animal were recorded directly after introduction into the terrarium as well as after several days of co-housing based on previous work5,15,25,26,27,28,29.

We expected to find a difference in social behaviour between Tph2 knockout and wildtype focal mice, particularly concerning aggressive behaviours, and we expected the combination of genotypes to affect social behaviour. To test these hypotheses, behaviour of the focal animal was analysed for an effect of the focal animal’s genotype, the partner animal’s genotype as well as interactions between these factors.

Animals, Materials and Methods

Animals and housing conditions

The study comprised 80 adult female Tph2 constitutive homozygous knockout (KO; n = 40) and wildtype (WT; n = 40) mice. They were derived from the internal stock of the Division of Molecular Psychiatry, University of Würzburg and transferred to the Department of Behavioural Biology in Münster, where the study was performed. During the experiment, the animals were kept in pairs in spacious and structured terraria (100 cm × 34 cm × 40.5 cm; “super-enriched terrarium”25; Fig. 1). Each terrarium included wire mesh on top and on one of the short sides, a second level that could be reached via two stairs and a passable standard polycarbonate cage type III equipped with standard mouse diet (Altromin 1324, Altromin GmbH, Lage, Germany) and tap water. Additionally, wood shavings as bedding material (Allspan, Höveler GmbH & Co. KG, Langenfeld, Germany), two semi-transparent red plastic houses (11.1 cm × 11.1 cm × 5.5 cm, Tecniplast Deutschland GmbH, Hohenpeißenberg, Germany), a wooden climbing frame and pressed cotton as nesting material were provided. The colony room was maintained at a temperature of about 22 °C, a relative humidity of about 50%, and a 12 h light-dark cycle. As all experimental animals were previously housed in groups of varying size and genotype composition (which might affect social behaviour), a “wash-out phase” preceded the experiment: for two weeks, each experimental mouse was housed together with two female mice of the C57BL/6J strain in a standard polycarbonate cage type III (37 cm × 21 cm × 15 cm). With this approach we aimed to standardize the housing conditions prior to the beginning of the experiment (i.e. no variation in group size, no variation in the cage-mates’ genotypes). C57BL/6J is the background strain of the Tph2 knockout mouse model. These mice express Tph2 and can thus produce brain serotonin.

Figure 1
figure 1

Super-enriched terrarium. Each terrarium (100 cm × 34 cm × 40.5 cm)25 included wire mesh on top and on one of the short sides, a second level that could be reached via two stairs and a passable standard polycarbonate cage type III equipped with food and water. Additionally, wood shavings, two semi-transparent red plastic houses, a wooden climbing frame and pressed cotton as nesting material were provided (photo of the two black mice: Dirk-Heinz Loddenkemper).

All procedures complied with the regulations covering animal experimentation within the EU (European Communities Council DIRECTIVE 2010/63/EU), were conducted in accordance with the institutional and national guidelines for the care and use of laboratory animals, and were approved by the local government authorities (Amt für Gesundheit, Veterinär- und Lebensmittelangelegenheiten, Münster, reference number: 39.32.7.1).

Experimental design

This study aimed to elucidate the effects of 5-HT deficiency caused by homozygous constitutive knockout of Tph2 on spontaneous social behaviour in a pair housing context. For this purpose, female Tph2 WT and KO mice were housed together in pairs with each one female mouse of either the same or the other genotype. As behaviour of two individuals in a cage is highly dependent, only behaviour of one mouse per terrarium was analysed (focal animal). The second mouse is referred to as “partner animal”. This approach thus resulted in four different combinations (focal animal/partner animal, n = 10 each): WT/WT, WT/KO, KO/WT and KO/KO (Fig. 2). Social behaviour of the focal animals was recorded during the first two hours after introduction on day 1 as well as during two hours on day 5. With this, we aimed to assess behaviour towards an unfamiliar individual (day 1) and towards a familiar individual (day 5). The 2 × 2 factorial design allowed analysing the behavioural data for an effect of the focal animal’s genotype (WT vs. KO), an effect of the partner animal’s genotype (WT vs. KO) as well as interactions between these factors. The decision to record only behaviour of one individual per cage was made to account for the dependencies of social behaviours between two individuals. Exclusively females were investigated as co-housing of unfamiliar male mice might lead to serious injuries due to their territoriality, especially as Tph2 knockout males have been found to be particularly aggressive21,22.

Figure 2
figure 2

Experimental design. Female Tph2 WT and KO focal mice were housed together in pairs with each one unfamiliar female partner mouse of either the same or the other genotype. Social behaviour of the focal animal (arrows) was recorded for two hours directly after introduction on day 1 as well as on day 5 (original photo of the two mice: Dirk-Heinz Loddenkemper).

Observation of social behaviour

Behaviour of the mice was filmed during the first two hours of the dark phase (i.e. the animals’ active period) on day 1 and day 5 using an infra-red camera system (EH1000H-4 Nano cameras, AVer Information Inc., Taiwan). To allow individual identification, mice were colour-marked on the tail one day prior to the introduction into the terraria.

Videos were analysed by an experienced observer (N.K.) and social behaviour was recorded using the software Observer XT 7.0 (Noldus Information Technology, Wageningen, The Netherlands). During analysis, the observer was blind to the mice’s genotype. Focal sampling and continuous recording30 were applied to record frequencies and durations of social behaviour. Based on previous work5,15,25,26,27,28,29, the following behaviours were defined and categorized: social interest (approaching, facial/body sniffing, ano-genital sniffing, following), aggressive behaviour (chasing, attacking, escalated fighting), defensive behaviour (avoiding, fleeing, defensive upright posture), threat behaviour (tail rattling), and other behaviour (mounting) (for definitions see Table 1). Please note that defensive upright posture and tail rattling were only measured qualitatively.

Table 1 Description of behavioural patterns.

Statistical analyses

The analysis of the behavioural data is split into two parts. As the data were clearly not normally distributed, non-parametric statistics were applied. As a first step to avoid the so-called inflation of the alpha, we implemented a global statistical test in which we analysed whether there was a general effect of the focal animal’s genotype to at least one of the behaviours performed by the focal animals. In the second part, we then implemented local descriptive tests in which we analysed the behavioural data for an effect of the focal animal’s genotype, the partner animal’s genotype and for an effect of interactions between these factors on each single focal animal’s behaviour. A rejection of the global test’s null hypothesis in the first part justified the use of local tests in the second part. Statistical tests were performed and graphs were created using the software R31. For a detailed description of the statistics, please see Supplementary Methods (M1).

Results

All twelve defined behaviours of different behavioural systems including social exploration, aggressive behaviour, defensive behaviour and threat behaviour were shown at least once by a KO as well as a WT mouse. The permutation test revealed a significant global effect of the focal animal’s genotype on social behaviour on day 1 (p = 0.0155; Figs 35). If, instead of using a permutation test, we had used the approximate F-distribution of the \({T}^{2}\)-statistics, we would get the same result: the global effect remained significant (p = 0.0454).

Figure 3
figure 3

Social interest. Female Tph2 wildtype (WT) and knockout (KO) mice (focal animals) were housed together with a female of the same or the other genotype (partner animal), resulting in four combinations (focal animal/partner animal, n = 10 each): WT/WT, WT/KO, KO/WT and KO/KO. Behaviour was recorded on day 1 and day 5. Data are presented as boxplots with medians and 25–75% quartiles (box). The whiskers range from the minimum to the maximum of those observations with a distance from the box of less than 1.5 times the length of the box. Observations outside this area are represented as circles. Statistics: see Methods section; (a) approaching on day 1; (b) approaching on day 5; (c) ano-genital sniffing on day 1; (d) ano-genital sniffing on day 5.

Figure 4
figure 4

Aggressive behaviour. Female Tph2 wildtype (WT) and knockout (KO) mice (focal animals) were housed together with a female of the same or the other genotype (partner animal), resulting in four combinations (focal animal/partner animal, n = 10 each): WT/WT, WT/KO, KO/WT and KO/KO. Behaviour was recorded on day 1 and day 5. Data are presented as boxplots with medians and 25–75% quartiles (box). The whiskers range from the minimum to the maximum of those observations with a distance from the box of less than 1.5 times the length of the box. Observations outside this area are represented as circles. Statistics: see Methods section; (a) chasing on day 1; (b) chasing on day 5; (c) escalated fighting on day 1; (d) escalated fighting on day 5.

Figure 5
figure 5

Defensive behaviour. Female Tph2 wildtype (WT) and knockout (KO) mice (focal animals) were housed together with a female of the same or the other genotype (partner animal), resulting in four combinations (focal animal/partner animal, n = 10 each): WT/WT, WT/KO, KO/WT and KO/KO. Behaviour was recorded on day 1 and day 5. Data are presented as boxplots with medians and 25–75% quartiles (box). The whiskers range from the minimum to the maximum of those observations with a distance from the box of less than 1.5 times the length of the box. Observations outside this area are represented as circles. Statistics: see Methods section; (a) avoiding on day 1; (b) avoiding on day 5; (c) fleeing on day 1; (d) fleeing on day 5.

Effects of the focal animal’s genotype

There was a pronounced difference between KO and WT females in terms of aggressive behaviours on both day 1 and day 5: KO females attacked their partners more frequently (day 1: p = 0.0230; day 5: p = 0.0040), chased them for longer durations (day 1: p = 0.0048; day 5: p = 0.0014) and engaged more often in escalated fighting (day 1 and 5: p < 0.0001; Fig. 4). Additionally, KO mice approached the partners more frequently on day 1 (p = 0.0098; day 5: p = 0.0622; Fig. 3). The focal animal’s genotype did overall not affect the frequency or duration of the other measured behaviours (Table 2).

Table 2 Results of the statistical tests (see Methods section) for effects of the focal animal’s genotype, the interaction partner’s genotype and interactions between these factors.

Effects of the partner animal’s genotype

The genotype of the partner animal affected the amount of defensive behaviour displayed by the focal animal: on day 1 as well as day 5, females housed together with a KO avoided the latter more frequently (day 1: p = 0.0006; day 5: p = 0.0110) and displayed longer durations of fleeing (day 1: p = 0.0004; day 5: p = 0.0010; Fig. 5). Furthermore, females housed together with a KO more often engaged in escalated fighting (day 1 and 5: p < 0.0001). None of the other measured behaviours were affected by the genotype of the partner animal (Table 2).

Interactions between the focal animal’s genotype and the partner animal’s genotype

There was an interaction between the focal animal’s genotype and the partner animal’s genotype concerning avoiding (day 1, p < 0.0001; day 5: p = 0.0086), fleeing (day 1: 0.0002; day 5: p = 0.0004) as well as escalated fighting (day 1 and 5: p < 0.0001; for the other behaviours see Table 2). Graphical examination indicates that in all these parameters the effect of the partner animal’s genotype (higher duration/frequencies of the behaviours displayed towards KO females) was more pronounced in KO than in WT mice.

Discussion

The most outstanding result of this study was: while WT females were rather docile, KO individuals displayed considerable amounts of aggressive behaviours in a pair-housing context. This confirms our hypothesis and corresponds to the untypically high levels of aggression displayed by Tph2 KO females towards unfamiliar WT females when tested in a resident intruder paradigm21,23. Furthermore, Tph2 KO females in the present study performed aggressive behaviours not only directly after introduction into the terraria but still after several days of co-housing. This indicates that the higher aggressiveness of KO mice is not only shown in response to an unfamiliar intruder in a test situation but also spontaneously in the housing context and might thus represent a generalized form of aggression.

As outlined in the introduction, the “serotonin deficiency hypothesis” links lower 5-HT concentrations to higher aggressiveness. However, there is also contradicting evidence32,33 and a recent meta-analysis in humans found a rather weak negative correlation between brain 5-HT concentrations and aggression3. Nevertheless, our results correspond to a dampening effect of 5-HT on aggression and would thus be in line with several previous studies in rodents10,11,12,13,14,15,16.

The overwhelming majority of studies on the connection between 5-HT and aggression has been performed in male rodents. The few studies investigating females have mainly focussed on maternal aggression: 5-HT has been found to affect aggression in lactating female rats16 and mice6,34. Beyond that, our data indicate a role of serotonergic neurotransmission also in the regulation of female aggression outside of a reproductive context.

Yet, although this study could clearly link Tph2 knockout to higher aggression, it is still possible, that the described behavioural effects were not directly caused by the absence of serotonergic neurotransmission35. Constitutive Tph2 KO mice lack 5-HT throughout development and compensatory mechanisms as well as a disturbed tryptophan metabolism may also play a role. In a previous study, however, restored brain serotonin synthesis in male Tph2 KO mice by supplement of 5-HT’s immediate precursor (5-hydroxytryptophan) resulted in decreased aggression in a resident intruder paradigm12.

Beyond the higher aggression, social behaviour did not seem to be affected profoundly by 5-HT deficiency. First of all, the repertoire of social behaviours was exactly the same in both genotypes. Moreover, despite a higher rate of approaching on day 1, KO females did overall not differ from WT females concerning levels of social interest, defensive behaviour and mounting. Thus, our results do not support the notion of decreased social interest in female Tph2 KO mice that was found in a social interaction test23. This different outcome, however, can be explained by the fact that in the previous study only social exploration of non-aggressive KO females was analysed, which might have affected the results.

In line with our second hypothesis, the genotype of the partner animal had a considerable effect on defensive behaviour of the focal animal on both time points: corresponding to the higher aggression in KO females, avoiding as well as fleeing were more frequently displayed towards mice of this genotype. Additionally, focal animals were more often involved in escalated fighting when housed together with KO individuals. This finding highlights the fact that agonistic behaviour is not only influenced by internal factors, but additionally depends on situational conditions, in this case the genotype of the interaction partner. This has also been described in previous studies performed in male mice varying in brain 5-HT concentrations15.

As expected, some behaviours were also affected by an interaction between the genotypes. Escalated fighting was considerably more frequent in the KO/KO condition than in the mixed genotype condition and did never occur between WT females, thus further underlining the higher aggressiveness of KO females. Additionally, such an interaction also existed concerning defensive behaviours: the effect of the partner animal’s genotype on the focal animal’s behaviour were stronger in KO than in WT females. Thus, despite their higher aggressiveness, KO females were not only able to display fleeing and avoiding when meeting an aggressive opponent (particularly a KO) but they did so even more than WT females. As discussed by Angoa-Pérez and colleagues21, the higher aggressiveness in 5-HT deficient individuals might result from high impulsivity due to a lack of behavioural inhibition. In this context, our results indicate that this lack of inhibition in 5-HT deficient mice is not restricted to offensive aggression but also concerns defensive behaviours. This suggests a general increase of impulsivity and loss of behavioural control in KO females during agonistic social encounters.

Conclusion

In this study, changes in social behaviour caused by Tph2 knockout were mainly related to aggressive behaviour: KO females displayed untypically high levels of aggressive behaviour even after several days of co-housing. Furthermore, they showed increased levels of defensive behaviours in response to aggressive partner animals. Thus, the results of this study on the one hand show, that 5-HT is not required to display the full range of social behaviours investigated in this study. On the other hand, they indicate an involvement of serotonergic neurotransmission in the control of female aggression and a prominent role in the fine-tuning of behaviour during agonistic social interactions. Future research will show, whether the observed behavioural effects are indeed directly caused by the lack of serotonin or by potential compensatory mechanisms.