Serotonin control of sleep-wake behavior

Summary

Based on electrophysiological, neurochemical, genetic and neuropharmacological approaches, it is currently accepted that serotonin (5-HT) functions predominantly to promote wakefulness (W) and to inhibit REM (rapid eye movement) sleep (REMS). Yet, under certain circumstances the neurotransmitter contributes to the increase in sleep propensity. Most of the serotonergic innervation of the cerebral cortex, amygdala, basal forebrain (BFB), thalamus, preoptic and hypothalamic areas, raphe nuclei, locus coeruleus and pontine reticular formation comes from the dorsal raphe nucleus (DRN). The 5-HT receptors can be classified into at least seven classes, designated 5-HT_1–7. The 5-HT_1A and 5-HT_1B receptor subtypes are linked to the inhibition of adenylate cyclase, and their activation evokes a membrane hyperpolarization. The actions of the 5-HT_2A, 5-HT_2B and 5-HT_2C receptor subtypes are mediated by the activation of phospholipase C, with a resulting depolarization of the host cell. The 5-HT₃ receptor directly activates a 5-HT-gated cation channel which leads to the depolarization of monoaminergic, aminoacidergic and cholinergic cells. The primary signal transduction pathway of 5-HT₆ and 5-HT₇ receptors is the stimulation of adenylate cyclase which results in the depolarization of the follower neurons. Mutant mice that do not express 5-HT_1A or 5-HT_1B receptor exhibit greater amounts of REMS than their wild-type counterparts, which could be related to the absence of a postsynaptic inhibitory effect on REM-on neurons of the laterodorsal and pedunculopontine tegmental nuclei (LDT/PPT). 5-HT_2A and 5-HT_2C receptor knock-out mice show a significant increase of W and a reduction of slow wave sleep (SWS) which has been ascribed to the increase of catecholaminergic neurotransmission involving mainly the noradrenergic and dopaminergic systems. Sleep variables have been characterized, in addition, in 5-HT₇ receptor knock-out mice; the mutants spend less time in REMS that their wild-type counterparts. Direct infusion of the 5-HT_1A receptor agonists 8-OH-DPAT and flesinoxan into the DRN significantly enhances REMS in the rat. In contrast, microinjection of the 5-HT_1B (CP-94253), 5-HT_2A/2C (DOI), 5-HT₃ (m-chlorophenylbiguanide) and 5-HT₇ (LP-44) receptor agonists into the DRN induces a significant reduction of REMS. Systemic injection of full agonists at postsynaptic 5-HT_1A (8-OH-DPAT, flesinoxan), 5-HT_1B (CGS 12066B, CP-94235), 5-HT_2C (RO 60-0175), 5-HT_2A/2C (DOI, DOM), 5-HT₃ (m-chlorophenylbiguanide) and 5-HT₇ (LP-211) receptors increases W and reduces SWS and REMS. Of note, systemic administration of the 5-HT_2A/2C receptor antagonists ritanserin, ketanserin, ICI-170,809 or sertindole at the beginning of the light period has been shown to induce a significant increase of SWS and a reduction of REMS in the rat. Wakefulness was also diminished in most of these studies. Similar effects have been described following the injection of the selective 5-HT_2A receptor antagonists volinanserin and pruvanserin and of the 5-HT_2A receptor inverse agonist nelotanserin in rodents. In addition, the effects of these compounds have been studied on the sleep electroencephalogram of subjects with normal sleep. Their administration was followed by an increase of SWS and, in most instances, a reduction of REMS. The administration of ritanserin to poor sleepers, patients with chronic primary insomnia and psychiatric patients with a generalized anxiety disorder or a mood disorder caused a significant increase in SWS. The 5-HT_2A receptor inverse agonist APD-125 induced also an increase of SWS in patients with chronic primary insomnia. It is known that during the administration of benzodiazepine (BZD) hypnotics to patients with insomnia there is a further reduction of SWS and REMS, whereas both variables tend to remain decreased during the use of non-BZD derivatives (zolpidem, zopiclone, eszopiclone, zaleplon). Thus, the association of 5-HT_2A antagonists or 5-HT_2A inverse agonists with BZD and non-BZD hypnotics could be a valid alternative to normalize SWS in patients with primary or comorbid insomnia.

Introduction

Within the central nervous system serotonin (5-HT) participates in a great number of functions including sleep-wake behavior, cognition, affect, sexual function, thermoregulation and food intake. It has been established, in addition, that increases or decreases of 5-HT at central sites correlate with improved or worsened depression. In this respect, most of the drugs used to treat mood disorders including the selective serotonin reuptake inhibitors (SSRIs), the tricyclic antidepressants and the monoamine oxidase inhibitors increase synaptic levels of 5-HT. The SSRIs are also the first line drugs for the treatment of anxiety disorders such as generalized anxiety disorder, panic disorder and obsessive compulsive disorder¹.

Although many questions remain about the complicated role of 5-HT and its receptors in regulating sleep and wakefulness (W), recent studies have revealed much detailed information about this process. It should be mentioned that in the 1970s, based on lesion studies and neuropharmacological analysis, 5-HT was hypothesized to be responsible for the initiation and maintenance of slow wave sleep (SWS).² However, a series of findings from several laboratories seriously challenged the serotonergic hypothesis of sleep. Thus, the firing rate of 5-HT-containing dorsal raphe nucleus (DRN) neurons decreases during SWS relative to W³; reduction of brain 5-HT by p-chlorophenylalanine (PCPA) fails to disrupt sleep in humans, and in some studies, also in the rat4, 5; 5,7-dihydroxytryptamine, which produces a more selective and extensive depletion of brain 5-HT than the raphe lesions, does not significantly affect SWS⁶; and there is no significant difference between tryptophan-deficient and -replete rats in time spent in SWS.⁷ Concerning those studies showing that PCPA increases W in the rat,8, 9 it should be taken into consideration that this 5-HT synthesis inhibitor also induces increased responsiveness to noxious or neutral environmental stimuli and enhances motor activity, aggressivity, and sexual activity.10, 11, 12, 13 Therefore, insomnia after PCPA in the rat and the cat could be related to a general hyper-responsiveness to stimuli rather than to impairment of sleep regulation itself. Even in the studies in which insomnia followed PCPA administration, the effect was a transitory one, and SWS returned while 5-HT levels remained depleted.¹⁴ Moreover, in rats subjected to combined treatment with PCPA and sleep deprivation, delta activity reached the same high levels as after deprivation alone, which tends to indicate that SWS-regulating mechanisms are not disrupted in PCPA-treated rats.¹⁵ With regard to an alternative hypothesis,¹⁶ there is no firm evidence to support the proposal that 5-HT released during W might act as a neurohormone and induce the synthesis and/or release of hypnogenic factors secondarily responsible for SWS and REM (rapid eye movement) sleep (REMS) occurrence.

More recently it has been posed that 5-HT should not be considered either wake-promoting or SWS-promoting, and that the effect of 5-HT on sleep-wake behavior would depend upon the degree to which the serotonergic system is activated (systemic administration of low vs. high doses of the 5-HT precursor 5-hydroxytryptophan), and the time at which the activation occurs (light vs. dark period of the light-dark cycle).17, 18, 19 It has been hypothesized that the 5-hydroxytryptophan-related increase of SWS during the dark period depends upon the synthesis or release of as yet to be identified sleep-promoting factors.¹⁷ Alternatively, sleep occurrence might be associated with a 5-HT_1A receptor-mediated reduction in the activity of cholinergic waking-promoting neurons in the basal forebrain (BFB).20, 21

On the basis of results obtained from genetic, neurochemical, electrophysiological and neuropharmacological studies conducted over the past three decades it is proposed that 5-HT functions predominantly to promote W and to inhibit REMS. Notwithstanding this, under certain circumstances the indolamine seems to contribute to the increase in sleep propensity. The evidence that 5-HT is involved in the regulation of the behavioral state and its relationship to other sleep-wake regulatory neurotransmitters is the focus of this review.

The brain regions and neurotransmitter systems involved in the regulation of the behavioral state will be described next.

The brain regions involved in the promotion of the waking state are located in the brain stem, hypothalamus and BFB. The nuclei found in the brain stem include serotonin [5-HT: dorsal raphe nucleus (DRN), median raphe nucleus (MRN)]; norepinephrine [NE: locus coeruleus (LC)]; dopamine [DA: ventral tegmental area (VTA), substantia nigra pars compacta (SNc), ventral periaqueductal grey matter (VPGM)]; acetylcholine [ACh: laterodorsal and pedunculopontine tegmental nuclei (LDT/PPT); BFB]; and glutamate [GLU: medial pontine reticular formation (mPRF); BFB]-containing neurons. The structures located in the hypothalamus include histamine [HA: tuberomammillary nucleus (TM)]; and orexin [OX: posterior lateral hypothalamus (LH) around the fornix ]-containing cells. The cholinergic and glutamatergic neurons of the BFB involved in the regulation of the behavioral state are located predominantly in the medial septum and the diagonal band of Broca.22, *23

The neural structures that participate in the regulation of W give rise to mainly ascending projections. In this respect, a) 5-HT-, NE-, and HA-containing neurons send long ascending projections to the thalamus, cerebral cortex, and BFB; b) DA-containing cells project into the basal ganglia (caudate and putamen, external and internal globus pallidus, subthalamic nucleus) and the frontal cortex; c) cholinergic neurons from the midbrain tegmentum (LDT/PPT) project to the thalamus (ventromedial, intralaminar, and midline nuclei) and the BFB, and cholinergic BFB neurons have widespread rostral projections to the cerebral cortex and the hippocampus; d) orexin-containing neurons from the LH project to the entire forebrain and brain stem arousal systems; and e) glutamatergic neurons make up the projection neurons of the mPRF and thalamus.*23, *24 All these ascending projections into the forebrain follow a dorsal and a ventral route.²³ The dorsal route terminates in nonspecific thalamic nuclei, which in turn project to the cerebral cortex. The ventral route passes through the hypothalamus and continues into the BFB, where cells in turn project to the cerebral cortex and the hippocampus. In addition, a number of neural structures send descending projections to the spinal cord that modulate muscle tone.

Neurons of the preoptic area, anterior hypothalamus, and adjacent BFB constitute the sleep-inducing system.²⁵ Sleep active neurons of the preoptic area are mainly located in the ventrolateral preoptic area (VLPO). A majority of these neurons contain γ-aminobutyric acid (GABA) and galanin, and project to the BFB and to brain stem and hypothalamic areas involved in the promotion of W including the DRN, LC, LDT/PPT and LH neurons. During SWS, the neurons of the VLPO inhibit the W-producing structures. A similar role has been proposed for the melanin-concentrating hormone (MCH). Accordingly, MCH-containing neurons located in the zona incerta, perifornical nucleus and lateral hypothalamic area facilitate sleep occurrence by inhibiting 5-HT, NE, ACh and OX neurons involved in the promotion of W.²⁶ Of note, interleukin-1 (IL-1) increases SWS in several animal species, and the serotonergic system participates in the response. In this respect, the DRN contains IL-1 receptors; IL-1β inhibits the firing rates of DRN 5-HT neurons in a slice preparation; and microinjection of IL-1β into the DRN of freely behaving rats increases SWS.²⁷ It has been determined, in addition, that IL-1β inhibits DRN 5-HT cells by potentiating GABAergic inhibitory effects.²⁸

On the grounds of studies in which the cholinergic agonist carbachol was microinjected into rostral pontine structures of the cat, it has been proposed that the critical areas for REMS generation could be one or more of the following: a) the most ventral and rostral part of the pontine reticular nucleus; b) the perilocus coeruleus alpha nucleus (peri-LCα) of the mediodorsal pontine tegmentum; c) the rostral part of the rostral pontine tegmentum; d) the medial pontine reticular formation which encloses the rostral and caudal part of the pontine reticular nucleus and dorsally reaches the locus coeruleus alpha (LCα), or e) the ventral part of the pontine reticular nucleus.29, 30, 31 The reciprocal interaction hypothesis of REMS generation identifies cholinergic neurons of the LDT/PPT as promoting REMS and posits the inhibition of these neurons by, among others, serotonergic afferents from the DRN.³² The REMS induction region of the mPRF, one of the neuroanatomical structures proposed to be responsible for REMS generation, includes predominantly glutamatergic neurons, which are in turn activated by efferent connections of the LDT/PPT. The DRN provides the principal source of 5-HT innervation to the LDT/PPT and the REMS induction zone of the mPRF.³³ Accordingly, serotonergic terminals have been characterized that make synaptic contacts with the soma and the proximal dendrites of cholinergic tegmental neurons, and with non-cholinergic, presumptively glutamatergic, neurons of the REMS induction zone of the mPRF.³⁴ The DRN also sends ascending non-serotonergic projections to the LDT/PPT.35, 36