Using signals associated with safety in avoidance learning: computational model of sex differences

Simulation design

The lever press avoidance protocol used by Beck et al. (2011) consisted of an acquisition and extinction phase. The acquisition phase was divided into 240 trials, each including a 60-s warning period marked by a tone, that was followed by scrambled 0.5-s shocks at 1 mA delivered through the floor every 3.5 s for a maximum of 99 shocks. The warning stimulus remained on throughout the shock period. On a given trial, rats learned to press a lever to escape the shock once it was on, or avoid it altogether by pressing the lever during the warning period just prior to shock. Upon successful escape or avoidance, or after all 99 shocks were delivered, both the warning stimulus and the shock (if present) were terminated. This was followed by a safe inter-trial interval (ITI) that was, for half of the subjects, marked by the onset of a flashing light serving as an ITI signal. The extinction phase was identical to acquisition, except that the shock and ITI signal were never presented. A daily training session consisted of twenty trials, and began with a 60-s stimulus-free period in the testing chamber. Sessions took place every other day, 48 h apart, with animals removed to the home cage between sessions.

The procedure used to simulate this paradigm was similar to that of Myers et al. (2014). Stimuli were represented as binary inputs to both the actor and critic modules indicating if each stimulus was present (1) or absent (0) on a given timestep. The stimuli included warning signal, shock, ITI signal, testing context and home context. Context inputs represented the testing and home cage of an animal, respectively. There were 12 sessions in the acquisition phase and each began with a 6-timestep pre-stimulus period where only the testing context input was present (Fig. 2A). Each session was further divided into 20 trials. Every trial was divided into 54 timesteps, each representing 10 s of simulated time (Fig. 2B). The start of every trial was marked by the onset of the warning input, which continued for 6 timesteps. At that point the shock input also became active and could continue (along with the warning) for up to 30 timesteps. At each timestep, the actor could select a response from a set of possible actions that included lever press. Press responses during the warning period were scored as avoidance, while those during the shock interval were scored as escape. A lever press resulted in the immediate termination of both the warning signal and shock (if present) and transition to an 18-timestep ITI that, for some of the simulations, could be marked by the onset of the ITI signal. As a secondary measure of avoidance behavior, the latency to lever press was also recorded for each trial. If no press response occurred, the latency was scored as the maximum number of timesteps in the trial (i.e., 54). Finally, at the end of each session, an 18,000-timestep inter-session interval was simulated corresponding to the 48 h animals spent in the home cage prior to the next session. During this time, the only input presented to the model was the home context. All simulation figures report the mean of 10 simulations per group, and error bars represent standard error around the mean.

Actor module

At each timestep t, the actor selected a response r from a set of 100 possible actions. One of these actions was designated as the lever press while the remaining actions represented arbitrary behaviors, such as rearing or grooming, that an animal may engage in besides pressing the lever. The probability of selecting a response r at timestep t was defined as ${\displaystyle Pr\left(r\right)=\frac{f\left(\frac{M_r}{T}\right)}{\sum _{a}f\left(\frac{M_a}{T}\right)}}$ where f(x) = e^x for a = 1–100. The explore/exploit parameter T determined the tendency to repeat actions that were previously reinforced vs. explore the effect of new actions, and was set to 1.0 for all simulations. This value was chosen because it was found to best simulate SD rat behavior in prior modeling work (Myers et al., 2014). In contrast, lower values increase the tendency to repeat previously selected responses—a behavioral pattern associated with high behavioral inhibition and typical in the WKY rat strain (Myers et al., 2014). At each timestep t, the values M_a were defined as ${\displaystyle M_a=\sum _i{m}_{ai}{I}_i+c_{ai}P}$ where I_i was the current value of input i and m_ai was the strength of connection from input i to action a and c was a working memory trace that recorded prior actions in response to a particular configuration of inputs: c_ri = 1 for the action r executed at time t while c_ai = 0.95c_ai for all other actions a ≠ r. All values of m had an initial value of 0.01 while those of c started at 0 at the beginning of a simulation. Finally, the perseveration parameter P controlled the tendency to repeat a prior response and was set to 0.25 for all simulations.

Critic module

External reinforcement R was provided depending on the action r selected by the actor at timestep t. If there was shock at time t + 1 then R was set to R_shock, a large negative value (i.e., −4 or −8 depending on sex as described in the next section). Otherwise, R was set to 0 unless the selected action was lever press, in which case it was set to −0.2, a small negative value representing the energy cost associated with that response. Based on R, the critic module computed prediction error PE via ${\displaystyle PE=R+\gamma V-V^{\prime }}$ where γ was the discounting factor, V was the predicted future value of R, calculated as ${\displaystyle V=\sum _i{v}_i{I}_i}$ and where V′ was the value of V from the previous time t. The value of γ was held constant at 0.9 as in previous simulations of avoidance learning in SD rats using the same model (Myers et al., 2014). Selection of this value was motivated by prior research, which has estimated it to be closer to 1 based on how much dopamine neuron responses decrease with longer stimulus-reinforcement intervals (Suri & Schultz, 1999; Suri, 2002). Additionally, values closer to 1 allow more distant past reinforcement to influence future decisions, leading to learning that is based on both short- and long-term consequences of actions (Barto, 1995). In contrast, a value closer to 0 results in learning only based on short-term consequences, which carries several disadvantages, including reliance on immediate primary reinforcement, which may not be always available, and can preclude selecting actions that lead to greater future reinforcement (Barto, 1995). Finally, in calculating V, all values of v_i were set to 0 at the start of a simulation and were updated as ${\displaystyle \Delta v_i=\hspace{0.17em}\alpha \times PE\times I_i}$ where α was a learning rate that determined the rate of weight change in the critic, and was set to 0.01 or 0.05 depending on sex as described in the next section. Our prior simulations using this model showed that changes in α can affect the rate of extinction, largely independent of acquisition (Myers et al., 2014). The values of v_i could not exceed ±R_shock to prevent them from growing out of bounds.

The weights in the actor module m_ri for the chosen action r were also updated based on PE via ${\displaystyle \Delta m_{ri}=\epsilon \times \left(PE-m_{ri}\right)\times I_i}$ where ε was a learning rate that determined the rate of weight change in the actor, and was held constant at 0.005 as in previous simulations of avoidance learning in SD rats using the same model (Myers et al., 2014). The values of m could only be ≥0.

Simulating sex differences

The parameter values chosen for male simulations in the current study were identical to those in our previous simulations of male SD behavior (Myers et al., 2014). To simulate female sex, we varied the value of two parameters in the model—shock cost and the learning rate in the critic (α).

Shock cost was set to −8 to simulate females, compared to −4 to simulate males, where greater negative values represent more aversive shock. This manipulation is analogous to changes in shock intensity and was chosen because female rats appear to be more sensitive to shock than males (Pare, 1969) and these differences have also been associated with faster avoidance learning in females (Beatty & Beatty, 1970). Faster acquisition of avoidance behavior in females was also reported by Beck et al. (2011) and greater shock cost has been shown to speed up acquisition in prior simulations using this model (Myers et al., 2014).

In addition to shock cost, the learning rate in the critic module was set to 0.01 for female simulations, compared to 0.05 for males. This manipulation was chosen because male SD rats have greater D1 dopamine receptor density in the nucleus accumbens than female SD rats (Andersen et al., 1997), which could correspond to a higher learning rate in the model. While there are a number of sex differences in the release, uptake or sensitivity to dopamine in the ventral striatum (Becker, Perry & Westenbroek, 2012), we chose to focus on one to maintain simplicity, and because this difference also appears to have a direct correlate in the model. As discussed earlier, the prediction error signal computed by the critic module reproduces a number of temporal features of phasic dopamine neuron activity (Suri, 2002), and could correspond to dopaminergic inputs to the striatum (Collins & Frank, 2014; Moutoussis et al., 2008; Seymour et al., 2004). Also, since dopaminergic signals may serve a gating function (Montague, Hyman & Cohen, 2004), when the learning rate is higher, this could “open the gate”, allowing the current goal representation (or policy of the model) to be updated. In contrast, lower learning rates may preserve the current policy in the face of interference, consistent with the resistance to extinction observed in female SD rats (Beck et al., 2011).

Modeling methods

One set of simulations were trained following the avoidance protocol described in the general methods; half were “male” and half were “female,” based on shock cost and learning rates as defined earlier. Groups were further divided by ITI condition, where an ITI signal was either present or absent in acquisition. In either case, the signal was never present in extinction. A second set of simulations were trained following the same procedure as above; however, the ITI signal was now always present in extinction, whether or not it had occurred during acquisition. If the ITI signal only provides general information as an unexpected event, extinction should be faster when the ITI changes (added or deleted) from acquisition to extinction, compared to conditions where the ITI is always present or absent in both phases.

Results and discussion

A mixed-design ANOVA with within-subjects factor of session, and between-subjects factors of sex (male or female) and ITI signal condition (present or absent), was performed on the proportion of avoidance responses. The acquisition and extinction phases were analyzed separately. Separate analyses were also performed on simulations in which an ITI signal was either never presented (Fig. 4A), or always presented (Fig. 4B), in the extinction phase. For a complete summary of statistics for these simulations, refer to the Supplemental Information 2, Section 1. Here, and in all subsequent experiments, only the most relevant significant differences will be discussed in the main text.

Simulations in which an ITI signal was never presented during extinction (Fig. 4A) replicate three key aspects of the empirical data on SD avoidance (see Beck et al., 2011; Catuzzi & Beck, 2014). First, female simulations acquired the avoidance response faster than male simulations, and reached higher overall performance, regardless of condition. This was confirmed by a significant interaction between session and sex, F(3.91, 140.6) = 19.589, p < .001, ${\eta }_p^2=.352$. There were also main effects of session, F(3.91, 140.6) = 310.105, p < .001, ${\eta }_p^2=.896$, and sex, F(1, 36) = 161.711, p < .001, ${\eta }_p^2=.818$. Second, female simulations that acquired the avoidance response with an ITI signal extinguished faster when that cue was removed during extinction (Fig. 4A). However, omission of the ITI signal did not enhance the extinction rate for male simulations. This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 19.01, p < .001, ${\eta }_p^2=.346$. Third, male simulations extinguished faster than female simulations. This was not surprising considering males acquired avoidance to a lesser extent than females, and were therefore expected to extinguish faster. Main effects of session, F(11, 396) = 116.287, p < .001, ${\eta }_p^2=.764$, and sex, F(1, 36) = 213.264, p < .001, ${\eta }_p^2=.856$, were also found. Similar results were observed when the mean latency to press the lever was examined (Fig. 4C)—an alternative measure of avoidance behavior.

Unlike the empirical data, however, the ITI signal enhanced acquisition in the male, but not female simulations. This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 9.029, p = .005, ${\eta }_p^2=.201$. While there was a trend for the ITI signal to enhance acquisition in SD rats, regardless of sex (Beck et al., 2011), this also did not reach significance. The significant interaction in Fig. 4A could be due to a ceiling effect, precluding observing the effect of the ITI signal in female simulations. Faster acquisition of avoidance associated with an ITI signal is obtained in some studies (Bolles, Grossen & Bond, 1969) but not others (Katzev & Hendersen, 1971), also possibly due to ceiling effects (Galvani & Twitty, 1978). Finally, the discrepancy could also be due to greater variability in the empirical data than in the simulations or a limitation of using only two parameters to simulate sex differences.

Similar to the above, female simulations presented in Fig. 4B acquired the active avoidance response faster than male simulations, and reached higher overall performance. This was confirmed by a significant interaction between session and sex, F(4, 145.18) = 29.61, p < .001, ${\eta }_p^2=.451$. There were also main effects of session, F(4, 145.18) = 317.255, p < .001, ${\eta }_p^2=.898$, and sex, F(1, 36) = 228.857, p < .001, ${\eta }_p^2=.864$. However, these simulations failed to replicate the interaction between sex and ITI signal condition observed above, despite having an identical acquisition phase. This could be due to differences in the initial values of the weights assigned to each input by the critic and actor modules or the responses chosen by the actor early in training.

Unlike simulations where the ITI signal was never presented in the extinction phase (Fig. 4A), the interaction between sex and ITI signal condition failed to reach significance in simulations where the ITI signal was, instead, always present in extinction (Fig. 4B). Taken together, these results indicate that previous experience with the ITI signal is required for it to have an effect during extinction. Therefore, omission of the signal during extinction is not equivalent to presenting it for the first time during extinction. This result represents a novel prediction of the model that is inconsistent with an interpretation of the signal acting as a general indicator of change in the task, leading to generalization decrement and, therefore, faster extinction. Finally, as before, there was a significant interaction between session and sex, F(11, 396) = 8.286, p < .001, ${\eta }_p^2=.187$, indicating that male simulations extinguished faster than female simulations. There were also main effects of session, F(11, 396) = 157.948, p < .001, ${\eta }_p^2=.814$, and sex, F(1, 36) = 194.589, p < .001, ${\eta }_p^2=.844$.

One explanation of these results is that the ITI signal might acquire the properties of a conditioned reinforcer (Weisman & Litner, 1969; Dinsmoor & Sears, 1973; Fernando et al., 2014). Although the ITI signal was not contingent on avoidance in the current protocol, with training, avoidance comes to dominate the behavior. Therefore, on most trials, the ITI signal would follow avoidance, not shock, and could reinforce the avoidance response. This can account for why the ITI signal could, in some cases, enhance acquisition. If the ITI signal acquires reinforcing properties during acquisition, then its omission during extinction would be comparable to removal of reinforcement after avoidance responses, facilitating extinction as shown in Fig. 4A. In contrast, the ITI signal could not become a conditioned reinforcer if it was first presented in extinction (as in Fig. 4B).

An alternative explanation is that the ITI signal may become a positive occasion setter indicating that the warning signal would be followed by shock. The ITI signal could come to play this role through associations with the warning signal formed early in training, prior to high avoidance responding, when the warning signal was more likely to be followed by shock. This association could then be maintained through performance of the avoidance response if the absence of shock was attributed to that response. Subsequent omission of the ITI signal in the extinction phase would then indicate that the warning signal would not be followed by shock, facilitating extinction. This explanation predicts that the ITI signal should acquire properties similar to the warning signal.

To help test these two possible explanations, the mean values of the weights for each stimulus in the critic and actor modules, and how they change with every trial over the course of acquisition, were examined for the simulations presented in Fig. 4A. One set represents the value assigned to each stimulus by the critic component while the other represents the strength of association between each input with all possible actions in the actor component (refer back to Fig. 1). The weights in the actor have the most direct link to what action will be selected, and therefore to the overall behavior of the model. It is important to note that because the probability of selecting a particular action is calculated via an exponential function, even apparently small differences in the weights can have a large impact on behavior. On the other hand, positive and negative values of the weights in the critic module indicate how predictive of reward or punishment, respectively, a given input is Barto (1995). In contrast to the prediction of the occasion setting account, but consistent with the explanation based on conditioned reinforcement, the ITI signal was assigned a positive value by the critic during acquisition (Fig. 5A), suggesting it acquired the properties of a conditioned reinforcer.

In contrast to the positive values assigned to the ITI signal, the values of the inputs representing the warning signal and the testing context were negative in the critic module, first decreasing then reaching an asymptote equal to the value of the shock input, which is determined by the shock cost parameter (Fig. 5A). As described earlier, shock cost is one of the parameters we manipulated in order to simulate male or female-like behavior in the avoidance task. This parameter places a limit on how aversive the warning signal and testing context can become early in training due to their association with shock. As the model acquires the avoidance behavior, the values of the stimuli associated with the shock decrease back toward zero. That is because successful avoidance is never followed by shock, and these stimuli come to represent a less aversive state as avoidance is acquired.

Interestingly, female simulations trained with an ITI signal learned to assign a less negative value to the warning signal and a more negative value to the testing context relative to simulations trained without this signal (Fig. 5A). A similar pattern also occurred for male simulations trained with the ITI signal but the difference was smaller in magnitude. A different picture emerged in the weights to the lever press action in the actor module (Fig. 5B). These weights can only be positive with higher values indicating greater probability that the lever press action will be selected when that input is present. Unlike the weights in the critic, there is a smaller difference for weights in the actor between female simulations trained with and without the ITI signal (Fig. 5B). While the critic module of the female simulations trained with an ITI signal assigned a greater negative value to the testing context than to the warning signal, the actor did the opposite. That is, a lever press response was more likely to be chosen if presented with the warning signal instead of the testing context. Thus, the weights to the warning signal in the critic and actor during acquisition were dissociated. Similarly, the actor and critic also appeared to treat the ITI signal differently. While the ITI signal acquired a positive value in the critic that first increased but then decreased over the acquisition phase of training, the actor assigned little if any weight to this input. Thus, the ITI signal alone should not have the capacity to elicit a lever press response.

Nonetheless, the ITI signal is critical as it appears to modulate the weights assigned to the warning and testing context inputs by both the critic and the actor, and does so to a greater extent in the female simulations. This occurs at least in part because the learning rate parameter that determines the rate of change of the weights in the critic is, by definition, lower for female simulations. The most notable difference in the weights of the actor module is that females have a greater probability to select the lever press action than males, explaining why female simulations acquire avoidance faster than males. By the end of acquisition, male and female simulations appear to have similar weights in the actor irrespective of ITI condition.

However, moving into extinction when the ITI signal is omitted, the weights in the actor for female simulations trained with the signal in acquisition return to zero faster relative to female simulations trained without that signal (Fig. 6B). Because these weights were similar at the end of acquisition (Fig. 5B), this change can be attributed to the final values of the weights in the critic (those by the end of acquisition), and the changes they underwent in extinction. In particular, the value assigned to the warning input in the critic for female simulations trained with the ITI signal was both smaller at the end of acquisition (Fig. 5A) and was the fastest to return to zero in extinction (Fig. 6A). Since the values assigned by the critic are used to adjust the probabilities of actions, this could explain why female simulations trained with the ITI signal moved on to extinguish faster than female simulations trained without the signal, similar to the corresponding groups in Beck et al. (2011). While the ITI signal was also associated with changes in the weights for the testing context, the actor assigned less weight to that input than to the warning (Fig. 6B). Thus, changes in the actor weights for the testing context during extinction likely had less impact on behavior because their values were already near zero.

Modeling methods

Simulations were trained following the avoidance protocol described above and assigned to ITI signal present or absent conditions. However, instead of considering what we defined earlier as “male” and “female,” we examined the effect of changing only the shock value (−4 vs. −8), holding the learning rate in the critic constant at either 0.01 or 0.05. We also examined the effect of changing only the learning rate in the critic (0.01 or 0.05), with shock value held constant at −4 or −8. The ITI signal was never presented in extinction.

Results and discussion

For simulations where the learning rate in the critic module (α) was held constant (at either 0.01 or 0.05), separate mixed-design ANOVAs with a within-subjects factor of training session, and between-subjects factors of shock cost (−4 or −8) and ITI signal condition (present or absent), were performed on the proportion of avoidance responses. Similar analyses were performed on simulations where shock cost was held constant (at either −4 or −8), in which case α (0.01 or 0.05) was a between-subjects factor. In either case, the acquisition and extinction phase were also analyzed separately. For a complete summary of the statistics for these simulations, refer to the Supplemental Information 2, Section 2.

Simulations where sex differences were defined only by shock cost and α was held at 0.01 are presented in Fig. 7A. There was a significant interaction between session and shock cost, F(4.2, 152) = 28.434, p < .001, ${\eta }_p^2=.441$, confirming that simulations with higher shock cost acquired avoidance faster and to a greater extent than simulations with lower shock cost. The main effects of session, F(4.2, 152) = 347.208, p < .001, ${\eta }_p^2=.906$, and shock cost, F(1, 36) = 140.556, p < .001, ${\eta }_p^2=.796$, were also significant. Similar results were obtained for simulations where sex differences were defined by shock cost but α was held at 0.05 (Fig. 7B). Again, an interaction between session and shock cost, F(4.94, 177.67) = 22.434, p < .001, ${\eta }_p^2=.384$, and main effects of session, F(4.94, 177.67) = 279.428, p < .001, ${\eta }_p^2=.886$, and shock cost, F(1, 36) = 91.3, p < .001, ${\eta }_p^2=.717$, were found. Taken together, the results suggest shock cost had similar effects on acquisition irrespective of the value of α, with higher values leading to both faster acquisition and higher levels of avoidance.

Simulations where sex differences were defined only by the value of α, and shock cost was held at −4 are presented in Fig. 7C. There was a significant interaction between session and α, F(5.6, 200) = 2.207, p = .048, ${\eta }_p^2=.058$, suggesting that higher learning rates in the critic were associated with slower acquisition, in particular near the end of the acquisition phase. Simulations where shock cost was held at −8 (Fig. 7D), only showed a significant main effect of α, F(1, 36) = 14.711, p < .001, ${\eta }_p^2=.290$, but no interaction with session, indicating that lower learning rates were associated with higher levels of avoidance. Importantly, the size of the effect was smaller than that of the shock cost parameter. Taken together, these simulations suggest modulation of acquisition by the learning rate parameter was both smaller in magnitude and opposite in direction compared to that of the shock cost parameter.

For extinction, the results indicate that simulations where sex differences were defined by shock cost while α was held at 0.01 (Fig. 7A) lacked the interaction between shock cost and ITI signal. The signal, however, still enhanced extinction, confirmed by a significant main effect of ITI signal condition, F(1, 36) = 25.481, p < .001, ${\eta }_p^2=.414$. This effect was subsequently eliminated when α was held at 0.05 (Fig. 7B). Together, these results suggest that changes in shock cost alone, irrespective of the value of the learning rate in the critic module, are insufficient to simulate the empirical data, where omission of the ITI signal enhanced extinction in female, but not male, SD rats.

The ITI signal tended to enhance extinction for simulations where sex differences were defined only by the value of α, while shock cost was held at −4 (Fig. 7C). However, this was only at the lower value of α, confirmed by a significant interaction between α and ITI signal condition, F(1, 36) = 8.382, p = .006, ${\eta }_p^2=.189$. The main effect of ITI signal condition, F(1, 36) = 5.796, p = .021, ${\eta }_p^2=.139$, was also significant. While this suggests that manipulations of shock cost are not necessary to simulate this interaction, it is important to note that the corresponding acquisition phase did not provide a good fit to the empirical data, where female SD rats acquired avoidance faster than male SD rats (Beck et al., 2011; also see Catuzzi & Beck, 2014). The main effect of sex (as defined by α) on acquisition was not significant in these simulations (Fig. 7C), indicating that manipulations of this parameter alone are unable to simulate the observed sex differences.

Similar results were obtained in extinction for simulations where shock cost was held at −8 (Fig. 7D). However, the interaction between α and the ITI signal also depended on session, F(11, 396) = 2.322, p = .009, ${\eta }_p^2=.061$, suggesting that omission of the ITI signal could only facilitate extinction at the lower α level, in particular later in training (sessions 4–10). In contrast, at the higher α level, omission of the signal only appeared to enhance extinction early in training (sessions 2–3). These results once again suggest that manipulations of α alone are unable to simulate the observed sex differences. Although α did modulate acquisition in these simulations (Fig. 7D), those with the higher α level (corresponding to males) acquired avoidance much faster and to a greater extent (90% avoidance) than male SD rats, who only reached approximately 70% avoidance (Beck et al., 2011). Thus, these simulations also did not provide a good fit to the empirical data.

Overall, the results suggest changes in α level are only sufficient to account for sex differences in ITI signal effects on extinction. In contrast, manipulations of shock cost are only sufficient to simulate sex differences in acquisition rate. Therefore, changes in both parameters are required to simulate the pattern of avoidance learning observed in SD rats.

Modeling methods

Again, male and female simulations were trained with an ITI signal present or absent in acquisition. The ITI signal was never presented during extinction. All conditions received a context shift between acquisition and extinction.

Results and discussion

A mixed-design ANOVA with within-subjects factor of training session, and between-subjects factors of sex (male or female) and ITI signal condition (present or absent), was performed on the proportion of avoidance responses. The acquisition and extinction phases were analyzed separately. For a complete summary of the statistics, refer to the Supplemental Information 2, Section 3.

Results are shown in Fig. 8. As expected, acquisition (before the context shift) was similar to that observed in Experiment 1. Unlike the simulations presented in Fig. 4A, but similar to those in Fig. 4B, the interaction between sex and ITI signal condition was not significant. Instead, the ITI signal was associated with faster acquisition, confirmed by a significant main effect of ITI signal condition, F(1, 36) = 16.757, p < .001, ${\eta }_p^2=.318$, suggesting the ITI signal may have acquired conditioned reinforcing properties. As discussed earlier, there was a similar trend in SD rats that did not reach significance (Beck et al., 2011), possibly due to greater variability in the empirical data than in the simulations. Because the acquisition phase in this experiment was identical to that of Experiment 1, the discrepancies between the simulations could once again be due to differences in the initial values of the weights assigned to each input by the critic and actor modules or the responses chosen by the actor early in training. They may also reflect a limitation of using only two parameters to simulate sex differences in the model. Similar to Experiment 1, female simulations acquired avoidance faster than male simulations, indicated by a significant main effect of sex, F(1, 36) = 185.07, p < .001, ${\eta }_p^2=.837$.

Finally, there was a visible drop in avoidance for all groups as a result of the context shift between the end of acquisition and the start of extinction. Importantly, despite the context shift, omission of the ITI signal still facilitated extinction in female (but not male) simulations that were trained with that signal during acquisition. This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 11.536, p = .002, ${\eta }_p^2=.243$. Significant main effects of session, F(11, 396) = 56.898, p < .001, ${\eta }_p^2=.612$, and sex, F(1, 36) = 196.248, p < .001, ${\eta }_p^2=.845$, were also found. These results suggest that omission of the ITI signal does not speed up extinction simply due to associations with the testing context and its impact on response selection by the actor module. Similar results were found when the mean latency to press the lever was examined (Fig. 8B).

Modeling methods

Simulations (male and female, ITI signal present or absent) were trained with an ITI length of 30 s (reduced from 180 s). All other model parameters and the training procedure were the same as described in the general method. As in the previous simulations, as well as in the experimental protocol of Beck et al. (2011), the duration of the ITI was equal to that of the ITI signal.

Results and discussion

A mixed-design ANOVA identical to that of Experiment 4 was conducted on the proportion of avoidance responses. The acquisition and extinction phases were again analyzed separately. For a complete summary of the statistics, refer to the Supplemental Information 2, Section 4.

Interestingly, when ITI duration was shortened, the effect of the ITI signal was reversed (relative to Experiment 1, Fig. 4A) for both acquisition and extinction. The signal now tended to slow (rather than enhance) acquisition. This was confirmed by a significant main effect of ITI signal condition, F(1, 36) = 12.108, p = .001, ${\eta }_p^2=.252$. As before, significant main effects of session, F(4.39, 158.15) = 248.148, p < .001, ${\eta }_p^2=.873$, and sex, F(1, 36) = 123.247, p < .001, ${\eta }_p^2=.774$, were also found. Omission of the ITI signal now slowed extinction in female, but not male, simulations trained with the signal during acquisition (Fig. 9A). This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 12.723, p < .001, ${\eta }_p^2=.261$. There were also significant main effects of sex, F(1, 36) = 512.523, p < .001, ${\eta }_p^2=.934$, and ITI signal, F(1, 36) = 11.705, p = .002, ${\eta }_p^2=.245$. Similar results were found when the mean latency to press the lever was examined (Fig. 9B).

These results suggest that the removal of the ITI signal does not always speed up acquisition or extinction, and that its effect depends on features of the task related to the ITI. In contrast to simulations where omission of the ITI signal enhanced acquisition, in which the ITI signal was assigned positive values by the critic module (Fig. 5A), here the signal was assigned negative values (Fig. 9C). Therefore, the ITI signal may have acquired aversive properties allowing it to serve as a negative reinforcer of the avoidance behavior. Its subsequent omission may have reinforced avoidance, and led to resistance to extinction. As discussed earlier, the results of the previous simulations could be interpreted in terms of a positive occasion setting account, but there were difficulties with that explanation given that the ITI signal acquired a positive value in the critic. The current simulations pose further difficulties for that explanation. While the ITI signal was assigned a negative value (Fig. 9C) similar to the warning (Fig. 9D) as would be predicted if it was a positive occasion setter, the signal’s removal instead slowed extinction, inconsistent with that account.

Alternatively, it could be assumed that the ITI signal became a negative occasion setter given that once avoidance responding is high, presentation of the ITI signal will be followed by the warning, absent of shock. Previous studies have suggested that the avoidance response itself comes to play a similar role (De Houwer, Crombez & Baeyens, 2005), but see (Declercq & De Houwer, 2011). Omission of the negative occasion setter could then increase avoidance, as observed here. However, if the ITI signal had become a negative occasion setter, it should have been assigned a positive value in the critic because it would predict the absence of an aversive state. Instead it acquired a negative value, inconsistent with this account. Once again, an explanation based on conditioned reinforcement appears more likely.

The shorter ITI duration also reduced the opportunity for ITRs (Fig. 10A). ITRs may be one reason why omission of the signal modulates extinction as they can change the values of the weights acquired by both the critic and actor components of the model. It is important to note that when the length of the ITI was 180 s (Fig. 10B), the ITRs decreased across each minute of the ITI, a pattern also observed in the empirical data (Beck et al., 2011). This was confirmed by a significant main effect of ITI segment, F(2, 72) = 422.958, p < .001, ${\eta }_p^2=.922$. The model predicted that a similar pattern should be observed with shorter ITI duration (Fig. 10A), also confirmed by a significant main effect of ITI segment, F(2, 72) = 44.404, p < .001, ${\eta }_p^2=.552$, though the effect was notably smaller. Additionally, female simulations made more ITRs than male simulations—a trend also present in rats (Beck et al., 2011). The model predicted that this should be independent of ITI duration, confirmed by significant main effects of sex for simulations with an ITI duration of 30 s, F(1, 36) = 465.231, p < .001, ${\eta }_p^2=.928$, and 180 s, F(1, 36) = 213.928, p < .001, ${\eta }_p^2=.856$. Finally, when ITI duration was shorter (Fig. 10A), the ITI signal tended to reduce ITRs, confirmed by a significant main effect of ITI signal condition, F(1, 36) = 15.634, p < .001, ${\eta }_p^2=.303$. In contrast, when ITI duration was longer, the ITI signal reduced ITRs in female (but not male) simulations (Fig. 10B), suggested by a significant interaction between sex and ITI signal condition, F(1, 36) = 12.294, p = .001, ${\eta }_p^2=.255$. This was confirmed via Bonferroni corrected independent samples t-tests performed on ITRs averaged across the ITI, which only found a significant difference between female simulations trained with and without the signal, t(18) = − 3.044, p = .007, r = 0.58.

Modeling methods

Simulations were conducted as before (male and female, ITI signal present or absent); within each condition, half of the simulations received an ITI of 180 s and half received an ITI of 30 s. During the ITI in both acquisition and extinction, lever press was “disabled,” meaning that the actor module could not execute that response. This completely eliminated ITRs, since no lever press responses could be executed during the ITI. All other parameters and the training procedure were the same as described in the general method.

Results and discussion

A mixed-design ANOVA with within-subjects factor of session, and between-subjects factors of sex (male or female), ITI signal condition (present or absent), and ITI duration (30 or 180 s) was performed on the proportion of avoidance responses. The acquisition and extinction phases were analyzed separately. The interaction between session, sex, ITI signal, and ITI duration was significant in analyses of both acquisition, F(5.13, 369.66) = 5.518, p < .001, ${\eta }_p^2=.071$, and extinction, F(8.09, 582.75) = 2.998, p = .003, ${\eta }_p^2=.040$. To understand these interactions, simulations with different ITI durations were analyzed separately. For a complete summary of statistics, refer to the Supplemental Information 2, Section 5.

Although ITRs were completely eliminated in this experiment (data not shown), omission of the ITI signal still facilitated extinction in the female simulations with a 180-sec ITI, although this did not occur until later in the extinction phase (Fig. 11A) relative to simulations in which the lever press response was available (Fig. 4A). This was confirmed by a significant interaction between session, sex and ITI signal, F(11, 396) = 1.885, p = .040, ${\eta }_p^2=.050$. The main effect of sex was also significant, F(1, 36) = 505.715, p < .001, ${\eta }_p^2=.934$. In Experiment 1, Fig. 4A, when the response was available during the ITI, facilitation was evident by session 3, while in the current simulations it did not appear until session 5, and was most evident starting at session 6 of extinction. This suggests that while the ITRs themselves may not be critical for the effect of the ITI signal, they still modulate it. Similar results were found when the mean latency to press the lever was examined (Fig. 11C).

The simulations also predict that eliminating the ITRs should alter how the ITI signal is used when ITI duration is short, with similar results for both proportion avoidance responses (Fig. 11B) and latency to press the lever (Fig. 11D). While there was a trend for the effect of the ITI signal to reverse, similar to Experiment 4 where ITI duration was also 30 s but ITRs were allowed (Fig. 9A), in the current simulations, neither the main effect of the ITI signal, nor the interaction between sex and the ITI signal, reached significance, (p = .052 and .087, respectively). These trends suggest that omission of the ITI signal may slow extinction in female simulations when the ITI is shorter, but not until later in training, similar to when the length of the ITI is longer. As in Experiment 4, the ITI signal was assigned a positive value in the critic when the ITI was longer and its omission led to faster extinction (Fig. 12A), and was assigned a negative value when the ITI was shorter and it tended to slow extinction (Fig. 12B). Similar to before, the warning also acquired a less negative value (Fig. 12C) or a more negative value (Fig. 12D) in simulations trained with the ITI signal when the ITI was longer or shorter, respectively.

As in some of the previous experiments, the ITI signal facilitated acquisition in male simulations trained with the longer ITI (Fig. 11A), but impaired learning when ITI length was shorter (Fig. 11B). In either case, it had minimal impact on acquisition in female simulations. This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 7.768, p = .008, ${\eta }_p^2=.266$ and F(1, 36) = 10.559, p = .003, ${\eta }_p^2=.227$, for simulations with an ITI length of 180 and 30 s, respectively. Note that all possible main effects and all other possible interactions were also significant. Again, this could be explained if the ITI signal was capable of reinforcing or punishing avoidance responding, respectively, consistent with the positive (Fig. 12A) and negative values (Fig. 12B) it was assigned in the critic module. It had little impact on female simulations likely because its effects were masked by the high shock value, which led to very high avoidance rates. Finally, in this experiment, ITI duration was still equal to that of the ITI signal. Thus, the duration of the ITI signal alone, as well as its timing within the ITI, could also be important. The next experiment examined this possibility.

Modeling methods

Simulations were conducted with male and female parameters. As before, the ITI signal could be present or absent in acquisition, while it was never presented in extinction. For all simulations, the length of the ITI was held constant at 180 s while that of the ITI signal was reduced to 30 s. However, for half of the simulations the ITI signal appeared at the beginning of the ITI, and for half it appeared at the end, as illustrated in Fig. 13. All other aspects of the training procedure were identical to those in the general method.

Results and discussion

A mixed-design ANOVA with within-subjects factor of session, and between-subjects factors of sex (male or female), ITI signal condition (present or absent), and ITI signal timing (first or last 30 s of the ITI) was performed on the proportion of avoidance responses. The acquisition and extinction phases were analyzed separately. Female simulations acquired avoidance faster and to a greater extent than male simulations, confirmed by a significant interaction between session and sex, F(4.4, 313.8) = 41.342, p < .001, ${\eta }_p^2=.365$, and significant main effect of sex, F(1, 72) = 314.747, p = .001, ${\eta }_p^2=.814$. The interaction between sex and ITI timing was also significant, F(1, 72) = 4.676, p = .034, ${\eta }_p^2=.061$. In the analysis of extinction, there was a significant interaction between sex, ITI signal condition, and ITI signal timing, F(1, 72) = 8.056, p = .006, ${\eta }_p^2=.101$. To understand these interactions, simulations with different ITI signal timing were analyzed separately. Finally, because the interaction between sex, ITI signal condition and ITI signal timing only reached significance in extinction in the analysis of the latency to press the lever, F(1, 72) = 6.28, p = .014, ${\eta }_p^2=.080$, only that phase was examined further. For a complete summary of statistics, refer to the Supplemental Information 2, Section 6.

The results suggest that the timing of the ITI signal is critical for how its omission will affect extinction. Omission of the ITI signal when it was presented in the first 30 s of the ITI facilitated extinction for female, but not male, simulations (Fig. 14A). This was confirmed by a significant interaction between sex and ITI signal condition, F(1, 36) = 12.802, p < .001, ${\eta }_p^2=.262$. In contrast, omission of the ITI signal had no effect on extinction if it was presented in the last 30 s of the ITI (Fig. 14B). These results argue against the occasion setting account. It should have been easier for the ITI signal to become an occasion setter when presented in the last 30 s of the ITI where it would be immediately followed by the warning signal. Under these conditions, the ITI signal’s subsequent omission should have enhanced extinction to a greater extent, or at least as well as, when there was a gap between presentation of the ITI signal and the warning. Instead, omission of the ITI signal only affected extinction when it was presented in the first 30 s of the ITI. It is important to note that prior research has shown that the avoidance response itself could serve as a negative occasion setter (De Houwer, Crombez & Baeyens, 2005). While the role of the avoidance response was not considered here, occasion setting could still be important for avoidance learning. More recently, however, it has been argued that prior studies have not provided unique support for this occasion setting account (Declercq & De Houwer, 2011). The results of the current simulations also argue against an occasion setting account of avoidance learning, a prediction that could be tested by future empirical work.

As before, these results can be explained if it is assumed that the ITI signal became a positive reinforcer of the avoidance behavior (Weisman & Litner, 1969; Dinsmoor & Sears, 1973; Fernando et al., 2014). Consistent with this, the value of the ITI signal was positive in the critic module when it was presented in the first 30 s of the ITI (Fig. 15A). This was also when its omission enhanced extinction. In contrast, this value was negative when it was presented in the last 30 s of the ITI (Fig. 15B), when omission had no effect on extinction. As discussed earlier, the values assigned to inputs in the critic module can be thought of as the valence of those inputs. Thus, positive values could indicate that the ITI signal was able to reinforce avoidance responding.

Though the difference was small, when the ITI signal acquired a positive value, this was greater in female compared to male simulations (Fig. 15A). At the same time, the warning signal acquired a less negative value in the simulations trained with the ITI signal (relative to those trained without it), and this difference was larger in the female simulations (Fig. 15C). This suggests that the ITI signal may have been more reinforcing in the female simulations and was, therefore, also more likely to decrease avoidance behavior when omitted in the extinction phase, accounting for the observed sex difference in its effect on extinction. Consistent with this interpretation, the signal was also assigned a more positive value by the critic in the female simulations from Experiment 1. Additionally, the sex difference in the value was larger in that experiment (Fig. 5A), which was also coupled with a larger effect on avoidance during extinction (Fig. 4A) than that observed here.

In either case, when the ITI signal acquired positive values, the warning signal acquired less negative values. This could be because the warning had also become a predictor of what was now, in essence, an appetitive event (presentation of the ITI signal). This could have occurred since avoidance responding is high late in acquisition. Therefore, the warning signal was reliably followed by the ITI signal instead of by shock. In contrast, the negative values assigned to the ITI signal when it was presented in the last 30 s of the ITI (Fig. 15B) suggest that it had acquired aversive properties, similar to the warning signal (and due to its close association with that signal). However, in this case the ITI signal could not modulate avoidance as it did not follow the avoidance response. Likewise, the warning signal acquired similar values in female simulations irrespective of ITI signal condition (Fig. 15D). Finally, it is important to note that the current simulations do not directly test whether the ITI signal acquired the properties of a conditioned reinforcer, which could be confirmed by future work as suggested further below.