Search for heavy Majorana or Dirac neutrinos and right-handed W gauge bosons in final states with charged leptons and jets in pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector

Abstract

A search for heavy right-handed Majorana or Dirac neutrinos $N_{\textrm{R}}$ and heavy right-handed gauge bosons $W_{\textrm{R}}$ is performed in events with energetic electrons or muons, with the same or opposite electric charge, and energetic jets. The search is carried out separately for topologies of clearly separated final-state products (“resolved” channel) and topologies with boosted final states with hadronic and/or leptonic products partially overlapping and reconstructed as a large-radius jet (“boosted” channel). The events are selected from pp collision data at the LHC with an integrated luminosity of 139 fb$^{-1}$ collected by the ATLAS detector at $\sqrt{s} = 13$ TeV. No significant deviations from the Standard Model predictions are observed. The results are interpreted within the theoretical framework of a left-right symmetric model, and lower limits are set on masses in the heavy right-handed $W_{\textrm{R}}$ boson and $N_{\textrm{R}}$ plane. The excluded region extends to about $m(W_{\textrm{R}}) = 6.4$ TeV for both Majorana and Dirac $N_{\textrm{R}}$ neutrinos at $m(N_{\textrm{R}})<1$ TeV. $N_{\textrm{R}}$ with masses of less than 3.5 (3.6) TeV are excluded in the electron (muon) channel at $m(W_{\textrm{R}})=4.8$ TeV for the Majorana neutrinos, and limits of $m(N_{\textrm{R}})$ up to 3.6 TeV for $m(W_{\textrm{R}}) = 5.2$ (5.0) TeV in the electron (muon) channel are set for the Dirac neutrinos. These constitute the most stringent exclusion limits to date for the model considered.

1 Introduction

The very small mass of neutrinos is one of the biggest puzzles in the Standard Model (SM) of particle physics. The seesaw mechanism [1,2,3] is a proposed solution, in which the light neutrinos acquire their Majorana masses through heavy right-handed neutrinos. From the effective field theory point of view, this is equivalent to dimension-5 operators [4] through electroweak symmetry breaking. Several types of the seesaw mechanism are proposed; for example, Type-I with right-handed neutrinos [1,2,3], Type-II with a scalar triplet [5,6,7] and Type-III with at least two fermion triplets [5, 8] scenarios. Type-I and Type-II models can further be embedded into the Left-Right Symmetric Model (LRSM) [9,10,11]. The LRSM attempts to explain the broken parity symmetry of the weak interaction in the SM and can introduce, depending on the form of the model, right-handed counterparts to the W and Z bosons ($W_{\textrm{R}}$ and $Z_R),$ and right-handed heavy neutrinos $N_{\textrm{R}}$ as the parity gauge partners of the corresponding left-handed neutrino fields.

In this analysis, a search for $W_{\textrm{R}}$ bosons decaying to $N_{\textrm{R}}$ and a charged lepton $\ell ^\pm$ in proton–proton (pp) collisions at a centre-of-mass energy $\sqrt{s}=13~\text {TeV}$ with the ATLAS detector is presented, where $\ell ^\pm$ denotes an electron/positron $(e^{\pm })$ or a muon $(\mu ^{\pm }).$ The exact process of interest is the Keung–Senjanović (KS) process [11]. In the case where the mass of $W_{\textrm{R}}$ is larger than the $N_{\textrm{R}}$ mass, $m(W_{\textrm{R}})> m(N_{\textrm{R}}),$ $N_{\textrm{R}}$ decays into a charged lepton and an off-shell $W_{\textrm{R}}$ $(W_{\textrm{R}}^{*}).$ In the case of $m(W_{\textrm{R}})< m(N_{\textrm{R}}),$ a $W_{\textrm{R}}$ produced off-shell decays into a pair of $N_{\textrm{R}}$ and $\ell ,$ and the $N_{\textrm{R}}$ subsequently decays via the $W_{\textrm{R}}$ resonant state. The charge of the final state leptons is a key feature of the analysis, as it determines the Dirac or Majorana nature of $N_{\textrm{R}}$. The focus of the search is on hadronic decays of the final-state $W_{\textrm{R}}^{(*)}$ because of their high branching fractions. The leading-order Feynman diagrams for the KS process targeted by this analysis are shown in Fig. 1. Depending on the target mass range, two types of analyses are performed; one requiring that the two quarks in the final state are clearly separated geometrically and are reconstructed as two separate jets (hereafter labelled as the “resolved” channel), with the second one targeting the $m(W_{\textrm{R}})\gg m(N_{\textrm{R}})$ region where particles from the $N_{\textrm{R}}$ decay are merged due to the Lorentz boost, and are reconstructed as a single large-radius (large-R) jet (the “boosted” channel) [12, 13].

Fig. 1

The Keung–Senjanović process for the m($W_{\textrm{R}}$) > m($N_{\textrm{R}}$) (left) and the m($N_{\textrm{R}}$) > m($W_{\textrm{R}}$) (right) cases. The asterisk $(^*)$ on the $W_{\textrm{R}}$ denotes an off-shell particle. In the case of Majorana $N_{\textrm{R}}$, the final state appears with two same-sign charged leptons 50% of the time, violating the lepton number conservation

Multiple experimental results can give constraints on the target signal of this analysis [14, 15]. The LRSM can enhance some low-energy processes, such as K-${\overline{K}}$ and $B_{d,s}$-$\overline{{B}_{d,s}}$ oscillations, via $W_{\textrm{R}}$ contributions in the box diagram. The lower limit on $m(W_{\textrm{R}})$ from the latest mixing results is about 3 $\text {TeV}$ [16, 17], assuming equal mixing matrices for left- and right-handed quarks. A Majorana right-handed electron neutrino and $W_{\textrm{R}}$ boson can contribute to the neutrinoless double beta decay $(0\nu \beta \beta )$ diagram. Assuming purely right-handed contributions to the $0\nu \beta \beta$ decay, the non-observation of this lepton number violating process can be used to set limits on the $W_{\textrm{R}}$ and $N_{\textrm{R}}$ masses in the LRSM. As an example, at $m(W_{\textrm{R}})= 3~\text {TeV}$ (5 TeV), the upper limit on the $N_{\textrm{R}}$ mass is about 180 $\text {GeV}$ (20 $\text {GeV}$) [18, 19].

The pp collisions at the Large Hadron Collider (LHC) offer a window into a unique phase space for this search, allowing the exploration of $m(N_{\textrm{R}})$ in a range from O(100) $\text {GeV}$ to a few TeV. Searches by the ATLAS [20,21,22] and CMS [23,24,25,26,27] collaborations have excluded signals in the LRSM with $m(W_{\textrm{R}})$ up to about 4.7 $\text {TeV}$ and 5 $\text {TeV}$ in the electron and muon final states, respectively, for a $m(N_{\textrm{R}})$ range from 100 $\text {GeV}$ to 3 $\text {TeV}$.

2 Signal model

The theoretical framework of LRSM offers the prediction of a right-handed charged current and a mass-generating mechanism for light and heavy neutrinos. The small left-handed neutrino masses are naturally explained via right-handed neutrinos in the Type-I seesaw mechanism, or via SU(2)-triplet scalars in the Type-II seesaw model. Both Type-I and Type-II contributions can coexist in the LRSM.

In the minimal LRSM, the left-handed (i.e. SM-like) neutrinos as well as the right-handed neutrinos are predicted to be Majorana particles. The model thus features the violation of the global lepton number symmetry. In the target KS process, same- and opposite-sign lepton pairs would be equally observed in a 50–50% admixture of signal events. In the LRSM variants that include the inverse seesaw mechanism [28,29,30,31], the $N_{\textrm{R}}$ neutrinos are pseudo-Dirac particles formed by two Majorana particles with identical masses [32]. For simple versions of LRSMs incorporating the inverse seesaw mechanism, lepton-number-violating processes are not expected [33]. In this paper we explore both “Dirac” and “Majorana” interpretations of the LRSM.

3 ATLAS detector

The ATLAS experiment [34] at the LHC is a multipurpose particle detector with a forward–backward symmetric cylindrical geometry and a near $4\pi$ coverage in solid angle.¹ It consists of an inner tracking detector surrounded by a thin superconducting solenoid providing a 2T axial magnetic field, electromagnetic and hadron calorimeters, and a muon spectrometer (MS). The inner tracking detector covers the pseudorapidity range $|\eta | < 2.5$. It consists of silicon pixel, silicon microstrip, and transition radiation tracking detectors. Lead/liquid-argon (LAr) sampling calorimeters provide electromagnetic (EM) energy measurements with high granularity. A steel/scintillator-tile hadron calorimeter covers the central pseudorapidity range $(|\eta | < 1.7).$ The endcap and forward regions are instrumented with LAr calorimeters for both the EM and hadronic energy measurements up to $|\eta | = 4.9$. The MS surrounds the calorimeters and is based on three large superconducting air-core toroidal magnets with eight coils each. The field integral of the toroids ranges between 2.0 and 6.0 T m across most of the detector. The MS includes a system of precision tracking chambers and fast detectors for triggering. A two-level trigger system is used to select events. The first-level trigger (L1) is implemented in hardware and uses a subset of the detector information to accept events at a rate below 100 kHz. This is followed by a software-based high-level trigger (HLT) that reduces the accepted event rate to 1 kHz on average depending on the data-taking conditions. An extensive software suite [35] is used in the reconstruction and analysis of real and simulated data, in detector operations, and in the trigger and data acquisition systems of the experiment.

Table 1 Summary of the simulated background events used in this analysis, including name of the event generator, accuracy in QCD when the matrix element is calculated, parton shower algorithm, underlying event (UE) tune, and accuracy in QCD when the cross section is calculated. NLO, NNLO and NNLL denote the next-to-leading order, next-to-next-to-leading order, and next-to-next-to-leading logarithm, respectively, and $n_{\text {parton}}$ is the number of partons in the matrix-element calculation. V denotes a W or Z boson

4 Dataset and simulated event samples

The data used in this analysis were collected with the ATLAS detector between 2015 and 2018, and correspond to an integrated luminosity of 139 fb$^{-1}$. The average number of pp interactions per bunch crossing (“pile-up”) in the dataset is 33.7. Only high-quality data, collected when the LHC has declared the beams to be stable and all of the ATLAS subdetectors are reported to be operating well, are analysed [36, 37]. All events are required to have a vertex with at least two associated ID tracks with $p_{\text {T}} >500~\text {MeV}$ [38, 39]. The one with the highest $\sum p_{\text {T}} ^{2}$ of the associated tracks is selected as the primary vertex.

The dataset was collected by single or dilepton triggers with a variety of transverse momentum ($p_{\text {T}}$) and isolation requirements, which depend on the data-taking period. In both resolved and boosted channels, only events in the region of trigger efficiency plateau [40, 41] are used. Trigger thresholds are no higher than (24 GeV, 24 GeV) for dielectrons and (22 GeV, 8 GeV) for dimuons in the resolved channel, and 140 GeV for electrons and 50 GeV for muons in the boosted channel. The trigger efficiency for most signal points for both resolved and boosted channels ranges from 90 to 95%, dropping to 85% in the $m(W_{\textrm{R}}) \sim m(N_{\textrm{R}})$ region where the compressed phase space affects the charged lepton kinematics. For the resolved analysis, the $e\mu$ trigger is used for the $t$ $\bar{t}$ background estimation. In addition, for the study of mis-identified leptons in the resolved analysis (Sect. 6) some non-isolated, heavily-prescaled single lepton triggers were also used.

Monte-Carlo (MC) simulated events are used to optimise the event selections, and to estimate the background contribution and the systematic uncertainties. For all MC events, the response of the ATLAS detector is simulated using the Geant4 toolkit [42]. The same reconstruction and trigger algorithms are applied for data and simulated events, using the default ATLAS software [43]. Multiple overlaid pp collisions are simulated with the soft QCD processes of Pythia 8.230 [44] using the A3 set of tuned parameters [45] and the NNPDF2.3LO parton distribution function (PDF) set [46]. MC events are reweighted so that the distribution of the average number of interactions per bunch crossing agrees with the data (“pile-up reweighting”).

Table 1 summarises the simulation packages used for the SM background processes. More details are found in other recent ATLAS publications, for example Refs. [47,48,49,50]. The multijet and $\gamma$ + jets samples are used only in the boosted channel.

The signal events for $pp \rightarrow W_{\textrm{R}}^{(*)} \rightarrow \ell \ell qq'$ in the LRSM are generated at leading-order (LO) using FeynRules [51] implemented in MadGraph5_aMC@NLO [52] and further modified as described in Ref. [53], where $\ell$ includes the tau lepton. The generated events are interfaced with Pythia 8.230 [44] for parton showering and hadronisation. The A14 parameter set is used for tuning the shower [54]. The NNPDF3.1NLO [55] PDF set enters in the matrix element calculation and the NNPDF2.3LO is used in the parton shower.

Lepton flavour mixing, albeit possible in principle in the LRSM, is not considered in this analysis. The branching fractions for the electron, muon and tau channels for $W_{\textrm{R}}$ decays to $N_{\textrm{R}}$ of equal masses are assumed to be exactly one-third for each flavour due to lepton universality. For sufficiently heavy $W_{\textrm{R}}$ masses, $m(W_{\textrm{R}})> m_{t},$ the hadronic final state also includes the $W_{\textrm{R}}\ \rightarrow tb$ channel. However, b-jets are vetoed in this search to reduce the top background contamination, effectively limiting the phase space to hadronic $W_{\textrm{R}}$ decays excluding b-quarks. At very low $N_{\textrm{R}}$ masses of less than 50 $\text {GeV}$, the decay length of $N_{\textrm{R}}$ is greater than 1 mm [53]. This mass range can be explored by dedicated ATLAS and CMS analyses requiring a displaced vertex, and is beyond the scope of this paper. The $m(W_{\textrm{R}})$ and $m(N_{\textrm{R}})$ are sampled from 1 $\text {TeV}$ to 7 $\text {TeV}$ and from 50 $\text {GeV}$ to 4 $\text {TeV}$, respectively, at intervals of about 500 $\text {GeV}$. In the $m(W_{\textrm{R}})> 10 \times m(N_{\textrm{R}})$ region, which is the focus of the boosted channel, the $N_{\textrm{R}}$ mass is further sampled in steps of 100 GeV. The simulation assumes a Majorana $N_{\textrm{R}}$, giving a 50% mixture of same-sign and opposite-sign lepton pairs. For the Dirac neutrino case, only the opposite-sign events are used in the analysis as no other differences are expected, and the production cross section is adjusted appropriately.

5 Object reconstruction

The definitions of the physics objects used in this analysis are summarised in Table 2. Physics objects in the events (electrons, muons, jets, and missing transverse momentum) are separately defined for the resolved and boosted analyses. The orthogonality between the two channels is not enforced: the same event can be used in both analyses, which are not statistically combined.

Table 2 Definitions of the electrons, muons, small-R and large-R jets, used in this analysis. Optimisations of the object selections are performed separately for the resolved and boosted analyses. See text about the definitions of ‘Baseline’ and ‘Leading’ leptons as well as leptons for ‘Fake estimation’

Electrons are reconstructed as ID tracks, matched to energy clusters in the EM calorimeter within $\left| \eta \right| < 2.47$ [72]. The electron candidates in the crack region of the EM calorimeter $(1.37< \left| \eta \right| < 1.52)$ are discarded. The electron identification utilises a multivariate likelihood-based discriminant that exploits the shower shapes in the EM calorimeter and the associated track properties. There are ‘Loose’, ‘Medium’ and ‘Tight’ identification working points described in Ref. [72]. To further suppress the mis-identified electrons, an isolation criterion can be applied to the electron candidates. Several isolation working points are defined in Ref. [72] e.g. ‘Loose’ and ‘HighPtCaloOnly’. Slightly different $p_{\text {T}}$ thresholds are used across the resolved and boosted channels to ensure the operation at a region of constant trigger efficiency. For the resolved analysis, electrons are required to have $p_{\text {T}} >25~\text {GeV}$ and to satisfy Tight identification and Loose isolation criteria. For the boosted analysis, electrons must have $p_{\text {T}} >25~\text {GeV},$ and satisfy Medium identification and Loose isolation criteria. Additional cuts are applied for the highest-$p_{\text {T}}$ electron candidate in the boosted channel (Sect. 7.1), to ensure that $p_{\text {T}} >200~\text {GeV},$ and to satisfy the Tight identification and HighPtCaloOnly isolation criteria. Tighter isolation criteria are required in the boosted channel because the multijet process with a mis-identified electron is a major source of background.

Muons are reconstructed from MS tracks matching ID tracks in the $\left| \eta \right| < 2.5$ region. There are several muon identification working points described in Ref. [73], namely ‘Loose’, ‘Medium’ ,‘Tight’ and ‘High-$p_{\text {T}}$ ’. The resolved channel uses muons with $p_{\text {T}} >25~\text {GeV}$ satisfying the Medium identification working point. For muons with $p_{\text {T}} >300~\text {GeV},$ the High-$p_{\text {T}}$ working point requirements must be met. For the boosted channel, muons with $p_{\text {T}} >28~\text {GeV}$ satisfying the Medium working point are used. As with electrons, several muon isolation working points are defined in Ref. [73], e.g. ‘FixedCutTightTrackOnly’ and ‘Tight’. The resolved channel requires muons to satisfy the FixedCutTightTrackOnly working point. In the boosted channel, no isolation criterion is required on the muon from the $N_{\textrm{R}}\rightarrow \mu qq'$ decay, as it is not expected to be clearly isolated from the hadrons. Further selection cuts are applied for the highest-$p_{\text {T}}$ muon candidate in the event, namely to have $p_{\text {T}} >200~\text {GeV}$ and satisfy the Tight identification and Tight isolation criteria (Sect. 7.1).

For both electrons and muons, track-to-vertex association requirements are employed by using the impact parameter observables. The longitudinal impact parameter of the lepton track, $z_{0},$ is required to satisfy $\left| z_{0} \sin \theta \right| < 0.5$ mm, where $\theta$ is the polar angle of the track. In addition, the transverse impact parameter divided by its uncertainty, $|d_{0}|/\sigma (d_{0}),$ is required to be less than 5 (3) for electrons (muons). The leptons’ reconstruction, identification, isolation and trigger efficiencies differ slightly between simulation and data. The simulation is corrected with scale factors to match the data efficiencies [72, 73].

Apart from the baseline leptons described above, leptons with modified requirements are employed for control samples for the estimation of jets misidentified as leptons or non-prompt leptons from decays of hadrons. In the resolved channel, the isolation requirement is inverted and the Loose identification is employed. In the boosted channel, the fraction of events containing fake muons surviving the $p_{\text {T}} >200~\text {GeV}$ requirement for the leading lepton is negligibly small. To estimate the hadrons mis-identified as electrons, electron candidates satisfying the Loose but failing the HighPtCaloOnly isolation criterion are used.

Jets are reconstructed from particle-flow objects [74] using the anti-$k_{t}$ algorithm [75, 76] with radius parameter $R=0.4.$ They are referred to as ‘small-R jets’ hereafter. A detailed description of the calibration of these jets is found in Ref. [77]. Only small-R jets with $p_{\text {T}} >20~\text {GeV}$ and $\left| \eta \right| < 2.5$ are considered. Small-R jets with $p_{\text {T}} < 60~\text {GeV}$ and $\left| \eta \right| < 2.4$ from pile-up interactions are suppressed using a jet-vertex tagging (JVT) discriminant [78]. Small-R jets containing b-flavoured hadrons (b-jets) are identified with the DL1r multivariate b-tagging algorithms [79, 80]. A working point to achieve 77% efficiency for b-jets is employed, which has a rejection factor of 5 for c-jets and 169 for light jets. It is only used to veto events containing jets identified as b-jets.

The missing transverse momentum, with magnitude $E_{\text {T}}^{\text {miss}}$, is calculated using the baseline electrons and the small-R jets in each event as described in Refs. [81, 82] using the Tight working point. The particle-flow track-based soft term, built from tracks that are matched to the primary vertex but not associated with any other objects, is also used in the calculation. The $E_{\text {T}}^{\text {miss}}$ is used only in the electron channel, and muons are not considered in the calculation, since the muon momentum resolution is significantly degraded in the high-$p_{\text {T}}$ region and affects the $E_{\text {T}}^{\text {miss}}$ calculation.

To avoid cases where the detector response to a single physical object is reconstructed as two different final-state objects, e.g. an electron reconstructed as both an electron and a jet, several steps are followed, as summarised in Table 3. To improve the efficiency for muons from $N_{\textrm{R}}$ decays in the boosted analysis, the last step of muon-jet overlap removal is considered only for the highest-$p_{\text {T}}$ muon in the event.

Table 3 The order of overlap removal used in this analysis. $\Delta R$ is defined as $\Delta R = \sqrt{ \Delta y^{2} + \Delta \phi ^{2}},$ where $\Delta y$ and $\Delta \phi$ are the rapidity and azimuthal-angle differences between two objects. The variables $p_{\text {T}} (e),$ $p_{\text {T}} (\mu ),$ $p_{\text {T}} (j)$ and $p_{\text {T}} (\text {trk})$ are the transverse momenta of the corresponding electron, muon, small-R jets, and all tracks associated with the small-R jet. The baseline definitions of electrons and muons are used. In the boosted channel, the last step of muon-jet overlap removal is considered only for the highest-$p_{\text {T}}$ muon in the event. Large-R jets are reclustered from small-R jets after the overlap removal

In the boosted analysis, the hadronic jets from the $N_{\textrm{R}}$ decay are reconstructed as a single large-R jet. The large-R jet is reconstructed from the small-R jets described above, using the anti-$k_t$ algorithm with $R=1.0.$ For such large-R jets, ‘re-clustered’ from small-R jets, the small-R jet calibration can be propagated. Large-R jets with $p_{\text {T}} > 200~\text {GeV}$ and $\left| \eta \right| < 2.0$ are considered.

6 Resolved channel: event selection and background estimate

6.1 Event reconstruction and selection

The resolved channel targets signals with a mass splitting between $W_{\textrm{R}}$ and $N_{\textrm{R}}$, i.e. $\Delta m \equiv m(W_{\textrm{R}})- m(N_{\textrm{R}}),$ up to $4~\text {TeV}.$ Events passing the following requirements are filtered into datasets comprising signal regions (SRs), namely regions where the presence of signal is hypothesised by the theoretical models and the analysis is expected to have high sensitivity. The definitions of the SRs are summarised in Table 4. Events are required to have exactly two baseline, same-flavour charged leptons, with $p_{\text {T}} > 40~\text {GeV}$ for the leading lepton, and at least two small-R jets with $p_{\text {T}} > 100~\text {GeV}.$ To suppress the Z + jets background, the dilepton invariant mass must satisfy $m_{\ell \ell } > 400~\text {GeV}.$ Since the focus of this search is the high $W_{\textrm{R}}$ mass region, the following requirements are imposed on the dijet invariant mass $(m_{jj}),$ and the scalar sum of the transverse momentum of the leptons and the two leading small-R jets $(h_{\text {T}})$: $m_{jj}>110~\text {GeV}$ and $h_{\text {T}}>400~\text {GeV}.$ Events are further separated into two categories based on the charge product of the two leptons: same- (rSRSS) or opposite-sign (rSROS) lepton pairs, where r denotes the resolved channel and SR the classification as signal region. The relatively small SM background in rSRSS increases the search sensitivity for Majorana $N_{\textrm{R}}$ signals where half of the events are expected to appear with same-sign lepton pairs. The $\Delta R$ between the two leptons is required to be less than 3.9 in rSRSS to reduce some mismodelling of the simulated diboson background without a signal efficiency loss. Finally, the SRs are separated into electron and muon channels. The four resulting SRs (rSROS2e, rSROS2mu, rSRSS2e and rSRSS2mu) are orthogonal to each other and combined for a statistical analysis in the Majorana $N_{\textrm{R}}$ interpretation. For the Dirac $N_{\textrm{R}}$ interpretation, only rSROS2e and rSROS2mu are considered.

In rSROS, the $W_{\textrm{R}}$ mass is reconstructed by combining the two charged leptons and the two leading small-R jets $(m_{\ell \ell jj}),$ and used as the final discriminant for signal points in the $m(W_{\textrm{R}})> m(N_{\textrm{R}})$ region. The $0< m_{\ell \ell jj} < 5~\text {TeV}$ region is scanned with a step size of 500 $\text {GeV}$, with one additional overflow bin that includes all events with $m_{\ell \ell jj}>5~\text {TeV}.$ For signals with $m(W_{\textrm{R}})< m(N_{\textrm{R}}),$ the $m_{jj}$ is used instead as a discriminant, with a similar 500 $\text {GeV}$ binning in $0< m_{jj} < 3~\text {TeV},$ and one additional overflow bin for $m_{jj}>3~\text {TeV}.$ The significance of yields is checked in every bin for both discriminants $(m_{\ell \ell jj}$ and $m_{jj})$ and the appropriate one is used for each $(m(W_{\textrm{R}}),$ $m(N_{\textrm{R}}))$ hypothesis for the interpretation. In the smaller-background rSRSS subset, the variable $h_{\text {T}}$ offers slightly better sensitivity and is used instead as the final discriminant. This variable is segmented into five bins: 400–600 $\text {GeV}$, 600–1000 $\text {GeV}$, 1–1.5 $\text {TeV}$, 1.5–2.2 $\text {TeV}$, and $> 2.2~\text {TeV},$ which take into consideration the signal resolution and MC statistical uncertainty in each bin.

Table 4 Definition of signal regions in the resolved channel. The baseline definitions of electrons and muons are used

6.2 Background estimation

The composition of the SM background is substantially different between rSROS and rSRSS, requiring different estimation techniques in the two categories. SM backgrounds containing two prompt leptons are estimated using simulations in both cases. An overall normalisation correction obtained from the observed data in control regions (CRs) is used for the main background sources: Z + jets, diboson, and $t$ $\bar{t}$ processes. CRs are kinematically similar to SRs, and enriched in the process for which the normalisation correction is applied. Signal contamination is very low (typically 1–2%). A different normalisation correction factor is assigned to each process, and typically treated as a free parameter in the fit discussed in Sect. 9. The validity of the normalisation correction factor is confirmed in validation regions (VRs). The component containing non-prompt or mis-identified leptons is estimated separately by employing a data-driven method. The definitions of SRs, CRs and VRs for the resolved channel are visualised in Fig. 2.

Fig. 2

Resolved channel: Schematic view of SR/CR/VRs definitions. The grey coloured regions are contributing simultaneously to the final fit in the electron or muons channels. The remaining regions are used to verify the background estimations (see text for details)

6.2.1 Opposite-sign category

The main SM backgrounds contributing to rSROS are events with top quarks (mainly $t$ $\bar{t}$) and Z + jets production, with contributions of 41% and 45% respectively in rSROS2e and 41% and 48% in rSROS2mu. Minor contributions arise from diboson processes (mainly $ZW \rightarrow \ell \ell jj$ and $ZZ \rightarrow \ell \ell jj).$

The normalisation factor for Z + jets is estimated in a control region defined by the same selection as rSROS with a modification in the dilepton mass selection: $60< m_{\ell \ell } < 110~\text {GeV}$ (rCROS). There is a known issue in the jet multiplicity description of the Z + jets MC sample observed in a previous analysis [21], leading to the mis-modelling of the dijet invariant mass $m_{jj}.$ Following the prescription described therein, a data-driven method is employed to correct for this effect. Electron and muon channels are combined and used to derive the $m_{jj}$-dependent correction factor. The rCROS data is used after subtracting the top-quark (3%) and other minor prompt-lepton (2%) background contributions evaluated with MC simulation, leaving a distribution with a size of about 94% of the original dataset. The normalisation of the Z + jets MC sample is then scaled to match the resulting rCROS distribution. The data-to-Z + jets MC ratio, r,  reaches a maximum value of about 1.05 at $m_{jj}=300~\text {GeV},$ and decreases at the higher-mass region to reach a value of $\sim 0.7$ at $m_{jj}=3~\text {TeV}.$ The ratio r is parameterised by the Novosibirsk function, as discussed in Ref. [21]. The r is used to apply an event-by-event correction to the Z + jets simulated sample in the OS channel. The method corrects not only the $m_{jj}$ shape but also the normalisation of Z + jets in the simulation. Therefore, rCROS is not used in the final distribution fit discussed in Sect. 9.

The $t$ $\bar{t}$ background is estimated with the same selection as rSROS, but by requiring a different flavour for the two charged leptons i.e. $m_{e\mu } > 400~\text {GeV}$ (rCROSemu). The normalisation factor is treated as a free parameter in the final fit, but determined mainly in rCROSemu due to its high purity in $t$ $\bar{t}$ $(\sim$ 80%), with dibosons being the other main component $(\sim$ 19%).

The validity of the background estimations described above is checked in the validation region rVROS, defined in an intermediate $m_{\ell \ell }$ region between SR and CR, i.e. $110<m_{\ell \ell }<400~\text {GeV}.$ An additional validation region is defined with the same selection as rSROS but with an $e\mu$ pair instead of two same-flavour charged leptons (rVROSemu). Validation results are presented in Sect. 9.1.

6.2.2 Same-sign category

Diboson processes (mainly $WZ \rightarrow \ell \ell$ and same-sign $W^{\pm }W^{\pm } \rightarrow \ell ^{\pm }\nu \ell ^{\pm }\nu )$ are the main background source in rSRSS. In particular, about 83% of the background in rSRSS2mu can be attributed to dibosons. In addition, there are non-negligible contributions from $Z (\rightarrow ee)$ + jets and top-quark events with charge mis-identification in rSRSS2e. The background contributions from diboson and charge mis-identified processes in rSRSS2e are 29% and 39%.

The diboson background is estimated with the same selection as rSRSS, but by requiring $110< m_{ee} < 300~\text {GeV}$ or $60< m_{\mu \mu } < 300~\text {GeV}$ (rCRSS). To increase the CR dataset size, no selection is applied on $h_{\text {T}}$ or $m_{jj}.$ The validity of the background estimation technique is confirmed in the $300<m_{\ell \ell }<400~\text {GeV}$ validation region (rVRSS), with validation results shown in Sect. 9.1.

The charge-mis-identified background consists of events with two opposite-sign electrons, one of which has its charge sign mis-measured. The most frequent cause of electron charge flip is the bremsstrahlung effect. The probability of an electron undergoing charge mis-identification is measured in the data [83] and compared to the MC prediction. A correction factor is then derived for the charge flip probability and applied to simulated events featuring same-sign electron pairs. The method is validated in a same-sign dielectron region with $|m_{ee} - m_{Z}| < 15~\text {GeV},$ which does not overlap with the rCRSS region $(110< m_{ee} < 300~\text {GeV}).$ The data agrees well with the corrected prediction within uncertainties. This result confirms the validity of the correction factor for the charge-flip probability in a limited $m_{ee}$ range, but does not cover a possible mis-modelling of the extended $m_{ee}$ distribution. Unlike opposite-sign Z + jets, the normalisation of the same-sign Z + jet sample is therefore free to float and determined mainly in rCRSS2e.

Another smaller background contribution comes from mis-identified leptons originating from non-isolated, non-prompt electrons and muons produced by secondary decays of light- or heavy-flavour hadrons. Another significant component of fake electrons arises from photon conversions, as well as from jets that are misreconstructed as electrons. The contribution of events with mis-identified leptons is estimated by a data-driven technique, known as the “fake-factor” method. The fake factor F is defined as the ratio of mis-identified leptons satisfying the baseline lepton criteria (labelled as Tight in this section) to those satisfying the selection criteria for fake estimation (Loose). The factors are estimated separately for electrons and muons. The fake factors calculated in Ref. [84] are used in this analysis, as common object definitions are employed. Fake factors are measured in data as a function of lepton $p_{\text {T}}$ and $\eta$ in control samples that are orthogonal to the SR, VR, and CRs used in this analysis.

The contribution of mis-identified leptons in SR, VR, and CRs is estimated as follows. The calculated F is applied to events satisfying the same selection criteria as in SR, VR, and CRs but by lowering the lepton ID requirement from Tight to Loose for at least one lepton. The signal contamination in these regions is found to be small (typically 1–2%) and ignored here. Three categories are considered separately, depending on the identification criteria satisfied by the leading and sub-leading leptons: they are the LT (Loose–Tight), TL (Tight–Loose) and LL (Loose–Loose) categories. Data events are weighted with fake factors according to the loose-lepton multiplicity of the region:

$$\begin{aligned} N^{\text {fake}}= & {} [ F(N_{TL} + N_{LT}) - F^{2} N_{LL} ]_{\text {data}} \nonumber \\{} & {} - [ F(N_{TL} + N_{LT}) - F^2 N_{LL} ]_{\text {MC}}^{\text {prompt } \ell \text { only}} \end{aligned}$$

(1)

with $N_{TL},$ $N_{LT},$ $N_{LL}$ denoting the number of events in the corresponding category. The prompt lepton contribution in the events with Loose leptons is subtracted using the MC simulation to account for the prompt-lepton contamination in the given category.

The composition of rCRSS is dibosons (32%), charge mis-identified background (45%) and fakes (20%) in the electron channel, and dibosons (65%), fakes (27%) and charge mis-identified background (8%) in the muon channel.

7 Boosted channel: event selection and background estimate

7.1 Event reconstruction and selection

The boosted analysis is designed for signals in the $\Delta m > 4~\text {TeV}$ region by using the large-R jet to reconstruct the hadrons from the $N_{\textrm{R}}$ decay. The signal events have a very energetic charged lepton and $N_{\textrm{R}}$ from the $W_{\textrm{R}}$ decay produced back-to-back in the x–y plane. In the highest $\Delta m$ region, the large-R jet overlaps with the lepton from the $N_{\textrm{R}}$ decay. In the muon channel, such a muon inside the jet can be identified, and a single SR (bSR2mu) with two baseline muons is used to cover the full $\Delta m$ range. On the other hand, in the electron channel, calorimeter clusters from the electron from the $N_{\textrm{R}}$ decay overlap with other hadron activities, making its identification challenging. Two signal regions, bSR1e and bSR2e, are thus defined requiring exactly one and two baseline electrons in the event, respectively, to cover the higher and lower $\Delta m$ ranges. The $W_{\textrm{R}}$ mass is reconstructed from the large-R jet and the electron $(m_{eJ})$ in bSR1e, and from the large-R jet and the two charged leptons $(m_{eeJ}$ and $m_{\mu \mu J},$ or collectively $m_{\ell \ell J})$ in bSR2e and bSR2mu. Due to the relatively small background after the event selections, the sensitivity gain by separating SR into same- and opposite-sign events is marginal, hence the lepton charge is not considered in the boosted analysis.

For all three SRs, the requirement that only one large-R jet is reconstructed in the event is imposed, which is expected to cover about 90% of signal events. This selection cut is useful in suppressing the multijet background, and improving the fake-lepton, multijet and $\gamma$ + jets modelling by the LO simulation with the exception of an overall normalisation factor. Assuming well-isolated and highly-boosted leptons from the prompt $W_{\textrm{R}}$ decay, tighter identification and isolation criteria (as detailed in Sect. 5) are required for the lepton with the highest-$p_{\text {T}}$ in the event. The azimuthal difference between the large-R jet and the highest-$p_{\text {T}}$ lepton is required to be greater than 2. In addition, to suppress the top-quark backgrounds, the number of b-tagged small-R jets is required to be zero.

7.1.1 One-electron category (bSR1e)

Region bSR1e requires exactly one electron from the $W_{\textrm{R}}$ decay, with the large-R jet encapsulating all decay products of the heavy neutrino $N_{\textrm{R}}\rightarrow eqq^{\prime }.$ A $E_{\text {T}}^{\text {miss}} < 200~\text {GeV}$ selection is applied to suppress the main background process, $W (\rightarrow e\nu )$ + jets. The high-$p_{\text {T}}$ electron and low $E_{\text {T}}^{\text {miss}}$ requirements bias the W + jets events towards higher helicity angles, $|\cos \theta | \sim 1.0,$ where $\theta$ is the polar angle of the electron from the W boson decay in the rest frame of the W.² The $|\cos \theta | > 0.7$ region is used for the signal search. The $|\cos \theta | < 0.7$ region is used for the normalisation of the W + jets contribution. To further reduce the $\gamma$ + jets and dijet contributions, a selection on the $\eta$ difference between the large-R jet and the electron candidate, $\Delta \eta = \left| \eta (J) - \eta (e) \right| < 2$ is applied. The bSR1e definition is summarised in Table 5.

7.1.2 Two-electrons category (bSR2e)

Region bSR2e covers the intermediate $\Delta m$ region between bSR1e and the resolved analysis. The events are selected if they contain exactly two electrons and one large-R jet. The $E_{\text {T}}^{\text {miss}} < 200~\text {GeV}$ selection is applied to further reduce the $t$ $\bar{t}$ background. The dominant background source is $Z (\rightarrow ee)$ + jets. To remove contributions from the pole mass of the Z boson, a $m_{ee} > 200~\text {GeV}$ selection is applied. The bSR2e definition is summarised in Table 5. bSR1e and bSR2e are orthogonal to each other and are statistically combined to obtain the final results.

Table 5 Definition of the signal regions in the boosted analysis. The baseline definitions of electrons and muons are used. Two regions in the electron channel cover higher- and lower-$\Delta m$ regions, where $\Delta m$ denotes the mass difference between $W_{\textrm{R}}$ and $N_{\textrm{R}}$

7.1.3 Two-muons category (bSR2mu)

Events are selected in bSR2mu if they contain exactly two muons and one large-R jet. A $m_{\mu \mu } > 200~\text {GeV}$ selection is applied. The upper cut on $E_{\text {T}}^{\text {miss}}$ is not useful in this channel due to the deteriorated momentum resolution for very high-$p_{\text {T}}$ muons. Instead, the dimuon system’s $p_{\text {T}}$ is required to be greater than 200 $\text {GeV}$ to suppress the $t$ $\bar{t}$ background. The bSR2mu definition is also summarised in Table 5.

7.1.4 Event categorisation for the background estimation

The normalisations of the main background sources, $W(\rightarrow e\nu )$ + jets and $Z(\rightarrow \ell \ell )$ + jets, as well as for the processes with less reliable yield predictions, $\gamma$ + jets and multijet, are estimated from data in control regions and adjusted by using the SR-to-CR ratio in the simulation.

In each category, the reconstructed $W_{\textrm{R}}$ mass distribution is segmented into four bins; 1–2 $\text {TeV}$, 2–3 $\text {TeV}$, 3–4 $\text {TeV}$, and $>4~\text {TeV},$ having considered the search sensitivity for signals with various $W_{\textrm{R}}$ masses and the stability of the background estimation. High-mass $W_{\textrm{R}}$ signals are expected to be observed in the third and fourth bins, which are designated bins 1 and 2 of the SR for each channel (for example, bins 1 and 2 of bSR2mu for the two-muon case). The first bin of the $W_{\textrm{R}}$ mass histogram is used for the SM background determination in a region where the presence of lower-mass $W_{\textrm{R}}$ signals is negligibly small, given exclusion limits obtained in previous ATLAS analyses [22]. For the two-electron and two-muon categories, the first bin is used to estimate the Z + jets background normalisation in designated bCRZ2e and bCRZ2mu, respectively. In the one-electron case, the 1–2 $\text {TeV}$ $m(W_{\textrm{R}})$ bin is used to distinguish between $\gamma$ + jets and W + jets events. If there is an additional ID track with opposite charge within $\Delta R<0.3$ and $\left| \Delta z_{0} \sin \theta \right| < 0.5$ from the electron candidate track, the electron is possibly coming from a photon conversion and the events are categorised into bCR $\gamma$ 1e. The remaining events are categorised into bCRLow1e. The fractions of W + jets, Z + jets, multijet and $\gamma$ + jets are about 33%, 10%, 24% and 30%, respectively, in bCR $\gamma$ 1e, and approximately 73%, 17%, 3% and 1% in bCRLow1e.

The normalisation for $W(\rightarrow e\nu )$ + jets and multijet events is estimated in the dedicated control regions, bCRW1e and bCRFake1e, requiring $|\cos \theta | < 0.7$ and the inverted electron isolation cut, respectively, with the remaining selection identical to that of bCRLow1e. The purity of the W + jets and multijet events in bCRW1e and bCRFake1e is about 83% and 91%. One should note that the $\gamma$ + jets contribution in bCRFake1e is not large since mis-identified electrons from photon conversions can satisfy the tight isolation requirement.

The estimated CR normalisation factors are extrapolated to higher-mass bins. The validity of the method is first checked in the second bin $(m(W_{\textrm{R}})= [2, 3]~\text {TeV})$ in each channel: bVR2e, bVR2mu and bVR1e, for bSR2e, bSR2mu and bSR1e, before the third and fourth bins are tested. In addition, the regions with $\left| \cos \theta \right| <0.7$ and the inverted electron isolation in the one-electron channel are used to check the extrapolation of W + jets and multijet normalisations from the lower to the higher $W_{\textrm{R}}$ mass bins. The regions are segmented into 2–3 TeV and $>3~\text {TeV}$ and labelled as bVRW1e and bVRFake1e, respectively.

To check the extrapolation of the Z + jets normalisation from the lower to the higher $W_{\textrm{R}}$ mass bins, additional validation regions are defined (bVRZ2e and bVRZ2mu), but which are not used in the final fit. They require $120< m_{\ell \ell } < 200~\text {GeV}$ instead of $>200~\text {GeV}.$

The definitions of SRs, CRs and VRs for the boosted channel are visualised in Fig. 3. Validation results are presented in Sect. 9.2.

7.2 Background estimation

For the W + jets and Z + jets estimation in the boosted analysis, since the number of jets is forced to be one, the correction for jet multiplicity mis-modelling used in the resolved analysis is not required. The simulated $W_{\textrm{R}}$ mass distributions for W + jets in bVRW1e (with $|\cos \theta |<0.7)$ and those for Z + jets bVRZ2e and bVRZ2mu agree well with the observed data. The $W_{\textrm{R}}$ mass distributions in these regions are only used to confirm the normalisation methodology for each process and these are within the considered uncertainties.

The fake lepton contribution is negligibly small in bSR2e and bSR2mu after the requirement of a high-$p_{\text {T}}$ selection on the leading lepton. It is, however, not negligible at the higher $W_{\textrm{R}}$ mass tail in bSR1e. MC simulation is used to estimate the non-prompt electron contribution with a data-driven correction factor obtained in the CR. The simulated $W_{\textrm{R}}$ mass in bVRFake1e agrees well with the multijet sample within the uncertainties considered. In the final fits the normalisation is mainly constrained by bCRFake1e and its validity in the SR relies on the good simulation of the isolation efficiency for fake electrons. To check the validity of this approach, two additional regions beyond those shown in Fig. 3 are defined, corresponding to bSR1e and bCRFake1e: the electron candidate is forced to satisfy the Medium identification but fail the Tight identification criterion in order to be orthogonal to the SR, CRs and VR. The remaining selection is the same as in bSR1e (bCRFake1e), with the requirement to satisfy (fail) the HighPtCaloOnly isolation criterion. In both regions, the purity of the multijet process is greater than 90%. By comparing data to MC in both regions, it is found that the simulated isolation efficiency for fake electrons agrees with the data within the considered uncertainties. A small data/MC disagreement on $m(W_{\textrm{R}})$ distribution is considered as an additional uncertainty on the multijet estimation in bSR1e.

The $\gamma$ + jets contribution with isolated electrons from photon conversions is also estimated by MC simulation with a normalisation factor determined in a data-driven way. Additional validation regions are defined requiring exactly one reconstructed photon instead of an electron, with the remaining selection the same as in bSR1e. The purity of $\gamma$ + jets sample in this region is about 70%, and the fraction of multijet events is 30%. The reconstructed $W_{\textrm{R}}$ mass with the photon (in the place of the electron) and the large-R jet agrees between data and MC within the considered uncertainties. The $p_{\text {T}}$- and $\eta$-dependent photon conversion rate may change the reconstructed $W_{\textrm{R}}$ mass distribution for the $\gamma$ + jets process, but this rate mainly depends on the detector geometry and it is well modelled in the simulation. The normalisation of the $\gamma$ + jets contribution is estimated mainly from bCR $\gamma$ 1e, by requiring an additional ID track close to the electron. The modelling of the fraction of events with an ID track close to the electron in simulated W + jets and multijets can affect the $\gamma$ + jets estimation, so a comparison is carried out with data in bCRW1e and bCRFake1e. For the $\gamma$ + jets contribution, the same check is performed in the $\Delta \eta > 2.0$ region. The purity of $\gamma$ + jets in this region is about 50%, and a conservative uncertainty on the other background subtraction is included in the study.

Fig. 3

Boosted channel: Schematic view of SR/CR/VRs definitions for bSR1e (a), (b), bSR2e (c) and bSR2mu (d). a and b show the regions for the estimation of W + jets and multijet backgrounds, respectively. The labels bSR1e, bVR1e, bCRLow1e and bCR $\gamma$ 1e shown in bold in the two figures correspond to the same regions. The dark-grey coloured regions in a, b and c are contributing simultaneously to the final fit in the electron channel, while the light-grey coloured regions in d are used for the muon channel, separately. The remaining regions are used to verify the background estimations (see text for details)

8 Systematic uncertainties

Several experimental and theoretical systematic uncertainties are considered, affecting both the background and signal predictions as well as the total event yield. In addition, the statistical uncertainty of the MC simulated events is also considered. These uncertainties also affect the shape of the variables used in the fit, with the exception of the luminosity and cross-section uncertainties.

The uncertainty on the integrated luminosity is 1.7% [85], It is obtained using the LUCID-2 detector [86] for the primary luminosity measurements. The uncertainty due to the pile-up reweighting procedure is estimated by varying the amount of pile-up in the simulation to cover the uncertainty in the ratio of the predicted and measured inelastic cross section [87].

A set of experimental systematic uncertainties arise from the energy and momentum calibrations of leptons and jets, the lepton reconstruction, isolation and trigger efficiencies, and the jet vertex tagger and flavour-tagging efficiencies. The largest uncertainty in the total SM yield arises from the energy calibration and smearing of jets, derived in Ref. [77], and is between 1 and 6% in the SS and about 4% in the OS in the resolved channel, and between 5 and 12% in the boosted channel, depending on the signal region. Uncertainties associated with lepton reconstruction, identification, isolation and trigger efficiencies, as well as energy or momentum calibration [41, 72, 88] and b-jet tagging [79, 89], vary between 5% (OS) and 13% (SS) of the total SM yield in the resolved analysis, and between 2 and 5% in the boosted. These uncertainties are propagated to the $E_{\text {T}}^{\text {miss}}$ calculation. In addition, uncertainties on the scale and resolution of the ‘soft term’ of the $E_{\text {T}}^{\text {miss}}$ are taken into account [81]. In bSR2mu, the muon $p_{\text {T}}$ is further smeared by taking into account the impact of MS outliers on the muon resolution not described by the simulation. This additional effect is considered as the high-$p_{\text {T}}$ muon uncertainty.

There are three additional sources of systematic uncertainty associated with the background estimation techniques. In rSRSS, the uncertainty related to the charge mis-identification probability of electrons arises from the statistical uncertainty of the data and simulated samples of Z + jets events used in this measurement. The uncertainty is around 6.8% with a mild dependence on the electron $E_{\text {T}}$ and $\eta$ [90]. The uncertainty on the fake estimation arises from the limited knowledge in the composition of fakes, as well as from the statistical uncertainty and prompt lepton subtraction used to derive F in the fake-enriched regions. The uncertainty due to the composition of fakes is estimated by varying the nominal criteria defining the selection sample used in the fake-factor measurement [84]. The effect on the SM yields is 3.8% (electrons) and 0.8% (muons). In rSROS, an additional uncertainty is associated with the $m_{jj}$ reweighting of the Z + jets process. This is evaluated by comparing the shape difference between the reweighted simulated $m_{jj}$ distribution and the one measured in data, using rVROS: the uncertainty in the reweighting factor is found to vary between 0.4 and 0.8%, as a function of the dijet invariant mass. This observed difference covers all possible mismodellings of the Z + jets shape, hence additional theoretical uncertainties for the Z + jets process are not considered in the resolved channel.

In the boosted channel, the theory uncertainties estimated for the W + jets, Z + jets, diboson, multijet and $\gamma$ + jets background processes are considered, which include the choice of QCD renormalisation $(\mu _r)$ and factorisation $(\mu _f)$ scales, choice of the PDF set and $\alpha _S,$ as well as CKKW matching scale and the resummation scale. The QCD scale uncertainty is estimated by varying $\mu _r$ and $\mu _f$ to half and twice their nominal values. The PDF uncertainty is estimated using the envelope of the NNPDF3.0 PDF set, as recommended in Ref. [91]. In addition, the MMHT2014 [92] and CT14NNLO [93] PDF sets are used to estimate the uncertainty due to the PDF choice. Moreover, the uncertainty due to $\alpha _S$ is evaluated by varying its nominal value of 0.118 by $\pm 0.001.$ The CKKW matching scale is the parameter to remove the overlap between jets from the matrix element calculation and the parton shower algorithm. The nominal value is 20 GeV, and the uncertainty is estimated by varying it to 30 GeV and 15 GeV. The uncertainty on the resummation scale of soft gluon emission is estimated by varying the parameter to half and twice the nominal value. The largest theory uncertainties generally originate from the CKKW matching and Sherpa resummation scales $(\sim$ 20%), and QCD scales variations which range between 10% and 20%, depending on the simulated process and the mass of the target signal.

The theory uncertainties associated with $t$ $\bar{t}$ processes are as follows. The uncertainty from hard scatter generation is evaluated by comparing the Powheg-Box and MG5_aMC@NLO generators, both interfaced to the Pythia 8.186 [94] parton shower model. The uncertainty due to the hadronisation and fragmentation model is determined by comparing the nominal Powheg-Box + Pythia 8.186 generated sample with the one generated by Powheg-Box interfaced to Herwig [95] (version 7.13). The uncertainty related to the amount of initial- and final-state radiation (ISR and FSR) is assessed by varying parton shower settings. The largest theory uncertainty generally originates from the amount of initial- and final-state radiation and is between 0.4–0.5% (ISR) and 0.6% (FSR) of the $t$ $\bar{t}$ process yield.

In the boosted analysis, additional uncertainties are considered by comparing different MC samples on the possible variations of the reconstructed $m(W_{\textrm{R}})$, $\cos \theta$ and the isolation efficiency for the fake electrons. For W/Z + jets, the nominal Sherpa 2.2.11 samples are compared with the samples generated with MadGraph5_aMC@NLO+Pythia 8.186. For $\gamma$ + jets, the nominal Sherpa 2.1 sample is compared with the alternative sample generated with Pythia 8.235 [44]. The nominal Pythia 8.230 multijet MC sample is compared with the sample generated with Sherpa 2.2.5. For the multijet sample, an additional uncertainty on the $m(W_{\textrm{R}})$ shape is considered by taking the residual non-closure in studies employing different electron identification discussed in Sect. 7.2. The possible mis-modelling of the bCR $\gamma$ 1e to bCRLow1e ratio is evaluated by considering the observed discrepancy in the CRs.

The theory uncertainty of the signal efficiency times acceptance amounts to 20%. It is evaluated by varying renormalisation and factorisation scales as described above and by using alternative PDF sets, CTEQ6 [96] and MSTW [97]. The $\alpha _S$ emission scale factor is also varied to half and twice the nominal value. The uncertainty is dominated by the variation in factorisation scale. The variations are performed using SysCalc [98].

The systematic uncertainties in each SR are summarised in Fig. 4. Individual uncertainties can be correlated with others, and do not necessarily add in quadrature towards the total background uncertainty.

Fig. 4

Relative uncertainties in the total background yield post-fit estimates for the a resolved and b boosted channels. “Theory” indicates the theoretical uncertainty associated with the simulated physics processes (e.g. cross sections). “Normalisation” is the uncertainty associated with the yield variations of the dominant backgrounds in the fit. “Experimental” corresponds to the combined uncertainty on physics object efficiencies (such as trigger, identification or isolation) and uncertainties associated with $E_{\text {T}}^{\text {miss}}$ and pile-up. The total uncertainty takes into account any correlations among nuisance parameters

9 Statistical analysis and results

The search for a $W_{\textrm{R}}$ and $N_{\textrm{R}}$ signal is carried out using the theoretical model described in Sect. 2. The statistical treatment is done separately for samples containing electrons and muons since there is no theoretical motivation that the $N_{\textrm{R}}$ should have the same mass across different flavours. The resolved and boosted channels are analysed independently. The statistical interpretation is performed by carrying out binned maximum-likelihood fits using the HistFitter [99] framework. The likelihood is a product of Poisson probability density functions, describing the observed number of events in each bin of regions involved in the fit, and Gaussian distributions that describe the nuisance parameters associated with each of the systematic uncertainties. Systematic uncertainties that are correlated between different samples and different regions are accounted for via a common nuisance parameter.

Fig. 5

VR fits in the resolved channel: a $m_{\ell \ell jj}$ in rVROS2mu and b $h_{\text {T}}$ distribution in rVRSS2e. ‘Top’ refers to all processes containing at least one top quark. ‘Multiboson’ refers to VV and VVV processes, where $V = W$ or Z. The expected background is determined via a fit on the CR data. The hatched band (‘total SM’) includes all post-fit systematic uncertainties, having taken into account all correlations among various sources. A blue arrow indicates an out-of-range data point for a given bin

Fig. 6

CR/VR/SR fits in the resolved channel: Integrated number of events for observed data and expected background in $e\mu$ (OS), and ee and $\mu \mu$ (SS) CR, ee and $\mu \mu$ (OS), and ee,  $\mu \mu$ and $e\mu$ VR and ee and $\mu \mu$ (both OS and SS) SRs. ‘Top’ refers to all processes containing at least one top quark. ‘Multiboson’ refers to VV and VVV processes, where $V=W$ or Z. The expected background is determined via a fit on the CR data. The hatched bands (‘total SM’) include all post-fit systematic uncertainties, having taken into account all correlations among various sources

A “background-only” fit is carried out first by using the observed event yield in CRs assuming that no signal contributes in these regions, and by applying the resulting normalisation factor to the number of background events predicted by simulation for the equivalent process in the validation regions. The normalisation factors of the background processes are allowed to vary freely. The VRs are only employed to confirm the validity of the background modelling in the background-only CR fits, and do not contribute to the search results presented here.

Fig. 7

SR fits in the resolved channel: The $m_{\ell \ell jj}$ distributions in a rSROS2e and b rSROS2mu, the $m_{jj}$ distributions in c rSROS2e and d rSROS2mu, and the $h_{\text {T}}$ distributions in e rSRSS2e and f rSRSS2mu. ‘Top’ refers to all processes containing at least one top quark. ‘Multiboson’ refers to VV and VVV processes, where $V = W$ or Z. ‘Non-prompt’ and ‘Charge-Flip’ refer to processes containing a mis-identified electron and Z + jets with charge mis-identification. The background expectation is the result of the fit to the CR data. The hatched bands (‘total SM’) include all systematic uncertainties post-fit with the correlation between various sources taken into account. A blue arrow indicates an out-of-range data point for a given bin

Fig. 8

CR/VR/SR fits in the boosted channel: Integrated number of events for observed data and expected background in bCRW1e and bCRFake1e (a), and observed $m(W_{\textrm{R}})$ distributions in 1e (b), 2e (c) and 2mu (d) categories. For b–d, the first and second bins are used as CR and VR, and the third and fourth bins correspond to the SR. ‘Top’ refers to all processes containing at least one top quark. ‘Multiboson’ refers to VV and VVV processes, where $V=W$ or Z. The expected background is determined via a fit on the CR data. The hatched bands (‘total SM’) include all post-fit systematic uncertainties, having taken into account all correlations among various sources. A blue arrow indicates an out-of-range data point for a given bin

A non-zero signal contribution for each $m(W_{\textrm{R}})$ and $m(N_{\textrm{R}})$ mass point is then allowed involving both SR and CRs in the fit, as summarised in Figs. 2 and 3. The normalisation factors of the various (signal or background) processes are allowed to vary freely. Upper limits at 95% CL on the signal strength of the KS process are calculated using the CL$_{\text{ s }}$ method [100] and the profile likelihood-ratio as the test statistic. The asymptotic method [101] was used, but similar results were obtained with toy MC tests.

Fig. 9

Expected and observed 95% CL upper limits for the Majorana (top) and Dirac (bottom) neutrino interpretations in the electron (left) and muon (right) channels. Exclusion limits from previous ATLAS [21, 22] and CMS [27] searches are overlaid for comparison

9.1 Resolved channel

In the resolved channel, the $m_{\ell \ell jj}$ distribution is used in the rSROS fit when $m(W_{\textrm{R}})> m(N_{\textrm{R}}).$ In the $m(W_{\textrm{R}})< m(N_{\textrm{R}})$ case, the fit results are obtained by employing the $m_{jj}$ distribution. The $h_{\text {T}}$ distribution is used in the rSRSS fit. For the scenario in which the $N_{\textrm{R}}$ neutrino is a Majorana particle, the OS and SS channels are fitted simultaneously, whereas for the Dirac neutrino scenario, only the OS channel is used in the fit.

A combined “background-only” fit is first carried out in the rCROSemu, rCRSS2e and rCRSS2mu regions. The free normalisation factors for the $t$ $\bar{t}$ and diboson processes are estimated mainly from rCROSemu and rCRSS2mu. The validity of the evaluated normalisation factors for same-sign Z + jets, $t$ $\bar{t}$ and diboson samples is confirmed by applying them in the rVROS2e, rVROS2mu, rVRSSemu, rVRSS2e, and rVRSS2mu regions. All systematic uncertainties described in Sect. 8 are considered as nuisance parameters. Figure 5 shows an example of the post-fit $m_{\ell \ell jj}$ distribution in rVROS2mu and $h_{\text {T}}$ distribution in rVRSS2e, showing a good agreement between the observed data and the expected background.

The integrated event counts for the observed data and the estimated background for the CR, VR and SR regions are shown in Fig. 6. The post-fit $m_{\ell \ell jj}$ and $m_{jj}$ distributions in rSROS and $h_{\text {T}}$ distributions in rSRSS are shown in Fig. 7. In all cases, no significant deviation above the expected background is observed in the data. The largest excess is observed in rSRSS2e around $h_T \sim 1.0$ TeV (Fig. 7e): 39 events are observed in a region where $24.0 \pm 2.4$ background events are expected. The observed deviation is in a background-rich region and is not consistent with the shape of the signal.

Exclusion limits at 95% CL on the signal strength of the KS process as a function of $m(W_{\textrm{R}})$ and $m(N_{\textrm{R}})$ are calculated. The results are shown in Fig. 9. For electron Majorana neutrinos, the excluded region extends to a $W_{\textrm{R}}$ mass of 5.6 TeV and to a $N_{\textrm{R}}$ mass of 3.5 TeV. For muon Majorana neutrinos, $W_{\textrm{R}}$ masses of up to 5.7 TeV were excluded, and $N_{\textrm{R}}$ masses of up to 3.6 TeV. For electron Dirac neutrinos, the excluded region extends to a $W_{\textrm{R}}$ mass of 5.9 TeV and to a $N_{\textrm{R}}$ mass of 3.6 TeV. For muon Dirac neutrinos, $W_{\textrm{R}}$ masses of up to 5.8 TeV were excluded, and $N_{\textrm{R}}$ masses of up to 3.8 TeV. Compared to the previous ATLAS analysis in the resolved channel [21], the exclusion limits on the $W_{\textrm{R}}$ masses are extended by 0.8–1.2 $\text {TeV}$ in the case of $m(W_{\textrm{R}})> m(N_{\textrm{R}}).$

9.2 Boosted channel

In the boosted channel, the reconstructed $m(W_{\textrm{R}})$ distribution $(m_{eJ}$ or $m_{\ell \ell J})$ is used exclusively since $m(W_{\textrm{R}})\gg m(N_{\textrm{R}}).$ In the electron channel, the normalisations of W + jets, Z + jets, $\gamma$ + jets and multijet samples are simultaneously estimated by the fit in the CRs and SRs. In the muon channel, only the Z + jet contribution is estimated by the fit.

A combined “background-only” fit is performed separately in the electron and muon channels in the bCRW1e, bCRFake1e, bCR $\gamma$ 1e, bCRLow1e and bCRZ2e, and bCRZ2mu regions, respectively. The normalisation factors for W + jets, Z + jets, $\gamma$ + jets and multijet are free to float in the electron channel, while only the Z + jets normalisation is treated as a free parameter in the muon channel. The validity of the method is confirmed by applying the normalisation factors obtained with the fit in the bVR1e and bVR2e regions in the electron channel, and the bVR2mu region in the muon channel. As shown in Fig. 8, a good agreement between data and the estimated SM background is found in all CRs and VRs.

The post-fit $m(W_{\textrm{R}})$ distributions for 1e, 2e and 2mu categories are found in Fig. 8. The first bin in the 1e category shows the sum of bCRLow1e and bCR $\gamma$ 1e. Again, no significant deviation above the expected background is observed in the data.

Exclusion limits at 95% CL on the signal strength of the KS process as a function of $m(W_{\textrm{R}})$ and $m(N_{\textrm{R}})$ are calculated again, with the results shown in Fig. 9. For Majorana neutrinos, the most stringent lower limit on the $W_{\textrm{R}}$ mass is 6.4 $\text {TeV}$, observed in both electron and muon channels at $m(N_{\textrm{R}})<1~\text {TeV}.$ This result is an improvement over previous ATLAS searches, and extends the exclusion limits on $m(W_{\textrm{R}})$ by about 1.5 TeV. There is a particular improvement in the $m(N_{\textrm{R}})< 100~\text {GeV}$ region via the incorporation of the new bSR1e region and the optimisation of bSR2mu. The observed limit on $m(W_{\textrm{R}})$ is 6.2 (6.1) TeV in the electron (muon) channel at $m(N_{\textrm{R}})= 50~\text {GeV}.$

10 Conclusion

A search for right-handed $W_{\textrm{R}}$ bosons and heavy right-handed Majorana or Dirac neutrinos $N_{\textrm{R}}$ is presented. The analysis uses various final states depending on the mass difference between $W_{\textrm{R}}$ and $N_{\textrm{R}}$: two jets and a pair of charged leptons $(\ell \ell jj),$ one large-R jet and two charged leptons $(\ell \ell J),$ or one large-R jet and one electron (eJ),  with $\ell = e, \mu .$ It is performed with a 139 fb$^{-1}$ sample of pp collisions at $\sqrt{s}$ = 13 TeV recorded by the ATLAS detector at LHC. No evidence of $W_{\textrm{R}}$ bosons or Majorana or Dirac heavy neutrinos is found assuming the KS process, and lower limits are set on $m(W_{\textrm{R}})$ and $m(N_{\textrm{R}}),$ assuming equality of left- and right-handed weak couplings $(g_L = g_R).$ The excluded region for the Majorana neutrinos extends to about $m(W_{\textrm{R}})$ = 6.4 TeV for $m(N_{\textrm{R}})= 1$ TeV in both electron and muon channels. The $m(N_{\textrm{R}})$ limits reach about 3.5 TeV in the electron channel and 3.6 TeV in the muon channel (for $m(W_{\textrm{R}})= 4.8~\text {TeV}).$ For Dirac neutrinos, limits reach about $m(W_{\textrm{R}})= 6.4$ TeV for $m(N_{\textrm{R}})= 1$ TeV in both electron and muon channels. Limits of $m(N_{\textrm{R}})$ up to 3.6 TeV in the electron channel and 3.8 TeV in the muon channel for $m(W_{\textrm{R}})= 5.0$ TeV are set. The new results constitute the most stringent limits to date for the KS process.