1 Introduction

The extension of the Standard Model (SM) gauge symmetry based upon the higher weak-isospin symmetry \(SU(3)_L\) is best known for solving the number of observed fermion generations, generally called 3-3-1 model when including color and electric charges [1,2,3,4]. In order to cancel the \([SU(3)_L]^3\) anomaly [5,6,7,8], the traditional arrangement is that one of three quark generations transforms under \(SU(3)_L\) differently from remaining two quark generations, whereas all lepton generations transform identically under this group. However, recently Fonseca and Hirsch have made an interesting proposal where one of three lepton generations to transform differently from remaining two lepton generations, while all quark generations are identical under \(SU(3)_L\) [9]. In this way, the 3-3-1 matter content is flipped (i.e. reversed), called the flipped 3-3-1 model. The key for flipping is that the \([SU(3)_L]^3\) anomaly induced by a fermion sextet equals that induced by seven fermion triplets, where note that the color number is not counted. Furthermore, the authors have shown that the flipped 3-3-1 model only realizes a unique and minimal setup of fermion content (see also [10]). This flip converts the flavor matters in quark sector to the lepton sector as well as leads to the anomalies in interaction of neutrinos with matter [10]. The lepton flavor violating decays of charged leptons and the SM-like Higgs boson in this model were investigated, all of which were well below experimental bounds [9,10,11].

It is noteworthy that the electric charge Q and the baryon-minus-lepton charge \(B-L\) neither commute nor close algebraically with \(SU(3)_L\) [10]. To close the symmetries on which we can base our theory, the isospin symmetry \(SU(3)_L\) must be enlarged (i.e. flipped) to \(SU(3)_L \otimes U(1)_X \otimes U(1)_N\) by symmetry principles, called 3-3-1-1 when including the color group [12, 13]. Here the new abelian charges X and N are related to Q and \(B-L\) through the Cartan generators of \(SU(3)_L\), respectively. Hence, the fully flipping in gauge sector yields a complete gauge symmetry and it may of course contain the above flipped fermion content. In this work, we discuss a model that is based on this complete gauge symmetry with the flipped fermion content, called fully flipped 3-3-1-1 model. We show that the model supplies the novel schemes of single-component dark matter (DM).

The rest of this paper is organized as follows. In Sect. 2, we provide the features of the model. The mass spectra for the scalar and gauge boson sectors are considered in Sect. 3. In Sect. 4 we compute relevant interactions. Sect. 5 is devoted to the DM observables. Finally, we conclude this work in Sect. 6.

2 The fully flipped 3-3-1-1 model

As stated, the 3-3-1-1 gauge symmetry is given by

$$\begin{aligned} SU(3)_C\otimes SU(3)_L\otimes U(1)_X\otimes U(1)_N, \end{aligned}$$
(1)

where \(SU(3)_C\) is the ordinary color group. The new charges X and N respectively determine Q and \(B-L\) as related to the diagonal generators of the \(SU(3)_L\) group to be [10]

$$\begin{aligned} Q = T_3 + \frac{1}{\sqrt{3}} T_8 + X,\quad B-L = \frac{2}{\sqrt{3}} T_8 + N. \end{aligned}$$
(2)

The flipped fermion content under the gauge symmetry \(SU(3)_C\otimes SU(3)_L\otimes U(1)_X\otimes U(1)_N\) is arranged, such as

$$\begin{aligned}&\psi _{1L} = \left( \begin{array}{ccc} \xi ^{+} &{} \frac{1}{\sqrt{2}}\xi ^{0} &{} \frac{1}{\sqrt{2}} \nu _1 \\ \frac{1}{\sqrt{2}} \xi ^{0} &{} \xi ^{-} &{} \frac{1}{\sqrt{2}} e_1 \\ \frac{1}{\sqrt{2}} \nu _1 &{} \frac{1}{\sqrt{2}} e_1 &{} E_1 \\ \end{array}\right) _L \sim (1,6,-1/3,-2/3), \end{aligned}$$
(3)
$$\begin{aligned}&\psi _{\alpha L} = (\nu _\alpha \ e_\alpha \ E_\alpha )^T_{L} \sim (1,3,-2/3,-4/3), \end{aligned}$$
(4)
$$\begin{aligned}&Q_{aL} = (d_a\ -u_a\ U_a)^T_{L} \sim (3,3^*,1/3,2/3), \end{aligned}$$
(5)
$$\begin{aligned}&\nu _{aR} \sim (1,1,0,-1),\quad e_{aR} \sim (1,1,-1,-1),\nonumber \\&\quad E_{aR} \sim (1,1,-1,-2), \end{aligned}$$
(6)
$$\begin{aligned}&u_{aR} \sim (3,1,2/3,1/3),\quad d_{aR} \sim (3,1,-1/3,1/3),\nonumber \\&\quad U_{aR} \sim (3,1,2/3,4/3). \end{aligned}$$
(7)

Here \(\alpha =2,3\) and \(a=1,2,3\) are generation indices, the component fields \(\xi \)’s compose a real triplet of \(SU(2)_L\), and the fields EU have the same electric charge as eu, respectively. The above fermion content is free from all the anomalies, including the gravitational anomaly.

To break the gauge symmetry and generate appropriate masses for the particles, the scalar content is introduced as

$$\begin{aligned} \eta&= (\eta ^0_1\ \eta ^-_2\ \eta ^-_3)^T\sim (1,3,- 2/3,- 1/3), \end{aligned}$$
(8)
$$\begin{aligned} \rho&= (\rho ^+_1\ \rho ^0_2\ \rho ^0_3)^T\sim (1,3, 1/3,- 1/3), \end{aligned}$$
(9)
$$\begin{aligned} \chi&= (\chi ^+_1\ \chi ^0_2\ \chi ^0_3)^T\sim (1,3,1/3,2/3), \end{aligned}$$
(10)
$$\begin{aligned} S&= \left( \begin{array}{ccc} S^{++}_{11} &{} \frac{1}{\sqrt{2}}S^{+}_{12} &{} \frac{1}{\sqrt{2}}S^+_{13} \\ \frac{1}{\sqrt{2}}S^+_{12} &{} S^0_{22} &{} \frac{1}{\sqrt{2}} S^0_{23}\\ \frac{1}{\sqrt{2}}S^+_{13} &{} \frac{1}{\sqrt{2}} S^0_{23} &{} S^0_{33}\\ \end{array}\right) \sim (1,6, 2/3,4/3),\end{aligned}$$
(11)
$$\begin{aligned} \phi&\sim (1,1,0,2), \end{aligned}$$
(12)

which have the corresponding vacuum expectation values (VEVs) to be

$$\begin{aligned} \langle \eta \rangle&= \frac{1}{\sqrt{2}} \left( \begin{array}{c} u\\ 0\\ 0\\ \end{array} \right) , \quad \langle \rho \rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} 0\\ v\\ 0\\ \end{array} \right) , \quad \langle \chi \rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} 0\\ 0\\ w\\ \end{array} \right) , \end{aligned}$$
(13)
$$\begin{aligned} \langle S\rangle&= \frac{1}{\sqrt{2}} \left( \begin{array}{ccc} 0 &{} 0 &{} 0\\ 0 &{} \kappa &{} 0\\ 0 &{} 0 &{} \Lambda \\ \end{array}\right) , \quad \langle \phi \rangle =\frac{1}{\sqrt{2}}\Delta , \end{aligned}$$
(14)

where we assume \(\kappa \ll u,v\ll w,\Lambda \ll \Delta \). Notice that the other neutral scalars are \(W_P\)-odd, hence possessing vanished VEV due to the \(W_P\) conservation, as shown below.

The gauge symmetry is broken via three steps,

$$\begin{aligned} \begin{array}{c} SU(3)_C\otimes SU(3)_L\otimes U(1)_X \otimes U(1)_N\\ \\ \downarrow \Delta \\ \\ SU(3)_C\otimes SU(3)_L \otimes U(1)_X\otimes W'_P\\ \\ \downarrow w,\Lambda \\ \\ SU(3)_C\otimes SU(2)_L \otimes U(1)_Y\otimes W_P\\ \\ \downarrow u,v,\kappa \\ \\ SU(3)_C\otimes U(1)_Q\otimes W_P \end{array} \end{aligned}$$

Here \(W'_P\) is a residual gauge symmetry of \(U(1)_N\) which is continuously broken along with \(SU(3)_L\) defining a final residual symmetry \(W_P=(-1)^{3(B-L)}\) with \(B-L=(2/\sqrt{3})T_8+N\) [12, 13]. When including the spin symmetry \((-1)^{2s}\), we have

$$\begin{aligned} W_P=(-1)^{3(B-L)+2s}. \end{aligned}$$
(15)

This symmetry divides the model particles into two classes, as displayed in Table 1, where the new fermions and some other particles are odd under \(W_P\), while all the remaining particles, including the SM ones, are even under this symmetry. Since \(W_P\) is conserved, the odd particles are only coupled in pairs in interactions and the lightest odd particle is stabilized and thus can be a DM candidate if it is electrically neutral and colorless.

Table 1 The residual gauge symmetry \(W_P\) separates the model particles into two classes: even particles according to \(W_P=1\) and odd particles according to \(W_P=-1\)

The total Lagrangian consists of

$$\begin{aligned} {\mathcal {L}}={\mathcal {L}}_{\mathrm{kinetic}}+{\mathcal {L}}_{\mathrm{Yukawa}}-V. \end{aligned}$$
(16)

The first term composes kinetic terms and gauge interactions,

$$\begin{aligned} {\mathcal {L}}_{\mathrm{kinetic}}&= \sum _F i\bar{F}\gamma ^\mu D_\mu F +\sum _\Phi (D^\mu \Phi )^\dag (D_\mu \Phi )\nonumber \\&\quad - \frac{1}{4} G_{i\mu \nu }G_i^{\mu \nu }- \frac{1}{4} A_{i\mu \nu }A_i^{\mu \nu } -\frac{1}{4} B_{\mu \nu }B^{\mu \nu }- \frac{1}{4} C_{\mu \nu }C^{\mu \nu }, \end{aligned}$$
(17)

where F and \(\Phi \) run over all the fermion and scalar multiplets, respectively. Additionally, the covariant derivative and the field strength tensors are determined by

$$\begin{aligned} D_\mu&= \partial _\mu + ig_s t_i G_{i\mu } + ig T_i A_{i\mu } + ig_X X B_\mu + ig_N N C_\mu , \end{aligned}$$
(18)
$$\begin{aligned} G_{i\mu \nu }&= \partial _\mu G_{i\nu } - \partial _\nu G_{i\mu } - g_s f_{ijk} G_{j\mu }G_{k\nu }, \end{aligned}$$
(19)
$$\begin{aligned} A_{i\mu \nu }&= \partial _\mu A_{i\nu } - \partial _\nu A_{i\mu } - g f_{ijk} A_{j\mu }A_{k\nu }, \end{aligned}$$
(20)
$$\begin{aligned} B_{\mu \nu }&= \partial _\mu B_\nu - \partial _\nu B_\mu ,\quad C_{\mu \nu } = \partial _\mu C_\nu - \partial _\nu C_\mu , \end{aligned}$$
(21)

where we denote \((g_s,g,g_X,g_N)\), \((t_i,T_i,X,N)\), and \((G_i,A_i,B,C)\) to be the coupling constants, the generators, and the gauge bosons corresponding to the 3-3-1-1 subgroups, respectively, and \(f_{ijk}\) are SU(3) structure constants.

The second term of (16) contains Yukawa interactions,

$$\begin{aligned} {\mathcal {L}}_{\mathrm{Yukawa}}&= h^e_{\alpha a} \bar{\psi }_{\alpha L}\rho e_{aR} + h^E_{\alpha a} \bar{\psi }_{\alpha L}\chi E_{aR} \nonumber \\&\quad + h^E_{1a} \bar{\psi }_{1L}SE_{aR} + h^\xi \bar{\psi }^c_{1L}\psi _{1L}S \nonumber \\&\quad + h^u_{ab} \bar{Q}_{aL}\rho ^* u_{bR} + h^d_{ab} \bar{Q}_{aL}\eta ^* d_{bR}\nonumber \\&\quad + h^U_{ab} \bar{Q}_{aL}\chi ^* U_{bR}\nonumber \\&\quad + h^\nu _{\alpha b} \bar{\psi }_{\alpha L}\eta \nu _{bR} +h^R_{ab}\bar{\nu }^c_{aR}\nu _{bR}\phi + H.c., \end{aligned}$$
(22)

which were previously presented in [10]. There, all the fermion masses were properly produced, so we do not refer to them hereafter.

The last term of (16) is the scalar potential, taking the form

$$\begin{aligned} V =V (\eta ,\rho ,\chi ,S)+V(\phi ,{\mathrm {mix}}), \end{aligned}$$
(23)

where

$$\begin{aligned}&V(\eta ,\rho ,\chi ,S)\nonumber \\&\quad =\mu _1^2 \eta ^\dag \eta +\mu _2^2 \rho ^\dag \rho +\mu _3^2 \chi ^\dag \chi +\mu _4^2 \mathrm{Tr}(S^\dag S)\nonumber \\&\qquad +\lambda _1(\eta ^\dag \eta )^2+\lambda _2(\rho ^\dag \rho )^2+\lambda _3 (\chi ^\dag \chi )^2\nonumber \\&\qquad +\lambda _4\mathrm{Tr}^2(S^\dag S)+\lambda _5\mathrm{Tr}(S^\dag S)^2\nonumber \\&\qquad +\lambda _6(\eta ^\dag \eta )(\rho ^\dag \rho )+(\lambda _7\eta ^\dag \eta +\lambda _8\rho ^\dag \rho )(\chi ^\dag \chi )\nonumber \\&\qquad +(\lambda _9\eta ^\dag \eta +\lambda _{10}\rho ^\dag \rho +\lambda _{11}\chi ^\dag \chi )\mathrm{Tr}(S^\dag S)\nonumber \\&\qquad +\lambda _{12}(\eta ^\dag \rho )(\rho ^\dag \eta ) +\lambda _{13}(\eta ^\dag \chi )(\chi ^\dag \eta )+\lambda _{14} (\chi ^\dag \rho )(\rho ^\dag \chi )\nonumber \\&\qquad +\lambda _{15}(\chi ^\dag S)(S^\dag \chi )+\lambda _{16}(\eta ^\dag S) (S^\dag \eta )+\lambda _{17}(\rho ^\dag S)(S^\dag \rho )\nonumber \\&\qquad +[\lambda _{18} \eta \rho (S\chi ^*)+f_1 \chi ^T S^\dag \chi +f_2 \eta \rho \chi + {\mathrm {H.c}}.], \end{aligned}$$
(24)
$$\begin{aligned}&V(\phi ,{\mathrm {mix}})\nonumber \\&\quad =\mu _5^2 \phi ^\dag \phi +\lambda _{19}(\phi ^\dag \phi )^2+[\lambda _{20}\eta ^\dag \eta +\lambda _{21}\rho ^\dag \rho +\lambda _{22}\chi ^\dag \chi \nonumber \\&\qquad +\lambda _{23}\mathrm{Tr}(S^\dag S)](\phi ^\dag \phi ) +(\lambda _{24}\rho ^T S^\dag \rho \phi + {\mathrm {H.c}}.), \end{aligned}$$
(25)

where the parameters \(\mu \)’s and \(f_{1,2}\) have mass dimension, while the couplings \(\lambda \)’s are dimensionless. Furthermore, the necessary conditions for the scalar potential bounded from below and producing the gauge symmetry breaking are \(\mu ^2_{1,2,3,4,5}<0\), \(\lambda _{1,2,3,19}>0\), and \(\lambda _4+\lambda _5 >0\).

3 Scalar and gauge sectors

Because of the condition \(\Delta \gg w, V\), the field \(\phi \) can be integrated out from the low-energy effective potential of \(\eta ,\rho ,\chi \), and S. As a result, the scalar potential below \(\Delta \) has a form similar to \(V(\eta ,\rho ,\chi ,S)\) where its couplings become effective due to the modification of heavy particles. Indeed, let us expand \(\phi =\frac{1}{\sqrt{2}} (\Delta +H_C+iG_{C})\), where \(H_C\) and \(G_{C}\) are the new Higgs and Goldstone bosons associate to \(U(1)_N\), respectively. The masses of the new Higgs \(H_C\) and the new gauge C are proportional to \(\Delta \), \(m_{H_C}\simeq \sqrt{2\lambda _{19}}\Delta \) and \(m_{C}\simeq 2 g_N \Delta \), which are decoupled from the low energy particle spectra. Hence, we will neglect the \(U(1)_N\) sector.

Now, we consider the scalar potential (24). To obtain the potential minimum and physical scalar spectrum, we expand the neutral scalar fields around the VEVs as

$$\begin{aligned}&\eta =\left( \begin{array}{c} \frac{1}{\sqrt{2}}(u+S_1+iA_1)\\ \eta ^-_2\\ \eta ^-_3\\ \end{array} \right) ,\quad \rho =\left( \begin{array}{c} \rho ^+_1\\ \frac{1}{\sqrt{2}}(v+S_2+iA_2)\\ \frac{1}{\sqrt{2}}(S_3^\prime +iA_3^\prime )\\ \end{array}\right) , \end{aligned}$$
(26)
$$\begin{aligned}&\chi = \left( \begin{array}{c} \chi ^+_1\\ \frac{1}{\sqrt{2}}(S_2^\prime +iA_2^\prime )\\ \frac{1}{\sqrt{2}}(w+S_3+iA_3)\\ \end{array} \right) ,\nonumber \\&S=\left( \begin{array}{c@{\quad }c@{\quad }c} S^{++}_{11} &{} \frac{1}{\sqrt{2}}S^{+}_{12} &{} \frac{1}{\sqrt{2}}S^+_{13} \\ \frac{1}{\sqrt{2}}S^+_{12} &{} \frac{1}{\sqrt{2}}(S_4+iA_4) &{} \frac{1}{2}(S_1^\prime +iA_1^\prime )\\ \frac{1}{\sqrt{2}}S^+_{13} &{}\frac{1}{2}(S_1^\prime +iA_1^\prime ) &{} \frac{1}{\sqrt{2}}(\Lambda +S_5+iA_5) \\ \end{array}\right) , \end{aligned}$$
(27)

where the fields \(\rho ^0_3\), \(\chi ^0_2\), and \(S^0_{23}\) are odd under \(W_P\), cannot develop VEVs, as mentioned. Additionally, since \(\kappa \) constrained by the \(\rho \)-parameter is tiny, its contribution would be neglected.

Expanding all the terms of the potential (24) up to the second-order contributions of physical scalar fields, the scalar potential is resulted as \(V(\eta ,\rho ,\chi ,S)=V_{\mathrm{min}}+V_{\mathrm{linear}}+V_{\mathrm{mass}}+V_{\mathrm{interaction}}\), where the first term is the potential minimum independent of fields,

$$\begin{aligned} V_{\mathrm{min}}&=\frac{1}{4}[2(\mu _1^2 u^2+\mu _2^2 v^2 +\mu _3^2 w^2+\mu _4^2 \Lambda ^2)\nonumber \\&\quad +\lambda _1 u^4+\lambda _2 v^4 +\lambda _3 w^4+(\lambda _4+\lambda _5)\Lambda ^4\nonumber \\&\quad +\lambda _{6} u^2v^2+\lambda _{7} w^2u^2+\lambda _{8} w^2v^2\nonumber \\&\quad +(\lambda _9 u^2+\lambda _{10}v^2+\lambda _{11}w^2) \Lambda ^2+\lambda _{15} w^2\Lambda ^2\nonumber \\&\quad +2\bar{\Lambda } wuv+2\sqrt{2} f_1 w^2\Lambda ], \end{aligned}$$
(28)

where \(\bar{\Lambda }=\sqrt{2} f_2+\lambda _{18}\Lambda \). The second term \(V_{\mathrm{linear}}\) contains all the linear terms in fields, should be vanished due to the gauge invariance, leading to

$$\begin{aligned}&[2(\mu _1^2+\lambda _1 u^2)+\lambda _6 v^2 +\lambda _7 w^2 +\lambda _9 \Lambda ^2]u+\bar{\Lambda } wv =0, \end{aligned}$$
(29)
$$\begin{aligned}&[2(\mu _2^2+\lambda _2 v^2)+\lambda _6 u^2 +\lambda _8 w^2 +\lambda _{10} \Lambda ^2]v+\bar{\Lambda }wu =0, \end{aligned}$$
(30)
$$\begin{aligned}&[2(\mu _3^2+\lambda _3 w^2)+\lambda _7 u^2 +\lambda _8 v^2 +(\lambda _{11}+\lambda _{15}) \Lambda ^2]w\nonumber \\&\quad +\bar{\Lambda }uv+2\sqrt{2} f_1w \Lambda =0, \end{aligned}$$
(31)
$$\begin{aligned}&[2(\mu _4^2+\lambda _4\Lambda ^2+\lambda _5\Lambda ^2)+\lambda _9 u^2 +\lambda _{10} v^2 +(\lambda _{11}+\lambda _{15}) w^2]\Lambda \nonumber \\&\quad +\Omega w=0, \end{aligned}$$
(32)

where \(\Omega =\sqrt{2} f_1 w+\lambda _{18} uv\). These minimization conditions yield the VEVs \(u,v,w,\Lambda \) as desirable.

The mass part \(V_{\mathrm{mass}}\) consists of the quadratic terms in fields, grouped into \(V_{\mathrm{mass}}=V_{\mathrm{mass}}^{S}+V_{\mathrm{mass}}^{A}+V_{\mathrm{mass}}^{\mathrm{charged}}+V_{\mathrm{mass}}^{S^\prime }+V_{\mathrm{mass}}^{A^\prime }\). Here the charged term includes charged scalars, while the remaining terms describe CP-even and CP-odd scalar fields, and notice that the primed fields are decoupled from the normal fields due to the \(W_P\) conservation. That said, after integrating \(\phi \) out, the scalar potential (24) gives a mass spectrum of the scalar sector including 22 massive Higgs fields, summarized as \(H^0_{1,2,3,4}\), \(H^{0,0*}_5\), \({\mathcal {A}}^0_{1,2}\), \(H_{11}^{\pm \pm }\), \(H_{12}^\pm \), \(H^\pm _1\), \({\mathcal {H}}'^{\pm }_{1,2}\), and \(H'^{0,0*}_{1,2}\), where \(H_1^0\) is identified as the SM-like Higgs boson with the mass in the weak scale, while the others are new Higgs bosons with the masses at \(w,\Lambda \) scales. Additionally, there are 8 Goldstone bosons, determined as \(G_Z\), \(G_{Z'}\), \(G_W^\pm \), \(G_X^\pm \), and \(G_Y^{0,0*}\), correspondingly eaten by 8 massive gauge bosons.

The mentioned Higgs and Goldstone bosons are related to those in the gauge basis, such as

$$\begin{aligned}&\left( \begin{array}{c} S_1 \\ S_2 \\ \end{array}\right) = \left( \begin{array}{cc} c_{\alpha _1} &{} s_{\alpha _1} \\ s_{\alpha _1} &{} -c_{\alpha _1} \\ \end{array}\right) \left( \begin{array}{c} H_1 \\ H_2 \\ \end{array}\right) ,\quad \left( \begin{array}{c} S_3 \\ S_5 \\ \end{array}\right) = \left( \begin{array}{cc} c_{\alpha _2} &{} s_{\alpha _2} \\ -s_{\alpha _2} &{} c_{\alpha _2}\\ \end{array} \right) \left( \begin{array}{c} H_3 \\ H_4 \\ \end{array}\right) ,\nonumber \\&\left( \begin{array}{c} A_1 \\ A_2 \\ A_3 \\ A_5 \\ \end{array}\right) \simeq \left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} c_{\alpha _1} &{} \frac{s_{\alpha _1}c_{\alpha _1}v}{\sqrt{w^2+4\Lambda ^2}} &{} -\sqrt{\frac{w^2+4\Lambda ^2}{u^2+v^2}}s_{\alpha _1} s_{\epsilon _1} &{} \sqrt{\frac{w^2+4\Lambda ^2}{u^2+v^2}}s_{\alpha _1} c_{\epsilon _1} \\ -s_{\alpha _1} &{} \frac{s_{\alpha _1}c_{\alpha _1}u}{\sqrt{w^2+4\Lambda ^2}} &{} -\sqrt{\frac{w^2+4\Lambda ^2}{u^2+v^2}}c_{\alpha _1}s_{\epsilon _1} &{} \sqrt{\frac{w^2+4\Lambda ^2}{u^2+v^2}}c_{\alpha _1} c_{\epsilon _1} \\ 0 &{} -s_{\alpha _3} &{} c_{\epsilon _1}c_{\alpha _3}-s_{\epsilon _1}s_{\alpha _3}s_{\alpha _1}c_{\alpha _1} &{} c_{\epsilon _1}s_{\alpha _3}s_{\alpha _1}c_{\alpha _1}+s_{\epsilon _1}c_{\alpha _3}\\ 0&{} -c_{\alpha _3} &{} -c_{\epsilon _1}s_{\alpha _3}-s_{\epsilon _1}c_{\alpha _3}s_{\alpha _1}c_{\alpha _1} &{} c_{\epsilon _1}c_{\alpha _3}s_{\alpha _1}c_{\alpha _1}-s_{\epsilon _1}s_{\alpha _3} \\ \end{array}\right) \left( \begin{array}{c} G_Z \\ G_{Z'} \\ {\mathcal {A}}_1 \\ {\mathcal {A}}_2 \\ \end{array}\right) , \nonumber \\&\left( \begin{array}{c} \eta _2^\pm \\ \rho _1^\pm \\ \end{array}\right) \simeq \left( \begin{array}{c@{\quad }c} c_{\alpha _1} &{} s_{\alpha _1} \\ -s_{\alpha _1} &{} c_{\alpha _1}\\ \end{array}\right) \left( \begin{array}{c} G_W^\pm \\ H_1^\pm \\ \end{array}\right) , \nonumber \\&\left( \begin{array}{c} \eta _3^\pm \\ \chi _1^\pm \\ S_{13}^\pm \\ \end{array}\right) \simeq \left( \begin{array}{c@{\quad }c@{\quad }c} \frac{u}{\sqrt{w^2+2\Lambda ^2}} &{} c_{\epsilon _2} &{} s_{\epsilon _2} \\ -s_{\alpha _4}&{} \frac{s_{\alpha _4}c_{\epsilon _2}u}{\sqrt{w^2+2\Lambda ^2}} - c_{\alpha _4}s_{\epsilon _2} &{} c_{\alpha _4}c_{\epsilon _2} \\ -c_{\alpha _4} &{} \frac{c_{\alpha _4}c_{\epsilon _2}u}{\sqrt{w^2+2\Lambda ^2}} + s_{\alpha _4}s_{\epsilon _2} &{} - s_{\alpha _4}c_{\epsilon _2}\\ \end{array}\right) \left( \begin{array}{c} G_X^\pm \\ {\mathcal {H}}'^{\pm }_1 \\ {\mathcal {H}}'^{\pm }_2 \\ \end{array}\right) , \nonumber \\&\left( \begin{array}{c} \chi _2^0 \\ \rho _3^{0*} \\ S_{23}^0 \\ \end{array}\right) \simeq \left( \begin{array}{c@{\quad }c@{\quad }c} s_{\alpha _4} &{} c_{\alpha _4}s_{\epsilon _3}+\frac{v}{w}c_{\epsilon _3} &{} -c_{\alpha _4} c_{\epsilon _3} \\ -\frac{v}{\sqrt{w^2+2\Lambda ^2}} &{} c_{\epsilon _3} &{} s_{\epsilon _3}+\frac{v}{w}c_{\alpha _4}c_{\epsilon _3}\\ c_{\alpha _4} &{} -s_{\alpha _4}s_{\epsilon _3} &{} s_{\alpha _4} c_{\epsilon _3} \\ \end{array}\right) \left( \begin{array}{c} G_Y^0 \\ H'^{0}_1 \\ H'^{0}_2 \\ \end{array}\right) ,\nonumber \\&S_{22}^0 \equiv H_5^0,\quad S_{12}^\pm \equiv H_{12}^\pm , \quad S_{11}^{\pm \pm } \equiv H_{11}^{\pm \pm }, \end{aligned}$$
(33)

where we define \(s_{\alpha _1}\equiv \sin \alpha _1\), \(c_{\alpha _1}\equiv \cos \alpha _1\), \(\tan \alpha _1\equiv t_{\alpha _1}=v/u\), and

$$\begin{aligned} t_{\alpha _2}&= \frac{4[(\lambda _{11}+\lambda _{15})\Lambda +\sqrt{2} f_1]w\Lambda }{4(\lambda _4+\lambda _5)\Lambda ^3-(4\lambda _3\Lambda +\sqrt{2} f_1)w^2},\nonumber \\ t_{\alpha _3}&= \frac{w}{2\Lambda },\quad t_{\alpha _4} =\frac{w}{\sqrt{2}\Lambda },\nonumber \\ t_{2\epsilon _1}&\simeq \frac{2[\lambda _{18}(w^2+2\Lambda ^2)-2\sqrt{2} f_2\Lambda ]\Lambda uv \sqrt{u^2+v^2}}{[\sqrt{2} f_1(w^2+4\Lambda ^2)uv-\bar{\Lambda }w \Lambda (u^2+v^2)]\sqrt{w^2+4\Lambda ^2}},\nonumber \\ t_{2\epsilon _2}&\simeq \frac{2\sqrt{2}[(2\lambda _{13}-\lambda _{16})w\Lambda u +(\lambda _{18}w^2-2\sqrt{2} f_2\Lambda )v]\Lambda u}{\sqrt{w^2+2\Lambda ^2} \{[(2\lambda _{13}+\lambda _{15})w^2+2(\lambda _{15}+\lambda _{16})\Lambda ^2]\Lambda u -2\bar{\Lambda }w\Lambda v+2\sqrt{2} f_1(w^2+2\Lambda ^2)u\}},\nonumber \\ t_{2\epsilon _3}&\simeq \frac{-2\sqrt{2}[(\lambda _{15}+\lambda _{17})\Lambda v -\lambda _{18}wu+2\sqrt{2} f_1v]\Lambda v \sqrt{w^2+2\Lambda ^2}}{w[(2\lambda _{14} +\lambda _{15})w^2\Lambda v+2(\lambda _{15}+\lambda _{17})\Lambda ^3v-2\bar{\Lambda }w\Lambda u +2\sqrt{2} f_1(w^2+2\Lambda ^2)v]}. \end{aligned}$$
(34)

The masses of the mentioned Higgs fields (the masses of the mentioned Goldstone bosons vanish, thus unlisted) are given by

$$\begin{aligned} m^2_{H_1}&\simeq \frac{2}{u^2+v^2}(\lambda _1u^4+\lambda _2v^4+\lambda _6u^2v^2),\nonumber \\ m^2_{H_2}&\simeq -\frac{1}{s_{2\alpha _1}}(\sqrt{2} f_2+\lambda _{18}\Lambda )w, \end{aligned}$$
(35)
$$\begin{aligned} m^2_{H_{3,4}}&\simeq \left( \lambda _3-\frac{f_1}{2\sqrt{2} \Lambda }\right) w^2 +(\lambda _4+\lambda _5)\Lambda ^2\nonumber \\&\quad \mp \sqrt{[(\lambda _{11}+\lambda _{15})\Lambda +\sqrt{2} f_1]^2w^2 +\left[ (\lambda _4+\lambda _5)\Lambda ^2-\left( \lambda _3+\frac{f_1}{2\sqrt{2} \Lambda }\right) w^2\right] ^2}, \end{aligned}$$
(36)
$$\begin{aligned} m^2_{H_5}&=-\frac{1}{2\Lambda }\left[ (2\lambda _5\Lambda ^2+\lambda _{15}w^2-\lambda _{17}v^2)\Lambda \right. \nonumber \\&\quad \left. +(\sqrt{2} f_1 w +\lambda _{18}uv) w\right] , \end{aligned}$$
(37)
$$\begin{aligned} m^2_{{\mathcal {A}}_1}&\simeq -\frac{2\sqrt{2}}{c^2_{\alpha _3}} f_1\Lambda , \quad m^2_{{\mathcal {A}}_2}\simeq -\frac{1}{s_{2\alpha _1}}(\sqrt{2} f_2 + \lambda _{18}\Lambda )w, \end{aligned}$$
(38)
$$\begin{aligned} m^2_{H_{11}^{\pm \pm }}&=-\frac{1}{2\Lambda } \left[ (2\lambda _5\Lambda ^2+\lambda _{15}w^2-\lambda _{16}u^2)\Lambda \right. \nonumber \\&\quad \left. +(\sqrt{2} f_1 w +\lambda _{18}uv) w\right] , \end{aligned}$$
(39)
$$\begin{aligned} m^2_{H_{12}^{\pm }}&=-\frac{1}{4\Lambda }\left[ (4\lambda _5\Lambda ^2+2\lambda _{15}w^2 -\lambda _{16}u^2-\lambda _{17}v^2)\Lambda \right. \nonumber \\&\quad \left. +2(\sqrt{2} f_1 w +\lambda _{18}uv) w\right] , \end{aligned}$$
(40)
$$\begin{aligned} m^2_{H_1^\pm }&=\frac{1}{s_{2\alpha _1}}\left[ \lambda _{12}uv -(\sqrt{2} f_2 + \lambda _{18}\Lambda )w\right] , \end{aligned}$$
(41)
$$\begin{aligned} m^2_{{\mathcal {H}}'^{\pm }_1}&\simeq \frac{1}{2}\left[ \lambda _{13}w^2+\lambda _{16}\Lambda ^2 -(\sqrt{2} f_2 + \lambda _{18}\Lambda )w t_{\alpha _1}\right] , \end{aligned}$$
(42)
$$\begin{aligned} m^2_{{\mathcal {H}}'^{\pm }_2}&\simeq -\frac{1}{2c^2_{\alpha _4}} (\lambda _{15}\Lambda +2\sqrt{2} f_1)\Lambda , \end{aligned}$$
(43)
$$\begin{aligned} m_{H'_1}^2&\simeq \frac{1}{2}\left[ \lambda _{14}w^2+\lambda _{17}\Lambda ^2 -\frac{u}{v}(\sqrt{2} f_2 + \lambda _{18}\Lambda )w\right] , \end{aligned}$$
(44)
$$\begin{aligned} m_{H'_2}^2&\simeq -\frac{1}{2c^2_{\alpha _4}}(\lambda _{15}\Lambda +2\sqrt{2} f_1)\Lambda . \end{aligned}$$
(45)

For the gauge boson sector, recall that the \(U(1)_N\) gauge boson is heavy, which is integrated out as \(\phi \) is. Hence, the spectrum of the remaining gauge bosons is identical to those obtained in [10]. That said, we have two new non-Hermitian gauge bosons XY with the masses at the \(w, \Lambda \) scales, besides the W boson of the SM, as follows

$$\begin{aligned}&W^\pm _\mu = \frac{1}{\sqrt{2}} (A_{1\mu }\mp i A_{2\mu }), \nonumber \\&X^\pm _\mu = \frac{1}{\sqrt{2}} (A_{4\mu }\mp i A_{5\mu }), \nonumber \\&Y^{0,0*}_\mu = \frac{1}{\sqrt{2}} (A_{6\mu }\mp i A_{7\mu }), \end{aligned}$$
(46)
$$\begin{aligned}&m^2_W \simeq \frac{g^2}{4}(u^2+v^2), \nonumber \\&m^2_X = \frac{g^2}{4}(u^2+w^2+2\Lambda ^2), \nonumber \\&m^2_Y \simeq \frac{g^2}{4}(v^2+w^2+2\Lambda ^2). \end{aligned}$$
(47)

The mass of W implies \(u^2+v^2\simeq (246\ \hbox {GeV})^2\). For the neutral gauge sector, the physical fields are related to the gauge fields as \((A_{3\mu } \ A_{8\mu } \ B_{\mu })^T = U (A_{\mu } \ Z_{\mu } \ Z'_{\mu })^T\), with

$$\begin{aligned} U = \left( \begin{array}{c@{\quad }c@{\quad }c} s_W &{} c_W c_\varphi &{} c_W s_\varphi \\ \frac{1}{\sqrt{3}}s_W &{} - \frac{1}{\sqrt{3}} s_Wt_Wc_\varphi - \sqrt{1-\frac{1}{3} t^2_W}s_\varphi &{} -\frac{1}{\sqrt{3}} s_Wt_Ws_\varphi + \sqrt{1-\frac{1}{3} t^2_W}c_\varphi \\ \sqrt{1-\frac{4}{3} s^2_W} &{} \frac{t_W}{\sqrt{3}}(s_\varphi - \sqrt{3-4s^2_W}c_\varphi ) &{} -\frac{t_W}{\sqrt{3}}(c_\varphi + \sqrt{3-4s^2_W}s_\varphi ) \end{array}\right) , \end{aligned}$$
(48)

where \(s_W = \sqrt{3} t_X/\sqrt{3+4t^2_X}\), \(t_X=g_X/g\), and

$$\begin{aligned} t_{2\varphi }\simeq \frac{\sqrt{3-t^2_W}}{2c^3_W}\frac{u^2-v^2c_{2W}}{w^2+4\Lambda ^2} \end{aligned}$$
(49)

defines the Z-\(Z'\) mixing angle, \(\varphi \). Here A is the massless photon field, while Z is the SM neutral weak boson with mass \(m_Z\simeq m_W/c_W\). \(Z'\) is the new neutral gauge boson, obtaining a mass at \(w,\Lambda \) scale, such as

$$\begin{aligned} m^2_{Z'}\simeq \frac{g^2}{3-t^2_W}(w^2+4\Lambda ^2). \end{aligned}$$
(50)

The lower bound of \(w,\Lambda \) is given by the rho parameter constraint to be \(\sqrt{w^2+4\Lambda ^2}\sim 5{-}7~{\mathrm {TeV}}\) dependent on uv relation, as well as the LHC search for \(Z'\) through dilepton products makes a constraint on \(Z'\) mass, \(m_{Z'}> 2.25\)–2.8 TeV, which translates to

$$\begin{aligned} \sqrt{w^2+4\Lambda ^2}=m_{Z'}\sqrt{(3-t^2_W)/g}> 4.5{-}6~{\mathrm {TeV}}, \end{aligned}$$
(51)

in agreement to the rho parameter [10]. We made a study deduced from the LEPII bound for the process \(e^+e^-\rightarrow \mu ^+\mu ^-\) via \(Z'\) exchange [14] which reveals \(m_{Z'}> 2.7\ \hbox {TeV}\), implying a similar bound for \(w,\Lambda \).

4 Relevant interactions

4.1 Fermion and gauge boson interactions

The gauge interactions of fermions arise from,

$$\begin{aligned} {\mathcal {L}}_f = \sum _F i\bar{F}\gamma ^\mu D_\mu F. \end{aligned}$$
(52)

Using the result in (46), we get the charged current interactions of W, X, and Y to be

$$\begin{aligned} {\mathcal {L}}^C_f = J^{-\mu }_W W^+_\mu + J^{-\mu }_X X^+_\mu + J^{0*\mu }_Y Y^0_\mu + H.c., \end{aligned}$$
(53)

where the currents are given by

$$\begin{aligned} J ^{-\mu }_W&= -\frac{1}{\sqrt{2}}g (\bar{\nu }_{aL}\gamma ^\mu e_{aL} +\bar{u}_{aL}\gamma ^\mu d_{aL}), \end{aligned}$$
(54)
$$\begin{aligned} J^{-\mu }_X&= -\frac{1}{\sqrt{2}}g (\sqrt{2} \bar{\nu }_{1L}\gamma ^\mu E_{1L} +\bar{\nu }_{\alpha L}\gamma ^\mu E_{\alpha L} - \bar{U}_{aL}\gamma ^\mu d_{aL} \nonumber \\&\quad + \sqrt{2}\bar{\xi }^+_L\gamma ^\mu \nu _{1L} + \bar{\xi }^0_L\gamma ^\mu e_{1L}), \end{aligned}$$
(55)
$$\begin{aligned} J^{0*\mu }_Y&= -\frac{1}{\sqrt{2}}g (\sqrt{2} \bar{e}_{1L}\gamma ^\mu E_{1L} +\bar{e}_{\alpha L}\gamma ^\mu E_{\alpha L} \nonumber \\&\quad + \bar{U}_{aL}\gamma ^\mu u_{aL} + \bar{\xi }^0_L\gamma ^\mu \nu _{1L} +\sqrt{2} \bar{\xi }^-_L\gamma ^\mu e_{1L}). \end{aligned}$$
(56)

Using the result in (48), we get the neutral current interactions for the neutral gauge bosons,

$$\begin{aligned} {\mathcal {L}}^N_f&= -eQ(f)\bar{f}\gamma ^\mu f A_\mu \nonumber \\&\quad - \frac{g}{2c_W}\{\bar{f}\gamma ^\mu [g^Z_V(f)-g^Z_A(f)\gamma _5]f Z_\mu \nonumber \\&\quad +\bar{f} \gamma ^\mu [g^{Z'}_V(f)-g^{Z'}_A(f)\gamma _5]f Z'_\mu \}, \end{aligned}$$
(57)

where \(e=gs_W\) and f refers to every fermion of the model, except for the right-handed neutrinos. The vector and axial-vector couplings of Z are collected in Table 2 as put in Appendix A. Notice that all the interactions between Z boson and ordinary fermions are consistently recovered in the limit \(\varphi \rightarrow 0\).

The couplings of \(Z'\) with fermions can be obtained from those for Z, by replacing

$$\begin{aligned} g^{Z'}_{V,A}(f) = g^{Z}_{V,A}(f) (c_\varphi \rightarrow s_\varphi , s_\varphi \rightarrow -c_\varphi ), \end{aligned}$$
(58)

which need not necessarily to be determined, but pointed out in the Table 2 caption.

Notice that \(\nu _R\)’s have only gauge interaction with the \(U(1)_N\) gauge boson and are obviously integrated out as C is, since they possess a large mass proportional to \(\Delta \) via the coupling \(\nu _R\nu _R\phi \) [10].

Fig. 1
figure 1

Relevant contributions to the scalar DM pair annihilation into SM particles

4.2 Scalar and gauge boson interactions

The gauge interactions of the scalar fields arise from

$$\begin{aligned} {\mathcal {L}}_\Phi = \sum _\Phi (D^\mu \Phi )^\dag (D_\mu \Phi ). \end{aligned}$$
(59)

Substituting the physical fields from (33), (46), and (48) to this Lagrangian, we get the interactions between a gauge boson and two scalars, two gauge bosons and a scalar, and two gauge bosons and two scalars in the model, given in Tables 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12, which are gathered in Appendix A. At the leading order, we have verified that all the interactions between the SM-like Higgs boson and the gauge bosons are consistently recovered. However, only the interactions relevant to the following diagrams are needed, yielding the cross-sections as given.

5 Search for DM

In this model, the two scalars \(H^{\prime 0}_{1,2}\), the fermion \(\xi ^0\), and the gauge boson \(Y^0\), which all have masses at \(w,\Lambda \) scale, are \(W_P\)-odd particles and electrically neutral and colorless. Hence, they may be responsible for the DM candidate. As indicated in [12] and updated in [15], the relic density contribution of the gauge boson \(Y^0\) is small compared to the observed dark matter density, so we do not interpret the vector field as DM. Additionally, if the scalar \(H'^0_2\) that transforms as a \(SU(2)_L\) doublet has a correct relic density to be DM, it should be ruled out by the direct detection experiments due to a large scattering cross-section induced by Z [16, 17]. Therefore, in the following we study the DM phenomenology associated with the candidates, singlet scalar \(H^{\prime 0}_1\) and triplet fermion \(\xi ^0\). The presence of the triplet candidate, a result of the fully flipped, would make dark matter phenomena of the model completely different from the other theories of this type, such as the ordinary 3-3-1 [18, 19], 3-3-1-1 [20,21,22], 3-2-3-1 [23], and flipped trinification [24].

5.1 DM as a singlet scalar

We now consider the DM scenario where \(H'^0_1\) is the lightest particle among all of the \(W_P\)-odd particles. The DM pair annihilation into the SM particles, via the most relevant channels, is described in Fig. 1.

Corresponding to each process, the thermal average annihilation cross-section times the relative velocity is approximately given by

$$\begin{aligned}&\langle \sigma v\rangle _{H'_1H'_1\rightarrow H_1H_1}= \frac{1}{16\pi m^2_{H'_1}} \left\{ 2\lambda _2 s^2_{\alpha _1}+\lambda _6 c^2_{\alpha _1}\right. \nonumber \\&\quad + \frac{2[\sqrt{2} f_2 c_{\alpha _1}c_{\alpha _4}-ws_{\alpha _4}(0.5\lambda _{14} c_{\alpha _4}+\lambda _{18}c_{\alpha _1})]^2}{m^2_{H'_1}+m^2_{H'_2}}\nonumber \\&\quad +\frac{g^2 s^2_{\alpha _1}m^2_{H'_1}}{2(m^2_{H'_1}+m^2_Y)} \left( 5+\frac{m^2_{H'_1}}{m^2_Y}\right) \nonumber \\&\quad -3\left[ (2\lambda _1 -\lambda _6) v c^3_{\alpha _1} -(2\lambda _2-\lambda _6) u s^3_{\alpha _1}\right] \frac{\lambda _6 u s_{\alpha _1}-2\lambda _2 v c_{\alpha _1}}{4m^2_{H'_1}-m^2_{H_2}}\nonumber \\&\quad - \left[ (\lambda _7 c^2_{\alpha _1}+\lambda _8 s^2_{\alpha _1})w c_{\alpha _2}-(\lambda _9 c^2_{\alpha _1} +\lambda _{10} s^2_{\alpha _1})\Lambda s_{\alpha _2}\right. \nonumber \\&\quad \left. + s_{\alpha _1}c_{\alpha _1} (\lambda _{18}\Lambda c_{\alpha _2}-\lambda _{18}w s_{\alpha _2}+\sqrt{2} f_2 c_{\alpha _2})\right] \nonumber \\&\quad \times \frac{(\lambda _8+\lambda _{14})wc_{\alpha _2}-(\lambda _{10}+\lambda _{17}) \Lambda s_{\alpha _2}}{4 m^2_{H'_1}-m^2_{H_3}}\nonumber \\&\quad \left. + (m_{H_3}\leftrightarrow m_{H_4}, c_{\alpha _2} \leftrightarrow s_{\alpha _2}, s_{\alpha _2}\leftrightarrow -c_{\alpha _2})\right\} ^2, \end{aligned}$$
(60)
$$\begin{aligned}&\langle \sigma v\rangle _{H'_1H'_1\rightarrow W^+W^-} = \frac{(2\lambda _2 v^2+\lambda _6 u^2)^2}{8\pi (u^2+v^2)^2m^2_{H'_1}},\nonumber \\&\langle \sigma v\rangle _{H'_1H'_1\rightarrow ZZ} =\frac{(2\lambda _2 v^2+\lambda _6 u^2)^2}{16\pi (u^2+v^2)^2m^2_{H'_1}}, \end{aligned}$$
(61)
$$\begin{aligned}&\langle \sigma v\rangle _{H'_1H'_1\rightarrow tt^c}=\frac{3m_t^2}{\pi m^2_{H'_1}(u^2+v^2)^2}\nonumber \\&\quad \times \left[ \frac{(2\lambda _2 v^2+\lambda _6 u^2)^2}{16m^2_{H'_1}} +\frac{(2\lambda _2 -\lambda _6)^2m^2_{H'_1}u^4}{(4m^2_{H'_1}-m^2_{H_2})^2}\right] , \end{aligned}$$
(62)

where the annihilation channels into the SM-like Higgs bosons (\(H_1\)) involving the \(H'_1\), \(H_1\) propagators as well as the last annihilation channel exchanged by \(Z'\) are infinitesimal, as neglected, because of the small coupling or non-relativistic dark matter momentum suppression. Indeed, although the channels exchanged by \(H'_{1,2}\) are the same, the \(H_1 H'_1 H'^*_1\) coupling as proportional to uv is radically smaller than the \(H_1 H'_1 H'^*_2\) coupling as proportional to \(w,f_2\). So the channels exchanged by \(H'_1\) are suppressed as compared to those by \(H'_2\), for which only the \(H'_2\) contribution appears in Eq. (60). Note that the \(H_2 H'_1 H'^*_1\) coupling is also proportional to uv. Hence, for the same reason, the s-channel contributions of \(H_{1,2}\) to the process \(H'_1H'_1\rightarrow H_1 H_1\) are much smaller than those by \(H_{3,4}\). The SM Higgs (\(H_1\)) contribution is straightforwardly prevented, while the \(H_2\) one may become significant, as kept, due to a resonance in the relic density when the DM mass is close to its mass. The amplitude of the \(Z'\)-exchanged diagram is proportional to the DM momentum \(\mathbf {p}\simeq m_{H'_1}\mathbf {v}\) at the \(Z' H'_1 H'^*_1\) vertex, hence strongly-suppressed by \(|\mathbf {v}|\sim 10^{-3}\). Additionally, the DM stability requires \(m_{H'_1}<m_X\simeq m_Y<m_{Z'}\), where the last two comparisons are derived from Eqs. (47) and (50) for \(w,\Lambda \gg u,v\). The \(Z'\) amplitude is also suppressed by \(1/m^2_{Z'}\) through its propagation. With the two suppressions, this contribution is manifestly omitted in comparison to the dominant channels.Footnote 1 Note that Z does not couple to \(H'_1 H'^*_1\) in the effective limit, \((u,v)^2/(w,\Lambda )^2\ll 1\), hence it does not contribute to such channel too. Furthermore, at the effective limit, the \(SU(2)_L\) singlets, \(H_{3,4}\), do not couple to top quarks. They do not contribute to the annihilation channel to \(tt^c\), unlike the \(H_{1,2}\) in the above diagram. This \(H_2\) correction has a size similar to the \(H_2\) contribution in the \(H'_1 H'_1\rightarrow H_1 H_1\) process, as mentioned. On the other hand, since \(H_{2,3,4}\) do not couple to WW and ZZ, as seen in Appendix A, they do not contribute to the processes \(H'_1 H'_1\rightarrow WW(ZZ)\). There might exist t- and/or u-channel diagrams associate to the processes \(H'_1 H'_1\rightarrow WW(ZZ)\), as exchanged by XY, but they are all discarded due to the \((u,v)^2/m^2_{X,Y}\) suppression in comparison to the \(H_1\) contribution. Also, such corrections are significantly smaller than the Y-exchanged diagrams to \(H_1 H_1\), which result in Eq. (60). Finally, the relic density of the \(H'_1\) is \(\Omega _{H'_1}h^2 \simeq 0.1 {\mathrm {pb}}/\langle \sigma v\rangle \), where

$$\begin{aligned} \langle \sigma v\rangle&= \langle \sigma v\rangle _{H'_1H'_1\rightarrow H_1H_1} +\langle \sigma v\rangle _{H'_1H'_1\rightarrow W^+W^-}\nonumber \\&\quad +\langle \sigma v\rangle _{H'_1H'_1\rightarrow ZZ} +\langle \sigma v\rangle _{H'_1H'_1\rightarrow tt^c}. \end{aligned}$$
(63)

To study the direct detection for the \(H'_1\) via the spin-independent (SI) scattering on nuclei, we write the effective Lagrangian describing DM–nucleon interaction in the limit of zero-momentum transfer through the exchange of the Higgs boson \(H_1\) as follows

$$\begin{aligned} {\mathcal {L}}^{\mathrm {eff}}_{H'_1-{{\mathrm {quark}}}} = \frac{2\sqrt{2} m_q}{(u^2+v^2)m^2_{H_1}}(2\lambda _2 v^2+\lambda _6 u^2) H'_1H'_1\bar{q}q. \end{aligned}$$
(64)

The SI cross-section for the scattering of \(H'_1\) on a target nucleus is given by [25]

$$\begin{aligned} \sigma ^{\mathrm {SI}}_{H'_1N} = \left( \frac{2m_{H'_1N}}{m^2_{H_1}} \frac{m_p}{m_{H'_1}} C_N\right) ^2, \end{aligned}$$
(65)

where \(N=p,n\) and \(m_{H'_1N}=m_{H'_1}m_N/(m_{H'_1}+m_N)\simeq m_N\) is the DM-nucleon reduced mass. The nucleus factor, \(C_N\), is

$$\begin{aligned} C_N&= \frac{4\sqrt{2}}{27(u^2+v^2)}(2\lambda _2 v^2+\lambda _6 u^2)\nonumber \\&\quad \times \left\{ 3Af^p_{Tg}+\sum _{q=u,d,s}[Zf^p_{Tq}+(A-Z)f^n_{Tq}]\right\} , \end{aligned}$$
(66)

where Z and A correspond to the nucleus charge and the total number of nucleons in the nucleus, respectively, and [26]

$$\begin{aligned} f^{p(n)}_{Tu}&\simeq 0.020 \ (0.014), \quad f^{p(n)}_{Td}\simeq 0.026 \ (0.036), \end{aligned}$$
(67)
$$\begin{aligned} f^{p(n)}_{Ts}&\simeq 0.118 \ (0.118), \quad f^p_{Tg} = 1-\sum _{q=u,d,s} f^p_{Tq}. \end{aligned}$$
(68)

For numerical computation in this subsection and the next one for DM fermion, we take the following values of known parameters as [27]

$$\begin{aligned}&u = v \simeq 174~{\mathrm {GeV}},\quad s^2_W \simeq 0.231,\quad \alpha \simeq 1/128, \nonumber \\&g=\sqrt{4\pi \alpha }/s_W,\quad m_t \simeq 173.1~{\mathrm {GeV}},\quad m_{H_1} \simeq 125.3~{\mathrm {GeV}},\nonumber \\&\quad A = 131,\quad Z=54,\quad m_N\simeq 1~{\mathrm {GeV}}. \end{aligned}$$
(69)
Fig. 2
figure 2

The relic density of the scalar candidate as a function of the DM mass according to the several choices of \(w,\Lambda \), where the DM unstable region (light red) is the band on the right side of each panel

Since the superheavy \(U(1)_N\) sector is decoupled, the intermediate new physics regime set by the 3-3-1 breaking scales \(w,\Lambda \) is the most relevant to dark matter phenomena. As seen, the scalar dark matter observables are governed by the dark matter mass \(m_{H'_1}\) and the new Higgs portals, given through the new Higgs masses and couplings, that connect the dark sector to the normal sector. Additionally, the masses of all such dark matter and new Higgs particles are proportional to the scalar couplings and the scales \(w,\Lambda \). Hence, we will investigate two alternative benchmark choices of parameters, either fixing \(w,\Lambda \) while changing necessary couplings or varying \(w,\Lambda \) while fixing relevant couplings, which might lead to distinct new physics results.

5.1.1 Fixing \(w,\Lambda \) while varying necessary couplings

When the new physics scales \(w,\Lambda \) are fixed, the dark matter observables depend on the dimensionless (\(\lambda \)’s) and mass-dimension (f’s) scalar couplings. Among these couplings, we choose the two typical parameters, \(f_2\) and \(\lambda _2\), to be varied, while the remainders are fixed as

$$\begin{aligned}&\lambda _6 = -0.2,\quad \lambda _{5,14,17,18} = -0.01,\nonumber \\&\lambda _{3,4,7,8,9,10,11,12,13,15,16} = 0.5,\quad f_1=-w. \end{aligned}$$
(70)

Note that \(\lambda _1\) is related to \(\lambda _2\) via the SM Higgs mass. Although \(\lambda _2\) can be changed, we will choose only three values of it that characterize its viable range, simultaneously making the DM phenomenology suitable, as supplied in the following figures.

In Fig. 2, we depict the relic density as a function of DM mass which all vary upon the variation of \(f_2\) and simultaneously show the DM unstable regime (light red, when \(H'_1\) is not the lightest \(W_P\)-odd field) according to the several selections of the new physics scales w and \(\Lambda \). These selections are consistent with the constraints from the \(\rho \)-parameter, the LHC dilepton, as well as the charged lepton flavor violating processes and nonstandard neutrino interactions, as studied in [10]. Additionally, throughout this paper the choice \(\Lambda \sim w\) would characterize the strength of the \(SU(3)_L\) breaking down to \(SU(2)_L\), as assumed from the model setup. The viable DM mass regime is given below the correct abundance \(\Omega h^2<0.12\) and before the DM unstable regime. We see that each density curve contains three resonances, corresponding to \(m_{H'_1} = m_{H_2}/2\), \(m_{H'_1} = m_{H_3}/2\) and \(m_{H'_1} =m_{H_4}/2\), where the relic density is largely decreased. The variation of \(\lambda _2\) slightly separates the relic density and this is also valid for the whole viable range of \(\lambda _2\sim 0\)–0.7, since \(\lambda _1>0\). When \(w,\Lambda \) are large as in the right panel, the DM mass upper bound is increased but appearing an excluded intermediate region according to \(\Omega h^2>0.12\).

Fig. 3
figure 3

The masses of the new scalar fields and the odd gauge bosons as functions of the parameter \(f_2\) according to the several choices of \(w,\Lambda \), where the dark field lines are put as red and the DM stable regime (i.e., below the light red background) is bounded by the XY line

With the benchmark choices in Eq. (70), the DM mass variation in Fig. 2 is exclusively obtained by varying \(f_2\), as mentioned. But, such variation of \(f_2\) also leads to the changes of the masses of other fields, which are described in Fig. 3. Here we do not explicitly label the lines of the same field type for simplicity. Note also that the XY gauge bosons are almost degenerated in mass. Obviously, all the new scalar fields in the model except the dark matter obtain mass values safely above \({\mathcal {O}}(1)~\hbox {TeV}\). There is a range of \(f_2\) where \(H'_1\) is the lightest \(W_P\)-odd field, set by the XY mass, as imposed to Fig. 2. Note that in the unstable region of Fig. 2, the DM scalar proceeds to decaying into XY bosons through channels, \(H'_1\rightarrow YH_1, YZ, XW\), derived by the corresponding vertices in Appendix A, which would be kinematically suppressed. Last, the dark fermions \(\xi ,E,U\) have masses proportional to the relevant Yukawa couplings in Eq. (22) which can easily be chosen, such that these fermions are heavier than the scalar DM. So we need not refer to this case here.

Fig. 4
figure 4

The SI DM-nucleon scattering cross-section limit as a function of DM mass according to the several choices of \(\lambda _2\), where the excluded region (light red) lies above the experimental (green and yellow) bands

The SI DM-nucleon cross-section (65) depends only on the DM mass \(m_{H'_1}\) and the scalar coupling \(\lambda _2\), where the remaining parameters \(u,v,\lambda _6\) were fixed as mentioned. The choice, change, or variation of all the relevant parameters in \(m_{H'_1}\), such as \(w,\Lambda ,f_2\) and so on, effectively modifies only the DM mass. Hence, although we have several cases for parameter values, it all leads to the cross-section dependence only in terms of \(m_{H'_1},\lambda _2\), yielding a common result. In Fig. 4, we depict the SI scattering cross-section of \(H'_1\) on the nucleon according to the above choices of \(\lambda _2\) as well as show the experimental bounds from XENON1T [28], where the black line is the SI WIMP-nucleon cross-section limit at 90% confidence level, while the green and yellow bands are the \(1\sigma \) and \(2\sigma \) sensitivities, respectively. We see that the scalar DM mass below around 500 GeV is excluded by the direct detection experiment for \(\lambda _2=0.12\). Hence, in this case the viable scalar DM mass regime is around 500 GeV to a few TeV (cf. Fig. 2). However, the SI DM cross-section is quite separated for the variation of \(\lambda _2\). In the latter cases for \(\lambda _2=0.05\) and 0.01, the respective lower bounds of \(H'_1\) mass are above 1 TeV. If \(\lambda _2\) is bigger, i.e. \(\lambda _2>0.12\), the DM mass is close to the weak scale, and in this case the DM would be subject to the electroweak precision test and collider bounds, which is not studied in this work. Additionally, when \(\lambda _2\) is smaller than 0.01, the viable DM mass region is narrow as constrained by the relic density and the unstable regime.

Fig. 5
figure 5

The relic density of the scalar candidate as a function of the DM mass, which all vary according to the variation of w and \(\Lambda \), but with several relations as fixed, where the DM unstable region (light red) is the band on the left side of each panel

Fig. 6
figure 6

The masses of the new scalar fields and the odd gauge bosons as functions of w according to the several relations between w and \(\Lambda \), where the dark field lines are put as red and the DM unstable regime is input as light red background, which is bounded by XY line

Fig. 7
figure 7

Dominant contributions to the fermion DM pair annihilation into the SM particles

Fig. 8
figure 8

The relic density of the fermion candidate as a function of its mass according to the several choices of \(w,\Lambda \), where the DM unstable region is input as light red

5.1.2 Varying \(w,\Lambda \) while fixing relevant couplings

Since the new physics scales w and \(\Lambda \) govern all the DM annihilation channels, it is interesting to recast the above investigation in which w and \(\Lambda \) are varied instead of \(f_2\). For this aim, we choose \(f_2=-4\ \hbox {TeV}\), without loss of generality. As mentioned, \(w,\Lambda \) commonly set the strength of the \(SU(3)_L\) breaking, it is suitably to impose \(w\sim \Lambda \), while both these parameters are simultaneously run from a lower bound as supplied in Eq. (51). We also vary \(\lambda _2\) while fixing the values of the other parameters as in the previous case. The results of the relic density and the mass variations are shown in Figs. 5 and 6, respectively.

As we see from Fig. 6, all the new particle masses significantly change upon the new physics scales, apposite to the previous case as \(f_2\) varies. Additionally, the DM scalar is stabilized if \(w>7.75\) and 4.75 TeV corresponding to the cases in the left and right panels, respectively. This translates to the unstable regions as lower bounds on the DM mass according to the left and right panels of Fig. 5, unlike the previous case. With the choice of the parameters, there are only two resonances in the relic density as seen in Fig. 5 set by the \(H_{3,4}\) masses, i.e. \(m_{H'_1}=\frac{1}{2} m_{H_{3}}\) and \(m_{H'_1}=\frac{1}{2} m_{H_{4}}\), respectively. This can also be realized from Fig. 6 by comparing relevant masses. The splittings in the relic density and the SI DM cross-section according to \(\lambda _2\) are similar to the previous case. Here note that the SI DM cross-section remains the same previous case, since it depends only on the DM mass and \(\lambda _2\).

That said, the viable scalar DM mass regime is now either from 5.25 to 6.85 TeV according to \(\Lambda =1.1 w\) or from 3.65 to 4.2 TeV and from 5.4 to 5.67 TeV according to \(\Lambda =1.5 w\), as bounded by the unstable region and the relic density \(\Omega h^2<0.12\). Such ranges are appropriate to the SI DM cross-section bounds.

5.2 DM as a triplet fermion

The \(\xi \) triplet components have a degenerate mass,

$$\begin{aligned} m_\xi =-\sqrt{2}h^\xi \Lambda , \end{aligned}$$
(71)

at the tree level, as induced by the VEV of the scalar sextet. But, the loop effects of gauge bosons can make \(\xi ^\pm \) mass larger than \(\xi ^0\) mass by an amount,

$$\begin{aligned} m_{\xi ^\pm }-m_{\xi ^0}=166~{\mathrm {MeV}}, \end{aligned}$$
(72)

as shown in [16]. Hence, \(\xi ^0\) is first regarded as the lightest of the triplet components. We further assume that \(\xi ^0\) is the lightest particle among all of the other \(W_P\)-odd particles and thus \(\xi ^0\) is responsible for the DM candidate. This scenario was briefly discussed in [10], in which the field \(\xi ^0\) yielded the correct abundance and satisfied the direct detection bounds, provided that it had a mass \(m_{\xi ^0}\simeq 2.86\ \hbox {TeV}\). Here in the present work, we will explore the full viable mass region of \(\xi ^0\) and investigate physical density resonances which are crucial for the experimental detections.

It is obvious that the DM candidates mainly annihilate to the SM particles. The dominant channels for the fermion DM pair annihilation \(\xi ^0\) to the SM particles are presented in Fig. 7.

The thermal average annihilation cross-section times the relative velocity is approximated as follows

$$\begin{aligned}&\langle \sigma v\rangle _{\xi ^0\xi ^{0c}\rightarrow {\mathrm {SM\ SM}}}\nonumber \\&\quad =\frac{g^4m^2_{\xi ^0}}{\pi c^4_W} \left\{ \left[ g^{Z'}_V(\xi ^0)\right] ^2\left[ \sum _{f} N_C(f)\frac{[g^{Z'}_V(f)]^2+[g^{Z'}_A(f)]^2}{16(4m^2_{\xi ^0}-m^2_{Z'})^2} \right. \right. \nonumber \\&\qquad \left. + \frac{g^2_{Z'ZH_1}c^2_W}{32g^2m^2_Z (4m^2_{\xi ^0}-m^2_{Z'})^2} \right] \nonumber \\&\qquad + \left[ \frac{c^4_W}{(m^2_{\xi ^0}+m^2_Y)^2} -\frac{g^{Z'}_V(\xi ^0) g^{Z'}_V(\nu _1)c^2_W}{2(m^2_{\xi ^0}+m^2_Y) (4m^2_{\xi ^0}-m^2_{Z'})}\right. \nonumber \\&\qquad \left. \left. +(\nu _1\leftrightarrow e_1, m_Y\leftrightarrow m_X)\right] \right\} , \end{aligned}$$
(73)

where f denotes the SM fermions and the masses of the new gauge bosons are assumed to be much larger than the masses of the SM fermions. Note that \(\xi ^0\) can also annihilate to the SM Higgs boson via a s-channel diagram, exchanged by the new Higgs portal, \(H_{3,4}\), because \(H_{3,4}\) couples to \(\xi ^0\) via the coupling \(h^\xi \sim m_\xi /\Lambda \) and to \(H_1\) via a coupling in the form of \(\bar{\lambda }\times (w,\Lambda )\), where \(\bar{\lambda }\) is a combination of the known scalar self-couplings, \(\lambda \)’s. This contribution may become evident with the interesting resonances at \(m_{\xi ^0}=m_{H_{3,4}}/2\) in the relic density, similar to the scalar DM case, as studied above. In the present case, we are most favored in investigating the gauge portals, which are not significant in the scalar DM case. For this aim, the \(H_{3,4}\) contributions to the total annihilation cross-section can directly be suppressed, as omitted, by assuming that \(\bar{\lambda }\) is radically smaller than the gauge coupling constant, g.

Note that \(\xi ^0\) does not interact with Z at the effective limit \(u,v\ll w,\Lambda \). To investigate the direct detection of the field \(\xi ^0\) via the SI scattering on nuclei, we write the effective Lagrangian that describes the interactions of \(\xi ^0\) with fundamental level quarks, induced through the t-channel exchange of the field \(Z'\), such as

$$\begin{aligned} {\mathcal {L}}^{\mathrm{eff}}_{\xi ^0-{\mathrm {quark}}}&=\frac{g^2}{4c^2_W m^2_{Z'}} \bar{\xi }^0\gamma ^\mu g^{Z'}_V(\xi ^0)(1-\gamma _5)\xi ^0\bar{q}\gamma _\mu \nonumber \\&\quad \times [g^{Z'}_V (q)-g^{Z'}_A (q)\gamma _5]q. \end{aligned}$$
(74)

Thus, we obtain the SI cross-section for the scattering of the \(\xi ^0\) on a target nucleus [25],

$$\begin{aligned} \sigma ^{\mathrm{SI}}_{\xi ^0 N}&= \frac{g^4 m^2_{\xi ^0 N}}{16\pi c^4_W m^4_{Z'}} {[}g^{Z'}_V(\xi ^0)]^2 \nonumber \\&\quad \times \left| g^{Z'}_V (u) (Z+A)+g^{Z'}_V (d) (2A-Z)\right| ^2, \end{aligned}$$
(75)

where \(m_{\xi ^0 N}\simeq m_N\).

For numerical computation in this subsection, we take \(m_Z\simeq 91.187\ \hbox {GeV}\) and relevant parameters given in (69). In Fig. 8, we plot the relic density of the DM as a function of its mass according to the several choices of w and \(\Lambda \). We see that each density curve always contain a quite narrow resonance at \(m_{\xi ^0}=m_{Z'}/2\), at which the relic density is substantially reduced, tending to zero. Whereas, outside the resonance region, the relic density slowly increases. Since the gauge couplings are fixed and that the Z-\(Z'\) mixing angle is small, the relic density only depends on \(m_{\xi ^0}\) and \(w,\Lambda \) through \(m_{X,Y,Z'}\). It is proportional to \(m^4_{X,Y,Z'}/m^2_{\xi ^0}\) for \(m_{\xi ^0}\) close to the weak scale, while it is proportional to \(m^2_{\xi ^0}\) for \(m_{\xi ^0}\) much beyond \(m_{X,Y,Z'}\). Because the range of the DM mass considered in Fig. 8 is narrow, the corresponding relic density outside the resonance region is weakly changed as scaled by the resonance mass \(m_{Z'}\). To see a significant change, we make an estimation, \(\Omega h^2=2.8\), 0.7, and 3.75 for \(m_{\xi ^0}=1\), 20, and 50 TeV, respectively, according to the right panel. Whereas, \(\Omega h^2=0.6\) and 3.6 correspond to \(m_{\xi ^0}=20\) and 50 TeV, respectively, according to the left panel. In Fig. 9, we plot the SI scattering cross-section limit as a function of the DM mass according to the above choices. The limits are in good agreement the constraint from XENON1T [28]. Combining the results in Figs. 8 and 9, the viable DM mass regime is as follows: \(m_{\xi ^0}=1.3\)–3.2 TeV for \(w=5~\hbox {TeV}\), \(\Lambda =6~\hbox {TeV}\); \(m_{\xi ^0}=3.5\)–4.4 TeV for \(w=8\ \hbox {TeV}\), \(\Lambda =9\ \hbox {TeV}\); and \(m_{\xi ^0}=4.9\)–5.5 TeV for \(w=11\ \hbox {TeV}\), \(\Lambda =12\ \hbox {TeV}\).

Fig. 9
figure 9

The SI DM-nucleon scattering cross-section limit as a function of DM mass according to the several choices of \(w, \Lambda \), where the excluded region is input as light red

Table 2 The couplings of Z with fermions (\(f\ne \nu _R\)), whereas those for \(Z'\) are obtained by replacement: \(c_\varphi \rightarrow s_\varphi \) and \(s_\varphi \rightarrow -c_\varphi \)
Table 3 The interactions of a neutral gauge boson with two scalars
Table 4 The interactions of a charged gauge boson with two scalars
Table 5 The interactions of a scalar with two gauge bosons
Table 6 The interactions of a neutral and a charged gauge boson with two scalars (Table continued)

Last, but not least, the scalar sextet (S) and triplet (\(\chi \)) are decomposed as

$$\begin{aligned} S\sim 6=3\oplus 2\oplus 1,\quad \chi \sim 3=2\oplus 1, \end{aligned}$$
(76)

under the SM \(SU(2)_L\) symmetry at the effective limit. Here \(S_{33}\) and \(\chi _3\) transform as the \(SU(2)_L\) singlets respectively, hence they can develop the VEVs, \(\langle S_{33}\rangle =\Lambda /\sqrt{2}\) and \(\langle \chi _{3}\rangle =w/\sqrt{2}\), for breaking \(SU(3)_L\) down to \(SU(2)_L\) as conserved. At this stage, \(w,\Lambda \) contribute to all the new particle masses, as seen in the new gauge and Higgs bosons above. Additionally, w gives exotic quark mass, while \(\Lambda \) provides \(\xi \) mass, and both \(w,\Lambda \) supply E mass. All that implies a similar role between \(w,\Lambda \) as mutually contributing to the new physics and interacting of their fields with the SM. Although we have chosen \(\Lambda >w\) in interpreting the results, we reexamined that an opposite choice, \(w>\Lambda \), or including both cases, but always ensuring \(w\sim \Lambda \), lead to the same physics.

Table 7 The interactions of a neutral and a charged gauge boson with two scalars (Continued)
Table 8 The interactions of a neutral and a charged gauge boson with two scalars (Continued)

6 Conclusion

In this work, the fully flipped 3-3-1-1 model has been interpreted. The scalar sector has explicitly been diagonalized, yielding the appropriate particle spectrum. We have shown that the model naturally provides the two DM candidates, a singlet scalar and a triplet fermion, with the masses in TeV regime. We have determined the physical resonances in the dark matter relic density, which are set by the new neutral gauge boson and the new neutral Higgs boson according to the fermion and scalar densities, respectively. A further experimental search for these resonances is crucial to probe the existence of dark matter.