Heavy neutral leptons in gauged U(1)_Lµ−Lτ at muon collider

Heavy neutral leptons N are well motivated to explain the origin of tiny neutrino masses. Due to the singlet nature of heavy neutral leptons under the standard model gauge group, we can write a Majorana mass term $ m_N \overline{N^c}N $, which results in light Majorana neutrino masses as $ m_\nu\sim m_D^2/m_N $ via the type-I seesaw mechanism [1, 2]. To generate the sub-eV neutrino masses, the heavy neutral lepton masses should be at very high scale $ m_N\gtrsim10^{14} $ GeV if $ m_D $ is at the electroweak scale, which is far beyond current experimental reach. Alternatively, an electroweak scale $ m_N $ is usually assumed for phenomenological studies [3], which is then determined by the mixing parameter $ V_{\ell N} $.

To confirm the Majorana nature of neutrinos, lepton number violation signatures are expected. The most sensitive experiment is the neutrinoless double beta decay, which could probe the inverted mass ordering scenario in next-generation experiments [4]. Meanwhile, the detection of the lepton number violation signature at colliders could unravel the explicit mechanism of tiny neutrino masses [5]. For heavy neutral leptons in the type-I seesaw, the distinct signature at the hadron collider is $ pp\to W^{\pm*}\to \ell^\pm N\to\ell^\pm\ell^\pm jj $ [6]. However, the sensitive region of this signature heavily depends on a relatively large mixing parameter $ V_{\ell N} $ [7, 8], which is usually much higher than the natural seesaw prediction $ V_{\ell N}\sim\sqrt{m_\nu/m_N} $.

Besides the canonical seesaw, there are many theories with extended gauge groups, such as the $ U(1)_{B-L} $ model [9, 10] and left-right symmetric model [11, 12]. In these models, the heavy neutral leptons are charged under the extended gauge group, which opens new production mechanisms for heavy neutral leptons. Pair production of heavy neutral leptons via the decay of the $ Z' $ boson in the framework of the left-right symmetric model has been searched for recently by the CMS collaboration [13]. The region within $ m_N\lesssim1.4 $ TeV and $ m_{Z'}\lesssim4.4 $ TeV has been excluded by the dimuon channel. Meanwhile, the ATLAS collaboration searches for the right-handed $ W' $ boson decaying to heavy neural lepton N and leptons $ \ell $, which can exclude the region with $ m_N\lesssim3.8 $ TeV and $ m_{W'}\lesssim6.4 $ TeV [14].

The construction of a multi-TeV muon collider was proposed recently [15, 16]. Since then, searches for heavy neutral leptons at the multi-TeV muon collider have drawn increasing interest [17−21]. One promising signature is $ \mu^+\mu^-\to N\nu $, which could probe the mixing parameter $ |V_{\mu N}|^2\gtrsim10^{-6} $ at the 10 TeV muon collider [22−24]. The Majorana nature of heavy neutral leptons can be confirmed with double peaks of the rapidity distribution of the reconstructed N [25]. Another interesting signature is the lepton number violation signal via vector boson scattering process $ W^\pm Z/\gamma\to \ell^\pm N $ [26], via associated production process $ \mu^+\mu^-\to N W^\pm \ell^\mp $ [27], or at the same-sign muon collider via $ \mu^+\mu^+\to W^+W^+ $ process [28]. These studied signatures also require that the mixing parameter $ V_{\ell N} $ is not too small. Meanwhile, under the tight constraints from current experimental searches [13, 14], the first stage of a 3 TeV muon collider [29] is not promising to probe prompt heavy neutral leptons in the $ U(1)_{B-L} $ and left-right symmetric models.

The multi-TeV muon collider is a perfect machine to test the muon-philic forces. One attractive option is the gauged $ U(1)_{L_\mu-L_\tau} $ model [30−34]. Compared to $ U(1)_{B-L} $ or left-right symmetry, $ U(1)_{L_\mu-L_\tau} $ symmetry is less constrained due to the lack of direct couplings to electrons and quarks. For instance, there is only a loose constraint from neutrino trident production when $ m_{Z'}\gtrsim100 $ GeV [35]. Therefore, this model has been extensively studied to explain the anomaly of the muon magnetic moment [36, 37], B meson anomaly [38, 39], dark matter [40, 41], and neutrino masses [42, 43]. It is shown that a 3 TeV muon collider is powerful enough to probe a relatively large parameter space with $ m_{Z'}\lesssim10 $ TeV [44, 45]. In this paper, we consider the gauged $ U(1)_{L_\mu-L_\tau} $ model with three heavy neutral leptons $ N_e,N_\mu,N_\tau $. We then study the pair production of heavy neutral lepton N via $ Z' $ boson at a 3 TeV muon collider. To test the Majorana nature of neutrinos, we focus on the lepton number violation process $ \mu^+\mu^-\to Z^{\prime *}\to NN\to \mu^\pm\mu^\pm W^\mp W^\mp $ and $\mu^+\mu^-\to Z^{\prime} \gamma\to NN \gamma \to \mu^\pm\mu^\pm W ^\mp W^\mp \gamma$ with the hadronic decay of W.

The remainder of this paper is organized as follows. In Section II, we review the gauged $ U(1)_{L_\mu-L_\tau} $ model with three heavy neutral leptons and discuss relevant experimental constraints. The decay properties of gauge boson $ Z' $ and heavy neutral lepton N are also considered in this section. In Section III, we study the pair production of heavy neutral lepton at the 3 TeV muon collider. Analysis of the lepton number violation signatures is performed in Section IV. Finally, Section V summarizes our conclusions.

In this paper, we consider the anomaly-free gauged $ U(1)_{L_\mu-L_\tau} $ extension of the type-I seesaw mechanism. Three heavy neutral leptons $ N_e,N_\mu,N_\tau $ with $ U(1)_{L_\mu-L_\tau} $ charge $ (0,1,-1) $ are introduced to generate tiny neutrino masses. Predicting type C$ ^R $ of the two-zero minor [46, 47], the minimal model with only one scalar singlet $ \Phi_1 $, which carries $ U(1)_{L_\mu-L_\tau} $ charge $ +1 $, is now tightly constrained by the neutrino oscillation data and the sum of light neutrino masses [48, 49]. Thus, the second scalar singlet $ \Phi_2 $ with $ U(1)_{L_\mu-L_\tau} $ charge $ +2 $ is also employed to reduce the above conflict [50]. The Yukawa interactions and mass terms relevant to neutrino masses are given by [51]

where $ L_e,L_\mu,L_\tau $ are the lepton doublets, H is the standard model Higgs doublet, and $ \tilde{H}=i\tau_2 H^* $. After the spontaneous symmetry breaking, we can denote the vacuum expectation values of scalars as $ \langle H \rangle=v_0,\langle \Phi_1 \rangle=v_1,\langle \Phi_2 \rangle=v_2 $. Then, the Dirac neutrino mass matrix and heavy neutral lepton mass matrix are given by

Light neutrino masses are generated via the type-I seesaw formula

Without any specific structure of heavy neutral lepton mass matrix $ M_N $, the obtained light neutrino mass matrix $ M_\nu $ is a general symmetric matrix; thus, it can easily fit the neutrino oscillation data [52]. Conversely, we can use Eq. (3) to express the mass matrix $ M_N $ as [53]

where $ M_\nu=U^*\text{diag}(m_{\nu_1},m_{\nu_2},m_{\nu_3}) U^\dagger $, and U is the light neutrino mixing matrix. In this way, the mass matrix $ M_N $ is determined by the light neutrino oscillation data and the Yukawa coupling $ y_e,y_\mu,y_\tau $. The mass matrix $ M_N $ can be diagnalized by a unitarity matrix Ω as

Considering the mixing with light neutrinos, the heavy neutral leptons interact with the $ W^\pm,Z $ gauge boson and Higgs boson as

where the mixing matrix $ V\simeq M_D M_N^{-1} $. For electroweak-scale heavy neutral leptons, we assume a seesaw-induced mixing parameter $ V_{\ell N}\sim \sqrt{m_\nu/m_N}\sim10^{-6} $, which is allowed by current experimental limits [3].

The interactions of the new boson $ Z' $ with fermions are

In terms of mass eigenstates, the new neutral current of neutral leptons can be rewritten as

From Eqs. (6) and (8), it is obvious that the collider phenomenology of heavy neutral leptons depends on the mixing patterns, which makes the results model-dependent. In the following collider simulation, we consider a muon-philic heavy neutral lepton N, which couples exclusively to muons [54]. We also take the mixing parameter $ \Omega_{\mu N}^{*}\Omega_{\mu N}-\Omega_{\tau N}^{*}\Omega_{\tau N}=1 $ in the following calculation, so the general model-dependent results can be obtained by a simple rescaling.

In this section, we briefly summarize the constraints on the gauged $ U(1)_{L_\mu-L_\tau} $ model. One important motivation of this symmetry is to explain the anomaly of the muon magnetic moment $ \Delta a_\mu=(251\pm59)\times10^{-11} $ [36]. The new gauge boson $ Z' $ contributes to $ \Delta a_\mu $ at one loop level, which is evaluated as [37]

The viable parameter space to interpret $ \Delta a_\mu $ is shown in Fig. 1. For very light $ m_{Z'} $, we have $ \Delta a_\mu\simeq g^{\prime 2}/8\pi^2 $, which requires $ g'\sim4\times10^{-4} $ to explain $ \Delta a_\mu $. In the heavy $ Z' $ limit, $ \Delta a_\mu\simeq g^{\prime 2} m_\mu^2/12\pi^2 m_{Z'}^2 $; thus, the experimental discrepancy requires $ g'/m_{Z'}\sim 5\times10^{-3}\; \text{GeV}^{-1} $.

Figure 1.  (color online) Constraints on the gauged $ U(1)_{L_\mu-L_\tau} $. The blue band can explain the $ \Delta a_\mu $ result [36]. The gray, orange, red, purple, and pink regions are excluded by BBN [55], CCFR [35], NA64μ [56], BABAR [57], and LHC [58, 59], respectively.

For the gauge boson $ Z' $, one tight constraint is the dimuon production from the inelastic neutrino-nucleus scattering $ \nu_\mu N_u \to \nu_\mu N_u \mu^+\mu^- $, where $ N_u $ denotes the nucleus [60]. The CCFR experiment measured the neutrino trident production cross section, which is consistent with the standard model prediction as $ \sigma/\sigma_\text{SM}=0.82\pm0.28 $ [35]. This result has excluded the $ \Delta a_\mu $ favored region with $ m_{Z'}\gtrsim0.3 $ GeV. Recently, the NA64μ experiment has excluded $ g'\gtrsim6\times10^{-4} $ with $ m_{Z'}\lesssim0.1 $ GeV by the process of $ \mu N_u\to \mu N_u Z' \to \mu N_u \bar{\nu} \nu $ [56], which makes $ m_{Z'}\sim0.01 $ GeV the $ \Delta a_\mu $-favored region. A more stringent constraint comes from the $ e^+e^-\to \mu^+\mu^- Z' \to 4\mu $ searches at BABAR in the mass region of $ 0.212\; { \rm GeV}<m_{Z'}<10 $ GeV [57]. For $ Z' $ mass in the range of [5, 81] GeV, the LHC experiment has excluded $ g'\gtrsim0.003 $ to 0.2 (depending on $ m_{Z'} $) by the process $ pp\to Z'\mu^+\mu^-\to4\mu $ with 139 fb$ ^{-1} $ data [58]. Within the same mass region, a complement search by the process $ pp\to Z'\mu^\pm\nu\to 3\mu \nu $ with 140 fb$ ^{-1} $ data was also performed [59], which has better sensitivity in the light $ Z' $ region. In this paper, we consider the combined limit of the $ 3\mu $ and $ 4\mu $ channels at the LHC [59]. Meanwhile, for $ Z' $ lighter than approximately 6 MeV, it is disfavored by $ \Delta N_\text{eff} $ constraints from big-bang nucleosynthesis (BBN) [55], because decays of $ Z' $ could heat the neutrino population and delay the process of neutrino decoupling. In the following study, we consider $ m_{Z'}\gtrsim 100 $ GeV to avoid these tight constraints.

For the heavy neutral lepton N, the promising signature at the LHC is $ pp\to W^{\pm(*)}\to \ell^\pm N $, which depends on the mixing parameter $ V_{\ell N} $. When $ m_N<m_W $, N is produced from on-shell W boson decay; then, LHC could exclude $ |V_{\ell N}|^2\gtrsim2.9\times10^{-6} $ via the fully-leptonic decay channel $ N\to \ell \ell \nu $ [61]. Meanwhile, for light N with proper $ V_{\ell N} $, it becomes long-lived and leads to displaced vertex signature. Currently, the LHC has excluded $|V_{\mu N}|^2 > 4\times10^{-7}$ for $ m_N=10 $ GeV [62]. Above $ m_W $, production of N is mediated by off-shell W boson, so the LHC limit becomes much weaker, e.g., $ |V_{\mu N}|^2=1.1\times10^{-3} $ for $ m_N=100 $ GeV [61]. In this paper, we also consider $ m_N>100 $ GeV with seesaw-induced mixing $ |V_\ell N|^2\lesssim10^{-12} $ to satisfy current LHC limits.

Before studying the explicit signatures, we first review the decay properties of new gauge boson $ Z' $ and heavy neutral lepton N. The partial decay widths of $ Z' $ are calculated as

where $ \ell=\mu,\tau $, and we have assumed vanishing masses for $ \ell,\nu_\ell $ in the above calculations. For simplicity, we assume that the tree-level kinetic mixing between boson $ Z' $ and photon γ is absent. However, mixing between $ Z' $ and γ in the form of $ -\dfrac{\epsilon}{2} Z'_{\mu\nu} F^{\mu\nu} $ appears at one-loop level, where the kinetic mixing factor $ \epsilon $ is calculated as [55]

In this way, $ Z' $ also couples to electron e via mixing with γ. The $ Z'\to e^+e^- $ decay width is suppressed by the kinetic mixing factor $ \epsilon\simeq -g'/70 $, so it is not taken into account in this study.

After the production of heavy neutral lepton N, it decays via mixing with light neutrinos. When the heavy neutral lepton is heavier than the standard model gauge bosons, the two-body decays are the dominant modes. The partial decay widths are given by

In Fig. 2, we show the branching ratio (BR) of gauge boson $ Z' $ and heavy neutral lepton N. The dilepton mode $ Z'\to \mu^+\mu^-,\tau^+\tau^- $ is always the dominant decay channel of $ Z' $. Neglecting the final state's phase space effect, we have BR$ (Z'\to \nu_\ell \nu_\ell)= $ BR$ (Z'\to N_\ell N_\ell )=1/4 $ for two degenerate heavy neutral leptons. In the heavy $ m_N $ limit, we have BR$ (N \to \ell^\pm W^\mp) $: BR$ (N \to \nu Z) $: BR$ (N \to \nu h)= 2:1:1 $. To reconstruct the heavy neutral lepton mass, we focus on the decay mode $ N\to \ell^\pm W^\mp $ with $ W\to q \bar{q}' $ in the collider simulation.

Figure 2.  (color online) Branching ratio of gauge boson $ Z' $ (left) and heavy neutral lepton N (right).

Through mixing with light neutrinos, the pair production of heavy neutral leptons at the muon collider can occur through s-channel Z exchange and t-channel W exchange. However, the production cross section is proportional to the fourth power of the mixing parameter $ |V_{\ell N}|^4 $. Provided natural seesaw predicted value $V_{\ell N}\lesssim 10^{-6}$, the predicted cross section is negligible tiny.

The heavy neutral lepton N is charged under $ U(1)_{L_\mu-L_\tau} $ symmetry, which induces the pair production of heavy neutral lepton via $ Z' $ at the muon collider. With fixed collision energy, the direct pair production process $\mu^+\mu^-\to NN$ via the s-channel off-shell $ Z' $ is more promising when $ m_{Z'}>\sqrt{s} $. Meanwhile, when $ m_{Z'}<\sqrt{s} $, the new gauge boson $ Z' $ can be produced on-shell with an associated photon from the initial legs as $ \mu^+\mu^-\to Z' \gamma \to NN \gamma $, which also requires $ m_N<m_{Z'}/2 $ to make the decay $ Z'\to NN $ kinematically allowed.

We first consider the pair production of heavy neutral leptons without associated photon. The production cross section of $ \mu^+\mu^-\to Z^{\prime *}\to NN $ with a center of mass energy $ \sqrt{s} $ is calculated as

where $ \Gamma_{Z'} $ is the total decay width of $ Z' $.

The theoretical cross section of $ \mu^+\mu^-\to Z^{\prime *}\to NN $ at a 3 TeV muon collider for specific scenarios is shown in Fig. 3, where we directly use Eq. (17) without considering the initial state radiation effect. It is obvious that when $ m_{Z'}\simeq \sqrt{s} $, the on-shell production of $ Z' $ can greatly enhance the cross section. The maximum value is approximately $ 12\pi \text{BR}(Z'\to \mu^+\mu^-) \text{BR}(Z'\to NN)/m_{Z'}^2\approx 3\pi/(8m_{Z'}^2) $ [45], which could be over 50 pb. For a light $ Z' $, the cross section reduces to

Figure 3.  (color online) Cross section of $ \mu^+\mu^-\to Z^{\prime *}\to NN $ at a 3 TeV muon collider. We have fixed $ m_N=100 $ GeV in the left panel and $ m_{Z'}=2000 $ GeV in the right panel.

which is independent of $ Z' $ mass. Typically, for $ g'=0.3 $ and $ m_N=1000 $ GeV, we have $ \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN)\simeq4.7 $ fb. Meanwhile, a heavy $ Z' $ leads to

In this scenario, the cross section is suppressed by the large value of $ m_{Z'} $. While single production could probe $ m_N\lesssim \sqrt{s} $, the pair production of heavy neutral lepton is kinematically allowed when $ m_N<\sqrt{s}/2 $. Thus, the 3 TeV muon collider could only probe $ m_N\lesssim1500 $ GeV. For heavier $ m_N $, we require a more energetic muon collider.

The photon from initial state radiation carries part of the collision energy, which results in the gauge boson $ Z' $ produced on-shell when $ m_{Z'}\lesssim \sqrt{s} $. In this case, there is a resonance peak in the invariant mass spectrum of the dilepton [44]. Here, we study the pair production of heavy neutral leptons with associated photon process as

Here, the scenario with off-shell $ Z' $ contribution is also considered when $ m_{Z'}\leqslant 2 m_N $ or $ m_{Z'}\geqslant \sqrt{s} $.

In Fig. 4, we show the cross section of $ \mu^+\mu^-\to NN\gamma $ at a 3 TeV muon collider, where the numerical results are calculated by MadGraph. There is also a sharp peak around $ m_{Z'}\simeq \sqrt{s} $. To avoid the soft photon singularity, the detected photon is required to satisfy the following pre-selection cuts:

Figure 4.  (color online) Cross section of $ \mu^+\mu^-\to NN\gamma $ at a 3 TeV muon collider. We have fixed $ m_N=100 $ GeV in the left panel and $ m_{Z'}=2000 $ GeV in the right panel.

When $ m_{Z'}>2 m_N $, the on-shell production of $ Z' $ followed by the cascade decay $ Z'\to NN $ can notably enhance the cross section of $ \mu^+\mu^-\to NN\gamma $. Using the narrow-width approximation, the cross section can be expressed as [63]

Because the branching ratio of $ Z'\to NN $ is independent of the new gauge coupling $ g' $, the cross section of $ \mu^+\mu^-\to Z'\gamma\to NN\gamma $ is proportional to $ e^2g^{\prime 2} $, while the cross section of the off-shell process $\mu^+\mu^-\to Z ^{\prime (*)}\gamma \to NN\gamma$ is proportional to $ e^2g^{\prime 4} $. Thus, we observe that the enhancement effect becomes larger when the gauge coupling $ g' $ is smaller. For $g'=0.1,\; m_N=100$ GeV, and $ m_{Z'}=1000 $ GeV, we have $ \sigma(\mu^+\mu^-\to Z'\gamma\to NN\gamma)\sim 1 $ fb, while $ \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN) $ is less than 0.1 fb; therefore, the former process is expected to be more promising at the muon collider.

Pair production of heavy neutral leptons followed by various cascade decay modes could lead to many interesting signatures, such as monolepton, dilepton, and trilepton signals [64, 65]. In this paper, we focus on the most distinct lepton number violation, same-sign dimuon signal, which also has a much cleaner background compared with the lepton number conserving one.

The simulation procedure is as follows. The FeynRules package [66] is used to implement the model. The parton level events are simulated with MadGraph5_aMC@NLO [67]. We then use PYTHIA 8 [68] for parton shower. The detector effects are included by employing Delphes 3 [69] with the detector card of the muon collider. The W boson from the heavy neutral lepton decay could be highly boosted, so the two jets from cascade W decay merge into one fat-jet J. We use the Valencia algorithm [70] with $ R=1.2 $ to reconstruct the fat-jets.

The full production process of the same-sign dimuon signature without associated photon is

with the hadronic decays of W. The standard model backgrounds come from processes such as

The contributions of $ \mu^+\mu^- W^+W^- $ and $ \mu^+\mu^-W^+ W^-\gamma/Z $ are from lepton charge misidentification, which are suppressed by the misidentification rate 0.1% [17]. The $ W^\pm W^\pm W^\mp \mu^\mp \nu $ is dominant by the vector boson fusion process, where the two same-sign W bosons decay leptonically. There is also one possible background process $ \mu^+\mu^-\to W^+W^-jj $ with the light jets mistagged. However, by requiring the light jet masses close to the W mass, this background can easily be suppressed [17]. Meanwhile, the $ t\bar{t}W^\pm \mu^\mp\nu $ process also contributes to the same-sign dimuon signature, which can be further reduced by cut on the opposite-sign lepton [26]. Thus, we do not include contributions of the $ \mu^+\mu^-\to W^+W^- jj $ and $ t\bar{t}W^\pm \mu^\mp\nu $ processes in this paper.

We first apply the following pre-selection cuts on the transverse momentum and pseudorapidity of the muon and fat-jets:

In Fig. 5, we show the distributions of some variables for the $ \mu^\pm\mu^\pm JJ $ signal and backgrounds after applying the pre-selection cuts. We have set $ m_{N}=1000 $ GeV, $ m_{Z'}= $ 2500 GeV, and $ g'=0.6 $ as the benchmark point of the signal. The cut flow for the $ \mu^\pm\mu^\pm JJ $ signature and backgrounds are summarized in Table 1.

Figure 5.  (color online) Normalized distribution of transverse momentum of muon $ P_T(\mu^\pm) $ (up-left panel), transverse momentum of fat-jet $ P_T(J) $ (up-right panel), fat-jet mass $ m_J $ (down-left panel), and invariant mass of muon and fat-jet $ m_{\mu J} $ (down-right panel) for the $ \mu^\pm\mu^\pm JJ $ signature and corresponding backgrounds.

σ(fb)	$ \mu^\pm\mu^\pm W^\mp W^\mp $	$ \mu^+\mu^- W^+W^- $	$ \mu^+\mu^-W^+ W^-\gamma/Z $	$ W^\pm W^\pm W^\mp \mu^\mp \nu $
Pre-selection	68.19	21.57	2.13	3.11
$ N_{\mu^\pm}=2 $	41.38	$ 2.6\times10^{-2} $	$ 2.5\times10^{-3} $	$ 1.8\times10^{-1} $
$ N_J=2 $	24.87	$ 1.1\times10^{-2} $	$ 1.3\times10^{-3} $	$ 7.1\times10^{-2} $
50 GeV $ \leqslant m_J\leqslant $ 100 GeV	14.41	$ 3.2\times10^{-3} $	$ 2.1\times10^{-4} $	$ 2.0\times10^{-2} $
$ 0.8 m_N<m_{\mu J}<1.2 m_N $	13.62	$ 3.4\times10^{-4} $	$ 2.7\times10^{-5} $	$ 8.1\times10^{-4} $
Significance	476.6	Total Background		$ 1.2\times10^{-3} $

Table 1.  Cut flow table for the $ \mu^\pm\mu^\pm JJ $ signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated with Eq. (29) by assuming an integrated luminosity of 1000 fb$ ^{-1} $.

The lepton number violation signature is satisfied for events with two same-sign muons. In the single production of the heavy neutral lepton process [26, 27], only one boosted W is expected from N decay. To distinguish from the single production and also suppress the background from vector boson fusion processes, we require exactly two fat-jets in the final states

although tagging one fat-jet has greater efficiency for the signal. To be identified as W bosons, the fat-jet masses are also required in the following range:

Because there are two heavy neutral leptons in the signal, we reconstruct their masses through the two-muon and two-fat-jet system by minimizing $ \chi^2= (m_{\mu_1 J_1} - m_N)^2+(m_{\mu_2 J_2} -m_N)^2 $ [17]. We require that both the invariant mass of muon and fat-jet $ m_{\mu J} $ from reconstructed N satisfy

According to the distributions in Fig. 5, we may tighten the cuts on $ P_T(\mu^\pm) $ and $ P_T(J) $ to suppress the background. However, we can see from Table 1 that the cut on the invariant mass $ m_{\mu J} $ is relatively efficient to reduce the background. After applying all these cuts, the total cross section of the background is approximately $ 1.2\times10^{-3} $ fb. Thus, with an integrated luminosity of 1000 fb$ ^{-1} $, there are only 1.2 background events, while the signal events are over $ 10^4 $. In this manner, the significance will be 476.6 for the benchmark point with 1000 fb$ ^{-1} $ data, where the significance is calculated as [71]

Here, $ N_S $ and $ N_B $ are the event numbers of signal and background, respectively. To reach the $ 5\sigma $ discovery limit, we require only 0.11 fb$ ^{-1} $ data.

Based on the above analysis, we then explore the sensitivity of the $ \mu^\pm \mu^\pm JJ $ signature at the 3 TeV muon collider. The results are shown in Fig. 6. There are three free parameters $ g',m_{Z'} $, and $ m_N $ related to the collider phenomenology. With fixed mass relation $ m_{N} = m_{Z'}/3 $, we could probe $ m_{Z'} $ approximately in the range of [300, 4500] GeV. Around the resonance region $ m_{Z'}\sim3000 $ GeV, the gauge coupling $ g' $ could be down to approximately 0.025 with 1000 fb$ ^{-1} $ data. We also find that for $ m_N\lesssim300 $ GeV, the cut efficiency decreases quickly as $ m_N $ becomes smaller. One main reason for this is that the heavy neutral leptons are also highly boosted at the 3 TeV muon collider for such light $ m_N $, so the muons and W-jets from boosted N decays are mostly non-isolated [72]. One may use the substructure-based variables as lepton sub-jet fraction (LSF) and lepton mass drop (LMD) [73] to probe the light $ m_N $, which is beyond the scope of this study.

Figure 6.  (color online) Sensitivity of the $ \mu^\pm \mu^\pm JJ $ signature at the 3 TeV muon collider. In the upper-left panel, we fix the mass relation with $ m_N=m_{Z'}/3 $. In the upper-right panel, we set $ m_N=1000 $ GeV. In the lower panel, we fix the gauge coupling $ g'=0.6 $. The dark-green region corresponds to sensitivity with 1 fb $ ^{-1} $ data, while the light-green region corresponds to sensitivity with 1000 fb$ ^{-1} $ data. The orange region is excluded by neutrino trident production at CCFR [35]. The magenta region is not allowed kinematically. The gray line is the projected sensitivity at the HL-LHC [74]. The red line is the combined sensitivity of the 3 TeV muon collider via processes $ \mu^+\mu^-\to \ell^+\ell^-(\ell=\mu\; \text{or}\; \tau), $$ \ell\bar{\ell}\gamma(\ell=\mu\; \text{or}\; \tau\; \text{or}\; \nu_{\mu/\tau}) $ [44].

For pair production of heavy neutral leptons without associated photon signature, the gauge boson $ Z' $ does not need to be heavier than $ 2 m_N $. In the upper-right panel of Fig. 6, we have fixed $ m_N=1000 $ GeV for illustration. The production cross section of $ \mu^+\mu^-\to NN $ is approximately constant for light $ Z' $, so $ g'\sim0.13 $ is nearly independent of $ m_{Z'} $ when $ Z' $ is lighter than 1 TeV. This channel is most sensitive for the large $ m_{Z'} $ region. For instance, we may probe $ m_{Z'}\sim10 $ TeV with $ g'\gtrsim0.4 $ and $ m_N\sim1000 $ GeV at the 3 TeV muon collider. Near the resonance region with $ m_{Z'}\sim \sqrt{s} $, we can even have a promising signature when $ g'\gtrsim0.14 $ with only 1 fb$ ^{-1} $ data. Compared with the direct dilepton channels

the sensitivity region of lepton number violation signal from heavy neutral lepton pair at the muon collider is smaller due to lower tagging efficiency of N.

We then fix the gauge coupling $ g'=0.6 $ and explore the sensitivity region on the $ m_N-m_{Z'} $ plane. With 1 fb$ ^{-1} $ data, a large parameter space near the resonance region with 2 TeV $ \lesssim m_{Z'}\lesssim4 $ TeV can be probed. The most sensitive heavy neutral lepton mass is around 500 GeV. This is because a lighter $ m_N $ leads to smaller acceptance efficiency, while a larger $ m_N $ has a smaller production cross section. With an integrated luminosity of 1000 fb$ ^{-1} $, the region with $ m_{Z'}\lesssim 13 $ TeV and 150 GeV $ \lesssim m_N<1500 $ GeV is within the reach of the 3 TeV muon collider.

The full production process of the same-sign dimuon signature with associated photon is

with the hadronic decays of W. The standard model backgrounds come from processes such as

During the simulation, the pre-selection cuts on associated photon in Eq. (21) are also applied. Based on the cut-flow in Table 1, we expect that the cross section of the process $ \mu^+\mu^-W^+ W^-\gamma\gamma/Z $ with one additional $ \gamma/Z $ gauge boson is approximately an order of magnitude smaller than that of the process $ \mu^+\mu^- W^+W^-\gamma $. Therefore, $ \sigma(\mu^+\mu^-W^+ W^-\gamma\gamma/Z) $ is approximately $ 10^{-5} $ fb after all cuts, which will not be considered in the analysis of the $ \mu^\pm\mu^\pm JJ\gamma $ signal.

The cross sections of the background processes with associated photons are approximately one order of magnitude smaller than those without associated photons. If we apply exactly the same selection cuts as the previous $ \mu^\pm\mu^\pm JJ $ signal, the total cross section of background is expected to be of the order of $ 10^{-4} $ fb, which will have approximately 0.1 background events, even with a total of 1000 fb$ ^{-1} $ data. Therefore, we can loosen the selection cut to keep more signal events. In the analysis of the $ \mu^\pm\mu^\pm JJ\gamma $ signal, we consider that at least one fat-jet is detected in the final states

while we further require one detected photon in this signal.

In Fig. 7, we show the distributions of some variables for the $ \mu^\pm\mu^\pm JJ\gamma $ signal and backgrounds after applying the pre-selection cuts. The benchmark point for the $ \mu^\pm\mu^\pm JJ\gamma $ signal is the same as for the previous $ \mu^\pm\mu^\pm JJ $ signal. With relatively similar distributions of $ P_T(\mu^\pm) $, $ P_T(J) $, $ m_J $, and $ m_{\mu J} $, we then apply the same selection cuts for these variables as the $ \mu^\pm\mu^\pm JJ $ signal. The cut flows for the $ \mu^\pm\mu^\pm JJ\gamma $ signature and backgrounds are summarized in Table 2.

Figure 7.  (color online) Normalized distribution of transverse momentum of muon $ P_T(\mu^\pm) $ (upper-left panel), transverse momentum of fat-jet $ P_T(J) $ (upper-right panel), fat-jet mass $ m_J $ (middle-left panel), and invariant mass of muon and fat-jet $ m_{\mu J} $ (middle-right panel). Energy of photon $ E_\gamma $ (lower-left panel) and invariant mass of reconstructed heavy neutral lepton pair $ m_{NN} $ (lower-right panel) for the $ \mu^\pm\mu^\pm JJ \gamma $ signature and corresponding backgrounds.

σ/fb	$ \mu^\pm\mu^\pm W^\mp W^\mp \gamma $	$ \mu^+\mu^- W^+W^- \gamma $	$ W^\pm W^\pm W^\mp\gamma \mu^\mp \nu $
Pre-selection	24.91	1.15	0.14
$ N_{\gamma}=1,N_{\mu^\pm}=2 $	13.28	$ 1.1\times10^{-3} $	$ 7.4\times10^{-3} $
$ N_J\geq1 $	13.25	$ 1.0\times10^{-3} $	$ 7.2\times10^{-3} $
50 GeV $ \leqslant m_J\leqslant $ 100 GeV	9.61	$ 5.7\times10^{-4} $	$ 4.5\times10^{-3} $
$ 0.8 m_N<m_{\mu J}<1.2 m_N $	8.86	$ 2.6\times10^{-4} $	$ 1.2\times10^{-3} $
Significance	369.0	Total Background	$ 1.5\times10^{-3} $

Table 2.  Cut flow table for the $ \mu^\pm\mu^\pm JJ\gamma $ signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated by Eq. (29), assuming an integrated luminosity of 1000 fb$ ^{-1} $.

Although with one additional photon, the cross section of the $ \mu^\pm\mu^\pm JJ\gamma $ signal is of the same order as the $ \mu^\pm\mu^\pm JJ $ signal. After all selection cuts, the cross section of the $ \mu^\pm\mu^\pm JJ\gamma $ process is 8.86 fb. With a loose cut on the fat-jet number, the total cross section of the background is approximately $ 1.5\times10^{-3} $ fb after all cuts. With an integrated luminosity of 1000 fb$ ^{-1} $, the significance of the $ \mu^\pm\mu^\pm JJ\gamma $ signal could be approximately 369 for the benchmark point. Meanwhile, 0.19 fb$ ^{-1} $ data is sufficient to discover the benchmark point at the $ 5\sigma $ level.

Besides the invariant mass of dilepton $ m_{\ell^+\ell^-}(\ell=\mu,\tau) $, there are two more pathways to confirm the on-shell production of gauge boson $ Z' $ in the $ \mu^\pm \mu^\pm JJ\gamma $ signature. For the two-body process $ \mu^+\mu^-\to Z' \gamma $, the photon energy is connected to the gauge boson mass as [44]

For the benchmark point with $ m_{Z'}=2500 $ GeV at the 3 TeV muon collider, the peak value of $ E_\gamma $ is approximately 458 GeV. The sharp bump above the background is clearly shown in the lower-left panel of Fig. 7. Meanwhile, we can also measure $ m_{Z'} $ through the invariant mass of two reconstructed heavy neutral leptons $ m_{NN} $, which also has a resonance near $ m_{Z'} $. From the distribution of $ m_{NN} $ in the lower-right panel of Fig. 7, the background processes $ \mu^+\mu^-W^+W^-\gamma $ and $ W^\pm W^\pm W^\mp \gamma \mu^\mp \nu $ have peaks around 3 TeV and 1.5 TeV, respectively. Besides, with less than two background events after all cuts, the backgrounds have little impact on the distribution of $ m_{NN} $ for the signal.

The sensitivity of the $ \mu^\pm \mu^\pm JJ\gamma $ signature at the 3 TeV muon collider is explored with the cuts in Table 2, and Fig. 8 depicts the results. The upper-left panel shows the sensitivity region with fixed mass relation $ m_N=m_{Z'}/3 $, so the decay channel $ Z'\to NN $ is always allowed. In this way, we might probe $ m_{Z'} $ in the range of [300, 4300] GeV. Compared with the $ \mu^\pm \mu^\pm JJ $ signature, we find that this signal is more promising approximately in the region of [400, 3000] GeV. For example, with $ m_{Z'}=2000 $ GeV, the $ \mu^\pm \mu^\pm JJ\gamma $ signature could probe $ g'\gtrsim0.015 $, while the $ \mu^\pm \mu^\pm JJ $ signature could only reach $ g'\sim0.1 $. Near $ m_{Z'}\sim \sqrt{s}=3 $ TeV, we can even reveal $ g'\gtrsim0.005 $. When $ m_{Z'} $ is larger than the collision energy $ \sqrt{s} $, the associated photon with the off-shell contribution of $ Z' $ makes the $ \mu^\pm \mu^\pm JJ\gamma $ signature less detectable than the $ \mu^\pm \mu^\pm JJ $ signal.

Figure 8.  (color online) Same as Figure 6 but for the sensitivity of the $ \mu^+\mu^-JJ\gamma $ signal.

The upper-right panel of Fig. 8 shows the sensitivity region with fixed $ m_N=1000 $ GeV. When $ m_{Z'}\leqslant 2 m_N $, the decay channel $ Z'\to NN $ is not allowed, so production of $ NN\gamma $ is through the off-shell $ Z' $. With nearly a constant cross section, the $ \mu^\pm \mu^\pm JJ\gamma $ signature could test $ g'\gtrsim0.23 $ for $ m_{Z'}\leqslant 2000 $ GeV, which is clearly less promising than the $ \mu^\pm \mu^\pm JJ $ signature. However, once the decay mode $ Z'\to NN $ is open, the cross section of $ NN\gamma $ is enhanced, so the $ \mu^\pm \mu^\pm JJ\gamma $ signal becomes quite promising. According to our simulation, we find that this signature could probe $ g'\gtrsim0.11 $ with only 1 fb$ ^{-1} $ data when 2000 GeV $ \lesssim m_{Z'}\lesssim\sqrt{s} $.

The sensitivity region in $ m_N-m_{Z'} $ with fixed $ g'=0.6 $ is shown in the lower panel of Fig. 8. With 1 fb$ ^{-1} $ data, a large portion of the parameter space with $ 2m_N\lesssim m_{Z'}\lesssim \sqrt{s} $ could be tested. With a similar reason as the $ \mu^\pm \mu^\pm JJ $ signature, the most sensitive $ m_N $ of the $ \mu^\pm \mu^\pm JJ\gamma $ signature is also approximately 500 GeV. Outside the above resonance $ Z' $ region, the significance of the $ \mu^\pm \mu^\pm JJ\gamma $ signature decreases due to the associated photon. Although less promising than the $ \mu^\pm \mu^\pm JJ $ signature, the $ \mu^\pm \mu^\pm JJ\gamma $ signal could probe the region with $ m_{Z'}\lesssim8.3 $ TeV and 120 GeV $ \lesssim m_N\lesssim1470 $ GeV at the 3 TeV muon collider with an integrated luminosity of 1000 fb$ ^{-1} $.

Heavy neutral leptons N are introduced in the type-I seesaw mechanism to generate tiny neutrino masses. While electroweak scale N is extensively studied at colliders, a relatively large mixing parameter $ V_{\ell N} $ is usually required. In this paper, we considered the gauged $ U(1)_{L_\mu-L_\tau} $ extension of the type-I seesaw mechanism, where three heavy neutral leptons $ N_e,N_\nu,N_\tau $ with $ U(1)_{L_\mu-L_\tau} $ charge $ (0,1,-1) $ were introduced. Charged under the new gauged $ U(1)_{L_\mu-L_\tau} $ symmetry, there are new production channels of the heavy neutral lepton at colliders. We then investigated the lepton number violation signatures of heavy neutral lepton N in the gauged $ U(1)_{L_\mu-L_\tau} $ at the 3 TeV muon collider.

Mediated by the new gauge boson $ Z' $, the heavy neutral lepton N can be pair produced at the muon collider via the processes $ \mu^+\mu^-\to Z^{\prime *}\to NN $ and $\mu^+\mu^-\to Z^{\prime (*)}\gamma\to NN \gamma$. Cross sections of these processes are determined by the new gauge coupling $ g' $, gauge boson mass $ m_{Z'} $, and heavy neutral lepton mass $ m_N $, which are independent of the mixing parameter $ V_{\ell N} $. While both processes have a maximum cross section at $ m_{Z'}\simeq \sqrt{s} $, the former is more promising when $ m_{Z'}>\sqrt{s} $. For lighter new gauge bosons satisfying $ 2m_N<m_{Z'}<\sqrt{s} $, $ Z' $ is produced on-shell with an associated photon, and the decay mode $ Z'\to NN $ is also allowed. In this way, the cross section of $ \mu^+\mu^-\to NN\gamma $ can be enhanced.

Cascade decays of heavy neutral leptons can raise various interesting signatures. Provided the Majorana nature of heavy neutral leptons in the seesaw model, we focus on the lepton number violation signatures in this study. For illustration, we further assume that the heavy neutral lepton N preferentially couples to muons via mixing with light neutrinos. Thus, the dominant decay mode of heavy neutral leptons is $ N\to \mu^\pm W^\mp $. The hadronic decays of W bosons are considered to reconstruct $ m_N $, which are treated as one fat-jet J. The explicit signatures are $ \mu^+\mu^-\to NN\to \mu^\pm\mu^\pm JJ $ and $ \mu^+\mu^-\to NN\gamma\to \mu^\pm\mu^\pm JJ\gamma $.

With relatively clean backgrounds, a cut-based analysis was then performed, which indicates that a large part of the viable parameter space is within the reach of the 3 TeV muon collider. For example, $ \mu^\pm\mu^\pm JJ $ could probe $ m_{Z'} $ in the range of [300, 4500] GeV with fixed mass relation $ m_{N}=m_{Z'}/3 $ and an integrated luminosity of 1000 fb$ ^{-1} $. Around the resonance region $ m_{Z'}\sim\sqrt{s} $, the gauge coupling $ g' $ could be down to approximately 0.025. We can probe $ g'\gtrsim0.13 $ with $ m_N=1000 $ GeV and light $ Z' $. Otherwise, this signal could probe $ m_{Z'}\lesssim 13 $ TeV with $ g'=0.6 $. Meanwhile, if the gauge boson mass satisfies $ 2 m_N<m_{Z'}<\sqrt{s} $, the $ \mu^\pm\mu^\pm JJ\gamma $ signature would be more promising than the $ \mu^\pm\mu^\pm JJ $ signature. It is also notable that around the resonance region $ m_{Z'}\sim\sqrt{s} $, both signatures could probe a sizable parameter space even with only 1 fb$ ^{-1} $ data for $ g'=0.6 $.