1 Introduction and main results

The geometric inequalities and the related problems have received a great deal of attention in recent years. The inequalities such as the Sobolev inequality, the Hardy–Littlewood–Sobolev inequality (HLS for short) and the Stein–Weiss type inequality play an essential role in the theory of partial differential equations and geometric analysis. For instance, the HLS inequality has been widely applied to investigate the qualitative properties and the classification of solutions. Moreover, the HLS inequality implies the sharp Sobolev inequality, as well as Gross’s logarithmic Sobolev inequality, is the key ingredient in the study of Yamabe problem and Ricci flow problem [30]. Let us first recall the classical version of the Hardy–Littlewood–Sobolev inequality, which is established by Hardy, Littlewood and Sobolev in [32, 47].

Proposition 1.1

Let \(1<p,~q^{\prime }<\infty \), \(0<\mu <n\), \(f\in L^{p}(\mathbb {R}^n)\), and \(g\in L^{q^{\prime }}(\mathbb {R}^n)\). Then there exists a sharp constant \(C\left( p,q^{\prime }, \mu , n\right) \) such that

$$\begin{aligned} \int _{\mathbb {R}^{n}}\int _{\mathbb {R}^n}\frac{f(y)g(x)}{|x-y|^{\mu }}dxdy\le C(p,q^{\prime },n,\mu )\Vert f\Vert _{L^{p}(\mathbb {R}^{n})}\Vert g\Vert _{L^{q^{\prime }}(\mathbb {R}^{n})}, \end{aligned}$$
(1.1)

where \(\frac{1}{p}+\frac{1}{q^{\prime }}+\frac{\mu }{n}=2\) and \(C(p,q^{\prime },n,\mu )\) is independent of f and g. Moreover, the optimal constant satisfy

$$\begin{aligned} C\left( p,q^{\prime }, \mu , n\right) \le \frac{n}{(n-\mu )}\left( |\mathbb {S}^{n-1}|/n\right) ^{\frac{\mu }{n}}\frac{1}{pq^{\prime }}\left( \left( \frac{\mu /n}{1-1/p}\right) ^{\mu /n}+\left( \frac{\mu /n}{1-q^{\prime }}\right) ^{\mu /n}\right) , \end{aligned}$$

where we use \(q^{\prime }\) to stand for the dual index of q.

If one of p or \(q^{\prime }\) equals 2 or \(p=q^{\prime }\), the existence of extremals for the HLS inequality with optimal constant was discussed by Lieb [39]. However, if \(p\not =q^{\prime }\), neither the sharp constant nor the existence of extremals is known. Later, Frank and Lieb [27] explored the best constant and extremals of the inequality for the case that \(p=q^{\prime }=\frac{2n}{2n-\mu }\) by the reflection positivity of inversions in spheres. Carlen and Loss [6] also studied the problem via the competing symmetry argument. In [7], Carlen et al. simplified the proof and obtained the sharp version of inequality (1.1) with \(n\ge 3\) and \(\mu =n-2\). Frank and Lieb [28] investigated the optimal constant of (1.1) via the rearrangement free argument [29] and obtained the sharp constant of HLS on the Heisenberg group, which was established by Folland and Stein [26]. Dou and Zhu [23] investigated the reversed HLS inequality. Dou, Guo and Zhu [22] proved a version of the reversed HLS inequality on the upper half by the subcritical argument. Ngô and Nguyen [48] employed the layer cake representation to obtain the reversed HLS inequality.

In [38, 46] Stein and Weiss established the weighted HLS type inequality,

$$\begin{aligned} \int _{\mathbb {R}^{n}}\int _{\mathbb {R}^n}\frac{f(y)g(x)}{|x|^{\alpha }|x-y|^{\mu }|y|^{\beta }}dxdy\le C(p,q^{\prime },n,\mu ,\alpha , \beta )\Vert f\Vert _{L^{p}(\mathbb {R}^{n})}\Vert g\Vert _{L^{q^{\prime }}(\mathbb {R}^{n})}, \end{aligned}$$
(1.2)

where \(p,q^{\prime }>1\), \(0<\mu <n\), \(\alpha +\beta \ge 0\) and the double weights satisfy \(\frac{1}{p}+\frac{1}{q^{\prime }}+\frac{\alpha +\beta +\mu }{n}=2\) and \(1-\frac{1}{p}-\frac{\mu }{n}<\frac{\alpha }{n}<1-\frac{1}{p}\). If \(p<q\), Lieb [39] proved that the extremals for this inequality can be obtained under the restriction \(\alpha ,\beta \ge 0\), and he also studied the non-existence of extremal functions for inequality (1.2) in the case \(p=q\). The same result was obtained by Herbst [36] for the case \(\mu =n-1,p=q=2,\alpha =0\) and \(\beta =1\). In [4, 5], Beckner established a new equivalent formulation to study the best constant of inequality with \(p=q\). Later, Chen, Lu and Tao [8] employed the concentration compactness principle to classify the extremal functions of (1.2) under the conditions \(p<q\) and \(\alpha +\beta \ge 0\). Particularly, they also generalized the results onto the Heisenberg group. It is worth mentioning that Han et al.[35] also derived the Stein–Weiss type inequality on the Heisenberg group. Chen et al.[10] considered the reversed Stein–Weiss inequality (1.2). Furthermore, the existence and classification of solutions of the Euler–Lagrange equations related to the integral inequalities have also attracted a lot of interest. By using the moving plane argument and regularity lifting technique, Chen and Li [16] classified precisely the positive solutions to of the integral type Euler–Lagrange equations related to the HLS inequality (1.1). Meanwhile, the authors studied the qualitative property of the extremal functions of the Stein–Weiss inequality (1.2) in [14]. Bebernes et al. [3] established the asymptotic behavior of the solutions of the weighted integral systems. For recent development and applications of the HLS inequalities and the Stein–Weiss inequalities, we refer the readers to [41, 50] and the references therein.

In recent years, many people are interested in the integral inequalities on the upper half space. In [33], Hang, Wang and Yan established the following integral inequality with harmonic kernel,

$$\begin{aligned} \Vert \int _{\partial \mathbb {R}^{n}_{+}}P(x,y)f(y)dy\Vert _{L^{q}\left( \mathbb {R}^{n}_{+}\right) }\le C(n,p)\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) },~~x=\left( x^{\prime },~x_{n}\right) \in \mathbb {R}^{n}_{+},~ y\in \partial \mathbb {R}^{n}_{+}, \end{aligned}$$
(1.3)

where \(P(x,y)=c(n)\frac{x_{n}}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{n}{2}}}\) and \(q=\frac{np}{n-1}\). In fact, inequality (1.3) can be regarded as Carleman’s inequality in higher dimension, and it implies the sharp isoperimetric inequality, see[34]. The authors considered the existence of extremal function for inequality (1.3) through the method of symmetrization and the concentration compactness principle, and they also discussed qualitative properties of extremal functions including regularity and symmetry. Later, Chen [13] generalized the above inequality to the case with poly-harmonic extension. More precisely, the author obtained the following integral inequality on the upper half space,

$$\begin{aligned} \Vert P_{\mu }f\Vert _{L^{\frac{np}{n-1}}(\mathbb {R}^{n}_{+})}\le C(n,\mu ,p)\Vert f\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}, \end{aligned}$$

for all \(1<p<\infty \) and \(n\ge 2\), where

$$\begin{aligned} P_{\mu }(f)(x):=\int _{\partial \mathbb {R}^{n}_{+}}\frac{x_{n}^{\mu +1-n}}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}f(y)dy \end{aligned}$$

is the poly-harmonic extension operator. Moreover, the author classified the positive extremals via the rearrangement method for \(p=\frac{2(n-1)}{2n-2-\mu }\). Dou and Zhu [24] proved the existence of extremals and computed explicitly the sharp constant by Riesz’s rearrangement technique. Recently, Gluck [31] obtained the following sharp inequalities on the upper half space

$$\begin{aligned} \left| \int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}K\left( x^{\prime }-y,x_{n}\right) f(y)g(x)dxdy\right| \le C(n,\mu ,\lambda ,p)\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert g\Vert _{L^{q^{\prime }}\left( \mathbb {R}^{n}_{+}\right) }, \end{aligned}$$
(1.4)

where K is a family of kernels

$$\begin{aligned} K(x)=K_{\mu ,\lambda }(x)=\frac{x_{n}^{\lambda }}{\left( |x^{\prime }|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}. \end{aligned}$$

If \(\mu >0\), the kernel K includes the Riesz kernel and the classical Poisson kernel as special cases. By a subcritical method, Gluck computed the best constant for a family of HLS inequalities (1.4) and gave a precise classification of the related extremals via the method of moving sphere. Liu [43] generalized the Hardy-Littlewood-Sobolev inequality with general kernel in the conformal invariant case for all critical indices. For the case \(\mu <0\), the authors in [20] proved the reversed Hardy-Littlewood-Sobolev inequality with extended kernel K. For the doubly weighted case, the Stein–Weiss inequality on the half space was established by Dou [21], the author also studied the existence of extremal functions. If the kernel function in (1.3) is replaced by fractional Poisson kernel, Chen, Liu, Lu and Tao [12] investigated the following inequality with the double weights,

$$\begin{aligned} \int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P(x,y,\mu )f(y)g(x)|x|^{-\beta }dxdy\nonumber \\\le C(n,p,q^{\prime },\alpha ,\beta ,\mu )\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert g\Vert _{L^{q^{\prime }}\left( \mathbb {R}^{n}_{+}\right) }, \end{aligned}$$
(1.5)

where \(1<p,q^{\prime }<\infty \), \(2\le \mu <n\) satisfying

$$\begin{aligned} \begin{aligned} \alpha<\frac{n-1}{p^{\prime }}, \beta <\frac{n+q}{q}, \alpha +\beta \ge 0,\\ \frac{n-1}{n}\frac{1}{p}+\frac{1}{q^{\prime }}+\frac{\alpha +\beta +2-\mu }{n}=1, \end{aligned} \end{aligned}$$

and \(P(x,y,\mu )=\frac{x_{n}}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{n+2-\mu }{2}}}\). By applying the rearrangement approach and Lorentz interpolation inequality, they studied the existence of extremals for the sharp constant of inequality (1.5). Moreover, they classified the extremals of this inequality via the regularity lifting argument and Poho\(\check{z}\)aev type identity. Especially, if \(\alpha =\beta =0\) in (1.5), the optimal inequality with the fractional Poisson kernel was established by Chen et al. [9]. Meanwhile, Chen and his collaborators [11], Tao [49] considered the reversed Stein–Weiss type inequality on the upper half space respectively.

In the present paper, we are going to study the existence of extremal functions of the Stein–Weiss type inequality with a general kernel on the half space \(\mathbb {R}^{n}_{+}\). First of all, we will establish the Stein–Weiss type inequality with general kernels on the half space. The first main result of this paper is the following integral inequality.

Theorem 1.2

Let \(n\ge 3\), \(1<p, q^{\prime }<\infty \), \(\lambda \ge 0\), \(\frac{n-1}{n}\frac{1}{p}+\frac{1}{q^{\prime }}\ge 1\) and \(\mu <n-1+\lambda \) satisfies

$$\begin{aligned} \frac{n-1}{n}\frac{1}{p}+\frac{1}{q^{\prime }}+\frac{\alpha +\beta +\mu -\lambda }{n}=\frac{2n-1}{n} \end{aligned}$$
(1.6)

with \(\alpha <\frac{n-1}{p^{\prime }}\), \(\beta <\frac{n+q}{q}\) and \(\alpha +\beta \ge 0\). Then there exists some positive constant \(C(n,p,q^{\prime },\alpha ,\beta ,\lambda ,\mu )\) such that for any functions \(f\in L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) \) and \(g\in L^{q^{\prime }}\left( \mathbb {R}^{n}_{+}\right) \), such that

$$\begin{aligned} \int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )f(y)g(x)|x|^{-\beta }dxdy\nonumber \\\le C(n,p,q^{\prime },\alpha ,\beta ,\lambda ,\mu )\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert g\Vert _{L^{q^{\prime }}\left( \mathbb {R}^{n}_{+}\right) }, \end{aligned}$$
(1.7)

where \(P_{\lambda }(x,y,\mu )=\frac{x_{n}^{\lambda }}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}\) with \(x=(x^{\prime },x_{n})\in \mathbb {R}^{n-1}\times \mathbb {R}^{+}\).

In fact, by defining the following integral operator with double weights

$$\begin{aligned} V(f)(x):=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )f(y)|x|^{-\beta }dy,\ \ x\in \mathbb {R}^{n}_{+}, \end{aligned}$$

and

$$\begin{aligned} W(g)(y):=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )g(x)|x|^{-\beta }dx,\ \ y\in \partial \mathbb {R}^{n}_{+}, \end{aligned}$$

inequality (1.7) is equivalent to the following weighted inequality via a dual argument

$$\begin{aligned} \Vert V(f)\Vert _{L^{q}\left( \mathbb {R}^{n}_{+}\right) }\le C(n,p,q^{\prime },\alpha ,\beta ,\lambda ,\mu )\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) }, \end{aligned}$$
(1.8)

and

$$\begin{aligned} \Vert W(g)\Vert _{L^{p^{\prime }}\left( \partial \mathbb {R}^{n}_{+}\right) }\le C(n,p,q^{\prime },\alpha ,\beta ,\lambda ,\mu )\Vert g\Vert _{L^{q^{\prime }}\left( \mathbb {R}^{n}_{+}\right) }, \end{aligned}$$
(1.9)

where \(\frac{n-1}{n}\frac{1}{p}+\frac{\alpha +\beta +\mu -\lambda -n+1}{n}=\frac{1}{q}\) and \(\frac{1}{p^{\prime }}=\frac{n}{n-1}\frac{1}{q^{\prime }}+\frac{\alpha +\beta +\mu -\lambda -n}{n-1}\).

Next, based on symmetric rearrangement inequality, we shall prove the existence of extremal functions for inequality (1.7).

Theorem 1.3

Let \(n\ge 3\), \(1<p,q^{\prime }<\infty \), \(\lambda \ge 0\), \(\frac{n-1}{n}\frac{1}{p}+\frac{1}{q^{\prime }}\ge 1\) and \(\mu <n-1+\lambda \) satisfying

$$\begin{aligned} \frac{n-1}{n}\frac{1}{p}+\frac{1}{q^{\prime }}+\frac{\alpha +\beta +\mu -\lambda }{n}=\frac{2n-1}{n} \end{aligned}$$
(1.10)

with \(\alpha <\frac{n-1}{p^{\prime }}\), \(\beta <\frac{n+q}{q}\) and \(\alpha ,\beta \ge 0\). Then there exists some positive functions \(f\in L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) \) satisfying \(\Vert f\Vert _{L^{p}\left( \partial \mathbb {R}^{n}_{+}\right) }=1\) and \(\Vert V(f)\Vert _{L^{q}\left( \mathbb {R}^{n}_{+}\right) }=C(n,p,q^{\prime },\alpha ,\beta ,\lambda ,\mu )\). Moreover, if \(\left( f(y), g(x)\right) \) is a pair of maximizer of inequality (1.7), then f(y) is radially symmetric and monotone decreasing about the origin, and there exists some positive constant \(c_{0}\) such that \(g(x)=c_{0}V(f)(x)\).

Furthermore, we will study the properties of these extremal functions. For this goal, we may maximize the following functional

$$\begin{aligned} J(f,g)=\int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )f(y)g(x)|x|^{-\beta }dxdy, \end{aligned}$$
(1.11)

under the assumption \(\Vert f\Vert _{L^{p}({\partial \mathbb {R}^{n}_{+}})}=\Vert g\Vert _{L^{q^{\prime }}({\mathbb {R}^{n}_{+}})}=1\). According to the Euler–Lagrange multiplier theorem, we can deduce the following integral system with double weights

$$\begin{aligned} \left\{ \begin{aligned}&J(f,g)g(x)^{q^{\prime }-1}=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )f(y)|x|^{-\beta }dy,&x\in \mathbb {R}^{n}_{+},\\&J(f,g)f(y)^{p-1}=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )g(x)|x|^{-\beta }dx,&y\in \partial \mathbb {R}^{n}_{+}. \end{aligned} \right. \end{aligned}$$
(1.12)

In particular, we assume that \(u=c_{1}f^{p-1}\), \(v=c_{2}g^{q^{\prime }-1}\), \(q_{0}=\frac{1}{q^{\prime }-1}\) and \(p_{0}=\frac{1}{p-1}\) with two suitable constants \(c_{1}\) and \(c_{2}\) in (1.12), the system can be rewritten as

$$\begin{aligned} \left\{ \begin{aligned} u(y)&=\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|y|^{-\alpha }dx,&y\in \partial \mathbb {R}^{n}_{+},\\ v(x)&=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}}(y)|x|^{-\beta }dy,&x\in \mathbb {R}^{n}_{+}, \end{aligned} \right. \end{aligned}$$
(1.13)

where \(\alpha ,\beta \ge 0\), \(\mu <n-1+\lambda \) and \(p_{0}, q_{0}\) satisfies the identity \(\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}=\frac{\alpha +\beta +\mu -\lambda }{n}\).

By applying the regularity lifting arguments, we can derive the regularity results of the positive solutions for system (1.13) under integral assumptions.

Theorem 1.4

Let \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) be a pair of positive solutions of the integral system (1.13). Then \(\left( u, v\right) \in L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \) with

$$\begin{aligned}{} & {} \frac{1}{r}\in \left( \frac{\alpha }{n-1},\frac{\alpha +\mu -\lambda +1}{n-1}\right) \cap \left( \frac{1}{p_{0}+1} -\frac{n}{n-1}\frac{1}{q_{0}+1}+\frac{\beta -1}{n-1},\frac{1}{p_{0}+1}\right. \\{} & {} \qquad \left. -\frac{n}{n-1}\frac{1}{q_{0}+1} +\frac{\beta +\mu -\lambda }{n}\right) \end{aligned}$$

and

$$\begin{aligned}{} & {} \frac{1}{s}\in \left( \frac{\beta -1}{n},\frac{\beta +\mu -\lambda }{n}\right) \cap \left( \frac{1}{q_{0}+1} -\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{\alpha }{n},\frac{1}{q_{0}+1}\right. \\{} & {} \qquad \left. -\frac{n-1}{n}\frac{1}{p_{0}+1} +\frac{\alpha +\mu -\lambda +1}{n-1}\right) . \end{aligned}$$

Next, we are going to consider the asymptotic behavior of positive solutions for system (1.13). In light of the regularity lifting theorem, we derive the following result.

Theorem 1.5

Suppose that \(p_{0}, q_{0}>1\). Let \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) be a pair of positive solutions of the integral system (1.13).

  1. (1)

    If \(\frac{1}{q_{0}}-\frac{\mu +\beta -\lambda }{q_{0}n}>\frac{\beta -1}{n}\), then

    $$\begin{aligned} \lim \limits _{|y|\rightarrow 0}u(y)|y|^{\alpha }=\int _{\mathbb {R}^{n}_{+}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx. \end{aligned}$$
  2. (2)

    If \(\frac{1}{p_{0}}-\frac{\mu +\alpha -\lambda +1}{p_{0}(n-1)}>\frac{\alpha }{n-1}\), then

    $$\begin{aligned} \lim \limits _{|x|\rightarrow 0}\frac{v(x)|x|^{\beta }}{x_{n}^{\lambda }}=\int _{\partial \mathbb {R}^{n}_{+}}\frac{u^{p_{0}}(y)}{|y|^{\alpha +\mu }}dy. \end{aligned}$$

In the spirit of inequality (1.7) and the integral assumptions, we will apply the method of moving plane in integral form in [17, 18] to study the symmetry property of the solutions.

Theorem 1.6

Given \(p_{0}, q_{0}>1\). If \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) is pair of positive solution of integral system (1.13), where \(p_{0}\) and \(q_{0}\) satisfying \(\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}=\frac{\alpha +\beta +\mu -\lambda }{n}\), then u(y) and \(v(x)|_{\partial \mathbb {R}^{n}_{+}}\) must be radially symmetric and monotonicity decreasing about some point \(y_{0}\in \partial \mathbb {R}^{n}_{+}\).

Finally, as a special case, our results can be applied to the study of the following integral system with single weight

$$\begin{aligned} \left\{ \begin{aligned} u(y)&=\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)dx,&y\in \partial \mathbb {R}^{n}_{+},\\ v(x)&=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}}(y)dy,&x\in \mathbb {R}^{n}_{+}. \end{aligned} \right. \end{aligned}$$
(1.14)

By the Poho\(\check{z}\)aev identity, we study the necessary condition for the existence of non-trivial solutions for the single weighted system (1.14).

Theorem 1.7

Given \(\lambda \ge 0\), \(\mu <n-1+\lambda \) and \(0<p_{0}, q_{0}<\infty \). Let \(\left( u,v\right) \in C^{1}(\partial \mathbb {R}_{+}^{n})\times C^{1}(\mathbb {R}_{+}^{n})\) be a pair of non-negative solutions of the integral system (1.14) with

$$\begin{aligned} \left( u,v\right) \in L^{p_{0}+1}\left( |y|^{-\alpha }dy,\partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( |x|^{-\beta }dx,\mathbb {R}^{n}_{+}\right) . \end{aligned}$$

Then it must hold that

$$\begin{aligned} \frac{n-1-\alpha }{p_{0}+1}+\frac{n-\beta }{q_{0}+1}=\mu -\lambda . \end{aligned}$$

Clearly, according to the above theorem, we have the following Liouville type result for nonnegative solutions of the single weighted integral system (1.14).

Corollary 1.8

For \(\mu <n-1+\lambda \), assume that

$$\begin{aligned} \frac{n-1-\alpha }{p_{0}+1}+\frac{n-\beta }{q_{0}+1}\not =\mu -\lambda , \end{aligned}$$

then there are no non-trivial solutions \(\left( u,v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) of the integral system (1.14).

The second part of this paper is devoted to the study of some Hartree type elliptic equations on the half space. Consider

$$\begin{aligned} \left\{ \begin{aligned} -\Delta u(x)&=\left( \int _{\partial \mathbb {R}^{n}_{+}}\frac{F\left( u(y)\right) }{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dy\right) x_{n}^{\lambda }g\left( u(x)\right) ,\ \ x\in \mathbb {R}^{n}_{+}, \\ \frac{\partial u}{\partial \upsilon }(y)&=\left( \int _{\mathbb {R}^{n}_{+}}\frac{G\left( u(x)\right) x_{n}^{\lambda }}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dx\right) f\left( u(y)\right) ,\ \ y\in \partial \mathbb {R}^{n}_{+}, \end{aligned} \right. \end{aligned}$$
(1.15)

where the parameters \(\lambda ,\alpha ,\mu ,\beta \) and the functions GFfg satisfy some specific conditions. The Hartree type equation is very important in the study of the Hartree-Fock model. This nonlocal equation has been widely used in Bose-Einstein condensates theory to study the problem how to avoid collapse phenomena. Moreover, it is used to describe the source of dark matter in classical quantum mechanics. For convenience, the reader may turn to [37, 40] and the references therein for more backgrounds about the Hartree type equations.

The qualitative properties of the solutions, such as symmetry, monotonicity and non-existence have received a great deal of interest in the last years. By applying the various versions of the method of moving plane, Lei [42], Du and Yang [25] classifid the positive solutions of the Hartree type equation with critical exponent. In [19], the authors established the same non-existence results for Hartree type equation with the boundary conditions on the half space. As an application of inequality (1.7), we are ready to study the monotonicity and non-existence results of positive solutions for problem (1.15) via moving plane argument. To present our main results precisely, we first introduce the definition of the weak solution to the Hartree type equations (1.15).

Definition 1.9

We call that \(u\in W^{1,2}_{loc}(\mathbb {R}^{n}_{+})\cap C^{0}(\overline{\mathbb {R}^{n}_{+}})\) is a weak solution of Hartree type elliptic equations (1.15) if it satisfies for all \(\varphi \in C^{\infty }_{c}(\overline{\mathbb {R}^{n}_{+}})\),

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{n}_{+}}\nabla u(x)\nabla \varphi (x) dx&= \int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}\frac{F(u(y))x_{n}^{\lambda }g(u(x))\varphi (x)}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dxdy\\&\quad +\int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}\frac{G(u(x))x_{n}^{\lambda }f(u(y))\varphi (y)}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dxdy \end{aligned} \end{aligned}$$

on the upper half space.

Now we are in a position to state our main results.

Theorem 1.10

Assume that \(\lambda \ge 0\), \(\mu <n-1+\lambda \), \(\alpha \le \frac{n-\mu }{2}\), \(\beta \le \frac{2\lambda +4-\mu }{2}\) with \(\alpha +\beta \ge 0\). Let \(u\in W^{1,2}_{loc}(\mathbb {R}^{n}_{+})\cap C^{0}(\overline{\mathbb {R}^{n}_{+}})\) be a positive solution of the system (1.15). Suppose further the functions \(f(t),~g(t),~F(t),~G(t):\left[ 0,\infty \right) \rightarrow \left[ 0,\infty \right) \) are continuous in \(\left[ 0,\infty \right) \) satisfying the following conditions,

  1. (1)

    f(t), g(t), F(t) and G(t) are increasing in \(\left( 0,+\infty \right) \),

  2. (2)

    \(H(t)=\frac{F(t)}{t^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}\), \(K(t)=\frac{G(t)}{t^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}}\), \(k(t)=\frac{g(t)}{t^{\frac{n+2\lambda +2-(2\beta +\mu )}{n-2}}}\) and \(h(t)=\frac{f(t)}{t^{\frac{n-(2\alpha +\mu )}{n-2}}}\) are non-increasing in \(\left( 0,+\infty \right) \).

Then u depend only on \(x_{n}\).

It is worth observing that Theorem 1.10 provide a useful method to discuss the non-existence of positive solutions for Hartree type equations (1.15) with the special case \(\alpha =\beta =0\). The main content is the next result.

Corollary 1.11

Under the assumption of Theorem 1.10, Suppose that at least one of the functions h, k, H and K is not a constant in \(\Big (0, \sup \limits _{y\in \partial \mathbb {R}^{n}_{+}}u(y)\Big )\) or \(\Big (0,\sup \limits _{x\in \mathbb {R}^{n}_{+}}u(x)\Big )\). Then \(u=\tilde{c}\) with \(F(\tilde{c})=G(\tilde{c})=0\).

The paper is organized as follows. In Sect. 2, we establish the sharp Stein–Weiss type inequality with a general kernel on the upper half space. Then we apply Riesz’s rearrangement inequality and Lorentz norm to obtain the existence of extremals for this inequality. In Sect. 3, by applying the regularity lifting argument and the method of moving plane in integral form, we obtain the qualitative properties of the non-negative solutions to the integral system with double weights. In Sect. 4, by using the Poho\(\check{z}\)aev identity, we shall show the necessary condition for the existence of solutions of the single weighted integral system. In the last section, we study the weak solutions to Hartree type equation and prove the symmetry and non-existence results.

2 Stein–Weiss type inequality and sharp constant

In this section, we will establish Stein–Weiss type inequality (1.7) and study the existence of extremal functions for the sharp constant \(C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)\). For simplicity, we firstly introduce some symbols by defining

$$\begin{aligned} B_{R}(x)= & {} \left\{ \xi \in \mathbb {R}^{n}: |\xi -x|<R, x\in \mathbb {R}^{n} \right\} , \\ B_{R}^{n-1}(x)= & {} \left\{ \xi \in \partial \mathbb {R}^{n}_{+}: |\xi -x|<R, x\in \partial \mathbb {R}^{n}_{n} \right\} , \\ B^{+}_{R}(x)= & {} \left\{ \xi =(\xi _{1},\xi _{2},...,\xi _{n})\in B_{R}(x): \xi _{n}>0, x\in \mathbb {R}^{n}\right\} . \end{aligned}$$

In particular, we write C or \(C_{i}\) to denote different non-negative constants, where the value may be different from line to line.

2.1 Stein–Weiss type inequality

In this subsection, we will use the following integral estimates to prove Theorem 1.2. The method of these integral estimates on the upper half space was established in [21].

Lemma 2.1

Assume that W(x) and U(y) are two positive locally integrable functions defined on \(\mathbb {R}^{n}_{+}\) and \(\partial \mathbb {R}^{n}_{+}\) respectively, for \(1<p\le q<\infty \) and f is non-negative on \(\partial \mathbb {R}^{n}_{+}\), then

$$\begin{aligned} \left( \int _{\mathbb {R}^{n}_{+}}W(x)\left( \int _{B^{n-1}_{|x|}}f(y)dy\right) ^{q}dx\right) ^{\frac{1}{q}}\le C(p,q)\left( \int _{\partial \mathbb {R}^{n}_{+}}f^{p}(y)U(y)dy\right) ^{\frac{1}{p}}, \end{aligned}$$
(2.1)

holds if and only if

$$\begin{aligned} A_{0}=\sup \limits _{R>0}\left\{ \left( \int _{|x|\ge R}W(x)dx\right) ^{\frac{1}{q}}\left( \int _{|y|\le R}U^{1-p^{\prime }}(y)dy\right) ^{\frac{1}{p^{\prime }}}\right\} <\infty . \end{aligned}$$
(2.2)

While,

$$\begin{aligned} \left( \int _{\mathbb {R}^{n}_{+}}W(x)\left( \int _{\partial \mathbb {R}^{n}_{+}\backslash B^{n-1}_{|x|}dy}f(y)\right) ^{q}dx\right) ^{\frac{1}{q}}\le C(p,q)\left( \int _{\partial \mathbb {R}^{n}_{+}}f^{p}(y)U(y)dy\right) ^{\frac{1}{p}}, \end{aligned}$$
(2.3)

holds if and only if

$$\begin{aligned} A_{1}=\sup \limits _{R>0}\left\{ \left( \int _{|x|\le R}W(x)dx\right) ^{\frac{1}{q}}\left( \int _{|y|\ge R}U^{1-p^{\prime }}(y)dy\right) ^{\frac{1}{p^{\prime }}}\right\} <\infty . \end{aligned}$$
(2.4)

By applying the above Lemma, we are ready to complete the proof of Theorem 1.2.

Proof of Theorem 1.2

Without loss of generality, we may suppose that function f is positive. In addition, we consider the double weighted integral operator related to a general kernel as follows

$$\begin{aligned} P_{\lambda }(f)(x)=\int _{\partial \mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )f(y)dy. \end{aligned}$$

Obviously, we note that inequality (1.7) is equivalent to the following inequality,

$$\begin{aligned} \Vert P_{\lambda }(f)|x|^{-\beta }\Vert _{L^{q}(\mathbb {R}^{n}_{+})}\le C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })\Vert f|y|^{\alpha }\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}. \end{aligned}$$

Since \(q>1\), we may split the integral items into the following three parts, that is

$$\begin{aligned} \Vert P_{\lambda }(f)|x|^{-\beta }\Vert _{L^{q}(\mathbb {R}^{n}_{+})}\lesssim P_{\lambda ,1}+P_{\lambda ,2}+P_{\lambda ,3}, \end{aligned}$$

where

$$\begin{aligned} \begin{aligned}&P_{\lambda ,1}=\int _{\mathbb {R}^{n}_{+}}\left( |x|^{-\beta }\int _{B^{n-1}_{\frac{|x|}{2}}}\frac{x_{n}^{\lambda }f(y)}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}dy\right) ^{q}dx,\\&P_{\lambda ,2}=\int _{\mathbb {R}^{n}_{+}}\left( |x|^{-\beta }\int _{\partial \mathbb {R}^{n}_{+}\backslash B^{n-1}_{2|x|}}\frac{x_{n}^{\lambda }f(y)}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}dy\right) ^{q}dx, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} P_{\lambda ,3}=\int _{\mathbb {R}^{n}_{+}}\left( |x|^{-\beta }\int _{B^{n-1}_{2|x|}\backslash B^{n-1}_{\frac{|x|}{2}}}\frac{x_{n}^{\lambda }f(y)}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}dy\right) ^{q}dx. \end{aligned}$$

Based on the above analysis, we only need to prove

$$\begin{aligned} P_{\lambda ,i}\le C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })\Vert f|y|^{\alpha }\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}^{q}, \quad i=1,2,3. \end{aligned}$$

Firstly, we estimate \(P_{\lambda ,1}\). From the definition of \(P_{\lambda ,1}\), we get

$$\begin{aligned} \begin{aligned} P_{\lambda ,1}&=\int _{\mathbb {R}^{n}_{+}}\left( |x|^{-\beta }\int _{B^{n-1}_{\frac{|x|}{2}}}\frac{x_{n}^{\lambda }f(y)}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}dy\right) ^{q}dx\\&\lesssim \int _{\mathbb {R}^{n}_{+}}|x|^{-\beta q-(\mu -\lambda )q}\left( \int _{B^{n-1}_{\frac{|x|}{2}}}f(y)dy\right) ^{q}dx. \end{aligned} \end{aligned}$$
(2.5)

Set \(W(x)=|x|^{-\beta q-(\mu -\lambda )q}\) and \(U(y)=|y|^{\alpha p}\), according to Lemma 2.1, to verify

$$\begin{aligned} P_{\lambda ,1}\le C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })\Vert f|y|^{\alpha }\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}, \end{aligned}$$

one only need to show that W(x) and U(y) satisfy (2.2). In fact, since \(\alpha <\frac{n-1}{p^{\prime }}\), for any \(R>0\), we have

$$\begin{aligned} \begin{aligned} \int _{|x|\ge R}W(x)dx&=\int _{|x|\ge R}|x|^{-\beta q-(\mu -\lambda )q}dx\\&=\int _{\partial B_{1}^{+}}dy\int _{R}^{\infty }r^{-\beta q-(\mu -\lambda )q}dr\\&=C(n,\beta ,\lambda ,\mu ,q)R^{n-\beta q-(\mu -\lambda )q}, \end{aligned} \end{aligned}$$
(2.6)

and

$$\begin{aligned} \begin{aligned} \int _{|y|\le R}U^{1-p^{\prime }}(y)dy&=\int _{|y|\le R}\left( |y|^{\alpha p}\right) ^{1-p^{\prime }}dy \\&=\int _{s^{n-2}}d\nu \int _{0}^{R}r^{\alpha p(1-p^{\prime })}dr&=C(n,\alpha , p)R^{\alpha p(1-p^{\prime })+n-1}. \end{aligned} \end{aligned}$$
(2.7)

It follows from (2.6), (2.7) and (1.7) that

$$\begin{aligned} \begin{aligned}&\left( \int _{|x|\ge R} W(x)dx\right) ^{\frac{1}{q}}\left( \int _{|y|\le R}U^{1-p^{\prime }}(y)dy\right) ^{p^{\prime }}\\ {}&<C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })R^{-\beta -(\mu -\lambda )+\frac{n}{q}+\frac{\alpha p(1-p^{\prime })+n-1}{p^{\prime }}}\\&=C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime }). \end{aligned} \end{aligned}$$

Next, we consider \(P_{\lambda ,2}\). Noticing that \(|y|\ge 2x\), we know \(|y-x|\ge \frac{|x|}{2}\). Setting \(W(x)=|x|^{(-\beta +\lambda )q}\) and \(U(y)=|y|^{(\mu +\alpha )p}\) in (2.3), we know

$$\begin{aligned} P_{\lambda ,2}\lesssim \int _{\mathbb {R}^{n}_{+}}|x|^{(-\beta +\lambda )q}\left( \int _{\partial \mathbb {R}^{n}_{+}\backslash B^{n-1}_{2|x|}}f(y)|y|^{-\mu }dy\right) ^{q}dx\\\le C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })\Vert f|y|^{\alpha }\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}. \end{aligned}$$

We only need to check that W(x) and U(y) satisfy the condition (2.4). Since \(\beta <\frac{n}{q}+\lambda \), for \(R>0\) we have

$$\begin{aligned} \begin{aligned} \int _{|x|\ge R}W(x)dx&=\int _{|x|\ge R}|x|^{(-\beta +\lambda )q}dx=\int _{\partial B_{1}^{+}}d\nu \int _{R}^{\infty }r^{(-\beta +\lambda )q}dr\\&=C(n,\beta ,\lambda ,q)R^{(-\beta +\lambda )q+n}, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} \int _{|y|\le R}U^{1-{p^{\prime }}}(y)dy&=\int _{|y|\le R}|y|^{\left( (\mu +\alpha )p\right) 1-{p^{\prime }}}dy=\int _{s^{n-2}}d\nu \int _{0}^{R}r^{\left( \mu +\alpha \right) p(1-{p^{\prime }})}dr\\&=C(n,\alpha \mu , p)R^{(\mu +\alpha )p(1-p^{\prime })+n-1}. \end{aligned} \end{aligned}$$

Combining the above estimates, it’s easy to find the condition (2.4) holds.

Finally, we estimate \(P_{\lambda ,3}\). By virtue of \(\frac{|x|}{2}<|y|<2|x|\) and \(\alpha +\beta \ge 0\), it follows that

$$\begin{aligned} |x-y|^{\alpha +\beta }< 3^{\alpha +\beta }|y| ^{\alpha +\beta }\le 3^{\alpha +\beta }2^{\beta }|x|^{\beta }|y|^{\alpha }. \end{aligned}$$

Furthermore, we get

$$\begin{aligned} \begin{aligned} P_{\lambda ,3}&=\int _{\mathbb {R}^{n}_{+}}\left( |x|^{-\beta }\int _{B^{n-1}_{2|x|}\backslash B^{n-1}_{\frac{|x|}{2}}}\frac{x_{n}^{\lambda }f(y)}{\left( |x^{\prime }-y|^{2}+x_{n}^{2}\right) ^{\frac{\mu }{2}}}dy\right) ^{q}dx\\&\le \int _{\mathbb {R}^{n}_{+}}\left( \int _{B^{n-1}_{2|x|}\backslash B^{n-1}_{\frac{|x|}{2}}}\frac{x_{n}^{\lambda }f(y)|y|^{\alpha }}{|x-y|^{\mu +\alpha +\beta }}dy\right) ^{q}dx\\&\le \int _{\mathbb {R}^{n}_{+}}\left( \int _{\partial R^{n}_{+}}\frac{x_{n}^{\lambda }f(y)|y|^{\alpha }}{|x-y|^{\mu +\alpha +\beta }}dy\right) ^{q}dx. \end{aligned} \end{aligned}$$

Under the assumptions of Theorem 1.2, we know \(\mu +\alpha +\beta <n-1+\lambda \). Together with the results in [43], we deduce

$$\begin{aligned} P_{\lambda ,3}\le C(n,\alpha ,\beta ,\lambda ,\mu , p,q^{\prime })\Vert f|y|^{\alpha }\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}. \end{aligned}$$

Therefore, the proof is completed. \(\square \)

2.2 The extremal functions for inequality (1.7)

In this subsection, we will prove the existence of extremal functions for inequality (1.7) obtained in the previous subsection. More precisely, we point out that the study of the existence of extremal functions of the sharp constant is related to the following variational problem

$$\begin{aligned} C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n) :=\sup \left\{ \Vert V(f)\Vert _{L^{q}(\mathbb {R}^{n}_{+})}: f\ge 0,\ \ \Vert f\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=1\right\} . \end{aligned}$$
(2.8)

We shall study the existence of maximiziers to the above supreme problem via the Riesz rearrangement and Lorentz norm, which implies that the extremal functions of inequality (1.7) are radially symmetric and decreasing about some point. For a measurable function f on \(\partial \mathbb {R}^{n}_{+}\), and we introduce the Lorentz norm with \(0<r,s<+\infty \) as follows

$$\begin{aligned} \Vert f\Vert _{L^{r,s}(\partial \mathbb {R}^{n}_{+})} :=\left\{ \begin{aligned}&\left( \int _{0}^{\infty }\left( t^{\frac{1}{r}}f^{*}(t)\right) ^{s}\frac{dt}{t}\right) ^{\frac{1}{s}},&\text{ if }\ \ s<\infty ,\\&\ \ \sup _{t>0}t^{\frac{1}{r}}f^{*}(t),&\text{ if }\ \ s=\infty , \end{aligned} \right. \end{aligned}$$

where \(f^{*}(t)\) denote the decreasing and radially symmetric rearrangement function to f.

Clearly, for \(1\le p\le \infty \), given some positive functions f, g and h, then the following Riesz rearrangement inequality holds (See [2, 38])

$$\begin{aligned} I(f,g,h)\le I(f^{*},g^{*},h^{*}) \end{aligned}$$
(2.9)

with \(I(f,g,h):=\displaystyle \int _{\partial \mathbb {R}^{n}_{+}}\int _{\mathbb {R}^{n}_{+}}f(x)g(x-y)h(y)dxdy\).

In the following, we are ready to investigate the existence of the extremal functions.

Proof of Theorem 1.3

Suppose that \(\left\{ f_{j}\right\} _{j}\) is a maximizing sequence of problem (2.8), i.e.

$$\begin{aligned} \Vert f_{j}\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=1\quad \text{ and } \quad \lim _{j\rightarrow +\infty }\Vert V(f_{j})\Vert _{L^{q}(\mathbb {R}^{n}_{+})}= C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n). \end{aligned}$$

It follows from (2.9) that

$$\begin{aligned} \Vert f^{*}_{j}\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=\Vert f_{j}\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=1,\ \ \text{ and } \lim _{j\rightarrow +\infty }\Vert V(f_{j})\Vert _{L^{q}(\mathbb {R}^{n}_{+})}\le \lim _{j\rightarrow +\infty }\Vert V(f^{*}_{j})\Vert _{L^{q}(\mathbb {R}^{n}_{+})}, \end{aligned}$$

since \(\alpha , \beta \ge 0\). As a consequence, we know that \(\left\{ f_{j}\right\} _{j}\) is a positive radially non-increasing sequence. Next we take any \(f\in L^{p}(\partial \mathbb {R}^{n}_{+})\) and set \(f^{\kappa }_{j}=\kappa ^{-\frac{n-1}{p}}f(\frac{y}{\kappa })\) with \(\kappa >0\). It’s suffice to find that

$$\begin{aligned} \Vert f^{\kappa }_{j}\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=\Vert f_{j}\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}\ \ \text{ and }\ \ \lim _{j\rightarrow +\infty }\Vert V(f_{j}^{\kappa })\Vert _{L^{q}(\mathbb {R}^{n}_{+})}\le \lim _{j\rightarrow +\infty }\Vert V(f^{\kappa }_{j})\Vert _{L^{q}(\mathbb {R}^{n}_{+})}, \end{aligned}$$

which implies that \(\left\{ f_{j}^{\kappa }\right\} _{j}\) is still a maximizing sequence to problem (2.8). Furthermore, we write

$$\begin{aligned} e_{1} :=\left( 1,0,...,0\right) \in \mathbb {R}^{n-1},\ \ A_{j} :=\sup _{\kappa>0}f_{j}^{\kappa }(e_{1})=\sup _{\kappa >0}\kappa ^{-\frac{n-1}{p}}f_{j}\left( \frac{e_{1}}{\kappa }\right) . \end{aligned}$$

By direct calculation, we get

$$\begin{aligned} 0\le f_{j}(y)\le A_{j}|y|^{-\frac{n-1}{p}}\ \ \text{ and }\ \ \Vert f_{j}\Vert _{L^{p,\infty }(\partial \mathbb {R}^{n}_{+})}\le w_{n-2}^{\frac{1}{p}}A_{j}. \end{aligned}$$
(2.10)

Moreover, according to the Marcinkiewicz interpolation [44, 45] with (1.8), we can deduce the following inequality

$$\begin{aligned} \Vert V(f)\Vert _{L^{q}(\mathbb {R}^{n}_{+})}\le C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)\Vert f\Vert _{L^{p,q}(\partial \mathbb {R}^{n}_{+})}. \end{aligned}$$

Thus, we have

$$\begin{aligned} \begin{aligned} \Vert V(f_{j})\Vert _{L^{q}(\mathbb {R}^{n}_{+})}&\le C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)\Vert f_{j}\Vert _{L^{p,q}(\partial \mathbb {R}^{n}_{+})}\\&\le C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)\Vert f_{j}\Vert _{L^{p,\infty }}^{1-\frac{p}{q}}\Vert f_{j}\Vert _{L^{p}}^{\frac{p}{q}}\\&\le C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)A_{j}^{1-\frac{p}{q}}, \end{aligned} \end{aligned}$$
(2.11)

which immediately indicates that \(A_{j}\ge c_{0}\) for some positive constant \(c_{0}\).

On one hand, by choosing \(\kappa _{j}>0\) satisfies \(f_{j}^{\kappa _{j}}(e_{1})\ge c_{0}\). We replace \(\left\{ f_{j}\right\} _{j}\) with \(\left\{ f_{j}^{\kappa _{j}}\right\} _{j}\), and denoted as \(\left\{ f_{j}\right\} _{j}\), then for any j, we have \(\left\{ f_{j}\right\} _{j}\ge c_{0}\). For any \(R>0\), it holds that

$$\begin{aligned} \begin{aligned} \omega _{n-1}f_{j}^{p}(R)R^{n-1}&\le \omega _{n-2}\int _{0}^{R}f_{j}^{p}(r)r^{n-2}dr\le \omega _{n-2}\int _{0}^{\infty }f_{j}^{p}(r)r^{n-2}dr\\&=\int _{\partial R^{n}_{+}}f_{j}^{p}(y)dy=1. \end{aligned} \end{aligned}$$

Based on the arguments above, we obtain

$$0\le f_{j}(y)\le \omega _{n-1}^{-\frac{1}{p}}|y|^{-\frac{n-1}{p}}.$$

According to Lieb’s results based on the Helly theorem as in [39], we infer that there exists a positive, radially non-increasing function f such that

$$\begin{aligned} f_{j}\rightarrow f,\ \ \text{ a.e. }\ \ \partial \mathbb {R}^{n}_{+}. \end{aligned}$$

It’s clear that for \(|y|\le 1\) and \(\Vert f\Vert _{L^{p}(\partial \mathbb {R}_{+}^{n})}\le 1\), we have \(f(y)\ge c_{0}\). Furthermore, according to the Brezis-Lieb Lemma [1], it holds that

$$\begin{aligned} \begin{aligned} \lim _{j\rightarrow +\infty }\Vert f_{j}-f\Vert ^{p}_{L^{p}(\partial \mathbb {R}^{n}_{+})}&=\lim _{j\rightarrow +\infty }\Vert f_{j}\Vert ^{p}_{L^{p}(\partial \mathbb {R}^{n}_{+})}-\Vert f\Vert ^{p}_{L^{p}(\partial \mathbb {R}^{n}_{+})}\\&=1-\Vert f\Vert ^{p}_{L^{p}(\partial \mathbb {R}^{n}_{+})}. \end{aligned} \end{aligned}$$
(2.12)

For some constant \(C>0\), we have from (2.10),

$$\begin{aligned} V(f_{j})(x)\le C |x|^{-\beta }\int _{\partial \mathbb {R}^{n}_{+}}\frac{x_{n}^{\lambda }}{|y|^{\alpha }\left( |x^{\prime }-y|+x_{n}\right) ^{\frac{\mu }{2}}}\frac{1}{|y|^{-\frac{n-1}{p}}}dy. \end{aligned}$$
(2.13)

In view of the assumptions of Theorem 1.3, it’s easy to find that the integral is finite. Therefore, according to the dominated convergence theorem, for \(x\in \mathbb {R}^{n}_{+}\), we deduce that \(\lim \limits _{j\rightarrow +\infty }V(f_{j})(x)=V(f)(x)\).

By virtue of the Brezis-Lieb Lemma, we obtain

$$\begin{aligned} \begin{aligned}&\lim _{j\rightarrow +\infty }\Vert V(f_{j})\Vert ^{q}_{L^{q}(\mathbb {R}^{n}_{+})}=\Vert V(f)\Vert ^{q}_{L^{q}(\mathbb {R}^{n}_{+})}+\lim _{j\rightarrow +\infty }\Vert V(f_{j})-V(f)\Vert ^{q}_{L^{q}(\mathbb {R}^{n}_{+})}\\&\quad \le C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)^{q}\Vert f\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}+C(\alpha ,\beta ,\mu ,\lambda ,p,q^{\prime },n)^{q}\lim _{j\rightarrow +\infty }\Vert f_{j}-f\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}. \end{aligned} \end{aligned}$$
(2.14)

Combining (2.12) and (2.14), lt holds that

$$\begin{aligned} 1\le \Vert f\Vert ^{q}_{L^{p}(\partial \mathbb {R}^{n}_{+})}+\left( 1-\Vert f\Vert ^{p}_{L^{p}(\partial \mathbb {R}^{n}_{+})}\right) ^{\frac{q}{p}}. \end{aligned}$$

Since \(p<q\) and \(f\ne 0\), we deduce that \(\Vert f\Vert _{L^{p}(\partial \mathbb {R}^{n}_{+})}=1\). With all the analysis above, it’s clear that f is a maximizer to the problem (2.8). The proof is completed. \(\square \)

3 Qualitative analysis of the positive solutions

In this section, we study the qualitative properties of positive solutions to integral equations (1.13). More precisely, we are ready to obtain the regularity, asymptotic behaviors and symmetry. First of all, we introduce some basic definitions. Suppose that V is a topological vector space, and we define two fundamental norms \(\Vert \cdot \Vert _{X}\) and \(\Vert \cdot \Vert _{Y}\) on V,

$$\begin{aligned} \Vert \cdot \Vert _{X}, \Vert \cdot \Vert _{Y}: V\rightarrow [0,\infty ]. \end{aligned}$$

Let

$$\begin{aligned} X:=\{v\in V:\Vert v\Vert _{X}<\infty \}, Y:=\{v\in V:\Vert v\Vert _{Y}<\infty \}. \end{aligned}$$

We recall that the operator \(T:X\rightarrow Y\)

  • is said to be contracting if for any \(f,h\in X\), there exists some constant \(\varrho \in (0,1)\) such that

    $$\begin{aligned} \Vert T(f)-T(h)\Vert _{X}\le \varrho \Vert f-h\Vert _{Y}. \end{aligned}$$
  • is said to be shrinking if for any \(h\in X\), there exists some constant \(\delta \in (0,1)\) such that

    $$\begin{aligned} \Vert T(h)\Vert _{X}\le \delta \Vert h\Vert _{Y}. \end{aligned}$$

Clearly, it is not difficult to see that, a linear shrinking operator must be contracting.

Next, we recall the regularity lifting lemma in [15], which will play an essential role in the discussion.

Lemma 3.1

[15] Let T be a contraction map from \(X\rightarrow X\) and \(Y\rightarrow Y\), \(f\in X\) and there exists a function \(g\in X\cap Y\) such that \(f=Tf+g\) in X. Then \(f\in X\cap Y\).

Proof of Theorem 1.4

For any constant \(A>0\), we define

$$\begin{aligned} \begin{aligned}&u_{A}(y)= {\left\{ \begin{array}{ll} u(y),\ \ |u(y)|>A~\text{ or }~|y|>A,\\ 0,~~~~~~\text{ otherwise }, \end{array}\right. }&v_{A}(x)= {\left\{ \begin{array}{ll} v(x),\ \ |v(x)|>A~\text{ or }~|x|>A,\\ 0,~~~~~~\text{ otherwise }, \end{array}\right. } \end{aligned} \end{aligned}$$

\(u_{B}(y)=u(y)-u_{A}(y)\) and \(v_{B}(x)=v(x)-v_{A}(x)\). Define the linear operator \(T_{1}\) as

$$\begin{aligned} T_{1}(h)(y)=\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }P_{\lambda }(x,y,\mu )v_{A}^{q_{0}-1}(x)h(x)|y|^{-\alpha }dx,\ \ y\in \partial \mathbb {R}^{n}_{+}, \end{aligned}$$

and

$$\begin{aligned} T_{2}(h)(x)=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u_{A}^{p_{0}-1}h(x)|x|^{-\beta }dy,\ \ x\in \mathbb {R}^{n}_{+}. \end{aligned}$$

Noticing that \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) is a pair of non-negative solutions of the integral system (1.13), we have

$$\begin{aligned} \begin{aligned} u(y)&=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\alpha }dx\\&=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )\left( v_{A}(x)+v_{B}(x)\right) ^{q_{0}-1}v(x)|x|^{-\alpha }dx\\&=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v_{A}^{q_{0}-1}(x)v(x)|x|^{-\alpha }dx+\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v_{B}^{q_{0}}(x)|x|^{-\alpha }dx\\&:=T_{1}(v)(y)+F(y). \end{aligned} \end{aligned}$$

Similarly,

$$\begin{aligned} \begin{aligned} v(x)=&\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}}(y)|x|^{-\beta }dy\\ =&\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )\left( u_{A}(y)+u_{B}(y)\right) ^{p_{0}-1}u(y)|x|^{-\beta }dy\\ =&\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u_{A}^{p_{0}-1}(y)u(y)|x|^{-\beta }dy\\&\,+\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u_{A}(y)+u_{B}^{p_{0}}(y)|x|^{-\beta }dy\\ :=&T_{2}(u)(x)+G(x), \end{aligned} \end{aligned}$$

where

$$\begin{aligned}{} & {} F(y)=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v_{B}^{q_{0}}(x)|x|^{-\alpha }dx,\\{} & {} G(x)=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u_{A}(y)+u_{B}^{p_{0}}(y)|x|^{-\beta }dy. \end{aligned}$$

Furthermore, we may define the operator \(T:L^{r}(\partial \mathbb {R}^{n}_{+})\times L^{s}(\mathbb {R}^{n}_{+})\),

$$\begin{aligned} T(h_{1},h_{2})=\left( T_{1}(h_{2}), T_{2}(h_{1})\right) \end{aligned}$$

with norm \(\Vert \left( h_{1},h_{2}\right) \Vert _{r,s}=\Vert h_{1}\Vert _{L^{r}(\partial \mathbb {R}^{n}_{+})}+\Vert h_{2}\Vert _{L^{s}(\mathbb {R}^{n}_{+})}\). Obviously, it holds that

$$\begin{aligned} \left( u,v\right) =T(u,v)+(F,G). \end{aligned}$$

To take advantage of the regularity lifting argument via contracting map, we set the parameters r and s satisfy

$$\begin{aligned} \frac{1}{s}+\frac{n-1}{n}\frac{1}{p_{0}+1}=\frac{1}{q_{0}+1}+\frac{n-1}{n}\frac{1}{r}. \end{aligned}$$

Noticing that under the hypothesis of Theorem 1.4, the existence of parameters r and s can be ensured. To prove that \(\left( u, v\right) \in L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \), we only need to show that, for A sufficiently large, the following hold.

  1. (1)

    T is shrinking from \(L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) to \(L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \).

  2. (2)

    T is shrinking from \(L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \) to \(L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \).

  3. (3)

    \(\left( F,G\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \cap L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \).

For the proof of (1). In fact, by applying the weighted integral inequality (1.8) and the Hölder inequality, for \((h_{1},h_{2})\in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \), we know

$$\begin{aligned} \begin{aligned} \Vert T_{1}(h_{2})\Vert _{L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) }&\le C_{1}\Vert v_{A}^{q_{0}-1}\Vert _{L^{\frac{q_{0}+1}{q_{0}-1}}\left( \mathbb {R}^{n}_{+}\right) }\Vert h_{2}\Vert _{L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) }\\&\le C_{1}\Vert v_{A}\Vert ^{q_{0}-1}_{L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) }\Vert h_{2}\Vert _{L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) }, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} \Vert T_{2}(h_{1})\Vert _{L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) }&\le C_{2} \Vert u_{A}^{p_{0}-1}\Vert _{L^{\frac{p_{0}+1}{p_{0}-1}}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert h_{1}\Vert _{L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) }\\&\le C_{2}\Vert u_{A}\Vert ^{p_{0}-1}_{L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert h_{1}\Vert _{L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) }, \end{aligned} \end{aligned}$$

where constant \(C_{1}, C_{2}>0\).

Notice that since the integrability \(L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \), for A sufficiently large, we deduce

$$\begin{aligned} \Vert T\left( h_{1},h_{2}\right) \Vert _{p_{0}+1,q_{0}+1}=\Vert T_{1}(h_{2})\Vert _{p_{0}+1}+\Vert T_{2}(h_{1})\Vert _{q_{0}+1}\le \frac{1}{2}\Vert \left( h_{1},h_{2}\right) \Vert _{p_{0}+1,q_{0}+1} \end{aligned}$$

which immediately implies that T is a shrinking operator from \(L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \) to itself.

Next, we will use Stein–Weiss type inequality (1.7) with general kernels to prove (2), which is similar to the arguments for (1). For convenience, we only verify that \(\Vert T_{2}(h_{1})\Vert _{ L^{s}\left( \mathbb {R}^{n}_{+}\right) }\le \frac{1}{2}\Vert h_{1}\Vert _{L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) }\), since \(\Vert T_{1}(h_{2})\Vert _{L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) }\le \frac{1}{2}\Vert h_{2}\Vert _{ L^{s}\left( \mathbb {R}^{n}_{+}\right) }\) can be proved in same way.

Actually, there exists positive constant C such that

$$\begin{aligned} \begin{aligned} \Vert T_{2}(h_{1})\Vert _{ L^{s}\left( \mathbb {R}^{n}_{+}\right) }&\le C\Vert u_{A}^{p_{0}-1}h_{1}\Vert _{L^{z}(\partial \mathbb {R}^{n}_{+})}\\&\le C\Vert u_{A}\Vert ^{p_{0}-1}_{L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) }\Vert h_{1}\Vert _{L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) }. \end{aligned} \end{aligned}$$
(3.1)

In view of \(u\in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \), by selecting A sufficiently large in (3.1), we have

$$\begin{aligned} \Vert T_{2}(h_{1})\Vert _{ L^{s}\left( \mathbb {R}^{n}_{+}\right) }\le \frac{1}{2}\Vert h_{1}\Vert _{L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) },\quad \forall h_{1}\in L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) . \end{aligned}$$

In conclusion, we require the parameters r, s and z satisfy

$$\begin{aligned} \frac{1}{z}-\frac{1}{r}=\frac{p_{0}-1}{p_{0}+1}, \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} \frac{1}{s}&=\frac{\alpha +\beta +\mu -\lambda -n+1}{n}+\frac{n-1}{n}\frac{1}{z}\\&=\frac{\alpha +\beta +\mu -\lambda -n+1}{n}+\frac{n-1}{n}\left( \frac{p_{0}-1}{p_{0}+1}+\frac{1}{r}\right) \\&=\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}-\frac{n-1}{n}+\frac{n-1}{n}\left( \frac{p_{0}-1}{p_{0}+1}+\frac{1}{r}\right) \\&=\frac{n-1}{n}\frac{1}{r}+\frac{1}{q_{0}+1}-\frac{n-1}{n}\frac{1}{p_{0}+1}, \end{aligned} \end{aligned}$$

where we applied

$$\begin{aligned} \frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}=\frac{\alpha +\beta +\mu -\lambda }{n}. \end{aligned}$$

Based on the analysis above, we conclude that \(\Vert T(h_{1},h_{2})\Vert _{r,s}\le \frac{1}{2}\Vert h_{1},h_{2}\Vert _{r,s}\), which indicates that T is a shrinking operator from \(L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \) to itself.

Finally, we prove (3), that is \(\left( F,G\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \cap L^{r}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{s}\left( \mathbb {R}^{n}_{+}\right) \). It should be noted that, \(u_{B}\) and \(v_{B}\) are uniformly bounded by A. Thus the proof of Theorem 1.4 is completed by using the regularity lifting Lemma 3.1. \(\square \)

3.1 Asymptotic estimates

In this subsection, we are going to study the asymptotic behaviors of non-negative solutions of integral system (1.13).

Proof of Theorem 1.5

In order to prove that

$$\begin{aligned} \lim \limits _{|y|\rightarrow 0}u(y)|y|^{\alpha }=\int _{\mathbb {R}^{n}_{+}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx, \end{aligned}$$
(3.2)

first, we verify that \(\int _{\mathbb {R}^{n}_{+}}\frac{v^{q_{0}}(x)}{|x|^{\mu +\beta -\lambda }}dx<+\infty \). In fact, for any \(R>0\), we observe that

$$\begin{aligned} \int _{\mathbb {R}^{n}_{+}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx=\int _{B^{+}_{R}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx+\int _{\mathbb {R}^{n}_{+}\backslash B^{+}_{R}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx. \end{aligned}$$

On one hand, since \(u\in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \), then there exists \(y_{0}\in \partial \mathbb {R}^{n}_{+}\) satisfies \(|y_{0}|<\frac{R}{2}\) such that \(u(y_{0})<+\infty \). From the system (1.13),

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{n}_{+}\backslash B^{+}_{R}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx&\le C\int _{\mathbb {R}^{n}_{+}\backslash B^{+}_{R}}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx\\&\quad +\int _{B^{+}_{R}}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx\\&\le C\int _{\mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx=|y_{0}|^{\alpha }u(y_{0})<+\infty , \end{aligned} \end{aligned}$$

since \(|x-y|<|x|\).

On the other hand, by using the Hölder inequality, we get

$$\begin{aligned} \int _{B^{+}_{R}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx\le C\left( \int _{B^{+}_{R}}\left( \frac{1}{|x|^{\mu +\beta -\lambda }}\right) ^{t^{\prime }}dx\right) ^{\frac{1}{t^{\prime }}}\left( \int _{B^{+}_{R}}v^{q_{0}t}(x)dx\right) ^{\frac{1}{t}}. \end{aligned}$$

With the aim of \(\int _{\mathbb {R}^{n}_{+}}\frac{v^{q_{0}}(x)}{|x|^{\mu +\beta -\lambda }}dx<+\infty \), now we require that \(\left( \mu +\beta -\lambda \right) t^{\prime }<n\) and

$$\begin{aligned}{} & {} \frac{1}{q_{0}t}\in \left( \frac{\beta -1}{n},\frac{\beta +\mu -\lambda }{n}\right) \cap \left( \frac{1}{q_{0}+1} -\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{\alpha }{n},\frac{1}{q_{0}+1}\right. \\{} & {} \qquad \left. -\frac{n-1}{n}\frac{1}{p_{0}+1} +\frac{\alpha +\mu -\lambda +1}{n-1}\right) . \end{aligned}$$

Noting that \(\left( \mu +\beta -\lambda \right) t^{\prime }<n\), we have

$$\begin{aligned} \frac{\mu +\beta -\lambda }{q_{0}n}<\frac{1}{q_{0}}-\frac{1}{q_{0}t}. \end{aligned}$$

Therefore, since \(q_{0}>1\) and \(\frac{1}{p_{0}+1}>\frac{\alpha }{n-1}\), it is easy to find that

$$\begin{aligned} \begin{aligned} \frac{1}{q_{0}}-\frac{\mu +\beta -\lambda }{q_{0}n}&=\frac{1}{q_{0}}-\frac{1}{q_{0}}\left( \frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}-\frac{\alpha }{n}\right) \\&=\frac{1}{q_{0}+1}-\frac{1}{q_{0}}\left( \frac{n-1}{n}\frac{1}{p_{0}+1}-\frac{\alpha }{n}\right) \\&>\frac{1}{q_{0}+1}-\frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{\alpha }{n}, \end{aligned} \end{aligned}$$

here we applied the condition

$$\begin{aligned} \frac{n-1}{n}\frac{1}{p_{0}+1}+\frac{1}{q_{0}+1}=\frac{\alpha +\beta +\mu -\lambda }{n}. \end{aligned}$$

With the above analysis, we can select a suitable parameter t such that \(\left( \mu +\beta -\lambda \right) t^{\prime }<n\) and \(\Vert v\Vert _{L^{q_{0}t}(\mathbb {R}^{n}_{+})}<+\infty \), which immediately means that \(\int _{\mathbb {R}^{n}_{+}}\frac{v^{q_{0}}(x)}{|x|^{\mu +\beta -\lambda }}dx<+\infty \).

Next, we prove (3.2). Direct calculation yields that

$$\begin{aligned} \begin{aligned}&\left| \int _{\mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx-\int _{\mathbb {R}^{n}_{+}}\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}dx\right| \\&\quad \le \left| \int _{B^{+}_{\rho }}\left( P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }-\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}\right) dx\right| \\&\qquad +\left| \int _{\mathbb {R}^{n}_{+}\backslash B^{+}_{\rho }}\left( P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }-\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}\right) dx\right| \\&\quad :=A_{1}+A_{2}. \end{aligned} \end{aligned}$$
(3.3)

On one hand, for \(A_{1}\), we have

$$\begin{aligned} \begin{aligned} \int _{B^{+}_{\rho }}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx&\le \int _{B^{+}_{\rho }(y)}\frac{v^{q_{0}}(x)}{|x-y|^{\mu +\beta -\lambda }}dx+\int _{B^{+}_{\rho }}\frac{v^{q_{0}}(x)}{|x-y|^{\mu +\beta -\lambda }}dx\\&\le 2\Vert v^{q_{0}}\Vert _{L^{t}(\mathbb {R}^{n}_{+})}\Vert \frac{1}{|x|^{\mu +\beta -\lambda }}\Vert _{L^{t^{\prime }}(B^{+}_{\rho })}. \end{aligned} \end{aligned}$$
(3.4)

Taking the limit in (3.4), which leads to \(\lim \limits _{\rho \rightarrow 0}\lim \limits _{|y|\rightarrow 0} A_{1}=0\).

On the other hand, according to the Lebesgue dominated convergence theorem for \(A_{2}\), we get

$$\begin{aligned} \lim \limits _{|y|\rightarrow 0}\int _{\mathbb {R}^{n}_{+}\backslash B^{+}_{\rho }}\left( P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }-\frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}\right) dx=0. \end{aligned}$$
(3.5)

With the help of (3.4) and (3.5), we derive

$$\begin{aligned} \begin{aligned} \lim \limits _{|y|\rightarrow 0}&\left| \int _{\mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx-\int _{\mathbb {R}^{n}_{+}}\left( \frac{x_{n}^{\lambda }v^{q_{0}}(x)}{|x|^{\mu +\beta }}\right) dx\right| \\&=\lim \limits _{\rho \rightarrow 0}\lim \limits _{|y|\rightarrow 0}A_{1}+\lim \limits _{\rho \rightarrow 0}\lim \limits _{|y|\rightarrow 0}A_{2}=0. \end{aligned} \end{aligned}$$

For the other case, if \(\frac{1}{p_{0}}-\frac{\mu +\alpha -\lambda +1}{p_{0}(n-1)}>\frac{\alpha }{n-1}\), then

$$\begin{aligned} \lim \limits _{|x|\rightarrow 0}\frac{v(x)|x|^{\beta }}{x_{n}^{\lambda }}=\int _{\partial \mathbb {R}^{n}_{+}}\frac{u^{p_{0}}(y)}{|y|^{\alpha +\mu }}dy. \end{aligned}$$

Similarly, we can prove the second conclusion. In conclusion, the proof of Theorem 1.5 is completed. \(\square \)

3.2 Symmetry via the moving plane argument

In this subsection, we will prove the symmetry of positive solutions under the integral conditions. In order to apply the moving plane argument, we give some basic notations. For any \(\tau \in \mathbb {R}\), one write

$$\begin{aligned} y^{\tau }= \left( 2\tau -y_{1},...,y_{n-1}\right) ,\ \ x^{\tau }=\left( 2\tau -x_{1},...,x_{n}\right) ,\ \ u(y^{\tau })=u_{\tau }(y),\ \ v(x^{\tau })=v_{\tau }(x), \end{aligned}$$

and

$$\begin{aligned} T_{\tau }=\left\{ x\in \mathbb {R}^{n}: x_{1}=\tau \right\} ,\ \ \Sigma _{y,\tau }=\left\{ y\in \partial \mathbb {R}^{n}_{+}: y_{1}<\tau \right\} ,\ \ \ \ \Sigma _{x,\tau }=\left\{ x\in \mathbb {R}^{n}_{+}: x_{1}<\tau \right\} . \end{aligned}$$

Next, we will show the following equalities, which are useful in the method of moving plane.

Lemma 3.2

Supposed that \(\left( u,v\right) \) is a pair of non-negative solution of the integral system (1.13), for any \(y\in \partial \mathbb {R}^{n}_{+}\) and \(x\in \mathbb {R}^{n}_{+}\), it holds that

$$\begin{aligned} \begin{aligned} u(y)-u_{\tau }(y)&=\int _{\Sigma _{x,\tau }}P_{\lambda }(x,y,\mu )\left( |y|^{-\alpha }v^{q_{0}}(x)|x|^{-\beta }-|y^{\tau }|^{-\alpha }v_{\tau }^{q_{0}}(x)|x^{\tau }|^{-\beta }\right) dx\\&+\int _{\Sigma _{x,\tau }}P_{\lambda }(x^{\tau },y,\mu )\left( |y|^{-\alpha }v_{\tau }^{q_{0}}(x)|x|^{-\beta }-|y^{\tau }|^{-\alpha }v^{q_{0}}(x)|x|^{-\beta }\right) dx, \end{aligned} \end{aligned}$$
(3.6)

and

$$\begin{aligned} \begin{aligned} v(x)-v_{\tau }(x)=&\int _{\Sigma _{y,\tau }}P_{\lambda }(x,y,\mu )\left( |y|^{-\alpha }u^{p_{0}}(y)|x|^{-\beta }-|y^{\tau }|^{-\alpha }u_{\tau }^{p_{0}}(y)|x^{\tau }|^{-\beta }\right) dy\\&+\int _{\Sigma _{y,\tau }}P_{\lambda }(x^{\tau },y,\mu )\left( |y|^{-\alpha }u_{\tau }^{p_{0}}(y)|x|^{-\beta }-|y^{\tau }|^{-\alpha }u^{p_{0}}(y)|x|^{-\beta }\right) dy, \end{aligned} \end{aligned}$$
(3.7)

Proof

By direct computation, we know

$$\begin{aligned} \begin{aligned} u(y)&=\int _{\mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx\\&=\int _{\Sigma _{x,\tau }}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)|x|^{-\beta }dx+\int _{\Sigma _{x,\tau }}|y|^{-\alpha }P_{\lambda }(x^{\tau },y,\mu )v_{\tau }^{q_{0}}(x)|x^{\tau }|^{-\beta }dx, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} u_{\tau }(y)=&\int _{\mathbb {R}^{n}_{+}}|y^{\tau }|^{-\alpha }P_{\lambda }(x,y^{\tau },\mu )v^{q_{0}}(x)|x|^{-\beta }dx\\&=\int _{\Sigma _{x,\tau }}|y^{\tau }|^{-\alpha }P_{\lambda }(x,y^{\tau },\mu )v^{q_{0}}(x)|x|^{-\beta }dx\\&+\int _{\Sigma _{x,\tau }}|y^{\tau }|^{-\alpha }P_{\lambda }(x^{\tau },y^{\tau },\mu )v_{\tau }^{q_{0}}(x)|x^{\tau }|^{-\beta }dx. \end{aligned} \end{aligned}$$

Since \(P_{\lambda }(x^{\tau },y^{\tau },\mu )=P_{\lambda }(x,y,\mu )\) and \(P_{\lambda }(x,y^{\tau },\mu )=P_{\lambda }(x^{\tau },y,\mu )\), it’s not difficult to find that

$$\begin{aligned} \begin{aligned} u(y)-u_{\tau }(y)=&\int _{\Sigma _{x,\tau }}P_{\lambda }(x,y,\mu )\left( |y|^{-\alpha }v^{q_{0}}(x)|x|^{-\beta }-|y^{\tau }|^{-\alpha }v_{\tau }^{q_{0}}(x)|x^{\tau }|^{-\beta }\right) dx\\&+\int _{\Sigma _{x,\tau }}P_{\lambda }(x^{\tau },y,\mu )\left( |y|^{-\alpha }v_{\tau }^{q_{0}}(x)|x|^{-\beta }-|y^{\tau }|^{-\alpha }v^{q_{0}}(x)|x|^{-\beta }\right) dx. \end{aligned} \end{aligned}$$

Clearly, in the same ways, we get

$$\begin{aligned} \begin{aligned} v(x)-v_{\tau }(x)=&\int _{\Sigma _{y,\tau }}\left( |y|^{-\alpha }P_{\lambda }(x,y,\mu )|x|^{-\beta }-|y^{\tau }|^{-\alpha }P_{\lambda }(x,y^{\tau },\mu )|x|^{-\beta }\right) u^{p_{0}}(y)dy\\&+\int _{\Sigma _{y,\tau }}\left( |y|^{-\alpha }P_{\lambda }(x,y^{\tau },\mu )|x^{\tau }|^{-\beta }-|y^{\tau }|^{-\alpha }P_{\lambda }(x^{\tau },y^{\tau },\mu )|x^{\tau }|^{-\beta }\right) u_{\tau }^{p_{0}}(y)dy\\&=\int _{\Sigma _{y,\tau }}P_{\lambda }(x,y,\mu )\left( |y|^{-\alpha }u^{p_{0}}(y)|x|^{-\beta }-|y^{\tau }|^{-\alpha }u_{\tau }^{p_{0}}(y)|x^{\tau }|^{-\beta }\right) dy\\&+\int _{\Sigma _{y,\tau }}P_{\lambda }(x^{\tau },y,\mu )\left( |y|^{-\alpha }u_{\tau }^{p_{0}}(y)|x|^{-\beta }-|y^{\tau }|^{-\alpha }u^{p_{0}}(y)|x|^{-\beta }\right) dy. \end{aligned} \end{aligned}$$

This finishes the proof. \(\square \)

Proof of Theorem 1.6

We divide the proof into two steps.

Step 1. We claim that for \(\tau \) sufficiently negative, there holds

$$\begin{aligned} u_{\tau }(y)\ge u(y),\ \ v_{\tau }(x)\ge v(x),\ \ \forall x\in \Sigma _{x,\tau },\ \ y\in \Sigma _{y,\tau }. \end{aligned}$$
(3.8)

Let

$$\begin{aligned} \Sigma _{y,\tau }^{u}=\left\{ y\in \Sigma _{y,\tau }| u(y)>u_{\tau }(y)\right\} ,\ \ \Sigma _{x,\tau }^{v}=\left\{ x\in \Sigma _{x,\tau }| v(x)>v_{\tau }(x)\right\} . \end{aligned}$$

Thus, we will show that for \(\tau \) sufficiently negative, the sets \(\Sigma _{y,\tau }^{v}\) and \(\Sigma _{x,\tau }^{u}\) must have measure zero.

For any \(x\in \Sigma _{x,\tau }^{v}\) and \(y\in \Sigma _{y,\tau }^{u}\), since \(|x^{\tau }|<|x|\), we know from (3.6)

$$\begin{aligned} \begin{aligned} u(y)-u_{\tau }(y)&\le \int _{\Sigma _{x,\tau }}|y|^{-\alpha }\!\left( P_{\lambda }(x,y,\mu )\!-\!P_{\lambda }(x^{\tau },y,\mu )\right) \left( v^{q_{0}}(x)|x|^{-\beta }\!-\!v_{\tau }^{q_{0}}(x)|x^{\tau }|^{-\beta }\right) dx\\&\le \int _{\Sigma _{x,\tau }^{v}}|y|^{-\alpha }P_{\lambda }\left( v^{q_{0}}(x)-v_{\tau }^{q_{0}}(x)\right) dx. \end{aligned} \end{aligned}$$

Applying the Mean Value Theorem, we get

$$\begin{aligned} u(y)-u_{\tau }(y)\le q_{0}\int _{\Sigma _{x,\tau }^{v}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )v^{q_{0}-1}(x)\left( v(x)-v_{\tau }(x)\right) dx. \end{aligned}$$

Similarly,

$$\begin{aligned} v(x)-v_{\tau }(x)\le p_{0}\int _{\Sigma _{y,\tau }^{u}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}-1}(y)\left( u(y)-u_{\tau }(y)\right) dy. \end{aligned}$$

Using the Hölder inequality and Theorem 1.2, we obtain

$$\begin{aligned} \Vert u-u_{\tau }\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau }^{u})}\le C\Vert v\Vert ^{q_{0}-1}_{L^{q_{0}+1}(\Sigma _{x,\tau })}\Vert v-v_{\tau }\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau }^{v})}, \end{aligned}$$
(3.9)

and

$$\begin{aligned} \Vert v-v_{\tau }\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau }^{v})}\le C\Vert u\Vert ^{p_{0}-1}_{L^{p_{0}+1}(\Sigma _{y,\tau })}\Vert u-u_{\tau }\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau }^{u})}. \end{aligned}$$
(3.10)

In view of \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \), for \(\tau \) sufficiently negative, it holds that

$$\begin{aligned} \Vert v\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau })}\le \frac{1}{2},\ \ \Vert u\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau })}\le \frac{1}{2}, \end{aligned}$$

which indicates that

$$\begin{aligned} \Vert u-u_{\tau }\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau }^{u})}=0,\ \ \Vert v-v_{\tau }\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau }^{v})}=0. \end{aligned}$$
(3.11)

Therefore, we deduce that \(\Sigma _{y,\tau }^{u}\) and \(\Sigma _{x,\tau }^{v}\) must be empty set. This implies that (3.8) holds.

Step 2. Inequality (3.8) provides a starting point to move the plane \(T_{\tau }\). Furthermore, we move the plane \(T_{\tau }\) from \(-\infty \) to the right as long as (3.8) holds. One can denote

$$\begin{aligned} \tau _{0}=\sup \left\{ \tau |u_{\rho }(y)\ge u(y), v_{\rho }(x)\ge v(x), \rho \le \tau , \forall y\in \Sigma _{y,\tau }, x\in \Sigma _{x,\tau } \right\} . \end{aligned}$$
(3.12)

Assume that \(\tau _{0}<0\), then we will show that the solution u and v must be symmetric and monotone about the limiting plane. Namely,

$$\begin{aligned} u_{\tau _{0}}(y)\equiv u(y),\ \ v_{\tau _{0}}(x)\equiv v(x),\ \ \forall y\in \Sigma _{y,\tau _{0}}, x\in \Sigma _{x,\tau _{0}}. \end{aligned}$$
(3.13)

Suppose for such a \(\tau _{0}\), on \(\Sigma _{y,\tau _{0}}\) and \(\Sigma _{x,\tau _{0}}\), we get

$$\begin{aligned} u(y)\le u_{\tau _{0}}(y),\ \ v(x)\le v_{\tau _{0}}(x),\ \ \text{ but }\ \ u(y)\not =u_{\tau _{0}}(y),\ \ v(x)\not = v_{\tau _{0}}(x). \end{aligned}$$
(3.14)

We will show that the plane can be moved further to the right. To be precise, there exists a positive parameter \(\varepsilon \) such that for any \(x\in \Sigma _{x,\tau }\) and \(y\in \Sigma _{y,\tau }\),

$$\begin{aligned} u(y)\le u_{\tau }(y), v(x)\le v_{\tau }(x),\ \ \forall \tau \in \left[ \tau _{0},\tau _{0}+\varepsilon \right) , \end{aligned}$$
(3.15)

which would contradict with the definition of \(\tau _{0}\).

Combining with the first step and (3.14), we know that \(u(y)<u_{\tau _{0}}(y)\) and \(v(x)<v_{\tau _{0}}(x)\) in the interior of \(\Sigma _{y,\tau _{0}}\) and \(\Sigma _{x,\tau _{0}}\) respectively. Moreover, we define

$$\begin{aligned} \Sigma ^{\widetilde{v}}_{x,\tau _{0}}=\left\{ x\in \Sigma _{x,\tau _{0}}|v(x)\ge v_{\tau }(x)\right\} ,\ \ \Sigma ^{\widetilde{u}}_{y,\tau _{0}}=\left\{ y\in \Sigma _{y,\tau _{0}}|u(y)\ge u_{\tau }(y)\right\} . \end{aligned}$$

Therefore, it’s easy to find that \(\Sigma ^{\widetilde{v}}_{x,\tau _{0}}\) and \(\Sigma ^{\widetilde{u}}_{y,\tau _{0}}\) have measure zero, and \(\lim _{\tau \rightarrow \tau _{0}}\Sigma ^{u}_{y,\tau }\subset \Sigma ^{\widetilde{u}}_{y,\tau _{0}}\), \(\lim _{\tau \rightarrow \tau _{0}}\Sigma ^{v}_{x,\tau }\subset \Sigma ^{\widetilde{v}}_{x,\tau _{0}}\). Furthermore, By virtue of \(\left( u, v\right) \in L^{p_{0}+1}\left( \partial \mathbb {R}^{n}_{+}\right) \times L^{q_{0}+1}\left( \mathbb {R}^{n}_{+}\right) \), we can select a adequate small \(\varepsilon \) such that for any \(\tau \) in \(\left[ \tau _{0},\tau _{0}+\varepsilon \right) \),

$$\begin{aligned} \Vert v\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau })}\le \frac{1}{2},\ \ \Vert u\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau })}\le \frac{1}{2}. \end{aligned}$$

In fact, the estimate is similar to (3.11), it holds that

$$\begin{aligned} \Vert u-u_{\tau }\Vert _{L^{p_{0}+1}(\Sigma _{y,\tau }^{u})}=0,\ \ \Vert v-v_{\tau }\Vert _{L^{q_{0}+1}(\Sigma _{x,\tau }^{v})}=0. \end{aligned}$$

Based on the above discussion, we conclude that \(\Sigma ^{u}_{y,\tau }\) and \(\Sigma ^{v}_{x,\tau }\) must be empty set. This proves (3.15) and hence (3.13).

If \(\tau _{0}=0\), then we can repeat the arguments in the opposite direction. Indeed, we will get two situations. If the plane \(T_{\tau }\) stop at some point before the origin, we can deduce that u(y) and v(x) must be symmetric and monotone decreasing in the \(x_{1}\)-direction based on the previous analysis. If it stays at the origin again, we also have the symmetry and monotonicity result with \(x_{1}=0\). Since \(x_{1}\) direction can be chosen arbitrarily, we deduce that u(y) and \(v(x)|_{\partial \mathbb {R}^{n}_{+}}\) must be radially symmetry and monotone decreasing about some point \(y_{0}\in \partial \mathbb {R}^{n}_{+}\). The proof is accomplished. \(\square \)

4 The single weighted integral system

In this section, we will consider the necessary condition for the existence of non-negative solutions of the integral system (1.14) with single weight.

$$\begin{aligned} \left\{ \begin{aligned} u(y)&=\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }P_{\lambda }(x,y,\mu )v^{q_{0}}(x)dx,&y\in \partial \mathbb {R}^{n}_{+},\\ v(x)&=\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}}(y)dy,&x\in \mathbb {R}^{n}_{+}. \end{aligned} \right. \end{aligned}$$

Proof of Theorem 1.7

Firstly, according to the integration by parts, we get

$$\begin{aligned} \begin{aligned} \int _{B^{n-1}_{R}\backslash B^{n-1}_{\varepsilon }}&|y|^{-\alpha }u^{p_{0}}(y)\left( y\cdot \nabla u(y)\right) dy \\&=\frac{1}{p_{0}+1}\int _{B^{n-1}_{R}\backslash B^{n-1}_{\varepsilon }}|y|^{-\alpha }y\cdot \nabla \left( u^{p_{0}+1}(y)\right) dy\\&=\frac{R^{1-\alpha }}{p_{0}+1}\int _{\partial B^{n-1}_{R}}u^{p_{0}+1}(y)d\tau +\frac{\varepsilon ^{1-\alpha }}{p_{0}+1}\int _{\partial B^{n-1}_{\varepsilon }}u^{p_{0}+1}(y)d\tau \\&\quad -\frac{n-1-\alpha }{p_{0}+1}\int _{B^{n-1}_{R}\backslash B^{n-1}_{\varepsilon }}u^{p_{0}+1}(y)|y|^{-\alpha }dy. \end{aligned} \end{aligned}$$

Similarly, we know

$$\begin{aligned} \begin{aligned} \int _{B^{+}_{R}\backslash B^{+}_{\varepsilon }}&|x|^{-\beta }v^{q_{0}}(x)\left( x \cdot \nabla v(x)\right) dx\\&=\frac{1}{q_{0}+1}\int _{B^{+}_{R}\backslash B^{+}_{\varepsilon }}|x|^{-\beta }x\cdot \nabla \left( v^{q_{0}+1}(x)\right) dx\\&=\frac{R^{1-\beta }}{q_{0}+1}\int _{\partial B^{+}_{R}}v^{q_{0}+1}(y)d\tau +\frac{\varepsilon ^{1-\beta }}{q_{0}+1}\int _{\partial B^{+}_{\varepsilon }}v^{q_{0}+1}(x)d\tau \\&\quad -\frac{n-\beta }{q_{0}+1}\int _{B^{+}_{R}\backslash B^{+}_{\varepsilon }}v^{q_{0}+1}(x)|x|^{-\beta }dx. \end{aligned} \end{aligned}$$

In particular, under the assumptions of Theorem 1.7, we know \(\left( u,v\right) \in L^{p_{0}+1}\left( |y|^{-\alpha }dy,\partial R^{n}_{+}\right) \times L^{q_{0}+1}\left( |x|^{-\beta }dx,R^{n}_{+}\right) \), then there exist \(R_{i}\rightarrow +\infty \) and \(\varepsilon _{i}\rightarrow 0\) such that

$$\begin{aligned} R^{1-\alpha }_{i}\int _{\partial B^{n-1}_{R_{i}}}u^{p_{0}+1}(y)d\tau \rightarrow 0,\quad R^{1-\beta }_{i}\int _{\partial B^{+}_{R_{i}}}v^{q_{0}+1}(x)d\tau \rightarrow 0. \end{aligned}$$

Similarly, we obtain

$$\begin{aligned} \varepsilon ^{1-\alpha }_{i}\int _{\partial B^{n-1}_{\varepsilon _{i}}}u^{p_{0}+1}(y)d\tau \rightarrow 0,\quad \varepsilon ^{1-\beta }_{i}\int _{\partial B^{+}_{\varepsilon _{i}}}v^{q_{0}+1}(x)d\tau \rightarrow 0. \end{aligned}$$

Based on the above analysis, we get

$$\begin{aligned} \begin{aligned} \int _{\partial \mathbb {R}^{n}_{+}}&|y|^{-\alpha }u^{p_{0}}(y)\left( y\cdot \nabla u(y)\right) dy+\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }v^{q_{0}}(x)\left( x \cdot \nabla v(x)\right) dx\\&=-\frac{n-1-\alpha }{p_{0}+1}\int _{\partial \mathbb {R}^{n}_{+}}|y|^{-\alpha }u^{p_{0}+1}(y)dy-\frac{n-\beta }{q_{0}+1}\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }v^{q_{0}+1}(x)dx. \end{aligned} \end{aligned}$$
(4.1)

Noticing that the weighted integral equations (1.14), by direct calculation, we obtain

$$\begin{aligned} \begin{aligned} \nabla u(y)\cdot y&=\frac{du(\rho y)}{d\rho }|_{\rho =1}\\&=-\mu \int _{\mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )|x-y|^{-2}(y-x)\cdot y |x|^{-\beta }v^{q_{0}}(x)dx, \end{aligned} \end{aligned}$$
(4.2)

and

$$\begin{aligned} \begin{aligned} \nabla v(x)\cdot x&=\frac{dv(\rho x)}{d\rho }|_{\rho =1}\\&=-\mu \int _{\partial \mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )|x-y|^{-2}(x-y)\cdot x|y|^{-\alpha }u^{p_{0}}(y)dy\\&\quad +\lambda \int _{\partial \mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )|y|^{-\alpha }u^{p_{0}}(y)dy. \end{aligned} \end{aligned}$$
(4.3)

It follows from (4.2) and (4.3) that

$$\begin{aligned} \begin{aligned} \int _{\partial \mathbb {R}^{n}_{+}}&|y|^{-\alpha }u^{p_{0}}(y)\left( y\cdot \nabla u(y)\right) dy+\int _{\mathbb {R}^{n}_{+}}|x|^{-\beta }v^{q_{0}}(x)\left( x \cdot \nabla v(x)\right) dx\\&=-\left( \mu -\lambda \right) \int _{\mathbb {R}^{n}_{+}}\int _{\partial R^{n}_{+}}|y|^{-\alpha }P_{\lambda }(x,y,\mu )u^{p_{0}+1}(y)v^{q_{0}+1}(x)|x|^{-\beta }dy dx\\&=-\left( \mu -\lambda \right) \int _{\mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )v^{q_{0}+1}(x)|x|^{-\beta }dx\\&=-\left( \mu -\lambda \right) \int _{\partial \mathbb {R}^{n}_{+}}P_{\lambda }(x,y,\mu )u^{p_{0}+1}(y)|y|^{-\alpha }dy. \end{aligned} \end{aligned}$$

Combining (4.1) and the above identity, we know that \(\frac{n-1-\alpha }{p_{0}+1}+\frac{n-\beta }{q_{0}+1}=\mu -\lambda \), which completes the proof. \(\square \)

5 Application to nonlocal elliptic equation on the upper half space

In this section, as an application of Stein–Weiss type inequality (1.7), we are interested in studying the symmetry and non-existence of positive solutions for equations (1.15) by the method of moving plane in half space. In fact, since the nonlinear term is non-local, we fail to obtain the symmetry result for positive solutions via the maximum principle. Therefore, we provide a different proof using the integral inequality to establish the symmetry and non-existence of weak solutions for the equations (1.15). For convenience, we first write

$$\begin{aligned} m(x)=\int _{\partial \mathbb {R}^{n}_{+}}\frac{F(u(y))}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dy,\qquad \eta (y)=\int _{\mathbb {R}^{n}_{+}}\frac{G(u(x))x_{n}^{\lambda }}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dx, \end{aligned}$$
(5.1)

then the equations (1.15) can be rewritten by the following form,

$$\begin{aligned} \left\{ \begin{aligned} -\Delta u(x)&=x_{n}^{\lambda }m(x)g(u(x)),\ \ x\in \mathbb {R}^{n}_{+},\\ \frac{\partial u}{\partial \upsilon }(y)&=\eta (y)f(u(y)),\ \ y\in \partial \mathbb {R}^{n}_{+}. \end{aligned} \right. \end{aligned}$$
(5.2)

Next, due to the lack of the decay property of those solutions, it is not convenient to prove symmetry and nonexistence results for equations (1.15) via the moving plane arguments directly. To overcome this difficulty, we will introduce the Kelvin transform of centered at a point. Then let us take any point \(x_{p}\in \partial \mathbb {R}^{n}_{+}\) and define the Kelvin transform of u(x), m(x) and \(\eta (y)\) as follows.

$$\begin{aligned} \begin{aligned}&v(x)=\frac{1}{|x-x_{\rho }|^{n-2}}u\left( \frac{x-x_{\rho }}{|x-x_{\rho }|^{2}}+x_{\rho }\right) ,\quad \omega (x)=\frac{1}{|x-x_{\rho }|^{2\beta +\mu }}m\left( \frac{x-x_{\rho }}{|x-x_{\rho }|^{2}}+x_{\rho }\right) ,\\&z(y)=\frac{1}{|y-y_{\rho }|^{2\alpha +\mu }}\eta \left( \frac{y-y_{\rho }}{|y-y_{\rho }|^{2}}+y_{\rho }\right) . \end{aligned} \end{aligned}$$

Obviously, by the above definition, for \(|x-x_{p}|\ge 1\), we have

$$\begin{aligned} v(x)\le \frac{C}{|x-x_{p}|^{n-2}},\ \ \omega (x)\le \frac{C}{|x-x_{p}|^{2\beta +\mu }}, \end{aligned}$$

and

$$\begin{aligned} z(y)\le \frac{C}{|y-y_{p}|^{2\alpha +\mu }}. \end{aligned}$$
(5.3)

It’s easy to find that \(\left( v,\omega ,z\right) \) has the singularities at point \(x_{p}\). Here and the rest of this paper, without loss of generality, we take \(x_{p}=0\).

In addition, by a straightforward computation, we obtain \(v(x),~\omega (x),~z(y)\) satisfies the following elliptic system

$$\begin{aligned} \left\{ \begin{aligned} -\Delta v(x)&=x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}},\ \ x\in \mathbb {R}^{n}_{+},\\ \frac{\partial v}{\partial \upsilon }(y)&=z(y)h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}},\ \ y\in \partial \mathbb {R}^{n}_{+}\backslash \left\{ 0\right\} ,\\ \omega (x)&=\int _{\partial \mathbb {R}^{n}_{+}}\frac{H(|y|^{n-2}v(y))}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}dy,\ \ x\in \mathbb {R}^{n}_{+},\\ z(y)&=\int _{\mathbb {R}^{n}_{+}}\frac{K(|x|^{n-2}v(x))x_{n}^{\lambda }}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}v(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}dx,\ \ y\in \partial \mathbb {R}^{n}_{+}\backslash \left\{ 0\right\} . \end{aligned} \right. \end{aligned}$$
(5.4)

Now we turn to the symmetry and monotonicity of v(x). We introduce the following notation. For \(\delta >0\), we define

$$\begin{aligned} \Sigma _{\delta }=\left\{ x\in \mathbb {R}^{n}_{+}|x_{1}>\delta \right\} ,\ \ \partial \Sigma _{\delta }=\left\{ x\in \partial \mathbb {R}^{n}_{+}|x_{1}>\delta \right\} ,\ \ T_{\delta }=\left\{ x\in \mathbb {R}^{n}_{+}|x_{1}=\delta \right\} , \end{aligned}$$

and we also denote the reflected point and functions relate to the hyperplane \(T_{\delta }\) by

$$\begin{aligned} x^{\delta }=\left( 2\delta -x_{1},...,x_{n}\right) ,\ \ v_{\delta }(x)=v(x^{\delta }), p^{\delta }=\left( 2\delta ,0,...,0\right) . \end{aligned}$$

Furthermore, we write

$$\begin{aligned} \Sigma _{\delta }^{v}=\left\{ x\in \Sigma _{\delta }|v(x)>v_{\delta }(x)\right\} ,\ \ \partial \Sigma _{\delta }^{v}=\left\{ x\in \partial \Sigma _{\delta }|v(x)>v_{\delta }(x)\right\} . \end{aligned}$$

In light of the above preparations, we shall give two basic inequality, which is useful in the later proof.

Lemma 5.1

Assume that v(x) is non-negative weak solution of the equations (1.15), then we have

$$\begin{aligned} \omega (x)-\omega _{\delta }(x)\le \int _{\partial \Sigma _{\delta }^{v}}\frac{H\left( |y|^{n-2}v(y)\right) }{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}\left( v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}-v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}\right) dy, \end{aligned}$$
(5.5)

and

$$\begin{aligned} z(y)-z_{\delta }(y)\le \int _{\Sigma _{\delta }^{v}}\frac{K\left( |x|^{n-2}v(x)\right) x_{n}^{\lambda }}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}\left( v(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}-v_{\delta }(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}\right) dx. \end{aligned}$$
(5.6)

Proof

By direct calculation, we get

$$\begin{aligned} \omega (x)=\int _{\partial \Sigma _{\delta }^{v}}\frac{H\left( |y|^{n-2}v(y)\right) }{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}dy\\+\int _{\partial \Sigma _{\delta }^{v}}\frac{H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) }{|x|^{\beta }|x-y^{\delta }|^{\mu }|y^{\delta }|^{\alpha }}v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}dy, \end{aligned}$$

and

$$\begin{aligned} \omega _{\delta }(x)= & {} \int _{\partial \Sigma _{\delta }^{v}}\frac{H\left( |y|^{n-2}v(y)\right) }{|x^{\delta }|^{\beta }|x^{\delta }-y|^{\mu }|y|^{\alpha }}v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}dy\\{} & {} +\int _{\partial \Sigma _{\delta }^{v}}\frac{H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) }{|x^{\delta }|^{\beta }|x^{\delta }-y^{\delta }|^{\mu }|y^{\delta }|^{\alpha }}v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}dy. \end{aligned}$$

Since \(x\in \Sigma _{\delta }\) and \(y\in \partial \Sigma _{\delta }\), then we get

$$\begin{aligned} \begin{aligned}&\omega (x)-\omega _{\delta }(x)=\int _{\partial \Sigma _{\delta }}\frac{1}{|x-y|^{\mu }}\left( \frac{H\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}{|x|^{\beta }|y|^{\alpha }}\right. \\&\qquad \left. -\frac{H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}{|x^{\delta }|^{\beta }|y^{\delta }|^{\alpha }}\right) dy\\&\qquad +\int _{\partial \Sigma _{\delta }}\frac{1}{|x^{\delta }-y|^{\mu }}\left( \frac{H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}{|x|^{\beta }|y^{\delta }|^{\alpha }}-\frac{H\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}{|x|^{\beta }|y|^{\alpha }}\right) dy\\&\quad \le \int _{\Sigma _{\delta }}\frac{1}{|x|^{\beta }|y|^{\alpha }}\left( \frac{1}{|x-y|^{\mu }}-\frac{1}{|x^{\delta }-y|^{\mu }}\right) \\&\qquad \cdot \left( H\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}-H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}\right) dy. \end{aligned} \end{aligned}$$

If \(y\in \partial \Sigma _{\delta }^{v}\), together with the monotonicity of H, then we know

$$\begin{aligned} H\left( |y|^{n-2}v(y)\right) \le H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) . \end{aligned}$$

If \(y\in \partial \Sigma _{\delta }\backslash \partial \Sigma _{\delta }^{v}\), then we have

$$\begin{aligned} \begin{aligned} H\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}&=\frac{F\left( |y|^{n-2}v(y)\right) }{|y|^{2(n-1)-(2\alpha +\mu )}}\le \frac{F\left( |y|^{n-2}v_{\delta }(y)\right) }{|y|^{2(n-1)-(2\alpha +\mu )}}\\&\le \frac{F\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) }{\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) ^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}}v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}\\&=H\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}. \end{aligned} \end{aligned}$$

From the analysis above, we immediately get the identity (5.5).

Similarly, from

$$\begin{aligned} z(y)=\int _{\Sigma _{\delta }^{v}}\frac{K\left( |x|^{n-2}v(x)\right) x_{n}^{\lambda }}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}v(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}dx\\+\int _{\Sigma _{\delta }^{v}}\frac{K\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) x_{n}^{\lambda }}{|x^{\delta }|^{\beta }|x^{\delta }-y|^{\mu }|y|^{\alpha }}v_{\delta }(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}dx, \end{aligned}$$

and

$$\begin{aligned} z_{\delta }(y)=\int _{\Sigma _{\delta }^{v}}\frac{K\left( |x|^{n-2}v(x)\right) x_{n}^{\lambda }}{|x|^{\beta }|x-y^{\delta }|^{\mu }|y^{\delta }|^{\alpha }}v(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}dx\\+\int _{\Sigma _{\delta }^{v}}\frac{K\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) x_{n}^{\lambda }}{|x^{\delta }|^{\beta }|x^{\delta }-y^{\delta }|^{\mu }|y^{\delta }|^{\alpha }}v_{\delta }(x)^{\frac{2n+2\lambda -(2\beta +\mu )}{n-2}}dx, \end{aligned}$$

we arrive at (5.6). The proof is completed. \(\square \)

Lemma 5.2

Under the conditions of Theorem 1.10, for any fixed parameter \(\delta >0\), then it holds that

  1. (1)

    \(v(x)\in L^{\frac{2n}{n-2}}(\Sigma _{\delta })\cup L^{\infty }(\Sigma _{\delta })\),

  2. (2)

    \((v-v_{\delta })^{+}\in L^{\frac{2n}{n-2}}(\Sigma _{\delta })\cup L^{\infty }(\Sigma _{\delta })\).

Moreover, there exists positive constant \(C_{\delta }\), which is non-increasing in \(\delta \), such that

$$\begin{aligned} \begin{aligned} \int _{\Sigma _{\delta }}&|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\\&\le C_{\delta }\left[ \Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}+\Vert \omega (x)\Vert _{L^{\frac{2n}{(2\beta +\mu )-2\lambda }}(\Sigma _{\delta }^{v})}\Vert v(x)\Vert ^{\frac{2\lambda +4-(2\beta +\mu )}{2n}}_{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}\right. \\&~~\left. +\Vert z(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\Vert v(y)\Vert ^{\frac{2-(2\alpha +\mu )}{2(n-1)}}_{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\right] \left( \int _{\Sigma _{\delta }^{v}}|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\right) . \end{aligned} \end{aligned}$$
(5.7)

Proof

Actually, because of \(\delta >0\), there exists a parameter \(r>0\) such that \(\Sigma _{\delta }\subset \mathbb {R}^{n}_{+}\backslash B_{r}^{+}(0)\). Therefore, in light of the definition and decay property of v(x), we have

$$\begin{aligned} v(x),~(v-v_{\delta })^{+}\in L^{\frac{2n}{n-2}}(\Sigma _{\delta })\cup L^{\infty }(\Sigma _{\delta }), \end{aligned}$$

where we denote \(B_{r}^{+}(0)=\left\{ x\in \mathbb {R}^{n}_{+}| |x|<r\right\} \).

Next, in order to remove the singularity of v(x), \(\omega (x)\) and z(y), we need to introduce a cut-off function \(\phi =\phi _{\varepsilon }(x)\in C^{1}(\mathbb {R}^{n}, \left[ 0,1\right] )\) as below

$$\begin{aligned} \phi _{\varepsilon }(x)= \left\{ \begin{aligned}&1,\ \ 2\varepsilon \le |x-p^{\delta }|\le \frac{1}{\varepsilon },\\&0,\ \ |x-p^{\delta }|<\varepsilon ,\ \ |x-p^{\delta }|>\frac{2}{\varepsilon }. \end{aligned} \right. \end{aligned}$$

Furthermore, we require that \(|\nabla \phi |\le \frac{2}{\varepsilon }\) for \(\varepsilon<|x-p^{\delta }|<2\varepsilon \) and \(|\nabla \phi |\le 2\varepsilon \) for \(\frac{1}{\varepsilon }<|x-p^{\delta }|<\frac{2}{\varepsilon }\). In addition, we also define two functions \(\psi (x)\) and \(\varphi (x)\) satisfy \(\varphi =\varphi _{\varepsilon }=\phi _{\varepsilon }^{2}\left( v-v_{\delta }\right) ^{+}\) and \(\psi =\psi _{\varepsilon }=\phi _{\varepsilon }\left( v-v_{\delta }\right) ^{+}\) respectively, it’s easy to find that

$$\begin{aligned} |\nabla \psi |^{2}=\nabla \left( v-v_{\delta }\right) ^{+}\nabla \varphi +\left[ \left( v-v_{\delta }\right) ^{+}\right] ^{2}|\nabla \phi |^{2}. \end{aligned}$$

Now, based on the above preparation, we deduce from (5.4)

$$\begin{aligned} \begin{aligned}&\int _{\Sigma _{\delta }\cup \left\{ 2\varepsilon \le |x-p^{\delta }|\le \frac{1}{\varepsilon }\right\} }|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\\&\quad \le \int _{\Sigma _{\delta }^{v}}|\nabla \psi (x)|^{2}dx\le \int _{\Sigma _{\delta }}\nabla \left( v(x)-v_{\delta }(x)\right) ^{+}\nabla \varphi dx+\int _{\Sigma _{\delta }^{v}}\left[ \left( v(x)-v_{\delta }(x)\right) ^{+}\right] ^{2}|\nabla \phi _{\varepsilon }|^{2}dx\\&\quad =\int _{\Sigma _{\delta }^{v}}-\Delta \left( v-v_{\delta }\right) \varphi (x)dx+\int _{\partial \Sigma _{\delta }^{v}}\frac{\partial \left( v-v_{\delta }\right) (y)}{\partial \upsilon }\varphi (y)dy+\int _{\Sigma _{\delta }^{v}}\left[ \left( v(x)-v_{\delta }(x)\right) ^{+}\right] ^{2}|\nabla \phi _{\varepsilon }|^{2}dx\\&\quad =\int _{\Sigma _{\delta }^{v}}\left[ x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-x_{n}^{\lambda }\omega (x^{\delta })k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\right. \\&\qquad \times \left. \left( v(x)-v_{\delta }(x)\right) ^{+}\phi _{\varepsilon }^{2}(x)dx\right] -\int _{\partial \Sigma _{\delta }^{v}}\left[ z(y)h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\right. \\&\qquad -z(y^{\delta })h\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\\&\qquad \left. \times \left( v(y)-v_{\delta }(y)\right) ^{+}\phi _{\varepsilon }^{2}(y)dy\right] +\int _{\Sigma _{\delta }^{v}}\left[ \left( v(x)-v_{\delta }(x)\right) ^{+}\right] ^{2}|\nabla \phi _{\varepsilon }|^{2}dx\\&\quad :=I_{1}+I_{2}+I_{3}. \end{aligned} \end{aligned}$$
(5.8)

Apparently, we are going to consider the three integrals above. For integrals \(I_{1}\), if \(x\in \Sigma _{\delta }^{v}\), then we have

$$\begin{aligned} k\left( |x|^{n-2}v(x)\right) \le k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) , \end{aligned}$$

where we used the monotonicity of k. Meanwhile, we treat the domain \(\Sigma _{\delta }^{v}\) as

$$\begin{aligned} \Omega _{1}=\left\{ x\in \Sigma _{\delta }^{v}|\omega (x)>\omega _{\delta }(x)\right\} \end{aligned}$$

and

$$\begin{aligned} \Omega _{2}=\left\{ x\in \Sigma _{\delta }^{v}|\omega (x)\le \omega _{\delta }(x)\right\} . \end{aligned}$$

If \(x\in \Omega _{1}\), we know

$$\begin{aligned} \begin{aligned} x_{n}^{\lambda }&\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-x_{n}^{\lambda }\omega (x^{\delta })k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&=\left[ \omega (x)-\omega (x^{\delta })\right] k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&\quad +x_{n}^{\lambda }\omega (x^{\delta })\left[ k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\right] \\&\le x_{n}^{\lambda }\left[ \omega (x)-\omega (x^{\delta })\right] k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&\quad +x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) \left[ v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\right] . \end{aligned} \end{aligned}$$
(5.9)

If \(x\in \Omega _{2}\), then we have

$$\begin{aligned} \begin{aligned} x_{n}^{\lambda }&\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-x_{n}^{\lambda }\omega (x^{\delta })k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&\le x_{n}^{\lambda }\omega (x)\left[ k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-k\left( |x^{\delta }|^{n-2}v_{\delta }(x)\right) v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\right] \\&\le x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) \left[ v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}-v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\right] . \end{aligned} \end{aligned}$$
(5.10)

Clearly, integral \(I_{2}\) can be estimated in the same way. Therefore, for \(y\in \partial \Sigma _{\delta }^{v}\), it follow from the monotonicity of h that,

$$\begin{aligned} h\left( |y|^{n-2}v(y)\right) \le h\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) . \end{aligned}$$

Similarly, we can divide domain \(\partial \Sigma _{\delta }^{v}\) into

$$\begin{aligned} \Omega _{3}=\left\{ y\in \partial \Sigma _{\delta }^{v}|z(y)>z_{\delta }(y)\right\} \end{aligned}$$

and

$$\begin{aligned} \Omega _{4}=\left\{ y\in \partial \Sigma _{\delta }^{v}|z(y)\le z_{\delta }(y)\right\} . \end{aligned}$$

If \(y\in \Omega _{3}\), then we have

$$\begin{aligned} \begin{aligned}&z(y)h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}-z(y^{\delta })h\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\\&\quad \le z(y)h\left( |y|^{n-2}v(y)\right) \left[ v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}-v_{\delta }(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\right] \\&\qquad +h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left[ z(y)-z(y^{\delta })\right] , \end{aligned} \end{aligned}$$
(5.11)

it also holds that for \(y\in \Omega _{4}\),

$$\begin{aligned} \begin{aligned} z(y)&h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}-z(y^{\delta })h\left( |y^{\delta }|^{n-2}v_{\delta }(y)\right) v_{\delta }(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\\&\le z(y)h\left( |y|^{n-2}v(y)\right) \left[ v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}-v_{\delta }(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\right] . \end{aligned} \end{aligned}$$
(5.12)

Consequently, inserting (5.9), (5.10), (5.11) and (5.12) into (5.8), we must have

$$\begin{aligned} \begin{aligned}&\int _{\Sigma _{\delta }\cup \left\{ 2\varepsilon \le |x-p^{\delta }|\le \frac{1}{\varepsilon }\right\} }|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\\&\quad \le I_{3}+C\int _{\Sigma _{\delta }^{v}}x_{n}^{\lambda }\left[ \omega (x)-\omega (x^{\delta })\right] k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}\phi _{\varepsilon }^{2}(x)dx\\&\qquad +C\int _{\Sigma _{\delta }^{v}}x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +4-(2\beta +\mu )}{n-2}}\left[ \left( v(x)-v_{\delta }(x)\right) ^{+}\right] ^{2}\phi _{\varepsilon }^{2}(x)dx\\&\qquad +C\int _{\partial \Sigma _{\delta }^{v}}z(y)h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2-(2\alpha +\mu )}{n-2}}\left[ \left( v(y)-v_{\delta }(y)\right) ^{+}\right] ^{2}\phi _{\varepsilon }^{2}(y)dy\\&\qquad +C\int _{\partial \Sigma _{\delta }^{v}}h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left[ z(y)-z(y^{\delta })\right] \left( v(y)-v_{\delta }(y)\right) ^{+}\phi _{\varepsilon }^{2}(y)dy\\&\quad =I_{3}+A_{1}+A_{2}+A_{3}+A_{4}. \end{aligned} \end{aligned}$$
(5.13)

In fact, based on the discussion above, we focus on estimating the five integrals in the remaining of proof.

Firstly, to estimate integrals \(I_{3}\) accurately, we write \(M_{1}=\left\{ x\in \Sigma _{\delta }|\varepsilon<|x-p^{\delta }|<2\varepsilon \right\} \) and \(M_{2}=\left\{ x\in \Sigma _{\delta }|\frac{1}{\varepsilon }<|x-p^{\delta }|<\frac{2}{\varepsilon }\right\} \), then we have

$$\begin{aligned} \int _{M_{1}}|\nabla \phi |^{n}dx\le C\frac{1}{\varepsilon ^{n}}\cdot \varepsilon ^{n}=C. \end{aligned}$$

Similarly,

$$\begin{aligned} \int _{M_{2}}|\nabla \phi |^{n}dx\le C\frac{1}{\varepsilon ^{n}}\cdot \varepsilon ^{n}=C. \end{aligned}$$

As a consequence, when \(\varepsilon \rightarrow 0\), by the Hölder inequality and \((v-v_{\delta })^{+}\in L^{\frac{2n}{n-2}}(\Sigma _{\delta })\), we conclude

$$\begin{aligned} I_{3}\le \left( \int _{M_{1}\subset M_{2}}\left[ (v-v_{\delta })^{+}\right] ^{\frac{2n}{n-2}}dx\right) ^{\frac{n-2}{n}}\left( \int _{\Sigma _{\delta }}|\nabla \phi |^{n}dx\right) ^{\frac{2}{n}}\rightarrow 0. \end{aligned}$$

Secondly, for integral \(A_{1}\), according to Theorem 1.2, we infer from (1.11) and the Hölder inequality that

$$\begin{aligned} \begin{aligned} A_{1}&\le C_{\delta }\int _{\Sigma _{\delta }^{v}}x_{n}^{\lambda }\left[ \omega (x)-\omega (x^{\delta })\right] v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}dx\\&\le C_{\delta }\int _{\Sigma _{\delta }^{v}}\int _{\partial \Sigma _{\delta }^{v}}\frac{x_{n}^{\lambda }v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}\left( v(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}-v_{\delta }(y)^{\frac{2(n-1)-(2\alpha +\mu )}{n-2}}\right) }{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dxdy\\&\le C_{\delta }\Vert v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\left( v(x) -v_{\delta }(x)\right) ^{+}\Vert _{L^{\frac{2n}{2n+2\lambda -(2\beta +\mu )}}(\Sigma _{\delta }^{v})}\Vert \\&\quad v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left( v(y)-v_{\delta }(y)\right) ^{+}\Vert _{L^{\frac{2(n-1)}{2n-2-(2\alpha +\mu )}}(\partial \Sigma _{\delta }^{v})}\\&\le C_{\delta }\Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert \left( v(x)-v_{\delta }(x)\right) \Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}\Vert \\&\quad v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}\Vert \left( v(y)-v_{\delta }(y)\right) \Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}. \end{aligned} \end{aligned}$$
(5.14)

Next, for integral \(A_{2}\), combining the Hölder inequality with the decay estimate of v, there exists a positive constant \(C_{\delta }\), which is non-increasing in \(\delta \), such that

$$\begin{aligned} \begin{aligned} A_{2}&\le \int _{\Sigma _{\delta }^{v}}x_{n}^{\lambda }\left[ \omega (x)-\omega (x^{\delta })\right] k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}dx\\&\le C_{\delta }\left( \int _{\Sigma _{\delta }^{v}}\omega (x)^{\frac{2n}{(2\beta +\mu )-2\lambda }}dx\right) ^{\frac{(2\beta +\mu )-2\lambda }{2n}}\left( \int _{\Sigma _{\delta }^{v}}v(x)^{\frac{2n}{n-2}}dx\right) ^{\frac{2\lambda +4-(2\beta +\mu )}{2n}}\\&\quad \left( \int _{\Sigma _{\delta }^{v}}\left[ \left( v(x)-v_{\delta }(x)\right) ^{+}\right] ^{\frac{2n}{n-2}}dx\right) ^{\frac{n-2}{n}}, \end{aligned} \end{aligned}$$
(5.15)

where we used the monotonicity of k.

Furthermore, for integral \(A_{3}\), in view of the Hölder inequality, we have

$$\begin{aligned} \begin{aligned} A_{3}&\le \int _{\partial \Sigma _{\delta }^{v}}z(y)h\left( |y|^{n-2}v(y)\right) v(y)^{\frac{2-(2\alpha +\mu )}{n-2}}\left[ \left( v(y)-v_{\delta }(y)\right) ^{+}\right] ^{2}dy\\&\le C_{\delta }\left( \int _{\partial \Sigma _{\delta }^{v}}v(y)^{\frac{2(n-1)}{n-2}}dy\right) ^{\frac{2-(2\alpha +\mu )}{2(n-1)}}\left( \int _{\partial \Sigma _{\delta }^{v}}z(y)^{\frac{2(n-1)}{2\alpha +\mu }}dy\right) ^{\frac{2\alpha +\mu }{2(n-1)}}\\&\quad \left( \int _{\partial \Sigma _{\delta }^{v}}\left[ \left( v(y)-v_{\delta }(y)\right) ^{+}\right] ^{\frac{2(n-1)}{n-2}}dy\right) ^{\frac{n-2}{n-1}}. \end{aligned} \end{aligned}$$
(5.16)

Finally, we estimate integral \(A_{4}\). According to Theorem 1.2, it follows from (1.12) and the Hölder inequality that

$$\begin{aligned} A_{4}{} & {} \le C_{\delta }\int _{\partial \Sigma _{\delta }^{v}}v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left[ z(y)-z(y^{\delta })\right] \left( v(y)-v_{\delta }(y)\right) ^{+}dy\nonumber \\{} & {} \le C_{\delta }\int _{\Sigma _{\delta }^{v}}\int _{\partial \Sigma _{\delta }^{v}}\frac{x_{n}^{\lambda }v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left( v(y)-v_{\delta }(y)\right) ^{+}v(x)^{\frac{n+2+2\lambda -(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}}{|x|^{\beta }|x-y|^{\mu }|y|^{\alpha }}dxdy\nonumber \\{} & {} \le C_{\delta }\Vert v(y)^{\frac{n-(2\alpha +\mu )}{n-2}}\left( v(y)-v_{\delta }(y)\right) ^{+}\Vert _{L^{\frac{2(n-1)}{2n-2-(2\alpha +\mu )}}(\partial \Sigma _{\delta }^{v})}\Vert \nonumber \\{} & {} \quad v(x)^{\frac{n+2+2\lambda -(2\beta +\mu )}{n-2}}\left( v(x)-v_{\delta }(x)\right) ^{+}\Vert _{L^{\frac{2n}{2n+2\lambda -(2\beta +\mu )}}(\Sigma _{\delta }^{v})}\nonumber \\{} & {} \le C_{\delta }\Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert \left( v(x)-v_{\delta }(x)\right) \Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}\Vert \nonumber \\{} & {} \quad v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}\Vert \left( v(y)-v_{\delta }(y)\right) \Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}. \end{aligned}$$
(5.17)

Hence, insertting (5.14), (5.15), (5.16) and (5.17) into (5.13) and taking \(\varepsilon \rightarrow 0\), then by Lebesgue’s dominated convergence theorem, the Sobolev trace inequality and the Sobolev inequality, it holds that

$$\begin{aligned} \begin{aligned}&\int _{\Sigma _{\delta }}|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\\&\quad \le C_{\delta }\left[ \Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}+\left( \int _{\Sigma _{\delta }^{v}}\omega (x)^{\frac{2n}{(2\beta +\mu )-2\lambda }}dx\right) ^{\frac{(2\beta +\mu )-2\lambda }{2n}}\right. \\&\left( \int _{\Sigma _{\delta }^{v}}v(x)^{\frac{2n}{n-2}}dx\right) ^{\frac{2\lambda +4-(2\beta +\mu )}{2n}}\\&\quad ~~~\left. +\left( \int _{\partial \Sigma _{\delta }^{v}}v(y)^{\frac{2(n-1)}{n-2}}dy\right) ^{\frac{2-(2\alpha +\mu )}{2(n-1)}}\left( \int _{\partial \Sigma _{\delta }^{v}}z(y)^{\frac{2(n-1)}{2\alpha +\mu }}dy\right) ^{\frac{2\alpha +\mu }{2(n-1)}}\right] \\ {}&\cdot \left( \int _{\Sigma _{\delta }^{v}}|\nabla \left( v-v_{\delta }\right) ^{+}|^{2}dx\right) , \end{aligned} \end{aligned}$$

which leads to (5.7). This accomplishes the proof. \(\square \)

Next, based on Lemmas 5.1 and 5.2, we give the following frequently useful lemma.

Lemma 5.3

Under the hypothesis of Theorem 1.10, there exists a non-negative constant \(\delta _{0}\), such that for any \(\delta \ge \delta _{0}\), \(x\in \Sigma _{\delta }\) and \(y\in \partial \Sigma _{\delta }\), we have

$$\begin{aligned} \omega (x)\le \omega (x^{\delta }),\ \ z(y)\le z(y^{\delta }),\ \ v(x)\le v_{\delta }(x). \end{aligned}$$
(5.18)

Proof

Actually, since \(\omega (x)\), z(y) and v(x) possess better decay properties, we can take \(\delta _{0}\) sufficiently large, such that for \(\delta >\delta _{0}\), it holds that

$$\begin{aligned} \begin{aligned} C_{\delta }&\left( \Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}+\Vert \omega (x)\Vert _{L^{\frac{2n}{(2\beta +\mu )-2\lambda }}(\Sigma _{\delta }^{v})}\Vert v(x)\Vert ^{\frac{2\lambda +4-(2\beta +\mu )}{2n}}_{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}\right. \\&\left. +\Vert z(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\Vert v(y)\Vert ^{\frac{2-(2\alpha +\mu )}{2(n-1)}}_{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\right) \le \frac{1}{2}. \end{aligned} \end{aligned}$$

Further, according to Lemma 5.1 and Lemma 5.2, for any \(x\in \Sigma _{\delta }\) and \(y\in \partial \Sigma _{\delta }\), we obtain \(\omega (x)\le \omega (x^{\delta })\), \(z(y)\le z(y^{\delta })\) and \(v(x)\le v_{\delta }(x)\). \(\square \)

Based on the above analysis, we find the starting point to move plane \(T_{\delta _{0}}\). Now we are ready to move the plane from the right to the left provided (5.18). To state the process accurately, we write

$$\begin{aligned} \delta _{1}=\inf \left\{ \delta |\omega (x)\le \omega (x^{\delta }),~ z(y)\le z(y^{\delta }),~ v(x) \le v_{\delta }(x),~\forall x\in \Sigma _{\delta },~ \forall y\in \partial \Sigma _{\delta }\right\} . \end{aligned}$$
(5.19)

Then we deduce the following important result.

Lemma 5.4

If \(\delta _{1}>0\), then for any \(x\in \Sigma _{\delta _{1}}\) and \(y\in \partial \Sigma _{\delta _{1}}\), it holds that \(\omega (x)\equiv \omega (x^{\delta _{1}})\), \(z(y)\equiv z(y^{\delta _{1}})\) and \(v(x)\equiv v_{\delta _{1}}(x)\).

Proof

Suppose that \(\omega (x)\not \equiv \omega (x^{\delta _{1}})\), \(z(y)\not \equiv z(y^{\delta _{1}})\) and \(v(x)\not \equiv v_{\delta _{1}}(x)\). On one hand, by means of the continuity of \(\omega (x)\), v(x) and z(y), for any \(x\in \Sigma _{\delta _{1}}\) and \(y\in \partial \Sigma _{\delta _{1}}\), we have

$$\begin{aligned} \omega (x)\le \omega (x^{\delta _{1}}),~ v(x)\le v_{\delta _{1}}(x),~z(y)\le z(y^{\delta _{1}}). \end{aligned}$$
(5.20)

Moreover, from the monotonicity of g and k, we know

$$\begin{aligned} \begin{aligned}&x_{n}^{\lambda }\omega (x)k\left( |x|^{n-2}v(x)\right) v(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}=x_{n}^{\lambda }\omega (x)\frac{g\left( |x|^{n-2}v(x)\right) }{|x|^{2\lambda +n+2-(2\beta +\mu )}}\\&\le x_{n}^{\lambda }\omega (x^{\delta _{1}})\frac{g\left( |x|^{n-2}v_{\delta _{1}}(x)\right) }{|x|^{2\lambda +n+2-(2\beta +\mu )}}\\&\le x_{n}^{\lambda }\omega (x^{\delta _{1}})\frac{g\left( |x|^{n-2}v_{\delta _{1}}(x)\right) }{\left( |x|^{n-2}v_{\delta _{1}}(x)\right) ^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}}v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&\le x_{n}^{\lambda }\omega (x^{\delta _{1}})\frac{g\left( |x|^{n-2}v_{\delta _{1}}(x)\right) }{\left( |x^{\delta _{1}}|^{n-2}v_{\delta _{1}}(x)\right) ^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}}v_{\delta }(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\\&=x_{n}^{\lambda }\omega (x^{\delta _{1}})k\left( |x^{\delta _{1}}|^{n-2}v_{\delta _{1}}(x)\right) v_{\delta _{1}}(x)^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}, \end{aligned} \end{aligned}$$

which immediately implies that

$$\begin{aligned} -\Delta v(x)\le -\Delta v(x^{\delta _{1}}),\ \ x\in \Sigma _{\delta _{1}}. \end{aligned}$$

Furthermore, according to the strong maximum principle, for any \(x\in \Sigma _{\delta _{1}}\) and \(y\in \partial \Sigma _{\delta _{1}}\), we obtain \(\omega (x)<\omega (x^{\delta _{1}})\), \(v(x)<v_{\delta _{1}}(x)\) and \(z(y)<z(y^{\delta _{1}})\).

On the other hand, when \(\delta \rightarrow \delta _{1}\), we get \(\frac{1}{|x|^{2n}}\chi _{\Sigma _{\delta }^{v}}{\mathop {\longrightarrow }\limits ^{a.e.}}0\) and \(\frac{1}{|y|^{2(n-1)}}\chi _{\partial \Sigma _{\delta }^{v}}{\mathop {\longrightarrow }\limits ^{a.e.}}0\). Therefore, there exists \(\tau >0\) such that for \(\delta \in \left[ \delta _{1}-\tau ,\delta _{1}\right] \), \(\frac{1}{|x|^{2n}}\chi _{\Sigma _{\delta }^{v}}\le \frac{1}{|x|^{2n}}\chi _{\Sigma _{\delta _{1}-\tau }^{v}}\) and \(\frac{1}{|y|^{2(n-1)}}\chi _{\partial \Sigma _{\delta }^{v}}\le \frac{1}{|y|^{2(n-1)}}\chi _{\partial \Sigma _{\delta _{1}-\tau }^{v}}\). Here, applying Lebesgue’s dominated convergence theorem, when \(\delta \rightarrow \delta _{1}\), we know

$$\begin{aligned} \int _{\partial \Sigma _{\delta }^{v}}\frac{1}{|x|^{2n}}dx\rightarrow 0,\ \ ~~~~ \int _{\partial \Sigma _{\delta }^{v}}\frac{1}{|y|^{2(n-1)}}dy\rightarrow 0. \end{aligned}$$

Finally, using the decay properties of \(\omega (x)\), z(y) and v(x), we point out that there exists \(\tau _{0}\) such that for any \(\delta \in \left[ \delta _{1}-\tau _{0},\delta _{1}\right] \), it holds

$$\begin{aligned} \begin{aligned} C_{\delta }&\left( \Vert v(x)\Vert _{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}^{\frac{2\lambda +n+2-(2\beta +\mu )}{n-2}}\Vert v(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}^{\frac{n-(2\alpha +\mu )}{n-2}}+\Vert \omega (x)\Vert _{L^{\frac{2n}{(2\beta +\mu )-2\lambda }}(\Sigma _{\delta }^{v})}\Vert v(x)\Vert ^{\frac{2\lambda +4-(2\beta +\mu )}{2n}}_{L^{\frac{2n}{n-2}}(\Sigma _{\delta }^{v})}\right. \\&\left. +\Vert z(y)\Vert _{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\Vert v(y)\Vert ^{\frac{2-(2\alpha +\mu )}{2(n-1)}}_{L^{\frac{2(n-1)}{n-2}}(\partial \Sigma _{\delta }^{v})}\right) < \frac{1}{2}. \end{aligned} \end{aligned}$$

Therefore, by using Lemmas 5.1 and 5.2, we have for any \(\delta \in \left[ \delta _{1}-\tau _{0},\delta _{1}\right] \),

$$\begin{aligned} \omega (x)\le \omega (x^{\delta }),~ v(x)\le v_{\delta }(x),~z(y)\le z(y^{\delta }),\ \ x\in \Sigma _{\delta },~~ y\in \partial \Sigma _{\delta }, \end{aligned}$$

which contradicts the definition of \(\delta _{1}\). The proof is finished. \(\square \)

It’s clear that, the proof of Theorem 1.10 relies on the above results.

Proof of Theorem 1.10

In fact, we can move plane \(T_{\delta }\) from \(\infty \) to the left, and continue this proof until \(\delta =\delta _{1}\). If \(\delta _{1}>p_{1}\), then we obtain

$$\begin{aligned} \omega (x)\equiv \omega (x^{\delta _{1}}),~z(y)\equiv z(y^{\delta _{1}}),~v(x)\equiv v_{\delta _{1}}(x),\ \ x\in \Sigma _{\delta _{1}},~y\in \partial \Sigma _{\delta _{1}}. \end{aligned}$$

However, this is impossible. Therefore, we get \(\delta _{1}\le p_{1}\). Similarly, we can move the plane from \(-\infty \) to the right as we did in the previous discussion, then we derive \(\delta _{1}^{\prime }\) and \(\delta _{1}^{\prime }\ge p_{1}\). Furthermore, we conclude \(\delta _{1}=\delta _{1}^{\prime }=p_{1}\). Since \(x_{1}\) direction can be taken arbitrarily, then the fact implies that, \(\omega (x)\), v(x) and z(y) are symmetric with respect to any plane, which is passing through p and perpendicular to \(x_{i}~\left( i=1,2...n-1\right) \) axis.

Finally, for any \(p\in \partial \mathbb {R}^{n}_{+}\), v(x), \(\omega (x)\) and z(y) are symmetric about the plane that passing through p and is parallel to \(x_{n}\) axis. Hence we derive that u(x) and m(x) depend only on \(x_{n}\) and \(\eta \) is constant. This proof is accomplished. \(\square \)

Next, we will pay attention to studying the existence of positive solution for Hartree type equations (1.15). By virtue of Theorem 1.10, we give the proof of Corollary 1.11.

Proof of Corollary 1.11

Naturally, according to Theorem 1.10, we conclude that m(x), u(x) depend only \(x_{n}\) and \(\eta (y)\) are constant function. As a consequence, we can rewrite the equations (1.15) in the following form

$$\begin{aligned} \left\{ \begin{aligned}&-\frac{d^{2}u(x_{n})}{dx_{n}^{2}}=\left( \int _{\partial \mathbb {R}^{n}_{+}}\frac{F\left( u(0)\right) }{|x-y|^{\mu }}dy\right) x_{n}^{\lambda }g\left( u(x_{n})\right) ,\ \ x_{n}>0,\\&-\frac{\partial u}{\partial x_{n}}(0)=\left( \int _{\mathbb {R}^{n}_{+}}\frac{G\left( u(x_{n})\right) x_{n}^{\lambda }}{|x-y|^{\mu }}dx\right) f\left( u(0)\right) . \end{aligned} \right. \end{aligned}$$

It is worth noting that the first equation yield u(x) is concave function. Moreover, from the later equation, we have

$$\begin{aligned} \frac{d u}{d x_{n}}(0)=\left( \int _{\mathbb {R}^{n}_{+}}\frac{G\left( u(x_{n})\right) x_{n}^{\lambda }}{|x-y|^{\mu }}dx\right) f\left( u(0)\right) \ge 0. \end{aligned}$$

Thus, we conclude that u(x) is concave and decreasing unless \(u\equiv \tilde{c}\) with \(F(\tilde{c})=G(\tilde{c})=0\). On the one hand, if u is strictly decreasing with respect to \(x_{n}\), then we have \(\frac{d u}{d x_{n}}(0)\le 0\), which is impossible. The fact implies that this case will not happen. Hence we immediately deduce the desired result. The proof is accomplished. \(\square \)