An Uncertain Assessment Compatible Incentive Mechanism for Eliciting Continual and Truthful Assessments of Cloud Services

Abstract

The evaluation of dynamic performance of cloud services relies on continual assessments from cloud users, e.g., ordinary consumers and testing parties. In order to elicit continual and truthful assessments, an effective incentive mechanism in cloud environments should allow users to provide uncertain assessments when they are not sure about the real performance of cloud services, e.g., when users do not access cloud services on time, rather than providing untruthful or arbitrary assessments. Different from all prior works, we propose a novel uncertain assessment compatible incentive mechanism. Under this mechanism, a user not only has sufficient incentives to continually provide truthful assessments, but also would prefer providing uncertain assessments over untruthful or arbitrary assessments since uncertain assessments can bring more benefits than untruthful or arbitrary assessments. We theoretically analyze the proposed incentive mechanism and evaluate it through simulations under different circumstances. The theoretical analysis demonstrates the effectiveness of our approach. Moreover, the experimental results based on simulations strongly support the results from the theoretical analysis.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

As cloud services have become increasingly popular, reliable service evaluation is quite important for cloud consumers. Cloud service evaluation is usually based on cloud users’ assessments, which can be either subjective (e.g., user ratings) or objective (e.g., QoS monitoring or testing). No matter what types of assessments are used, the trustworthiness of users’ assessments has a great impact on the reliability of cloud service evaluation. For avoiding ambiguity, all the parties which provide cloud assessments are called cloud users in this paper, e.g., ordinary consumers or testing organizations.

In cloud environments, service performance may vary substantially and frequently due to the dynamic nature of cloud services. Thus, continual assessments over time are needed to effectively reflect the dynamic performance of services. Recent studies [18, 25] point out that continual assessments are vital for not only evaluating cloud users’ trustworthiness of providing assessments, but also predicting the dynamic performance of cloud services. However, eliciting continual and truthful assessments in cloud environments is still a challenging problem since it is usually hard to make self-interested users behave cooperatively in an online community [1]. A cloud user usually does not have sufficient incentives to regularly provide assessments of cloud services on time. To motivate users, an effective incentive mechanism should be designed. A common solution is that a cloud user can be paid if it provides assessments on scheduled time. The monetary rewards¹ could be provided by some professional cloud evaluation organizations, such as CloudReviews², the aim of which is to provide cloud selection services to potential cloud consumers based on cloud users’ assessments and thus earn profits from the potential consumers. Nevertheless, such a simple mechanism cannot prevent a user from “free-riding” (i.e., providing arbitrary assessments) [10, 24]. Moreover, sometimes an honest user may also provide arbitrary assessments in order to obtain monetary rewards when it does not really know the real performance of cloud services (e.g., a user does not consume services on the scheduled time while a user is required to provide an assessment). Such arbitrary assessments may be erroneous and misleading, and therefore greatly affect the effectiveness of service evaluations. To avoid the submission of arbitrary assessments, an effective incentive mechanism should motivate users to always tell the truth, i.e., motivating users to provide truthful assessments, and allowing users to provide uncertain assessments to express their uncertainty about service performance when necessary. However, there are no such incentive mechanisms in the literature, which considers uncertain assessments.

In our prior work [16], we proposed a basic framework supporting our incentive mechanism, which can take uncertain assessments into account. Based on this framework, the proposed incentive mechanism in this paper makes a further step, i.e., presenting the theoretical analysis of the effective incentive design and the optimal incentive design in our framework as well as discussing the whitewashing problem (See Sect. 3.6). The features and contributions of our work are summarized as follows:

• (1) Under our proposed mechanism, a user is considered honest if it gives truthful assessments most of the time, but may give a small proportion of uncertain assessments once it is not sure about the real performance of a service. The word “honest” indicates such a user always tells the truth. Thus, a UAC (uncertain-assessment-compatible) assessment scheme is first proposed. In particular, the new scheme can be extended from any type of ordinary (subjective or objective) assessment systems, but includes an extra uncertain state (see Sect. 3.1). Then the behaviors of users providing assessments are modeled using a repeated game framework (see Sect. 3.2).

• (2) A user can receive monetary rewards from a professional organization (called a broker) mentioned above for regularly providing assessments on schedule for the cloud services it consumes. In order to control the monetary rewards for the incentive mechanism design, we propose an assessment scoring scheme (see Sect. 3.3). In a nutshell, truthful assessments would bring the most rewards; uncertain assessments would bring less rewards; untruthful or arbitrary assessments would bring the very least rewards. Through our proposed mechanism, a rational user would choose its best option, i.e., providing truthful assessments. Once it is not sure about service performance, there still exists a second-best option, i.e., providing uncertain assessments.

• (3) In order to build an effective incentive mechanism, we present the theoretical analysis (see Sects. 3.4) of the scoring scheme according to the different strategies of users (i.e., providing truthful/uncertain/untruthful/arbitrary assessments). Moreover, we discuss how to build an optimal incentive mechanism in our framework (see Sect. 3.5) and the feasibility of solving the whitewashing problem [4] based on our proposed mechanism (see Sect. 3.6).

• (4) The results from the theoretical analysis show that our approach is effective in most circumstances (see Sect. 4.1). Furthermore, in order to evaluate the feasibility of our approach, we carry out simulation experiments under different situations. The results from the simulation strongly support the results from the theoretic analysis (see Sect. 4.2).

2 Related Work

In the literature, incentive mechanisms for eliciting truthful information are usually modeled in a seller-buyer scenario, where speaking the truth is an equilibrium for buyers. According to the applied techniques, those mechanisms can generally be classified into two groups: peer-prediction based approaches and reputation-based approaches. In addition to these approaches, some recent studies of incentive mechanisms in crowdsourcing environments are proposed for eliciting effective contributions of workers. In general, all these approaches are proposed for eliciting the cooperation of users, and thus are related to our work.

Peer-Prediction Based Approaches: Miller et al. [13] propose the pioneering “Peer-Prediction” method for eliciting truthful feedback. In their work, every user can obtain monetary payment from an authorized center. The amount of payment depends on how well a user can predict the signal from some other user (called a reference user) based on its own signal. Their work is feasible based on several common knowledge assumptions, e.g., product type distributions and conditional distributions of signals. However, there is a drawback in Miller et al.’s work, i.e., there may exist lying equilibria that can bring higher expected payoffs than the truthful equilibrium [13]. To overcome this drawback, Jurca and Faltings [7, 8] propose a collusion-resistant feedback payment approach, in which several reference reports are applied in the scoring rules instead of the one-reference-report scheme in the prior work. They prove that speaking the truth is the unique equilibrium if at least three reports are used.

In later studies, Witkowski [19] point out that the quality of goods or services provided by sellers is assumed fixed in prior works. However, in many real-world situations, the quality is inherently dynamic. Thus, he proposes a payment mechanism based on the hidden Markov setting to deal with such dynamics. It is worth noting that all these peer-prediction-based incentive mechanisms make strong common knowledge assumptions. To lift these assumptions, Witkowski and Parkes [20] propose peer prediction without a common prior. Their mechanism allows participants to adopt subjective and private priors instead of a common prior by asking a participant to offer two reports (one before the transaction and one afterwards), and their approach is proved to provide strict incentives for truthful reports. Compared to the peer-prediction-based approaches, our work needs fewer knowledge assumptions and no extra belief report submission.

Reputation Based Approaches: some incentive mechanisms focus on evaluating participants’ reputations on how truthfully they provide assessments or do something they have committed to. And the reputation would influence a participant’s future opportunities of obtaining profits. Jurca and Faltings [6] propose an incentive-compatible reputation mechanism, which allows sellers to “confess” when they did not provide the goods or services as those they have committed. Due to such a confession, a seller can prevent further losses for his/her cheating, which give sellers incentives to speak the truth. Papaioannou and Stamoulis [14], propose a reputation-based incentive mechanism in a peer-to-peer system to motivate peers for truthful reporting. In their work, a non-credibility metric is designed for controlling a peer’s punishment of having disagreed transaction feedback with other peers. Zhang et al. [22] propose a trust-based incentive mechanism, which is an extension of their prior work [21], in a reverse auction scenario. In this mechanism, a seller whose reputation is below a threshold is forbidden to participate in future auctions and therefore suffers a loss.

Incentive Mechanism Studies for Crowdsourcing: incentive mechanisms are employed in crowdsourcing environments for motivating users’ effective contributions. Mason and Watts [12] study the relationship between financial incentives and working performance, and argue that increasing financial incentives could only bring more workers, but not a working quality improvement as expected. A similar conclusion can be found in DiPalantino and Vojnovic’s work [2]. They argue that worker participation rates logarithmically increase with monetary rewards. Zhang and van der Schaar [24] focus on solving workers’ “free-riding” problem and requesters’ false-report problem. They designed optimal and sustainable incentive protocols based on social norms [9]. After that, they propose a generic rating protocol for online communities [23].

In our prior work [16], a basic framework for uncertain-assessment-compatible incentive mechanism is proposed without theoretical analysis. Different from all the above works, in this paper, we propose a novel incentive mechanism which is compatible with users’ uncertain assessments, and present the theoretical proofs of our work as well as the illustrative results, both of which demonstrate the feasibility of our work.

3 The Proposed Approach

The basic idea behind our approach is as follows: cloud users can get paid by selling their assessments for cloud services to a broker via a user agent system. Cloud users are allowed to provide uncertain assessments for the services when they are not sure about the real performance of the services. The cloud performance evaluation is carried out by the broker based on cloud users’ assessments, and the broker pays monetary rewards to the current cloud users for their assessments and obtains profits from potential cloud consumers by offering cloud selection services.

A user’s incentive is represented through its expected long-term payment. The long-term payment is composed of the payments obtained in the continual time windows, e.g., 9 am–10 am every day. Through an assessment scoring scheme, users’ participation of selling their assessments are controlled. In a nutshell, if a user is considered to submit a truthful assessment in a time window, it can keep on selling assessments until it is considered to have submitted an uncertain or untruthful assessment in a subsequent time window. Due to the submitted uncertain or untruthful assessment, the user would be isolated from selling assessments for a period of time, so that its long-term payment would suffer a loss because of such isolation. This is like fixed-term imprisonment. After the “imprisonment”, the user can still be involved in the subsequent assessment transactions. Hence, in a time window, the user would believe that truthful reporting can maximize its long-term payoff and an uncertain assessment would bring a larger payoff than an untruthful or arbitrary one, if the broker can correctly judge the truthfulness of an assessment with an overwhelming probability.

3.1 The UAC Assessment Schemes

A cloud user can give its own assessments for different performance aspects of the cloud services it consumes. For each aspect, such assessments can be expressed in any reasonable form including subjective or objective assessments. Taking service response time as an example, a cloud user can give its numerical ratings (e.g., “1”, “2” or “3”) or linguistic ratings (e.g., “poor”, “fair” or “good”) to express its subjective assessments. On the other hand, a user can also provide objective assessments according to QoS testing (e.g., 200 ms for response time). For any type of an assessment system, an uncertain state can be added into the system to express users’ uncertainty about service performance. For example, if a rating scheme consists of three states: “good”, “fair” and “poor”. The UAC assessment scheme, which can be applied in our incentive mechanism, is composed of four states, i.e., “good”, “fair”, “poor” and “uncertain”, where the first three are considered as certain assessments.

3.2 Game Setup

Broker and Payment Settings: the broker requires cloud users to provide continual assessments for services at regular time intervals. A user can get paid by providing an assessment in a scheduled time window. In each time window, only the latest assessment can be paid for by the broker. If the user misses a time window, it cannot give assessments until the next time window. In addition, we assume that the cloud users are long-lived, and care about their long-term payoffs of providing assessments.

In each time window, the broker must pay each user no matter what type of an assessment the user gives. The amount of payment has two levels. If a user gives a certain assessment, it would get a payment P regardless of the value of the assessment. Conversely, if a user gives an uncertain assessment, it would get a discounted payment $\lambda P$ for $\lambda \in [0,1]$. The reason for why a user can get such a discounted payment is that uncertain assessments cannot benefit the broker but the user still tells the truth without giving untruthful or arbitrary assessments which may even make the broker suffer losses by falsely evaluating the performance of cloud services. If a user does not provide any assessment in a time window, an uncertain assessment would be automatically submitted by a user agent instead.

The compulsory payment setting in our work aims to prevent the broker from “false-reporting” [3]. If the broker can afterwards decide whether to pay according to the quality of assessments, it would always have incentives to refuse to pay to users by cheating about the real quality of assessments. Thus, the payment from the broker in our framework can be considered “ex-ante” [24] with two amount levels. The compulsory payment and the judgement of certain or uncertain assessments can be supervised by a third-party authority (e.g., a payment management center). The authority can keep both levels of payment (for a/an certain or uncertain assessment) before each time window, and then transfers one level of payment to a user according to the certainty of its assessment, and returns the other level of payment to the broker. Therefore, the broker cannot deny that an assessment is certain or uncertain.

User Strategies: based on our framework, the payoff matrix between the broker and a user in a time window can be specified in Table 1. We follow the common assumption of incentive mechanisms made in the literature: a user is rational and self-interested, i.e., every user is motivated to maximize its own payoffs. A user would have three strategies of “cooperation”, “semi-cooperation” or “non-cooperation”. In our framework, cooperation for a user means giving a truthful assessment; semi-cooperation means giving an uncertain assessment; non-cooperation means giving an untruthful or arbitrary assessment (these two situations will be further discussed separately). B is the benefit a truthful assessment can create for the broker in a time window. P is the full payoff a user can obtain by giving a certain assessment. C is the cost of the effort for a user providing a truthful assessment. In the situations of semi-cooperation and non-cooperation, we consider that a user does not have any cost since it does not try to provide a truthful assessment. We follow the common assumption in the literature of incentive mechanisms, i.e., $B> P > C$. Here, we consider that all users are identical in terms of their knowledge and preference, thus B and C are constant, but P is adjustable. Note that, our work can be easily extended to a situation where there are different types of users by setting suitable system parameters for different users. Table 1 indicates that a user’s dominant strategy is to always behave non-cooperatively, which is not expected by the broker and cause quite negative effects in cloud performance evaluations.

Table 1. Payoff matrix in a time window

3.3 The Assessment Scoring Scheme

In order to make a user’s dominant strategy cooperation, we propose an assessment scoring scheme to control users’ participation in the transactions of selling their assessments. In our framework, a user has an assessment score to determine if it can sell its assessments to the broker in a time window. At the end of each time window, a new assessment score will be assigned to each user according to its current score and the submitted assessment. An assessment score $\theta$ is a positive integer from a nonempty finite set $\varTheta$ ($\theta \in \varTheta = \{0,1,2,\cdots ,L\}$), where L is the largest score.

At the end of each time window, the broker can judge whether an assessment is truthful or untruthful through some approaches (e.g., majority opinions). Then it reports its judgement for every user to the authority. According to the broker’s reports and users’ current assessment scores, the authority updates a new score for every user. Note that, the broker would always report the truth about a user’s assessments since the payment is ex-ante and the broker cannot lie about the certainty of an assessment in our framework. However, there may exist an error probability $\alpha$ of the broker falsely reporting without intention, e.g., a truthful assessment is reported as an untruthful one, and vice versa. And $\alpha$ should be smaller than the probability of random guessing, i.e., $\alpha \in [0,0.5]$.

Let $\tau (\theta , b)$ denote the assessment scoring scheme, and the new score of a user at the end of a time window is computed as follows:

$$\begin{aligned} \tau (\theta ,b) = \left\{ \begin{array}{ll} L, &{}\text {if}~~\theta =L~\text {and}~b=T, \\ h_U, &{}\text {if}~~\theta =L~\text {and}~b=U, \\ 0, &{}\text {if}~~\theta =L~\text {and}~b=UT,\\ \theta +1, &{}\text {if}~~\theta < L, \\ \end{array} \right. \end{aligned}$$

(1)

where $\theta$ is a user’s current score and b is its reported behavior. $h_U$ can be considered as a punishment level for users providing uncertain assessments. A user can be reported as having three types of behaviors, i.e., providing truthful (T), uncertain (U) or untruthful (UT) assessments. Figure 1 shows the scoring scheme. If a user having the largest score L is considered to have submitted a/an truthful/uncertain/untruthful assessment, its new score will be maintained at L, or become $h_U$ or 0 respectively, where $0< h_U < L$. If a user has a score less than L, its score will always increase by 1. Furthermore, the authority requires that only the users having the score L are allowed to submit and sell their assessments to the broker. This means that all users can be classified into two groups: active users and isolated users. If a user is considered to give a/an uncertain or untruthful assessment, it would be punished by being prohibited from selling assessments for a period of time. Thus it will suffer a loss in its future incomes. If a user is not be able to behave cooperatively for some reason, it has a second-best option, i.e., giving uncertain assessments. That is because giving uncertain assessments would cause a shorter period of isolation due to the requirement of $0< h_U < L$.

Fig. 1.

The assessment scoring scheme

3.4 Effective Incentive Mechanism Design

In order to build an effective incentive mechanism based on the proposed assessment scoring scheme, we need to analyze the long-term expected payoffs that an “honest” user can obtain and find out what values of L and $h_U$ are necessary for an effective incentive mechanism.

An honest user refers to a user who gives truthful assessments most of the time, but may give a small part of uncertain assessments. We apply the infinite-horizon discounted sum criterion to analyze an honest user’s long-term expected payoffs. Let $p(\theta '|\theta )$ denote the transition probability of an honest user’s assessment scores between two adjacent time windows, which is shown as follows:

$$\begin{aligned} \small p(\theta '|\theta ) = \left\{ \begin{array}{ll} (1-\alpha )(1-\beta ), &{}\text {if}~~\theta =L~\text {and}~\theta '=L, \\ \beta , &{}\text {if}~~\theta =L~\text {and}~\theta '=h_U, \\ \alpha (1-\beta ), &{}\text {if}~~\theta =L~\text {and}~\theta '=0, \\ 1, &{}\text {if}~~\theta <L~\text {and}~\theta '=\theta +1,\\ 0, &{}\text {otherwise,} \\ \end{array} \right. \end{aligned}$$

(2)

where $\theta$ represents the user’s current score and $\theta '$ is the user’s new score. $\alpha$ is the error probability of the broker making a false judgement about the user’s assessment. $\beta$ is the probability of the user giving an uncertain assessment in a time window. For an identical type of users and a broker, $\alpha$ and $\beta$ should be fixed in all time windows. Hence, an honest user’s long-term expected payoff in a time window can be computed by solving the following recursive equation:

$$\begin{aligned} v^\infty (\theta ) = v(\theta ) + \delta \sum _{\theta '}p(\theta '|\theta )v^\infty (\theta '), \text {for all }\theta \in \varTheta , \end{aligned}$$

(3)

where $v^\infty (\theta )$ denotes a user’s long-term payoff when it has the assessment score $\theta$ in a time window. And $v(\theta )$ denotes the user’s instant payoff after giving its assessment in the current time window. $\delta \in (0,1)$ represents a user’s patience about its future payoffs. A larger $\delta$ means that the user cares more about its future payoffs, and vice versa. Equation (3) indicates that an honest user’s long-term expected payoff consists of two parts, i.e., the instant payoff and the expected future payoff based on the score transition probability shown in Eq. (2). The notations of our approach are summarized in Table 2.

Theorem 1

(Existence of Long-term Expected Payoffs): Given the transition probabilities specified in Eq. (2), for any $\alpha \in [0,0.5]$, $\beta \in [0,1]$, $\sigma \in (0,1)$, $\lambda \in [0,1]$ and $P>C$, the recursive equation Eq. (3) has a unique positive solution.

Proof

All proofs in this paper are omitted due to the space limitation, and presented in [15]. $\Box$

Based on Theorem 1, we have the following property:

Table 2. The parameters of the incentive mechanism

Property 1:

The long-term expected payoffs defined in Eq. (3) satisfy the following conditions:

• (1)   $v^\infty (\theta + 1) > v^\infty (\theta ),~\text {for}~\forall \theta \in \varTheta - \{L\};$

• (2)   $v^\infty (\theta + 1) - v^\infty (\theta ) > v^\infty (\theta ) - v^\infty (\theta - 1),\ \text {for}~\forall \theta \in \varTheta - \{0,L\}.$ $\Box$

In Property 1, the statement (1) indicates that the higher the assessment score of a user, the more the long-term expected payoff. The statement (2) shows that the increase of the long-term expected payoff between two adjacent assessments scores becomes larger with the increase of users’ assessment scores. Property 1 demonstrates that an honest user always has incentives to pursue a higher score for obtaining a higher long-term payoff.

In our framework, there should be a dominant strategy for a user, and a second-best strategy if it cannot choose the dominant strategy for some reason. We expect the dominant strategy is to provide truthful assessments, and the second-best strategy is to provide uncertain assessments. As a user’s long-term expected payoffs can be computed in a recursive form, its strategy can be determined based on the one-shot deviation principle [5], i.e., if a user cannot increase its long-term expected payoff by choosing a strategy other than the dominant one in a time window, the user would not be able to increase the payoff by choosing any strategy other than the dominant one. The one-shot deviation principle can also be applied for the second-best strategy. Hence, we study an active (its assessment score is L) user’s long-term expected payoff³. If a user provides a truthful (T) assessment in a time window, and then its long-term expected payoff can be computed according to Eq. (3) as follows:

$$\begin{aligned} v_T^\infty (L) = P-C + \delta [(1-\alpha )v^\infty (L) + \alpha v^\infty (0)]. \end{aligned}$$

(4)

And if a user provides an uncertain (U) assessment, its payoff can be computed as follows:

$$\begin{aligned} v_U^\infty (L) = \lambda P + \delta [v^\infty (h_U)]. \end{aligned}$$

(5)

At last, if a user provides an untruthful (UT) assessment, its payoff can be computed as follows:

$$\begin{aligned} v_{UT}^\infty (L) = P + \delta [\alpha v^\infty (L) + (1-\alpha )v^\infty (0)]. \end{aligned}$$

(6)

In order to determine the unique dominant strategy and the second-best strategy, a user’s long-term expected payoff should satisfy the constraints: $v_T^\infty (L)> v_U^\infty (L) > v_{UT}^\infty (L)$, i.e.,

$$\begin{aligned} \small {\begin{matrix} \delta [(1-\alpha )v^\infty (L) + \alpha v^\infty (0) - v^\infty (h_U)] + (1-\lambda )P - C> 0,\\ \delta [v^\infty (h_U) - \alpha v^\infty (L) - (1-\alpha )v^\infty (0)] + (\lambda -1)P > 0.~~~~~~ \end{matrix}} \end{aligned}$$

(7)

An assessment scoring scheme satisfying Eq. (7) indicates that a user can obtain the most long-term expected payoffs when giving a truthful assessment, and the second-best expected payoffs when giving an uncertain assessment.

Strategic Users: In Eq. (7), we consider that a user only has three kinds of behaviors: providing truthful, uncertain or untruthful assessments. However, there may be strategic users who believe that they can guess the real performance of cloud services without actually knowing it. Even for the users who provide arbitrary assessments, there should be a small probability that they can guess the right results, so that they would not be punished for “free-riding”. The free-riders can be considered as a kind of strategic users. To solve the strategic user problem, we need to reconsider the constraints in Eq. (7) for an effective incentive mechanism in our framework.

For strategic users, the computations of the long-term expected payoff of giving a/an truthful or uncertain assessment in a time window are the same as Eqs. (4) and (5). Let $\gamma$ denote the probability that a strategic user (S) guesses the right result of cloud performance. The long-term payoff the user can obtain by giving a strategic assessment in a time window is computed as follows:

$$\begin{aligned} {\begin{matrix} v_{S}^\infty (L) = P + \delta \{ \gamma [(1-\alpha )v^\infty (L) + \alpha v^\infty (0)] +\\ (1-\gamma )[\alpha v^\infty (L) + (1-\alpha )v^\infty (0)]\}. \end{matrix}} \end{aligned}$$

(8)

Note that, we only consider the most beneficial case for a strategic user, i.e., a strategic assessment would not incur any cost of effort. Hence, without the consideration of the broker’s payoffs, an incentive mechanism is said to be effective if it satisfies all the following constraints:

$$\begin{aligned} {\begin{matrix} \small v_T^\infty (L)> v_U^\infty (L), v_U^\infty (L)> v_{S}^\infty (L)~\text {and}~v_U^\infty (L) > v_{UT}^\infty (L). \end{matrix}} \end{aligned}$$

(9)

Through straightforward calculations, $v_S^\infty (L) > v_{UT}^\infty (L)$ if and only if $\gamma \alpha < \frac{1}{2}$. In practice, $\alpha$ should usually be in the range of (0, 0.5) (0.5 for random guessing), thus the third constraint in Eq. (9) can be omitted in most cases.

3.5 Optimal Incentive Mechanism

For a type of users and a broker, there may be many assessment scoring schemes with different parameters L and $h_U$ to satisfy the constraints in Eq. (9). In order to find out which parameters are optimal, the total payoffs obtained by both the broker and a user should be analyzed. As only the users having the assessment score L can participate in the transactions of assessments, the total payoffs depend on the proportion of the active users in all users. Let $\eta (\theta )$ denote the proportion of the users having the score $\theta$. Because a user’s score is updated at the end of each time window, $\eta (\theta )$ would vary over time. As we assume that users care about their long-term payoffs, we analyze the stationary distribution of $\eta (\theta )$ for $\forall \theta \in \varTheta$ if all users are honest. Hence, the stationary distribution can be defined according to the score transition probability in Eq. (2) as follows:

$$\begin{aligned} \small \begin{aligned}&\eta (L) ~~= \eta (L-1) + (1-\alpha )(1-\beta )\eta (L), \\&\eta (\theta ) ~~~= \eta (\theta -1),~\text {if}~~h_U< \theta< L, \\&\eta (h_U) = \eta (h_U-1) + \beta \eta (L), \\&\eta (\theta ) ~~~= \eta (\theta -1),~\text {if}~~0< \theta < h_U, \\&\eta (0) ~~~= \alpha (1-\beta )\eta (L), \\&\sum _\theta \eta (\theta ) = 1~~\text {and}~~\eta (\theta ) \geqslant 0,~~\text {for}~\forall \theta .\\ \end{aligned} \end{aligned}$$

(10)

Theorem 2

(Existence of a Stationary Distribution): Given the transition probabilities specified in Eq. (2), for any $\alpha \in [0,0.5]$, $\beta \in [0,1]$ and $L> h_U > 0$, there exists a unique stationary distribution satisfying Eq. (10). $\Box$

Based on Theorem 2, we have the following property:

Property 2:

Given the stationary distribution specified in Eq. (10), $\eta (L)$ monotonically increases with $h_U$ and monotonically decreases with L. $\Box$

Property 2 indicates that adjusting L and $h_U$ can change the proportion of active users. The proportion can affect the broker and users’ total benefits.

The expected total payoffs obtained by the broker and an honest user in a time window can be computed as follows:

$$\begin{aligned} \small \begin{aligned}&U^* = \eta (L) \times [(1 - \beta )(B - P + P - C) + \beta (-\lambda P + \lambda P)]\\&~~~~~= \eta (L) \times (1 - \beta )(B - C). \end{aligned} \end{aligned}$$

(11)

Equation (11) illustrates that U monotonically increases with $\eta (L)$ and decreases with $\beta$. In addition, the expected payoff the broker can obtain from an honest user in a time window can be computed as follows:

$$\begin{aligned} \small \begin{aligned}&U = \eta (L) \times [(1 - \beta )(B - P) - \beta \lambda P].\\ \end{aligned} \end{aligned}$$

(12)

Hence, an effective incentive mechanism in our framework should satisfy the constraints specified in Eq. (9) and ensure that the broker can obtain a positive expected payoff in a time window, which is defined as follows:

Definition 1

(Effective Incentive Mechanism): An incentive mechanism with the adjustable parameters L, $h_U$, $\lambda$ and P is considered effective if it satisfies the following constraints:

$$\begin{aligned} \small \begin{aligned} v_T^\infty (L)> v_U^\infty (L), v_U^\infty (L)> v_{S}^\infty (L), v_U^\infty (L)> v_{UT}^\infty (L)\\ \text {and}~U > 0. \end{aligned} \end{aligned}$$

(13)

In this paper, we consider maximizing the total payoffs $U^*$ for an optimal incentive mechanism. Thus, we have the following definition:

Definition 2

(Optimal Incentive Mechanism): An effective incentive mechanism is considered optimal if $U^*$ is the maximum for some L, $h_U$, $\lambda$ and P.

Note that our work can be simply adjusted for satisfying other targets in any situation, e.g., maximizing the broker’s payoff U.

3.6 Whitewashing

Whitewashing is a common problem for the reputation or score based incentive mechanisms [4, 23], which refers to the situation where a user can reset its reputation or score by repeatedly re-participating in the activity with new identities. In our scenario, if a user having a score less than L is isolated from assessment transactions, it may try to create a new identity for transactions and expect to come back sooner from the isolation. Here, we assume that a user cannot hold multiple identities at the same time.

By finding out suitable mechanism parameters (i.e., L, $h_U$ and $\lambda$), our approach can prevent users from whitewashing. In order to solve this problem, a new user should not enter the assessment transactions instantly. It needs to wait for a period of time as an initializing period, and therefore cannot obtain any benefits. For a new user, an initial assessment score I is assigned. In order to prevent whitewashing, the initial score should satisfy the following constraint:

$$\begin{aligned} v^\infty (I) - v^\infty (\theta ) \leqslant c_w,~\text {for}~\forall \theta \in \varTheta ~\text {and}~I \in \varTheta , \end{aligned}$$

(14)

where $c_w \geqslant 0$ is the cost of a user whitewashing, e.g., the cost of creating a new identity. The expression $v^\infty (I) - v^\infty (\theta )$ indicates the expected long-term gain of a user with the assessment score $\theta$ whitewashing. If the gain is no larger than the cost, a user would have no motivation to reset its score. Considering the worst case for preventing whitewashing, i.e., $c_w = 0$, as $v^\infty (0)$ is the smallest long-term expected payoff according to the statement (1) of Property 1, $I = 0$ (lowest) is always a solution of Eq. (14). Assigning the lowest score to a new user means it can only enter assessment transactions after an initializing period. That means a user with any assessment score cannot gain more payoffs by carrying out whitewashing.

4 Illustrative Results and Simulation Results

4.1 Parameter Analysis

In our framework, the parameters of an incentive mechanism (see Table 2) can be grouped into two classes. The first class includes the intrinsic parameters $\alpha$, $\beta$, $\gamma$, $\delta$, B and C. For a type of users and a broker, the intrinsic parameters should be fixed. Thus, an incentive mechanism designer cannot adjust these parameters for an optimal incentive mechanism. The second class includes the adjustable parameters P, $\lambda$, L and $h_U$, where P and $\lambda$ may need to be conditionally adjusted according to the broker’s requirement since they can affect the broker’s payoffs. Due to space limitations, we only illustrate several main results.

Figure 2 illustrates the impact caused by $\alpha$. The vertical axis of the left sub-figure represents the percentage of effective incentive mechanisms in the total number of solutions. Here, we set that L is adjusted from 2 to 10 and $\lambda$ increases from 0 to 1 by steps of 0.05. The vertical axis of the right sub-figure represents the stationary percentage of active users in the corresponding optimal incentive mechanism. Figure 2 shows that the number of effective incentive mechanisms and active users decrease with $\alpha$. When $\alpha$ approaches nearly 0.4, there would not be any possible assessment scoring scheme which can be applied to building an effective incentive mechanism, thus the optimal total payoffs ($U^*$) would be 0. In addition, a larger $\beta$ would bring a smaller number of active users since an honest user would more often be punished for giving more uncertain assessments. Note that, the maximum possible value of $\alpha$ should only be 0.5 (random guessing) and be much smaller in most of practical cases. In the literature, many approaches are proposed to improve the accuracy of judging assessments for service evaluation, e.g., [11, 17]. Thus, the assumption of the error probability $\alpha$ in our approach is reasonable, so that our work can be applied in most circumstances.

Likewise, Fig. 3 shows that the number of effective incentive mechanisms decreases as $\gamma$ increases. Even if $\gamma$ reaches a very large value near 0.8, there still exist effective incentive mechanisms, but in those situations, $U^*$ would become very low since the punishment for a strategic user with a high correctness probability should be more serious to prevent its guessing.

Figure 4 demonstrates the results when the price P is adjusted between C and B. When P is near C, the constraints specified in Eq. (9) can be hardly satisfied. Conversely, U would be negative when P reaches close to B. Thus, the number of active users would reach the maximum when $\frac{P-C}{B-C}$ is between 0.4 and 0.7 since more effective incentive mechanisms can be built based on such P.

Fig. 2.

Incentive mechanisms affected by $\alpha$

Fig. 3.

Incentive mechanisms affected by $\gamma$

4.2 Simulation Experiments

Setting: since there are no suitable real environments supporting our framework, we have carried out simulation experiments and compared the simulation results with our theoretical results. We have simulated a cloud service environment containing many users, in which a user has its own strategies to provide assessments. Then, we set the same intrinsic parameters for both the simulation environment and the theoretical analysis, and compared the similarity between the two kinds of results. In the simulation experiments, a setting of the adjustable parameters is considered to build an effective incentive mechanism if, after a number of transactions of selling assessments, a user providing a smaller number of uncertain assessments would get a higher long-term payoff, and a user providing a proportion of uncertain assessments would get a higher long-term payoff than another user providing the same proportion of untruthful or strategic assessments.

Results and Analysis: the left sub-figure of Fig. 5 illustrates that the optimal L and $h_U$ between the two kinds of results are very similar when adjusting $\lambda$. In some cases, L and $h_U$ in these two kinds of results are not exactly equal since there are unavoidable computational errors in the simulation experiments when taking an action according to a specific probability. If some values of the constraints in Definition 1 are very small but still positive in some assessment scoring schemes, such schemes may be evaluated not to be able to make an effective incentive mechanism in the simulation experiments. Thus, the number of effective incentive mechanisms in the theoretical analysis is usually larger than that in the simulation experiments. According to the experimental results, the average rate between the latter number and the former one is approximately $75\,\%$. Likewise, if the values of the constraints are negative but very near 0, such a scheme may be considered to be effective for an incentive mechanism. Even so, the experimental results show that at least $93\,\%$ of the effective incentive mechanisms in the simulation experiments are the same as those from the theoretical analysis. The right sub-figure of Fig. 5 shows such results when P is adjusted between C and B.

Fig. 4.

Incentive mechanisms affected by P

Fig. 5.

Comparison between simulated results and theoretical results

5 Conclusion

This paper has proposed a novel incentive mechanism for eliciting continual and truthful assessments in cloud environments. The main novelty is that, different from prior works, our incentive mechanism is compatible with uncertain assessments. Hence, it can protect a user’s honesty by allowing it to give uncertain assessments in unavoidable situations. Through a suitable assessment scoring scheme, a user would have a dominant strategy (giving truthful assessments) and a second-best strategy (giving uncertain assessments). Meanwhile, the total payoffs of transacting assessments would be maximized. We have theoretically analyzed our approach and carried out simulation experiments. The proposed theoretical analysis indicates that our approach is feasible in most circumstances. The simulation experimental results strongly support the theoretical analysis.