Towards understanding and mitigating unintended biases in language model-driven conversational recommendation

https://doi.org/10.1016/j.ipm.2022.103139Get rights and content

Highlights

  • We study bias in a simple language model-driven recommender (LMRec).

  • LMRec demonstrates substantial price and category shifts based on race and gender.

  • LMRec appears to pick up on indirect mentions of homosexual relations.

  • LMRec demonstrates substantial price shift based on location and religion mentions.

  • Train side masking and test side neutralization maintain accuracy and nullify bias.

Abstract

Conversational Recommendation Systems (CRSs) have recently started to leverage pretrained language models (LM) such as BERT for their ability to semantically interpret a wide range of preference statement variations. However, pretrained LMs are prone to intrinsic biases in their training data, which may be exacerbated by biases embedded in domain-specific language data (e.g., user reviews) used to fine-tune LMs for CRSs. We study a simple LM-driven recommendation backbone (termed LMRec) of a CRS to investigate how unintended bias — i.e., bias due to language variations such as name references or indirect indicators of sexual orientation or location that should not affect recommendations — manifests in substantially shifted price and category distributions of restaurant recommendations. For example, offhand mention of names associated with the black community substantially lowers the price distribution of recommended restaurants, while offhand mentions of common male-associated names lead to an increase in recommended alcohol-serving establishments. While these results raise red flags regarding a range of previously undocumented unintended biases that can occur in LM-driven CRSs, there is fortunately a silver lining: we show that train side masking and test side neutralization of non-preferential entities nullifies the observed biases without significantly impacting recommendation performance.

Introduction

With the prevalence of language-based intelligent assistants such as Amazon Alexa and Google Assistant, conversational recommender systems (CRSs) have attracted growing attention as they can dynamically elicit users’ preferences and incrementally adapt recommendations based on user feedback (Gao et al., 2021, Jannach et al., 2021). As one of the most crucial foundations of CRSs, Natural Language Processing (NLP) has witnessed several breakthroughs in the past few years, including the use of pretrained transformer-based language models (LMs) for downstream tasks (Otter, Medina, & Kalita, 2020). Numerous studies have shown that these transformer-based LMs such as BERT (Devlin, Chang, Lee, & Toutanova, 2019), RoBERTa (Liu, Lin, Shi, & Zhao, 2021) and GPT (Radford, Narasimhan, Salimans, & Sutskever, 2018) pretrained on large corpora can learn universal language representations and are extraordinarily powerful for many downstream tasks via fine-tuning (Qiu et al., 2020). Recently, CRSs have started to leverage pretrained LMs for their ability to semantically interpret a wide range of preference statement variations and have demonstrated their potential to build a variety of strong CRSs (Hada and Shevade, 2021, Malkiel et al., 2020, Penha and Hauff, 2020).

However, pretrained LMs are well-known for exhibiting unintended social biases involving race, gender, or religion (Liang et al., 2021, Lu et al., 2020, Sheng et al., 2019). These biases result from unfair allocation of resources (e.g., policing, hospital services, or job availability) (Hutchinson et al., 2020, Zhang et al., 2020), stereotyping that propagates negative generalizations about particular social groups (Nadeem, Bethke, & Reddy, 2021), text that misrepresents the distribution of different social groups in the population (Liang et al., 2021), or language that is denigrating to particular social groups (Guo & Caliskan, 2021). Moreover, these biases may also be exacerbated by biases in data used for domain-specific LM fine-tuning for downstream tasks (Jin et al., 2021, Nadeem et al., 2021).

In this paper, we study a simple LM-driven recommendation backbone (termed LMRec) for CRSs to investigate how unintended bias manifests in substantially shifted price and category distributions of restaurant recommendations. Specifically, we generate templates with placeholders (a.k.a. template-based result generation) indicating non-preferential information such as names or relationships that implicitly indicate race, gender, sexual orientation, geographical context, and religion, and study how different substitutions for these placeholders modulate price and category distributions (a.k.a. attribute-based analysis) with the proposed metrics. To this end, we make the following technical contributions:

  • The proposed investigation methodology extends the template-based analysis from research works on bias in language models (Kurita et al., 2019a, May et al., 2019, Sheng et al., 2019, Tan and Celis, 2019) and the attribute-based analysis from the literature on fair recommender systems (Deldjoo et al., 2021, Mansoury et al., 2019, Tsintzou et al., 2019) to generate conversational recommendation results and to perform user–item attribute fairness analysis in language-based conversational recommender systems.

  • Our proposed methodology for user-item attribute bias analysis in conversational recommender systems provides novel techniques and metrics for use in fair recommender systems research.

Through the application of the above technical methodology and proposed metrics, we make the following key observational contributions:

  • LMRec recommends significantly more low-priced establishments when a black- vs. white-associated name is mentioned.

  • LMRec recommends significantly more alcohol-serving venues when a male- vs. female-associated name is mentioned.

  • LMRec picks up indirect mentions of homosexual relations (e.g. “my brother and his boyfriend”) as indicated by the elevation of “gay bar” in the recommendations vs. a heterosexual relation (e.g., “my brother and his girlfriend”).

  • Mentioning visits to professional locations (a “fashion studio” or “law office”) or a “synagogue” lead to a higher average price range of LMRec recommendations compared to mentioning a visit to the “convenience store” or a “mosque”.

While these results raise red flags regarding a range of previously undocumented unintended biases that can occur in LM-driven CRSs, there is fortunately a silver lining: we show that combining train side masking and test side neutralization of non-preferential entities nullifies the observed biases without hurting recommendation performance. Hence, with future language model-driven CRS assistants having a potential reach of hundreds of millions of end-users, the results of this work present an important step forward in identifying and mitigating potential sources of bias in CRSs that align with general goals of inequality reduction in society (Desa et al., 2016).

Section snippets

Related work

This section briefly summarizes how fairness/bias issues have been analysed in two requisite elements of language model-driven recommender systems: recommendation systems and language models. Following this, we review conversational recommender systems, where there is a notable lack of work on bias in LM-driven CRSs.

Methodology

In this section, we first provide a brief overview of BERT, followed by the description of LMs for Recommendation (LMRec) and technical details. Finally, we will outline our template-based methodology for exploring unintended bias in LMRec.

Experimental results

We now conduct several experiments to (1) evaluate the recommendation performance of LMRec and (2) identify and measure the unintended biases (e.g., via Percentage Score and Association Score). We aim to answer the following key research questions:

  • RQ1: How does LMRec perform and does test-side neutralization degrade performance with and without train-side masking?

  • RQ2: What ways may unintended racial bias appear?

  • RQ3: What ways may unintended gender bias appear?

  • RQ4: What ways may unintended

Limitations

We now proceed to outline some limitations of our analysis that might be explored in future work:

  • Choice of model: As discussed in Section 3.3, the recommendation results for this work are based purely on the context of language requests at test time and are not personalized to individual users. Therefore, future work can investigate the existence of unintended biases in a personalized version of LMRec although this extension of LMRec would be a novel contribution itself. Due to this

Conclusion

Given the potential that pretrained LMs offer for CRSs, we have presented the first quantitative and qualitative analysis to identify and measure unintended biases in language model-driven recommendation. We observed that the LMRec model exhibits various unintended biases without involving any preferential statements nor recorded preferential history of the user, but simply due to an offhand mention of a name or relationship that in principle should not change the recommendations. Fortunately,

CRediT authorship contribution statement

Tianshu Shen: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Jiaru Li: Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Mohamed Reda Bouadjenek: Conceptualization, Software, Validation, Investigation, Data curation, Writing – original draft, Visualization. Zheda Mai:

Uncited references

Goldstein, 1969

References (111)

  • AbdollahpouriHiman et al.

    Multistakeholder recommendation: Survey and research directions

    User Modeling and User-Adapted Interaction

    (2020)
  • AbdollahpouriHiman et al.

    Multi-stakeholder recommendation and its connection to multi-sided fairness

  • BadjatiyaPinkesh et al.

    Stereotypical bias removal for hate speech detection task using knowledge-based generalizations

  • BarikeriSoumya et al.

    RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models

  • BertrandMarianne et al.

    Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination

    American Economic Review

    (2004)
  • BhardwajRishabh et al.

    Investigating gender bias in bert

    Cognitive Computation

    (2021)
  • BolukbasiTolga et al.

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings

    Advances in Neural Information Processing Systems

    (2016)
  • BorgesRodrigo et al.

    On mitigating popularity bias in recommendations via variational autoencoders

  • BrandAlexander et al.

    Paying the pink tax on a blue dress-exploring gender-based price-premiums in fashion recommendations

  • BravemanPaula A. et al.

    Socioeconomic disparities in health in the United States: what the patterns tell us

    American Journal of Public Health

    (2010)
  • BrownellKelly D. et al.

    Food fight: The inside story of the food industry, America’s obesity crisis, and what we can do about it

    (2004)
  • BurkeRobin

    Multisided fairness for recommendation

  • BurkeRobin et al.

    Balanced neighborhoods for multi-sided fairness in recommendation

  • BursteinPaul

    Jewish educational and economic success in the United States: A search for explanations

    Sociological Perspectives

    (2007)
  • CaliskanAylin et al.

    Semantics derived automatically from language corpora contain human-like biases

    Science

    (2017)
  • ChenJiawei et al.

    Bias and debias in recommender system: A survey and future directions

    (2020)
  • ChristakopoulouKonstantina et al.

    Towards conversational recommender systems

  • CuiRuomeng et al.

    Wholesale price discrimination in global sourcing

    Manufacturing & Service Operations Management

    (2021)
  • DashAbhisek et al.

    When the umpire is also a player: Bias in private label product recommendations on e-commerce marketplaces

  • DeldjooYashar et al.

    A flexible framework for evaluating user and item fairness in recommender systems

    User Modeling and User-Adapted Interaction

    (2021)
  • DeldjooYashar et al.

    Recommender systems fairness evaluation via generalized cross entropy

  • DeldjooYashar et al.

    A survey of research on fair recommender systems

    (2022)
  • DesaUN

    Transforming our world: The 2030 agenda for sustainable development

    (2016)
  • DevlinJacob et al.

    BERT: Pre-training of deep bidirectional transformers for language understanding

  • DuesterhausMegan et al.

    The cost of doing femininity: Gendered disparities in pricing of personal care products and services

    Gender Issues

    (2011)
  • EdizelBora et al.

    FaiRecSys: mitigating algorithmic bias in recommender systems

    International Journal of Data Science and Analytics

    (2020)
  • EkstrandMichael D. et al.

    The demographics of cool

  • EkstrandMichael D. et al.

    All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness

  • EvansDavid S. et al.

    Matchmakers: The new economics of multisided platforms

    (2016)
  • FerraroAndres

    Music cold-start and long-tail recommendation: bias in deep representations

  • FinkelJenny Rose et al.

    Incorporating non-local information into information extraction systems by gibbs sampling

  • FriedmanBatya et al.

    Bias in computer systems

    ACM Transactions on Information Systems (TOIS)

    (1996)
  • FryerRoland G. et al.

    The causes and consequences of distinctively black names

    Quarterly Journal of Economics

    (2004)
  • FuZuohui et al.

    Fairness-aware explainable recommendation over knowledge graphs

  • FusterAndreas et al.

    Predictably unequal? The effects of machine learning on credit markets

    The Journal of Finance

    (2022)
  • GandalNeil et al.

    Obesity and price sensitivity at the supermarket

  • GaoRuoyuan et al.

    Addressing bias and fairness in search systems

  • GeYingqiang et al.

    Towards long-term fairness in recommendation

  • GeyikSahin Cem et al.

    Fairness-aware ranking in search & recommendation systems with application to linkedin talent search

  • GoldsteinSidney

    Socioeconomic differentials among religious groups in the United States

    American Journal of Sociology

    (1969)
  • Cited by (14)

    View all citing articles on Scopus
    View full text