Towards understanding and mitigating unintended biases in language model-driven conversational recommendation
Introduction
With the prevalence of language-based intelligent assistants such as Amazon Alexa and Google Assistant, conversational recommender systems (CRSs) have attracted growing attention as they can dynamically elicit users’ preferences and incrementally adapt recommendations based on user feedback (Gao et al., 2021, Jannach et al., 2021). As one of the most crucial foundations of CRSs, Natural Language Processing (NLP) has witnessed several breakthroughs in the past few years, including the use of pretrained transformer-based language models (LMs) for downstream tasks (Otter, Medina, & Kalita, 2020). Numerous studies have shown that these transformer-based LMs such as BERT (Devlin, Chang, Lee, & Toutanova, 2019), RoBERTa (Liu, Lin, Shi, & Zhao, 2021) and GPT (Radford, Narasimhan, Salimans, & Sutskever, 2018) pretrained on large corpora can learn universal language representations and are extraordinarily powerful for many downstream tasks via fine-tuning (Qiu et al., 2020). Recently, CRSs have started to leverage pretrained LMs for their ability to semantically interpret a wide range of preference statement variations and have demonstrated their potential to build a variety of strong CRSs (Hada and Shevade, 2021, Malkiel et al., 2020, Penha and Hauff, 2020).
However, pretrained LMs are well-known for exhibiting unintended social biases involving race, gender, or religion (Liang et al., 2021, Lu et al., 2020, Sheng et al., 2019). These biases result from unfair allocation of resources (e.g., policing, hospital services, or job availability) (Hutchinson et al., 2020, Zhang et al., 2020), stereotyping that propagates negative generalizations about particular social groups (Nadeem, Bethke, & Reddy, 2021), text that misrepresents the distribution of different social groups in the population (Liang et al., 2021), or language that is denigrating to particular social groups (Guo & Caliskan, 2021). Moreover, these biases may also be exacerbated by biases in data used for domain-specific LM fine-tuning for downstream tasks (Jin et al., 2021, Nadeem et al., 2021).
In this paper, we study a simple LM-driven recommendation backbone (termed LMRec) for CRSs to investigate how unintended bias manifests in substantially shifted price and category distributions of restaurant recommendations. Specifically, we generate templates with placeholders (a.k.a. template-based result generation) indicating non-preferential information such as names or relationships that implicitly indicate race, gender, sexual orientation, geographical context, and religion, and study how different substitutions for these placeholders modulate price and category distributions (a.k.a. attribute-based analysis) with the proposed metrics. To this end, we make the following technical contributions:
- •
The proposed investigation methodology extends the template-based analysis from research works on bias in language models (Kurita et al., 2019a, May et al., 2019, Sheng et al., 2019, Tan and Celis, 2019) and the attribute-based analysis from the literature on fair recommender systems (Deldjoo et al., 2021, Mansoury et al., 2019, Tsintzou et al., 2019) to generate conversational recommendation results and to perform user–item attribute fairness analysis in language-based conversational recommender systems.
- •
Our proposed methodology for user-item attribute bias analysis in conversational recommender systems provides novel techniques and metrics for use in fair recommender systems research.
Through the application of the above technical methodology and proposed metrics, we make the following key observational contributions:
- •
LMRec recommends significantly more low-priced establishments when a black- vs. white-associated name is mentioned.
- •
LMRec recommends significantly more alcohol-serving venues when a male- vs. female-associated name is mentioned.
- •
LMRec picks up indirect mentions of homosexual relations (e.g. “my brother and his boyfriend”) as indicated by the elevation of “gay bar” in the recommendations vs. a heterosexual relation (e.g., “my brother and his girlfriend”).
- •
Mentioning visits to professional locations (a “fashion studio” or “law office”) or a “synagogue” lead to a higher average price range of LMRec recommendations compared to mentioning a visit to the “convenience store” or a “mosque”.
While these results raise red flags regarding a range of previously undocumented unintended biases that can occur in LM-driven CRSs, there is fortunately a silver lining: we show that combining train side masking and test side neutralization of non-preferential entities nullifies the observed biases without hurting recommendation performance. Hence, with future language model-driven CRS assistants having a potential reach of hundreds of millions of end-users, the results of this work present an important step forward in identifying and mitigating potential sources of bias in CRSs that align with general goals of inequality reduction in society (Desa et al., 2016).
Section snippets
Related work
This section briefly summarizes how fairness/bias issues have been analysed in two requisite elements of language model-driven recommender systems: recommendation systems and language models. Following this, we review conversational recommender systems, where there is a notable lack of work on bias in LM-driven CRSs.
Methodology
In this section, we first provide a brief overview of BERT, followed by the description of LMs for Recommendation (LMRec) and technical details. Finally, we will outline our template-based methodology for exploring unintended bias in LMRec.
Experimental results
We now conduct several experiments to (1) evaluate the recommendation performance of LMRec and (2) identify and measure the unintended biases (e.g., via Percentage Score and Association Score). We aim to answer the following key research questions:
- •
RQ1: How does LMRec perform and does test-side neutralization degrade performance with and without train-side masking?
- •
RQ2: What ways may unintended racial bias appear?
- •
RQ3: What ways may unintended gender bias appear?
- •
RQ4: What ways may unintended
Limitations
We now proceed to outline some limitations of our analysis that might be explored in future work:
- •
Choice of model: As discussed in Section 3.3, the recommendation results for this work are based purely on the context of language requests at test time and are not personalized to individual users. Therefore, future work can investigate the existence of unintended biases in a personalized version of LMRec although this extension of LMRec would be a novel contribution itself. Due to this
Conclusion
Given the potential that pretrained LMs offer for CRSs, we have presented the first quantitative and qualitative analysis to identify and measure unintended biases in language model-driven recommendation. We observed that the LMRec model exhibits various unintended biases without involving any preferential statements nor recorded preferential history of the user, but simply due to an offhand mention of a name or relationship that in principle should not change the recommendations. Fortunately,
CRediT authorship contribution statement
Tianshu Shen: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Jiaru Li: Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Mohamed Reda Bouadjenek: Conceptualization, Software, Validation, Investigation, Data curation, Writing – original draft, Visualization. Zheda Mai:
Uncited references
Goldstein, 1969
References (111)
- et al.
Consumer-lending discrimination in the FinTech era
Journal of Financial Economics
(2022) - et al.
Fast food, race/ethnicity, and income: a geographic analysis
American Journal of Preventive Medicine
(2004) - et al.
Food cravings, binge eating, and eating disorder psychopathology: Exploring the moderating roles of gender and race
Eating Behaviors
(2016) - et al.
What does ”occupation” represent as an indicator of socioeconomic status?: Exploring occupational prestige and health
Social Science & Medicine
(2010) - et al.
Advances and challenges in conversational recommender systems: A survey
AI Open
(2021) Fried chicken and fresh apples: racial segregation as a fundamental cause of fast food density in black neighborhoods
Health & Place
(2008)- et al.
Bias-aware hierarchical clustering for detecting the discriminated groups of users in recommendation systems
Information Processing & Management
(2021) - et al.
Neighborhood characteristics associated with the location of food stores and food service places
American Journal of Preventive Medicine
(2002) Food cravings in young and elderly adults
Appetite
(1997)- et al.
Food cravings in a college population
Appetite
(1991)
Multistakeholder recommendation: Survey and research directions
User Modeling and User-Adapted Interaction
Multi-stakeholder recommendation and its connection to multi-sided fairness
Stereotypical bias removal for hate speech detection task using knowledge-based generalizations
RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models
Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination
American Economic Review
Investigating gender bias in bert
Cognitive Computation
Man is to computer programmer as woman is to homemaker? debiasing word embeddings
Advances in Neural Information Processing Systems
On mitigating popularity bias in recommendations via variational autoencoders
Paying the pink tax on a blue dress-exploring gender-based price-premiums in fashion recommendations
Socioeconomic disparities in health in the United States: what the patterns tell us
American Journal of Public Health
Food fight: The inside story of the food industry, America’s obesity crisis, and what we can do about it
Multisided fairness for recommendation
Balanced neighborhoods for multi-sided fairness in recommendation
Jewish educational and economic success in the United States: A search for explanations
Sociological Perspectives
Semantics derived automatically from language corpora contain human-like biases
Science
Bias and debias in recommender system: A survey and future directions
Towards conversational recommender systems
Wholesale price discrimination in global sourcing
Manufacturing & Service Operations Management
When the umpire is also a player: Bias in private label product recommendations on e-commerce marketplaces
A flexible framework for evaluating user and item fairness in recommender systems
User Modeling and User-Adapted Interaction
Recommender systems fairness evaluation via generalized cross entropy
A survey of research on fair recommender systems
Transforming our world: The 2030 agenda for sustainable development
BERT: Pre-training of deep bidirectional transformers for language understanding
The cost of doing femininity: Gendered disparities in pricing of personal care products and services
Gender Issues
FaiRecSys: mitigating algorithmic bias in recommender systems
International Journal of Data Science and Analytics
The demographics of cool
All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness
Matchmakers: The new economics of multisided platforms
Music cold-start and long-tail recommendation: bias in deep representations
Incorporating non-local information into information extraction systems by gibbs sampling
Bias in computer systems
ACM Transactions on Information Systems (TOIS)
The causes and consequences of distinctively black names
Quarterly Journal of Economics
Fairness-aware explainable recommendation over knowledge graphs
Predictably unequal? The effects of machine learning on credit markets
The Journal of Finance
Obesity and price sensitivity at the supermarket
Addressing bias and fairness in search systems
Towards long-term fairness in recommendation
Fairness-aware ranking in search & recommendation systems with application to linkedin talent search
Socioeconomic differentials among religious groups in the United States
American Journal of Sociology
Cited by (14)
Transfer learning for collaborative recommendation with biased and unbiased data
2023, Artificial IntelligenceRecognizing misogynous memes: Biased models and tricky archetypes
2023, Information Processing and ManagementFairness in recommender systems: research landscape and future directions
2024, User Modeling and User-Adapted Interaction