A step toward building a unified framework for managing AI bias

Realistic case study of AI bias

The previous 10 years have seen AI’s widespread adoption and popularity, allowing it to permeate practically every aspect of our daily lives. However, safety and fairness concerns have prompted practitioners to prioritise them while designing and engineering AI-based applications (Chouldechova et al., 2018; Howard & Borenstein, 2018; Osoba & Welser, 2017). Researchers have enumerated a few applications that, because of biases, have a detrimental impact on people’s lives, such as biometrics apps, autonomous vehicles, AI chat-bots, robotic systems, employment matching, medical aid, and systems for children’s welfare.

The US judiciary uses the correctional offender management profiling for alternative sanctions (COMPAS) (Pagano et al., 2023), an AI-based tool, to identify offenders more likely to commit crimes again. However, the Pulitzer Prize-winning nonprofit news organisation ProPublica (Angwin et al., 2016) discovered that COMPAS was racially prejudiced and that black criminals were at high risk (Dressel & Farid, 2018). PredPol, or predictive policing, is an AI-enabled software to predict the next crime location based on the number of arrests and frequency of calls to the police regarding different offences. PredPol was criticised for its prejudiced behaviour as it targeted racial minorities (Benbouzid, 2019).

An AI algorithm known as the Amazon recruiting engine was developed to evaluate the resumes of job applicants applying to Amazon and then shortlist eligible candidates for further interviews and consideration. However, it turned out that the Amazon algorithm was discriminatory regarding hiring women (Hofeditz et al., 2022). A labelling tool in Google Photos adds a label to a photo that corresponds to whatever is seen in the image. When it referred to images of a black software developer and his friend as gorillas, it was determined to be racist (González Esteban & Calvo, 2022). The well-known programme StyleGAN (Karras, Laine & Aila, 2019) automatically creates eerily realistic human faces and produces white faces more frequently than racial minorities.

Biases based on racial, gender, and other demographic factors restrict communities from using AI-automated technologies for regular tasks in the health and education sectors (Egan et al., 1996; Brusseau, 2022). Organisations must know the different types of biases in their data/algorithm that can affect their machine learning (ML) models. Ultimately, identifying and mitigating biases that skewed or produced undesirable outcomes and impeded the advancements made by AI for the everyday person is more than necessary.

Assessment tools

A systematic effort has been made to provide appropriate tools for practitioners to adopt cutting-edge fairness strategies into their AI pipelines. Software toolkits and checklists are the two basic ways to ensure fairness (Richardson & Gilbert, 2021). Programming language functions that can be used to identify or lessen biases are known as toolkits. AI practitioners can employ checklists and detailed instructions by fairness specialists to ensure ethical consideration is incorporated across their pipelines (Bailey et al., 2019).

Fairness indicators and the What-If toolkit (WIT) are well-known tools that Google provides (Richardson et al., 2021). Fairness indicators are based on fairness metrics for binary and multiclass classifiers (Agarwal, Agarwal & Agarwal, 2022). The What-If tool is an interactive visual tool designed to examine, evaluate, and compare ML models (Richardson et al., 2021). Uchicago’s Aequitas is utilised to assess ML-based outcomes to identify various biases and make justified choices regarding the creation and implementation of such systems (Saleiro et al., 2018). IBM’s AI Fairness 360 is a toolkit providing fairness detection and mitigation strategies (Bellamy et al., 2019). LinkedIn’s fairness toolkit (LiFT) provides detection strategies for measuring fairness across various metrics (Brusseau, 2022). Microsoft’s Fairlearn, ML Fairness Gym, Scikit’s fairness tool, and PyMetrics Audit-AI are readily available tools to detect/mitigate (or both) bias and ensure fairness management. These assessment techniques expand practitioners’ options for developing ethical products and maintaining stakeholder confidence (Dankwa-Mullan & Weeraratne, 2022).

Despite the availability of tools that incorporate explainable machine learning methods and fair algorithms, none of them currently offers a comprehensive set of guidelines to assist users in effectively addressing the diverse fairness concerns that arise at different stages of the machine learning decision pipeline (Pagano et al., 2023). Moreover the responsibility for identifying and mitigating bias and unfairness is often entirely placed on the developer, who may not possess sufficient expertise in addressing these challenges and cannot be solely responsible for the same (Schwartz et al., 2022). There is lack of consensus on what constitutes fairness as there is no single definition of fairness. Moreover, these tools have limited ability to detect complex forms of bias as they are designed to detect bias due to protected attributes only and casual fairness demands to investigate underlying relations between different attributes. Fairness assessment tools can themselves be biased. For example, a tool that is designed to detect bias in natural language processing models may be biased towards certain languages or dialects. Sometimes they can be time-consuming and expensive to use for organizations to adopt these tools, especially small or resource-constrained organizations. One more notable objection regarding them is that it can be difficult to interpret the results which make it difficult for organization to take action to address any bias that is found. Majority of fairness assessment tools may not be able to identify the root cause of bias.

Fairness management research datasets

Several standard datasets are available to make research on bias and fairness easier. Each of them has sensitive or protected attributes that could be used to demonstrate unfair or biased treatment toward underprivileged groups or classes. Table 1 summarises some of these datasets’ characteristics. These datasets are used as a benchmark to evaluate and compare the effectiveness of bias detection and mitigation strategies.

Bias, discrimination and unfairness

The existence of bias, discrimination, and unfairness are related topics, but it is essential to distinguish them for further investigation/analysis (Future Learn, 2013). “Bias is the unfair inclination or prejudice in a judgement made by an AI system, either in favour of or against an individual or group” (Fenwick & Molnar, 2022). Discrimination can be considered a source of unfairness due to human prejudice and stereotyping based on sensitive attributes, which may happen intentionally or unintentionally. In contrast, bias can be considered a source of unfairness due to the data collection (Luengo-Oroz et al., 2021), sampling, and measurement (Li & Chignell, 2022). Discrimination is a difference in the treatment of individuals based on their membership in a group (Ethics of AI, 2020). Bias is a systematic difference in treating particular objects, people or groups compared to others. Unfairness is the presence of bias where we believe there should be no systematic difference (Zhao et al., 2018b). Bias is a statistical property, whereas fairness is generally an ethical issue.

Protected, sensitive and potentially biased attributes

Until we stop making it, machine bias will keep appearing everywhere: bias in, bias out (Alelyani, 2021). Protected attributes are characteristics that cannot be relied upon to make decisions and may be chosen based on the organisation’s objectives or regulatory requirements. Similarly, sensitive attributes are characteristics of humans that may be given special consideration for social, ethical, legal or personal reasons (Machine Learning Glossary: Fairness, (n.d.)). In literature, these terms are used interchangeably and in place of one another. It is a potentially biased attribute if changing the value of an attribute through the alternation function has an impact on prediction, such as altering a male attribute to a female one or a black attribute to a white one (Alelyani, 2021). Any one of these attributes in the dataset necessitates special consideration. Sex, race, color, age, marital status, family, religion, sexual orientation, political opinion, pregnancy, physical or mental disability, career responsibilities, social origins, and national extraction are a few attributes examples (Fair Work Ombudsman, (n.d.)). However, it has been observed that groups/individuals still face discrimination through proxy attributes (Chen et al., 2019), even in the absence of some protected or sensitive traits. Proxy attributes correlate with protected or sensitive attributes, such as zip code (linked with race) (Mehrabi et al., 2021).

Trio-bias feedback loop

Data is the primary driving force behind most AI systems and algorithms, so they need data to be trained. As a result, the functioning of these algorithms and systems is closely tied to data availability. Underlying training data bias will manifest through an algorithm’s predictions (trained on it). Furthermore, algorithms may reflect discriminatory behaviour based on specific design considerations even when the data is fair. The biased algorithm’s output can then be incorporated into the existing system and impact users’ choices, producing more biased data that can be used to train the new algorithm. Figure 2 depicts the feedback loop between data biases, algorithmic biases, and user involvement. Humans are involved in data preparation and algorithm design; therefore, whether bias results from data or an algorithm, humans are the root cause.

Categorisation of biases

The human bias has more than 180 varieties, for example Dabas (2021). Bias has been broadly divided into three major categories to accomplish fairness with a focus on detection and mitigation mechanisms and dispel confusion with many different types of bias (Gan & Moussawi, 2022). Bias categories include pre-existing, technical and emergent (Friedman et al., 2013).

A bias before the development of technology is referred to as pre-existing bias. This kind of bias has its roots in social structures and manifests itself in individual biases. The same is introduced into technology by people and organisations responsible for its development (Curto et al., 2022), whether explicitly or implicitly, consciously or unconsciously (Friedman et al., 2013). The AI literature identifies that the most typical biases are pre-existing (Gan & Moussawi, 2022). A few examples of pre-existing bias are the wrong model of Microsoft bot (Victor, 2016), Duane Buck’s murder (O’Neil, 2017), the stance on fairness (O’Neil, 2017), criminal justice models (Bughin et al., 2018), and the understanding of concepts and reasonability of CO₂ emissions (Luengo-Oroz, 2019).

Technical bias refers to concerns about a product’s technological design, such as technical limitations or decisions (Friedman et al., 2013). Technical issues faced in prominent software like IMPACT, Tech, LSI-R, Kyle’s job application and hiring algorithms are because of technical bias. Emergent bias emerges after the practical use of a design as a result of a shift in social awareness or cultural norms (Friedman et al., 2013). PredPol, St. George’s model, COMPAS, and facial analysis tech were adversely affected due to emergent bias.

Origins of biases within the development cycle

The literature regarding the causes of biases and potential mitigation strategies is still scattered, and work on a systematic methodology for dealing with potential biases (Nascimento et al., 2018) and building well-established ML frameworks is in progress (Suresh & Guttag, 2021; Ricardo, 2018; Silva & Kenney, 2019). In this article, we attempt to overcome confusion and enhance understanding regarding the origin of different categories of bias during the development cycle by mapping SDLC, MLLC and CRISP-DM on a single scale, as shown in Fig. 3.

Professional designers or developers introduce pre-existing bias into the technical process, which is why it drags into the modelling phase of the CRISP-DM cycle (Barton et al., 2019). Technical bias, as opposed to pre-existing bias, results from how problems in the technical design are resolved. The design process and model evaluation contain many areas where technical bias can be identified (Friedman et al., 2013; Ho & Beyan, 2020). Pre-existing and technical biases occur prior to and within the technical development process. However, emergent bias appears during the actual use of the technical product after development (Barton et al., 2019). Emergent bias can be detected before deployment (during testing) and is often the most obvious category of bias (Gan & Moussawi, 2022). The same type of bias has several names in the literature. Therefore, biases were descriptively synthesised and characterised based on their origins (Ho & Beyan, 2020; Fink, 2019). Based on a thorough understanding of the development cycles, commonly encountered distinct types of biases were assigned to the phases based on their origin, as shown in Fig. 4.

The bias type and subtype distribution across different phases will create an appropriate detection method (Fahse, Huber & Giffen, 2021). Equal opportunity, equalized odds, conditional demographic disparity, disparate impact, Euclidean distance, Mahalanobis distance, Manhattan distance, demographic disparity, and different tools and libraries are used for bias detection (Garg & Sl, 2021).

Bias minimising strategies

Value sensitive design (VSD) is a methodology that can contribute to understanding and addressing issues of bias in AI systems (Simon, Wong & Rieder, 2020) and to promote transparency (Cakir, 2020) and ethical principles in AI systems (Dexe et al., 2020). It is a framework that provides recommendations for minimising or coping with various biases (Gan & Moussawi, 2022; Friedman et al., 2013), is adopted in this research as a potential strategy to minimise the biases associated with AI. In the VSD study, “value” is a general phrase that relates to what user values in life (Atkinson, Bench-Capon & Bollegala, 2020). This theoretically-based approach provides three possible solutions based on the types of investigations to prevent problems and advocate AI-specific value-oriented metrics that all stakeholders mutually agreed on.

Conceptual, empirical and technical are three different kinds of investigations. A conceptual investigation primarily focuses on analysing or prioritising different stakeholders’ values in the design and use of technology (Barton et al., 2019; Floridi, 2010). Empirical investigation assesses a technical design’s effectiveness using factors like how people react to technological products (Barton et al., 2019; Floridi, 2010). It frequently entails observation and recording; quantitative and qualitative research methods are appropriate. Because of these characteristics, the phases are limited to data understanding (slightly), data preparation, modeling (to some extent), deployment and feedback.

The primary focus of the technical investigation is on technology while focusing on how technological characteristics and underlying mechanics promote or undermine human values and involves proactive system design to uphold values discovered in conceptual investigations (Gan & Moussawi, 2022; Friedman et al., 2013; Ho & Beyan, 2020). VSD has revealed the relationships between different types of investigations and types of AI biases (Umbrello & Van de Poel, 2021). Researchers concluded by recognising the value of conceptual and empirical investigation for addressing pre-existing bias, investigations for minimising technical bias, and technical and empirical investigation for tackling emerging bias (Gan & Moussawi, 2022).

Bias mitigation methods

Several methods can mitigate a single bias, and multiple biases can be mitigated by a single method. The socio-technical approach comprising technical and nontechnical methods is widely adopted to counterattack the harmful effects of biased decisions (Fahse, Huber & Giffen, 2021). This approach mitigates bias and prevents it from recurring in the future. Table 2 depicts which method effectively mitigates different types of bias whenever it appears in different stages of the AI development cycle. The mitigation process is not executed during Feedback. From risk management’s perspective, to evaluate an AI system’s performance, ‘Fairness’ outclass any other vital measure, i.e., dependability, efficiency, and accuracy (Barton et al., 2019; Suri et al., 2022).

Various strategies have been put forth by researchers and practitioners to address bias in AI. These strategies encompass data pre-processing, model selection, and post-processing decisions. However, each of these methods has its own set of limitations and difficulties, such as the scarcity of diverse and representative training data, the complexity of identifying and quantifying different forms of bias and the potential trade-offs between fairness and accuracy (Pagano et al., 2023). Additionally, ethical considerations arise when determining which types of bias to prioritize and which groups should be given priority in the mitigation process (Pagano et al., 2022). Bias mitigation methods can be complex, computationally expensive and can introduce new biases for example, a method that tries to balance the representation of different groups in a dataset may introduce a new bias in favor of the majority group. They can be brittle such as that they can be sensitive to changes in the data or the model. This can make it difficult to ensure that the model remains fair over time. Another notable aspect that needs to be addressed are that there is a lack of understanding of the long-term effects of bias mitigation methods & standardized evaluation metrics. These methods can be opaque as it can be difficult to understand how they work, subsequently it can make it difficult to trust these methods, and to ensure that they are not introducing new biases. They might feel it difficult to adapt to new tasks, as result of it, they may not be effective for all machine learning models.

Despite these obstacles, the mitigation of bias in AI is of utmost importance to establish just and equitable systems that benefit everyone in society (Balayn, Lofi & Houben, 2021). Continuous research and development of mitigation techniques are crucial to overcome these challenges and ensure that AI systems are employed for the welfare of all individuals (Huang et al., 2022).

Fairness formalisation

Constraints, measures, specifications and criteria to ensure fairness is defined during the business understanding and data understanding phases (Northeastern Global News, 2020), as depicted in Fig. 5. In fact, at this stage, the benchmarks for auditing or evaluating fairness are stated.

Fairness sampling

Fairness sampling generally refers to preprocessing skewed data through different methods, such as oversampling (Orphanou et al., 2021). Issues with data, i.e., inaccuracy, incompleteness, improperly labelling, too much/less, inconsistency, and silos, are addressed at this stage (Walch, 2020). Fair sampling is accomplished during the data understanding and data preparation phases.

Fairness learning

It is not always feasible to develop a fair model by eliminating the bias in the initial data before training. Designing a fair classifier that uses a fair algorithm is the solution in such scenarios (Acharyya et al., 2020). As a result, we can still use a biased dataset to train the model, and the fair algorithm still produces predictions through in-processing methods carried out during the modelling and evaluation phases (Acharyya et al., 2020).

Fairness certification

At the final stage of the testing and deployment phases, it is evaluated whether a prediction aligns with the criteria mentioned in the fairness formalisation phase by executing post-processing methods (Floridi, 2010). Fairness certification solutions verify that unfairness does not surface during the feedback phase.

Limitations and challenges of existing approaches

After thoroughly examining various aspects of tools, methodologies, frameworks, and fairness solution spaces, it is now appropriate to consolidate and summarize the overall major limitations and challenges encountered throughout this exploration:

1. One of the shortcomings of current approaches is the lack of transparency and interpretability (Srinivasu et al., 2022). Many tools and strategies employed to mitigate biases in various domains, such as machine learning algorithms or content moderation systems, often lack clear explanations of how they address bias (Feuerriegel, Dolata & Schwabe, 2020). This lack of transparency makes it difficult for users to understand the underlying biases being addressed and the effectiveness of the applied methods (Berente et al., 2019). Additionally, without proper transparency (Society to Improve Diagnosis in Medicine, 2021), it becomes challenging to identify potential unintended consequences or biases that may arise from the bias management methods themselves (Turney, 1995).

2. Another major shortcoming lies in the limited customization and adaptability (Basereh, Caputo & Brennan, 2021). Most tools and strategies are developed with a one-size-fits-all approach, which may not adequately account for the specific biases prevalent in different contexts or domains (Qiang, Rhim & Moon, 2023). This limits their effectiveness in managing biases that are nuanced and context-dependent. Furthermore, these methods often do not provide enough flexibility for users to customize and fine-tune the bias management mechanisms according to their specific needs and requirements.

3. Existing methods, tools, and strategies for bias management rely on manual intervention, which makes the process time-consuming and prone to human error. When bias management is carried out manually, it becomes difficult to ensure consistent and comprehensive coverage of all potential biases. Additionally, manual methods may lack scalability and efficiency, particularly when dealing with large datasets or complex models (Chhillar & Aguilera, 2022). Therefore, there is a need for more automated and robust approaches to bias management.

4. Bias can be unintentionally introduced when the dataset used to train AI models is not representative of the real-world population it is designed to serve (Schwartz et al., 2022). For example, if a facial recognition system is trained primarily on data from lighter-skinned individuals, it may exhibit higher error rates for darker-skinned individuals. This lack of diversity can perpetuate existing societal biases and lead to discriminatory outcomes (Delgado et al., 2022; Michael et al., 2022).

5. Existing methodologies tend to place a heavy emphasis on technical solutions for bias mitigation, often neglecting the importance of interdisciplinary collaboration (Madaio et al., 2022). Addressing bias in AI requires input from diverse stakeholders (Michael et al., 2022), including ethicists, social scientists, and policymakers. Without incorporating a multidisciplinary perspective, frameworks may overlook crucial ethical considerations and fail to account for the broader societal impact of AI systems.

6. Many existing strategies tend to oversimplify the concept of bias, reducing it to a binary problem. They often focus on mitigating only explicit biases while overlooking implicit biases, which are more subtle and deeply ingrained in societal structures (Peters, 2022). Addressing implicit biases requires a more nuanced understanding of the underlying social dynamics and power structures. Failure to consider these complexities can result in incomplete or ineffective bias mitigation strategies.

7. Identifying and mitigating the various interlinked biases that can arise in AI systems poses a significant challenge due to their diverse nature and complexity.

8. In some cases, there may be a potential trade-off between fairness and accuracy. For example, if an AI system is designed to be fair to all groups, it may not be as accurate as it could be.

9. There are ethical considerations around how to mitigate bias in AI systems (Straw, 2020). For example, should we prioritize fairness to individuals or to groups? Should we focus on mitigating historical bias or on preventing future bias.

10. The prevailing focus of most approaches lies in addressing bias reactively, leading to high costs associated with corrective measures. There is a pressing need to adopt a proactive stance and mitigate bias as soon as it is identified.

Lack of comprehensive evaluation frameworks is another hurdle in way to achieve fairness. While some methods may claim to address biases, there is often a lack of standardized evaluation frameworks to assess their effectiveness and potential trade-offs (Landers & Behrend, 2023). This absence of robust evaluation frameworks hinders the attainment of fairness on a global scale.

Establishing an agile approach to address bias in AI systems:

A framework for managing bias in AI systems should be created to encourage fairness, accountability, and transparency throughout the AI system lifetime while integrating software engineering best practices, in light of the challenges listed above. Effectively reducing bias requires a multifaceted, adaptable, transparent, scalable, accessible, interdisciplinary, and iterative approach.

Agile methods can be employed to address bias in AI systems. By adopting an agile approach, AI developers can continuously monitor and mitigate bias throughout the development lifecycle and all stakeholders will be well informed with current situation (Benjamins, Barbado & Sierra, 2019). Framework based on agile approach can not only mitigate bias at the spot in proactive manners but also eradicate it from appearing in future by eliminating all interlinked biases as (Landers & Behrend, 2023; Caldwell et al., 2022). To design a framework for fair AI several working variables play a key role. Let us explore some of these variables in detail:-

• Data collection and preprocessing

• The first working variable to consider is the data used to train AI models. It is essential to ensure that the data collected is representative and diverse (Zhao et al., 2018a), without any inherent biases (de Bruijn, Warnier & Janssen, 2022). Biases can emerge if the data reflects historical prejudices or imbalances (Clarke, 2019). Careful preprocessing is necessary to identify and address these biases to prevent unfair outcomes. For example, if a facial recognition system is trained predominantly on a specific demographic, it may exhibit racial or gender biases.

• Algorithmic transparency and explainability

• To design a fair AI framework, it is crucial to consider the transparency and explainability of the algorithms used (Umbrello & Van de Poel, 2021). Black-box algorithms that provide no insight into their decision-making processes can pose challenges in identifying and rectifying biases (Clarke, 2019). By promoting algorithmic transparency, stakeholders can understand how decisions are made and detect any unfairness in the system. Explainable AI techniques, such as providing interpretable explanations for decisions, can also enhance fairness and accountability (Toreini et al., 2020).

• Evaluation metrics

• Establishing appropriate evaluation metrics is another essential working variable in designing a fair AI framework. The metrics used to assess the performance of AI systems should go beyond traditional accuracy measures (Buolamwini & Gebru, 2018) and incorporate fairness considerations (Clarke, 2019). For instance, metrics like disparate impact, equal opportunity, and predictive parity can help identify and mitigate biases across different demographic groups. Evaluating AI systems on these fairness metrics ensures that fairness is a fundamental goal rather than an afterthought.

• Regular audits and monitoring

• A fair AI framework requires ongoing audits and monitoring to identify and rectify biases that may emerge over time. Regular assessments can help evaluate the fairness of AI algorithms and models in real-world scenarios (Landers & Behrend, 2023). It allows for continuous improvement and ensures that any biases are promptly addressed. Organizations should establish mechanisms to monitor the performance of AI systems, collect feedback from users, and engage in iterative improvements to enhance fairness (Saas et al., 2022). Moreover, encourage continuous learning within your team about bias, fairness, and ethical AI. Continuously stay informed about the most recent research and best practices in this domain and ensure ongoing updates.

• Stakeholder inclusivity

• Involving diverse stakeholders in the design and deployment of AI systems is a critical working variable for fair AI. Including representatives from different communities, demographic groups, and experts from various fields can help identify potential biases and ensure fairness (Clarke, 2019). It is crucial to consider the perspectives and experiences of those who may be disproportionately affected by AI systems. Such inclusivity can lead to a more comprehensive understanding of biases and result in fairer outcomes (Johansen, Pedersen & Johansen, 2021).

• Ethical guidelines and governance

• Ethical guidelines and governance play a vital role in shaping the design of a fair AI framework (Hildebrandt, 2021). Organizations should establish clear policies and guidelines that explicitly address fairness concerns. These guidelines should define what constitutes fairness and provide actionable steps to ensure it is upheld. Integrating ethical considerations into the design and decision-making processes can help prevent biases (Jobin, Ienca & Vayena, 2019). In nutshell, agile methods provide a framework for effectively addressing bias in AI systems. By adopting diverse teams, defining clear goals and metrics, conducting frequent reviews and iterations, ensuring transparency, curating unbiased datasets, monitoring and evaluating system performance, collaborating with stakeholders, and prioritizing ethics, organizations can develop and deploy AI systems that are fair, unbiased, and inclusive.