Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Standardizing fairness-evaluation procedures: interdisciplinary insights on machine learning algorithms in creditworthiness assessments for small personal loans

Standardizing fairness-evaluation procedures: interdisciplinary insights on machine learning... In the current European debate on the regulation of Artificial Intelligence there is a consensus that Artificial Intelligence (AI) systems should be fair. However, the multitude of existing indicators allowing an AI system to be labeled as “(un)fair” and the lack of standardized, application field specific criteria to choose among the various fairness-evaluation methods makes it difficult for potential auditors to arrive at a final, consistent judgment. Focusing on a concrete use case in the application field of finance, the main goal of this paper is to define standardizable minimal ethical requirements for AI fairness-evaluation. For the applied case of creditworthiness assessment for small personal loans, we highlighted specific distributive and procedural fairness issues inherent either to the computing process or to the system’s use in a real-world scenario: (1) the unjustified unequal distribution of predictive outcome; (2) the perpetuation of existing bias and discrimination practices; (3) the lack of transparency concerning the processed data and of an explanation of the algorithmic outcome for credit applicants. We addressed these issues proposing minimal ethical requirements for this specific application field: (1) regularly checking algorithmic outcome through the conditional demographic parity metric; (2) excluding from the group of processed param- eters those that could lead to discriminatory outcome; (3) guaranteeing transparency about the processed data, in addition to counterfactual explainability of algorithmic decisions. Defining these minimal ethical requirements represents the main contribution of this paper and a starting point toward standards specifically addressing fairness issues in AI systems for creditworthiness assessments aiming at preventing unfair algorithmic outcomes, in addition to unfair practices related to the use of these systems. As a final result, we indicate the next steps that can be taken to begin the standardization of the three use case-specific fairness requirements we propose. Keywords Artificial intelligence · Data science · Fairness · Fairness metric · Standardization * Sergio Genovesi Romina Kleiner genovesi@uni-bonn.de kleiner@time.rwth-aachen.de Julia Maria Mönig Lena Krieger moenig@uni-bonn.de Lena.Krieger@din.de Anna Schmitz Alexander Zimmermann anna.schmitz@iais.fraunhofer.de Alexander.Zimmermann@din.de Maximilian Poretschkin University of Bonn, Bonn, Germany maximilian.poretschkin@iais.fraunhofer.de Fraunhofer IAIS, Sankt Augustin, Germany Maram Akila maram.akila@iais.fraunhofer.de RWTH, University of Aachen, Aachen, Germany Manoj Kahdan DIN, Berlin, Germany kahdan@time.rwth-aachen.de Vol.:(0123456789) 1 3 AI and Ethics 1 Introduction2 Defining fairness In the current European debate on the regulation of Artificial 2.1 AI ethics Intelligence there is a consensus that Artificial Intelligence (AI) systems should be developed in a human centered way In the current AI ethics discussions, fairness is generally and should be “trustworthy” [23, 24, 31, 99]. According to framed in accounts of distributive justice and is broadly these documents, one value that constitutes “trustworthi- referred to as unbiased distribution of access to services ness” is fairness. Many current publications on AI fairness and goods—e.g., access to treatments in healthcare [39, 81] predominantly focus on avoiding or fixing algorithmic dis- or access to credit [65]—and as absence of discrimination, crimination of groups or individuals and on data-de-bias- understood as unjustified unequal treatment of groups or ing, offering different metrics as tools to evaluate whether individuals [72, 76]. groups or individuals are treated differently [8 , 71, 96]. Concerning distributive justice and non-discrimination Moreover, the International Standardization Organization as equal treatment, one of the primary contemporary phil- (ISO)/International Electrotechnical Commission (IEC) osophical references is the Rawlsian idea of equality of TR 24028:2020, Information technology—Artificial Intelli- opportunities. This idea requires that citizens having the gence—Overview of trustworthiness in Artificial Intelligence same talents and being equally motivated should receive lists fairness as an essential part for ensuring trustworthiness the same educational and economic opportunities regard- in AI (ISO/IEC 2020). However, the multitude of existing less of their wealth or social status [83] (p. 44). Since in the indicators allowing the labeling of an AI system as “(un) social praxis basic rights and liberties are neither accessible fair” and the lack of standardized, application field specific nor enjoyable in the same way for different citizens, soci- criteria to choose among the various fairness-evaluation ety should take adequate measures in order for all citizens methods makes it difficult for potential auditors to arrive to enjoy their rights and liberties. Rawls develops on this at a final, consistent judgment [24, 96, 98]. The increasing intuition stating that “the worth of liberty to persons and need for standardized methods to assess the potential risks of groups depends upon their capacity to advance their ends AI systems is also highlighted by the draft for an “Artificial within the framework the system defines. […] Some have Intelligence Act” suggested by the European Commission in greater authority and wealth, and therefore greater means to April 2021, which, in accordance with the so-called “New achieve their aims” [82] (p. 179). Consequently, to avoid the Legislative Framework,” ascribes a major role to “Stand- exaggeration of the unequal enjoyment of basic rights and ards, conformity assessment, certificates [and] registration” liberties, a fair society must enact compensation mechanisms (Chapter 5). to maximize their worth to the least advantaged [82] (ibid). Focusing on a concrete use case in the application field of It is essential to avoid the development of vicious circles of finance, the main goal of this paper is to define standardiz- (un)privilege-polarization in society due to the moral harm able minimal ethical requirements for AI fairness evaluation. they produce. Therefore, removing the opportunities of the In Sect. 2, we explore different understandings of fairness less privileged to truly benefit from their rights and liberties from three perspectives and address the different vantage results in a harmful form of negative discrimination that points of many stakeholders involved in the development, amplifies economic inequalities and undermines the chances commercialization, and use of AI systems. In Sect.  3, we for the less advantaged to live an autonomous life and set discuss the example of a risk scoring machine learning self-determined goals. This is a form of disrespect toward (ML) model for small personal loans. As a main contribu- their personhood [29, 68] and can amplify social resentment. tion of the paper, we suggest ethical minimal requirements The philosophical debate about Rawls’ theory and other that should be complied with when evaluating fairness and forms of “egalitarianism” could help clarify current emerg- highlight a preferred fairness metric for fairness-evaluation ing issues concerning algorithmic fairness [11]. Egalitari- purposes in this specific application field. In Sect.  4, we anism in this sense means that “human beings are in some investigate how to translate our research findings into stand- fundamental sense equal and that efforts should be made to ardization criteria to be used when assessing ML credit scor- avoid and correct certain forms of inequality” [11] (p. 2). ing systems for small personal loans. Many approaches try to determine the kind of equality that Noble highlights intersectional questions of fairness underlining the adverse effects that “algorithms of oppression” have on black women. In general, feminist scholars have stressed that unfairness and injus- tices usually go hand in hand with domination and oppression [75, 75]. 1 3 AI and Ethics should be sought and which inequalities should be avoided the UN’s Sustainable Development Goals (SDG)—also to in civil society to uphold the fundamental equality of human the future and not only limiting it to present generations. In beings: among others, equality of preference-satisfaction the practice, fairness toward future generations means act- [19], equality of welfare, income, and assets [28], and the ing sustainably. equality of capabilities to achieve their goals [87]. However, These considerations lead us to the following prelimi- regarding the application of these views on algorithmic deci- nary understanding of fairness in the context of an AI ethics sions, defending an equal opportunity approach rather than assessment. First, focusing on the unbiased distribution of an equal outcome is not always the most effective solution. access to services and goods and on the absence of discrimi- If, for candidate selection or calculation of insurance, focus- nation of groups or individuals, fairness means the equal ing on equal opportunity might lead to increase “economic treatment of people regardless of their sex, race, color, justice,” in other contexts, such as during airport security ethnic or social origin, genetic features, language, religion checks, equality of outcome in the form of “parity of impact” or belief, political or any other opinion, membership of a could help establish a sense of social solidarity avoiding national minority, property, birth, disability, age or sexual the over-examination of certain groups [11] (p. 7). Thus, orientation, when it comes to granting or denying access the choice of a specific approach to evaluate (in)equality to products, services, benefits, professional or educational depends on the specific application context. As Balayn and opportunities, and medical treatments based on an auto- Gürses claim, the regulation of AI must go “beyond de- mated evaluation and classification of individual or groups. biasing” [6]. Mere data-based or outcome-based solutions In addition, a fair system should not involve work exploita- trying to solve local distributive issues of a system, such as tion or the violation of human rights of any of the involved trying to solve a racial bias in image recognition software stakeholders during its life cycle. Moreover, the real-world by enlarging the data basis with pictures from people of all application of the system should not create or amplify power ethnic backgrounds, are not sufficient alone to address struc- unbalances between stakeholders, nor place specific stake- tural inequality issues at the root [60]. If decision-making holders’ groups in a disadvantaged position. processes that influence people’s access to opportunities are biased, the intersection between algorithmic fairness and 2.2 Data science structural (in)justice requires investigation [51]. In addition, other fairness issues that are indirectly related Fairness is discussed in the context of data and data-driven with the algorithmic outcome, but rather with the entire systems whose inherent patterns or statistical biases can be system design and application processes, as well as with interpreted as “unfair.” Here, it is important to emphasize their consequences for society, can occur. These aspects of that the evaluation of whether certain patterns are “fair” or fairness also overlap with other human rights and societal “unfair” transcends the specific expertise of data scientists values. For instance, a structural fairness issue of many ML and requires further legal, philosophical, political, and socio- systems is the phenomenon of “digital labor” [33], referring economic considerations. What is being explored in data (among other things) to the precarious work conditions and science under the term “fairness” are quantitative concepts the very low pay of many click workers generating train- to identify patterns or biases in data, in addition to technical ing data for ML systems. In addition, the commodification methods to mitigate them. of privacy related with the use of many digital services Data analysis and data-based modeling of real-world raises fairness issues since users are often kept unaware of relationships have progressed in recent years especially the exact use of their data, and they are not always in the through Machine Learning (ML). ML is a subdiscipline of position to defend their right to privacy [74, 91, 95]—being AI research in which statistical models are fitted to so-called so a disadvantaged stakeholder compared with the service training data, recognize patterns and correlations in this providers. Finally, addressing the sustainability concerns data, and generate predictions for new (input) data on this that emerged during the so-called “third wave” of AI ethics, basis. ML methods have become a particular focus of fair- global and intergenerational justice can be highlighted as ness research, as they provide everyday applications using fairness issues [37, 93]. Considering intergenerational jus- personal data, e.g., employment decisions, credit scoring, tice means to add a temporal, anticipatory dimension to our understanding of fairness and extend the claim for the equity of human living conditions—as, for instance, expressed in These are the protected attributes listed in Article 21 (Non-discrim- ination) of the EU Charter of Fundamental Rights. While the term bias is often connoted negatively in other disci- plines (with discriminatory effects, etc.), in this context (computer In addition, this cannot be the solution to this problem because it science) it merely means a statistical deviation from the standard [41] would feed even more data into the systems of the service providers [7]. Whether this constitutes a case of discrimination is another ques- and would therefore support their data hunger and business logic. tion. 1 3 AI and Ethics and facial recognition [71]. Furthermore, they pose the chal- from manual processes or, if possible, from the observation lenge that bias within the training data might lead to biased whether in the given examples the loans were repaid in full. model results. When building a ML model, the training data is used to adjust the internal model parameters which determine the 2.2.1 Short introduction to machine learning mapping through f. For instance, in a neural network, the weights assigned to the network’s edges are adjusted by the ML-based applications have enabled technological progress learning algorithm during model building, a phase which is which can particularly be attributed to the fact that their also called “training.” Overall, ML is an optimization pro- functionality is based on learning patterns from data. By this cedure that finds internal model parameters such that they means, ML methods provide approaches to solving tasks that optimize a defined performance metric on a training dataset. could not be effectively addressed by “traditional” software In this case, the performance metric specified as optimiza- fully specified by human rules. In particular, deep neural net- tion objective is referred to as “loss function.” For example, works, a type of ML method involving vast amounts of data, in a classification task, a quantitative measure of the distance have significantly advanced areas, such as image [26] and between ground-truth and model output could be used as a speech recognition [13], in addition to predicting complex loss function. Consequently, such a model would be gener- issues, for instance medical diagnostics [102] and predictive ated in training, which optimally approximates the relation- maintenance [17]. ships between x and y provided in the training data. ML methods are designed to learn from data to improve Underlying the ML approach of fitting a model to train- performance on a given task [41]. A task can be viewed as ing data is the idea that the model infers patterns which finding a mapping that, for an input x , assigns an output y help produce valuable outputs when applied to new data. which is useful for a defined purpose. One ML task that is The term “generalizability” is used to describe the aim that particularly relevant for fairness is classification. The pur - the model performs well on data not seen during training. pose of classification is to identify to which of a set of cat- Thus, for model evaluation, an additional test dataset differ - egories a given input belongs, for instance, whether a person ent from the training data is used. Given training and test is creditworthy or not. ML is about finding such a model data, according to Goodfellow et al. model quality is indi- f that solves a task effectively by y = f(x). To achieve this, cated by two quantities: (i) the training error measured by a learning algorithm adjusts parameters within the model. the loss function, and (ii) the difference between training and The fitness of the model for the given task can be evaluated test error [41]. A model with a large training error is called using quantitative measures. Such quantitative indicators of “underfitting," while one with a low training error but large model or data properties are generally referred to as “met- difference between training and test error is "overfitting" the rics.” For example, a typical performance metric for clas- training data. sification tasks is precision, which measures the proportion to which the classification to a certain category by the model 2.2.2 Meaning and challenges of “fairness” was correct. The data which ML methods use to build a model, called Data has a crucial impact on the quality of a ML model. In “training data,” is a collection of input examples that the computer science, data quality has already been researched model is expected to handle as part of the task. A single for “classical” information systems, where it is considered example in the data is called a datapoint. For classification especially regarding large amounts of stored operational and tasks, a datapoint in the training data of a ML model con- warehousing data, e.g., a company’s client database. Numer- tains, in addition to the example x, a “ground-truth” label ous criteria for data quality have been proposed, which can that specifies how the ML model should process the respec- be mapped within four dimensions: “completeness, unambi- tive input x. Following on the example of creditworthiness guity, meaningfulness, and correctness” [100]. Only recently classification, the training data for a ML model addressing has the operationalization of data quality specifically for ML this task may be drawn from previous credit applications, been explored [46]. The issue of data completeness, relating and the individual datapoints could include features, such to the training and test data sufficiently capturing the appli- as income, age, or category of work activity (e.g., self- cation domain, is particularly relevant in this context. There employed, employed). Moreover, each datapoint should is a high risk that an ML model has low statistical power on also contain a ground-truth label that could be derived data either not included or statistically insignificant in its training set. To prevent strong declines in a model’s produc- tive performance, measures are being researched for deal- ing with missing or underrepresented inputs [46] as well as 5 for detecting distribution skews, e.g., between training and Reinforcement learning methods that learn from interaction with production data [12]. systems or humans are not considered in this description. 1 3 AI and Ethics In certain tasks and application contexts, individuals are been presented [96], particularly in light of providing statis- affected by the outputs of a ML model. For example, ML tical evidence for unequal treatment in classification tasks. models are being used to support recruiting processes, deci- Corresponding to the approach of identifying and compar- sions on loan approval, and facial recognition [71]. Conse- ing groups for identifying bias, so-called “group fairness quently, it is essential that the model performs equally well metrics” constitute a large part of the fairness metrics pre- for all individuals. The research direction in data science sented to date. These metrics compare statistical quantities that addresses related issues from a technical perspective is regarding groups defined on the basis of certain attributes referred to under the term “fairness.” Clearly, the motivation in a dataset (e.g., a group could be defined by means of age, for “fairness” in data or ML models does not derive from a gender, or location if these attributes are provided in the technical perspective, nor does data science as a scientific data). Among the group fairness metrics, one can further discipline provide a sufficient basis for evaluating under distinguish between two types: i) metrics which compare the which circumstances these should be classified as “fair.” The distribution of outputs, and ii) metrics which compare the approaches and methods researched in this area are usually correctness of the outputs with respect to different groups. neutral, as they can be applied to structurally similar sce- An example of the first type is to measure the discrepancy narios that do not involve individuals. to which a certain output is distributed by percentage among Regarding model quality, the data are a central object of two different groups. This quantification approach is called study in fairness from two perspectives. First, aspects of data “statistical parity,” and Sect. 3.3. provides a detailed elabo- quality should not differ regarding particular groups of peo- ration. The second type of metrics focus on model quality ple. Regarding the dimension of completeness, for instance, and compare performance-related aspects with respect to certain population groups could be underrepresented in the different groups (e.g., specific error rates or calibration). For training data resulting in a lower performance of the ML instance, the metric “equal opportunity” [96] calculates the model with respect to these groups [14]. Another example difference between the true-positive rates of a model on the is that the ML model might infer biased patterns from the respective data subsets representing two different groups. training data if their representativeness is compromised by Such metrics can highlight model weaknesses by providing non-random sampling, for example, if predominantly posi- insight on where the model quality may be inconsistent. tive examples are selected from one population group but Besides group fairness metrics, further measures have negative examples are selected from another. Second, even been developed to disclose biases. Two examples are “indi- if data are of high quality from a technical perspective, they vidual fairness” [27] and “counterfactual fairness” [63]. may (correctly) reflect patterns that one would like to pre - “Individual fairness” is based on comparing individuals. vent from being reproduced by the ML model trained on Therefore, a distance metric is defined that quantifies the it. For instance, data might capture systemic bias rooted in similarity between two datapoints. The underlying idea of (institutional) procedures or practices favoring or disadvan- this approach is that similar model outputs should be gener- taging certain social groups [86]. The technical challenge ated for similar individuals. In addition, measurable indica- that arises here is fitting a model to the training data but tors for an entire data set have been derived using such a simultaneously preventing inferring certain undesirable pat- distance metric, for example, “consistency” [104]. Similarly, terns that are present. Overall, proceeding from the variety inequality indices from economics such as the generalized of biases identified to date, both measures that “detect” and entropy index have also been proposed as bias indicators for measures that “correct” (potentially unfair) patterns in data- datasets [89], which require a definition of individual prefer - sets and models are being explored [48] (p. 1175). ences. “Counterfactual fairness” considers individual data- points, similar to “individual fairness”; however, it examines 2.2.3 Measures that “detect” the effect of changing certain attribute values on model out- puts. This can be used to uncover if the model would have Aiming to “detect,” one research direction is concerned with generated a different output for an individual if they had a developing technical approaches to disclose and quantify different gender, age, or ethnicity, for example. Many of the biases in the first place. Numerous “fairness metrics” have presented bias quantification and detection approaches have been implemented in (partially) open-source packages and tools [3, 9, 40] and are likewise applicable to input–output- 6 mappings not based on ML. A variety of bias causes and types has been explored that cannot Different fairness metrics might pursue different target be fully mapped here. For a categorization, in line with the two view- points described, into computational as well as human and systemic states, e.g., balanced output rates between groups (statisti- bias, we refer to Schwartz et  al. [86]. For a categorization of biases cal parity) versus balanced error rates (equal opportunity). along the feedback cycle of data, algorithm and user interaction, see Therefore, they also differ greatly in their potential conflict [71], and for a mapping of biases to the life cycle of AI applications, with other performance goals. For instance, consider a see [90]. 1 3 AI and Ethics dataset in which Group A contains 30% positive ground- model output and certain attributes [59], or using optimiza- truth labels and Group B contains 60%. If the model is to tion constraints to align certain error rates among different reach a low value for a fairness metric that measures the dis- groups [103]. Another in-processing approach, which affects crepancy in the distribution of positive labels across groups, the entire model architecture, is to include an adversarial its outputs must deviate from ground-truth. In addition to network in the training that attempts to draw an inference sacrificing accuracy, this could also result in unbalanced about protected attributes from the model outputs [105]. The error rates. Thus, depending on the nature of the data, fair- model and its adversary are trained simultaneously, where ness metrics may be mutually exclusive [7]. the optimization goal of the original model is to keep the performance of the adversary as low as possible. In contrast, 2.2.4 Measures that “correct” post-processing refers to those measures that are applied to fully trained models. For example, corresponding methods Another direction of research is striving to develop tech- comprise calibration of outputs [78] and targeted threshold nical measures which can “correct” or mitigate detected setting (with thresholds per group, if applicable) to equalize bias. To this end, approaches along the different develop- error rates [49]. Many of the in- and post-processing meas- ment stages of ML models are being explored [35]. The ures are researched primarily for classification tasks, and underlying technical issue, especially when facing systemic the methods developed are typically tailored to one type of or historical bias, is to train a model by inferring correlations model for technical reasons. in data that performs well on a given task—but simultane- ously preventing learning of certain undesirable patterns 2.2.5 Outlook that are present in the data. An important starting point for addressing this apparent contradiction is the data itself. A In the fairness research field, a variety of approaches have basal pre-processing method that has been proposed is “Fair- been developed to better understand and control bias. ness through Unawareness,” meaning that those (protected) Beyond these achievements, still, open research questions attributes are removed from the data set for which correla- remain unaddressed from a technical and interdisciplinary tion with model output values is to be avoided [63], or whose perspective. Regarding the first, many metrics, in addition inclusion could be perceived as “procedurally unfair” [44]. to the methods that work toward their fulfillment, are appli- However, this method alone is not recognized as sufficiently cable to specific tasks only and impose strong assumptions. effective as correlated “proxies” might still be contained in Here, one challenge is to adapt specific measures from one the data [61], and many mitigation methods actively incor- use case to another. Regarding the latter, a central issue is porate the protected attributes to factor out bias [63]. Fur- that different fairness metrics pursue different target states ther examples of pre-processing methods range from tar- for an ML model (see “Measures to detect”), therefore a geted reweighing, duplication, or deletion of datapoints to choice must be made when assessing fairness in practice. modifying the ground-truth [57] or creating an entirely new Furthermore, the concrete configuration of specific metrics, (synthetic) data representation [104]. The latter are usually for example, how a meaningful similarity metric for assess- based on an optimization in which the original datapoints are ing “individual fairness” should be defined, remains unre - represented as a debiased combination of prototypes. While solved. The question of which fairness metrics and meas- these methods primarily aim at equalizing ground-truth val- ures are desirable or useful in practice must be addressed in ues among different groups, some optimization approaches interdisciplinary discussion. A concrete example is provided for generating data representations also include aspects of in Sect. 3.3. individual fairness [64]. Furthermore, to mitigate representa- tion or sampling bias, over-sampling measures to counteract 2.3 Management science class imbalance [16] are being researched [15]. In addition to algorithmic methods, documentation guidelines have been Management literature [4, 21, 38, 77, 88] divides fairness developed to support adherence to good standards, e.g., in into four types: distributive, procedural, interpersonal, and data selection [36]. informational fairness. While the data centrally influences the model results Distributive fairness refers to the evaluation of the out- and pre-processing methods offer the advantage that they come of an allocation decision [34]. Equity is inherent to can typically be selected independently of the model to distributive fairness [79]. Hence, to achieve distributive be trained, research is also being conducted on so-called fairness, participants must be convinced that the expected in- and post-processing measures. In-processing measures value created by the organization is proportionate to their are those that intervene in modeling. This can be realized, contributions [4]. Procedural fairness refers to the process for example, by supplementing the loss function with a of decision right allocation (i.e., how do the parties arrive at regularization term that reduces the correlation between a decision outcome? [62]). To achieve procedural fairness, 1 3 AI and Ethics Table 1 Procedural and Fairness Type Components Description distributive fairness measurement scales [22, 79] Procedural (Rules Process Control Procedures provide opportunities for voice taken from [67, 92]) Decision Control Procedures provide influence over outcomes Consistency Procedures are consistent across persons and time Bias Suppression Procedures are neutral and unbiased Accuracy Procedures are based on accurate information Correctability Procedures offer opportunities for appeals of outcomes Representativeness Procedures consider concerns of subgroups Ethicality Procedures uphold standards of morality Distributive (Rules Equity Outcomes are allocated according to contributions taken from [1, 66]) Equality Outcomes are allocated equally Need Outcomes are allocated according to need a fair assignment of decision rights is required [4, 70]. The discrimination, aiming to exploit the market potential. decision rights must ensure fair procedures and processes for Banks’ business model is to spread risks and price risks future decisions that influence value creation [4 , 70]. according to their default risk in order to achieve the best Interpersonal and informational fairness refer to interac- possible return on investment. For example, banks price tional justice, which is defined by the interpersonal treat- the default risk of loans variously and derive differentiated ment that people experience in decision-making processes prices. Here, procedural fairness is crucial in the overall [10]. Interpersonal fairness reflects the degree of respect fairness assessment, since procedurally unfair price settings and integrity shown by authority figures in the execution of lead to higher overall price unfairness [32]. Ferguson, Ellen, processes. Informational fairness is specified by the level of and Bearden highlight that random pricing is assessed to truthfulness and justification during the processes [20, 43]. be more unfair than possible cost-plus pricing (price is the To ensure a differentiated assessment of AI systems from sum of product costs and a profit margin) within the proce- a socioeconomic perspective, these four dimensions should dural fairness assessment [32]. Furthermore, they provide be included in the evaluation of fairness. In particular, pro- evidence that procedural and distributive fairness positively cedural and distributive fairness should be emphasized, as interact and thus, if implemented accordingly, can maximize the credit scoring assessment concentrates primarily on the the overall fairness. As described in Table 1, the presented credit-granting decision process. Table 1 provides an over- six procedural components and three distributional compo- view of fairness measurement scales in management science nents should be considered in the pricing process to achieve based on Colquitt and Rodell’s [22] and Poppo and Zhou’s strong overall fairness. From an organizational perspective, [79] work. financial institutions should ensure that their credit ratings Within the data science perspective, inequality indices are neutral and unbiased based on accurate information such as the generalized entropy (GE) index or the Gini [30]. Regarding erroneous data, customers should be able coefficient are widely accepted [25]. Both measures aim to to review and correct the data if necessary. Pricing should evaluate income inequality from an economic perspective. be consistent to avoid the impression of random pricing. However, they differ in their meaning, with the GE index Therefore, people with the same attribute characteristics providing more detailed insights by collecting information should always receive the exact credit pricing. Furthermore, on the impact of inequality across different income spec- the possible use of algorithms should not disadvantage cer- trums. The GE index is also used in an interdisciplinary tain marginalized groups. Moreover, to maximize the overall context. For example, in computer science, it is used to fairness, banks should include the distributive components measure redundancy in data, which is used to assess the in their fairness assessment. Haws and Bearden emphasize disparity within the data. In addition to the economic level, that customers assess high prices with unfairness and vice approaches to fairness measurement also exist at the corpo- versa [50]. Thus, Ferguson, Ellen, and Bearden argue that rate level. distributional fairness is given when customers receive an However, the operationalizability of the fairness types advantageous price [32]. Consequently, when pricing loans, described, especially procedural and distributive fairness, banks should always adhere to market conditions to achieve remains unclear. This paper aims to address this issue. a maximum overall fairness. A typical instrument in management practice is price 1 3 AI and Ethics are granted without a comprehensive check-up by the credit 2.4 Summary institute. During the credit application process, as a preliminary Fairness can be generally considered as the absence of unjustified unequal treatment. This broad understanding step, a bank requests customer information, such as address, income, employment status, and living situation, which it takes on different specific connotations that require consid- eration when evaluating AI systems. In our interdisciplinary feeds into its own (simple) credit scoring algorithm. As opposed to the application process for higher volume credits, overview, two main aspects of fairness were highlighted. Distributive fairness is one of these. It concerns how auto- extensive information on the overall assets and wealth of the applicant is not required. Regarding particularly small mated predictions impacting the access of individuals to products, services, benefits, and other opportunities are allo- lending, account statements might not even be necessary. In some cases, the authorization to conduct a solvency check cated. This algorithmic outcome can be analyzed through statistical tools to detect an eventual unequal distribution of through a credit check agency might be requested. If so, the credit check agency will process additional informa- certain predictions and to assess whether this is justified by the individual features of group members, or whether this is tion concerning, among other things, the credit history of the applicant and other personal information to produce a due to biases or other factors. Considering procedural fair- ness is also fundamental for the evaluation of AI systems. credit rating. Finally, based on the creditworthiness assess- ment, a bank clerk decides whether the small personal loan Therefore, it should be considered how a decision is reached for different stakeholders’ groups and how members of these is granted. In some instances, the rates might be raised in order to compensate the credit institutes for the potential groups are treated in the different stages of the product life cycle. illiquidity of individual customers. In the European framework, guidelines to improve institu- tions’ practices in relation to the use of automated models for credit-granting purposes have been produced [2, 5, 30]. 3 Evaluating fairness In the report Guidelines on loan origination and monitor- ing, the European Banking Authority (EBA) recommends 3.1 T he use case: creditworthiness assessment scoring for small personal loans that credit institutions should “understand the quality of data and inputs to the model and detect and prevent bias in the Based on the empirical evidence of perpetuation of preex- credit decision-making process, ensuring that appropriate isting discriminatory bias and of the discrimination risks for specific demographic groups, recent literature on credit scoring algorithms investigated gender-related [96] and In the report Guidelines on loan origination and monitoring, the race-related [65] fairness issues of ML systems, looking for European Banking Authority lists a set of customer information that suitable tools to detect and correct discriminatory biases are admissible and, where applicable, recommended to collect for and unfair prediction outcomes. Here we consider the case credit institutions [30]. These are: purpose of the loan, when relevant to the type of product; employment; source of repayment capac- of small personal loans. These are small volume credits to ity; composition of a household and dependents; financial commit- finance, for instance, the purchase of a vehicle or pieces of ments and expenses for their servicing; regular expenses; collateral furniture, or to cover the costs of expenses such a wedding (for secured lending); other risk mitigants, such as guarantees, when or a holiday. They typically range from 1.000 EUR to 80.000 available (85.a — 85.h). A more extensive list of possible private cus- tomer information that might be asked in different loan application EUR—in some cases they can be up to 100.000€, which scenarios is included in the Annex 2 of the same publication. The col- 7 lection of all the listed information point is not mandatory. Different banks set different limits to the maximum amount of a small personal loan. For the purpose of this paper, we considered the For instance, the Commerzbank don’t require these for credit lend- five German top banks for number of customers (https:// www. mobil ing up to 15.000 EUR (https:// www. comme rzbank. de/ kredit- finan ebank ing. de/ mag az in/ bank en- r anki ng- die- g r oes s ten- bank en- deuts zieru ng/ produ kte/ raten kredi te/ klein kredit/). chlan ds. html) and found the following ranges: Sparkasse, 1.000– 80.000 EUR (https:// www. skpk. de/ kredit/ priva tkred it. html); Volks- On its website, the German credit rating agency Schufa lists the bank, 1.000–50.000 EUR (https:// www. vr. de/ priva tkund en/ unsere- following applicant features as impact factors for credit score: number produ kte/ kredi te/ priva tkred it. html); ING, 5.000–75.000 EUR (https:// and dates of relocations; number of credit cards and opening date of www. ing. de/ kredit/ raten kredit/); Postbank 3.000–100.000 EUR the credit cards’ accounts; number and dates of online purchases on (https:// www. postb ank. de/ priva tkund en/ produ kte/ kredi te/ priva tkred it- account; payment defaults; existing loans; number and opening dates direkt. html); Deutsche Bank 1.000–80.000 EUR (https:// www. deuts of checking accounts; existing mortgage loans (https:// www. schufa. che- bank. de/ opra4x/ public/ pfb/ priva tkred it/#/ page-2-0). Using online de/ score check tools/ pt- einfl ussfa ktoren. html). platforms to compare different lenders such as check24 (https:// www. For instance, in the credit application form of the Deutsche Bank check 24. de/) or verivox (https:// www. veriv ox. de/ kredit/ klein kredit/), is stated that the interest rate depends on the credit rating of the credit we found out that no credit lenders in Germany offers more than check agency (https:// www. deuts che- bank. de/ opra4x/ public/ pfb/ priva 100.000 EUR. tkred it/#/ page-2-0). 1 3 AI and Ethics safeguards are in place to provide confidentiality, integrity it represents a violation of basic human rights, for struc- and availability of information and systems have in place” tural reasons, many individuals belonging to disadvantaged (53.e, see also 54.a and 55.a), take “measures to ensure the groups still struggle to access credit. The gender pay gap is traceability, auditability, and robustness and resilience of a concrete example: a woman working full-time in the same the inputs and outputs” (54.b, see also 53.c), and have in position as a male colleague might earn less [42], and be less place “internal policies and procedures ensuring that the creditworthy from the bank’s perspective. quality of the model output is regularly assessed, using Since ethics is supposed to influence shaping a fairer soci- measures appropriate to the model’s use, including back- ety, the definition of minimal ethical requirements for a fair testing the performance of the model” (54.c, see also 53.f ML system in a specific application field should consider and 55.b) [30]. In the white paper Big data and artificial whether and how technologies can help prevent unfairness, intelligence, the German Federal Financial Supervisory and aim at assisting disadvantaged groups. Considering our Authority (BaFin) also recommends principles for the use fairness understanding as the absence of unjustified unequal of algorithms in decision-making processes. These include: treatment of individuals or groups, the first question leading preventing bias; ruling out types of differentiation that are to a definition of minimal ethical requirements is the follow - prohibited by law; compliance with data protection require- ing: Are there individuals belonging to certain groups that ments; ensuring accurate, robust and reproducible results; are not granted loans although they share the same relevant producing documentation to ensure clarity for both internal parameters with other successful applicants belonging to and external parties; using relevant data for calibration and other groups? This should not be mistaken with the claim validation purposes; putting the human in the loop; hav- that every group should have the same share of members ing ongoing validation, overall evaluation and appropriate being granted a loan (group parity), since it is not in the adjustments [5]. interest of the person applying for a loan to be granted one if The present work follows on these recommendations and they are unable to pay it back. This would also be unethical contributes to the regulatory discussion by highlighting use because it would further compromise the financial stability case-specific operationalizable requirements that address and creditworthiness of the person, causing legal trouble and the issues emphasized by European financial institutions. moral harm. To answer this question, fairness metrics can be We focus specifically on small volume credit for two main a useful tool to detect disparities among groups. reasons. First, the pool of potential applicants is significantly larger than the one for higher volume credits such as mort- 3.2.1 Which metric(s) to choose? gage lending. While high volume credits usually require the borrower to pledge one or more assets as a collateral and The choice of one metric in particular is not value-neutral. to be able to make a down payment to cover a portion of Several factors should be considered when investigating fair- the total purchase price of an expensive good, these condi- ness metrics. Among others, there is an ethical multi-stake- tions do not apply to small personal loans, making these also holder consideration to be performed [47]. Certain metrics accessible for citizens without consistent savings or other can better accommodate the businesses’ needs and goals, assets. Since the overall personal wealth should not inu fl ence while others will better safeguard the rights of those being the decision outcome in small personal loans, this makes it ranked or scored by a software. For instance, while evaluat- a particularly interesting scenario to evaluate potential dis- ing the accuracy of a credit scoring system among protected crimination of individual belonging to disadvantaged groups groups, financial institutes will be primarily interested in that are not eligible for higher volume credits, but could optimizing (to a minimal rate) the number of loans granted be granted a small personal loan. Second, the amount of to people who will not repay the debt (false positive rate) applicant information processed for creditworthiness assess- for all groups. However, it is in the interest of solvent credit ment is signic fi antly lower than in the case of higher volume applicants to optimize to a minimal rate the number of credit credits, allowing a clearer analysis of the relevant parameters applicants to whom credit is denied although they could have and their interplay. repaid the debt (false negative rate) for all groups. Therefore, if asked to choose a metric to evaluate fairness, the former 3.2 Preliminary ethical analysis could opt for a predictive equality fairness metric, measur- ing the probability of a subject in the negative class to have Regarding credit access, structural injustice severely afflicts a positive predictive value [96]. However, someone repre- women and demographic minorities. Although in contempo- senting the latter could rather choose the equal opportunity rary liberal democracies explicitly preventing credit access based on gender, race, disability or religion is illegal because See, e.g., the EU Charter of Fundamental Right, §21, non-discrim- ination, and §23, equality between women and men. 1 3 AI and Ethics metric, the probability of a subject in a positive class to The VDE SPEC 90012 (2022) “VCIO-based description have a negative predictive value [96]. Therefore, the choice of systems for AI trustworthiness characterization” recom- of a specific metric is not value-neutral since it could bet- mends to audit working and supply chain conditions, data ter serve the interest of certain stakeholder groups. Among processing procedures, ecological sustainability, adequacy other things, the role of AI ethics and AI regulation should of the systems outcome’s explanation to inform the affected be to prevent a minority of advantaged stakeholders from persons [94]. NIST Special Publication “Towards a Standard receiving the majority of advantages at the expense of those for Identifying and Managing Bias in Artificial Intelligence” who are less advantaged. In this specific use case, this goal recommends considering “human factors, including soci- should be reached by considering the equal treatment of all etal and historic biases within individuals and organizations, applicants as the actual chance of getting a loan when the participatory approaches such as human centered design, financial requirements are met irrespectively of the appli- and human-in-the-loop practices” when addressing bias cant’s demographic group. in AI [86]. Madaio et al. designed a check-list intended to In their paper, “Why fairness cannot be automated,” San- guide the design of fair AI systems including “solicit input dra Wachter, Brent Mittelstadt, and Chris Russel, highlight on definitions and potential fairness-related harms from dif- the “conditional demographic parity” as a standard baseline ferent perspectives,” “undertake user testing with diverse statistical measurement that aligns with the European Court stakeholders,” and “establish processes for deciding whether of Justice “gold standard” for assessment of prima facie dis- unanticipated uses or applications should be prohibited” crimination. Wachter, Mittelstadt, and Russel argue that, if among the to-dos [69]. adopted as an evidential standard, conditional demographic In the credit lending scenario, certain applicants’ groups parity will help answer two key questions concerning fair- have been structurally disadvantaged in their history of ness in automated systems: access to credit and could still experience obstacles in suc- cessfully participating in the application process. Conse- 1. Across the entire affected population, which protected quently, it should be ensured that only parameters which groups could I compare to identify potential discrimina- are relevant to assess the applicant’s ability to repay the tion? loan are processed  —  e.g., bank statements or monthly 2. How do these protected groups compare to one another income — and that parameters that may lead to direct or in terms of disparity of outcomes? [98] indirect discrimination and bias perpetuation — e.g., postal code, gender, or nationality — are excluded. On this point, Here we follow their general proposal and suggest to use we follow the privacy preserving principle of “data mini- this specific metric as an evaluation tool for the specific case mization” as expressed in Art. 5.1.(c) and 25.1. GDPR. of creditworthiness assessment for small personal loans. In Assuming that there are different computing methods to Sect. 3.3., we show how this metric can be used to evaluate optimize the algorithmic outcome in order to avoid unjusti- the algorithmic outcome in our application field. fied unequal treatment of credit applicants, those methods processing less data should be preferred over those requiring 3.2.2 De‑biasing is not enough a larger dataset containing more information on additional applicant’s attributes. A fairness metric alone is insufficient to address fairness Moreover, to empower credit applicants from all groups, issues. If it becomes clear that group inequality is moti- the decision process should be made explainable so that vated by a structural reason, both the algorithmic outcome rejected applicants can understand why they were unsuc- and the parameters and steps behind the decision process, cessful. This would prevent applicants from facing black box this process requires questioning [45]. Different examples decisions that cannot be contested, therefore diminishing the of checklists addressing procedural aspects of AI systems’ bargaining power unbalance between applicants and credit design, development and application can be found in recent institutes. The decision process can be questioned through reports, white papers, and standard proposals. The Assess- “counterfactual” explanations stating how the world would ment List for Trustworthy AI (ALTAI) for self-assessment have to be different for a desirable outcome to occur [73, 97]. by the High Level Expert Group for Artificial Intelligence As remarked by Wachter et al., in certain cases, knowing of the EU includes “mechanisms to inform users about the what is “the smallest change to the world that can be made purpose, criteria and limitations of the decisions generated to obtain a desirable outcome,” is crucial for the discus- by the AI system,” “educational and awareness initiatives,” sion of counterfactuals and can help understand the logic “a mechanism that allows for the flagging of issues related to of certain decisions [97] (p. 845). In our specific case, to bias discrimination or poor performance of the AI system,” provide applicants with this knowledge, the decisive parame- and an assessment taking “the impact of the AI system on ters or parameter combination (e.g., insufficient income and/ the potential end-users and/or subjects into account” [53]. or being unemployed) that led to credit denial should be 1 3 AI and Ethics made transparent, and the counterfactual explanation should parity” metrics [96] in addition to “demographic parity” explain how these parameters should have differed in order [98], which are equivalent under certain circumstances, for the credit application to be approved. This would provide constitute basic representatives of group fairness metrics the applicant the opportunity to contest the algorithmic deci- that compare the (distribution of) outputs. These metrics sion, to provide supplementary information relevant to sup- are best applied to scenarios where there is a commonly port their application or, eventually, to successfully reapply preferred output from the perspective of the affected indi- for a smaller loan. viduals (e.g., “credit granted” in case of credit scoring or This transparency requirement also relates to the issue “applicant accepted” in case of automated processing of job concerning processing data that could result in direct or indi- or university applications), and they compare how this out- rect discrimination since credit scoring might be performed put is distributed. “Statistical parity” serves to compare the based on data belonging to credit check agencies which are proportions to which different groups, defined by a sensitive/ not made available for private citizens. protected attribute, are assigned a (preferred) output. Let us illustrate this on the example of credit scoring: Denote c = 1 3.3 T he “conditional demographic parity” metric the prediction/outcome that a credit is granted (c = 0 if the credit is not granted), and S the sensitive attribute sex with In order to conduct the statistical calculation concerning the S = m denoting a male applicant and S = f a female applicant potential existence of indirect discrimination, in the interdis- (for now, we reduce this example to the binary case both for ciplinary literature, the so-called “conditional demographic the output and the sensitive attribute). The “statistical par- parity” metric has been proposed [98] (p. 54 ff.). This met- ity” metric (with respect to the groups of female and male ric mirrors a statistical approach, which can be applied to applicants) is defined as the difference between the propor - examine potential discrimination in the context of the Euro- tion to which male applicants are granted a loan and the pean anti-discrimination laws [98]. This technique should proportion to which female applicants are granted a loan. not be confused with the second step, meaning the ques- As a formula: tion of justification of a particular disadvantage. Instead, applicants with c = 1 and S = m it only concerns the first step, which deals with the ques- applicants with S = m tion whether a particular disadvantage within the meaning (1) applicants with c = 1 and S = f of the definition of indirect discrimination is present. From applicants with S = f a computer science perspective, numerous approaches for measuring fairness have been presented which fit in the For instance, if 80% of male applicants and 60% of female fundamental conceptions of “individual fairness” or “group applicants are granted a loan, the statistical (dis-)parity is fairness.” Individual fairness relates to the idea of comparing |80%–60%|= 20%. two persons, which can be classified as similar apart from Other than contrasting the protected group (here mean- the sensitive attribute; it is infringed if these two persons ing the people falling under the sensitive feature in question are not treated correspondingly [48] (p. 1175). However, and being examined in the specific case) with the non-pro- group fairness statistically compares two groups of persons tected group (what the “statistical parity” metric does), one [48] (p. 1175). A case of direct discrimination constitutes a could also consider solely the protected group and compare breach of individual fairness; a case of indirect discrimina- group proportions along preferred and non-preferred out- tion contravenes group fairness [48] (p. 1175). puts. “Demographic parity” as described by [98] follows Group fairness metrics generally compare statistical the latter approach. This metric compares to what propor- quantities regarding defined groups in a dataset, e.g., the tion the protected group is represented among those who set of data samples with annual income over 50.000€ and received the preferred output and among those who received the group with income less or equal to 50.000€. “Statistical the non-preferred output. According to the description in [98], demographic disparity exists if a protected group is to Remark: Under the assumption that both the output and the sensi- a larger extent represented among those with non-preferred tive attribute are binary, one can show that the concepts of “statistical output than among those with preferred output. parity” and “demographic parity” are equivalent, meaning that statis- Returning to our credit scoring example, “demographic tical parity is satisfied if and only if equality in demographic parity holds (for a proof under the above assumptions, see annex 1 in [98]). parity” here measures the difference between the proportion However, each sensitive attribute can be simplified to the binary case of females within the group of persons whom a credit is by considering the protected group (e.g., “female applicants”) on the granted, and the proportion of females within the group of one hand and “all others” on the other (i.e., the male and all third persons whom a credit is not granted. As a formula, demo- genders are thrown together to form the second group). Although not comparing two specific groups anymore but the protected group with graphic disparity exists if “all others,” for this simplification, “statistical parity” and “demo- graphic parity” are equivalent though. 1 3 AI and Ethics configuration of A , c, and S, “conditional demographic par- applicants with c = 0 and S = f ity” compares the proportion to which successful applicants applicants with c = 0 (2) satisfying A are female with the proportion to which unsuc- applicants with c = 1 and S = f cessful applicants satisfying A are female. As a formula: applicants with c = 1 applicants with c = 0, S = f and A For instance, if 36% of the applicants being granted a applicants with c = 0 and A loan are female, but 51% of those not being granted a loan (3) applicants with c = 1, S = f and A are female, demographic disparity exists with a discrepancy applicants with c = 1 and A of |36%–51%|= 15%. Both the “statistical parity” and “demographic parity” Let us assume that female loan applicants have an income metrics provide a first indication of the “particular disad- under 50.000€ statistically more often than non-female vantage” within the definition of indirect discrimination applicants. Using fictitious numbers, let 90% of the female presented above. Moreover, they can be easily calculated applicants satisfy A but only 70% of the non-female appli- independent of the (potentially biased) ground-truth data. cants. “Conditional demographic parity” can now help us However, groups defined by only one sensitive attribute better understand whether this gender pay gap provides an (e.g., sex) can be large. Thus, the metrics presented might explanation why female applicants are being granted a loan be coarse and unable to capture potential disparity within less frequently, or whether there is additional discrimina- a group. For example, within the group of females, single tion not resulting from unequally distributed income. Using applicants from the countryside might have a far lower fictitious numbers again, let us assume that 60% of the approval rate than the average female applicant, while mar- applicants who are being granted a loan and satisfy A are ried applicants from the city might be granted credits almost female and 62% of the unsuccessful applicants satisfying as often as men. A are female. Thus, analyzing small income only (where Considerations such as the previous, which aim to under- female applicants represent a higher percentage), female stand given statistical or demographic disparity more deeply and non-female applicants are not treated significantly (e.g., by finding correlated attributes to explain the exist- differently. Complementary, one could examine whether ing bias), should be informed by statistical evidence. One there is unequal treatment in the high-income group. Let approach to provide more granular information on poten- B = (“annual income” > = 50.000€). Following our exam- tial biases is to include (a set of) additional attributes A, ple, 30% of the non-female applicants fall in category B but which do not necessarily need to be sensitive/protected. In only 10% of the female applicants. Let us now apply con- particular, the statistical quantities which are subject to the ditional demographic parity with respect to “high-income.” metrics presented can be calculated on subgroups which Using fictitious numbers, let 27% of the applicants who are are characterized by attributes A (additional to the sensitive being granted a loan and satisfy B be female and 25% of attribute). This enables a comparison of (more homogene- the unsuccessful applicants satisfying B be female. Again, ous) subgroups. Following this approach, an extension of the the representation of females in the high-income group is demographic parity metric has been presented : fairly equal among successful and unsuccessful applicants. “Conditional demographic parity” [98] is defined in the Overall, analyzing the subsets of applicants with small same way as “demographic parity” but restricted to a data income and high-income separately, female and non-female subset characterized by attributes A. In other words, “condi- applicants seem to be treated equally among these groups. tional demographic parity” is violated if, for a (set of) attrib- Thus, our fictitious numbers indicate that the bias in overall utes A, the protected group is to a larger extent represented acceptance rates results from female applicants being to a among those with non-preferred output and attributes A than larger extent represented in the small income group than among those with preferred output and attributes A. non-female applicants. Returning to the credit scoring example, let A = (“annual Regarding the examination under the European non-dis- income” < 50.000€) the attribute characterizing a person crimination laws, the following remarks can be made with as having an annual income lower than 50.000€. For this regard to the example above: In principle, trying to avoid the There is also an extension of the statistical parity metric called “conditional (non-)discrimination” [58], also referred to as “con- These numbers are inspired by Einkommen von Frauen und Män- ditional statistical parity” [96], which is defined in the same way as nern in Deutschland 2021 | Statista (https:// de. stati sta. com/ stati stik/ “statistical parity” but is supposed to be calculated on a subset of da ten / s t udie / 2 9039 9/ u mfr a g e / u mfr a g e - in - de uts c hla nd- zu m- e inko the data characterized by attributes A. Due to the fact that this paper mmen- von- frauen- und- maenn ern/). According to this source, in 2021 focuses on conditional demographic parity, this metric is not sup- in Germany, 91,1% of the women had a monthly net income between posed to be further discussed in the following. 0 and 2.500€ while this holds for 72,1% of the men. 1 3 AI and Ethics non-repayment of a credit constitutes a legitimate aim of the costs and total wealth and assets is not required. These banking institute [85] (p. 310). As to this, the financial capa- can therefore not be considered as conditionals. bility of the applicant in question is decisive. In this respect, (2) Ensure the relevance of the chosen indicators. Param- it is conceivable to consider the applicants’ income — mak- eters which are not directly relevant to assess the appli- ing it a suitable means to foster the legitimate aim. With cant’s ability to repay the loan shall not be processed. this in mind one can see that when considering the criterion These include attributes, such as postal code, nation- “income,” the percentage of women among the unsuccessful ality, marital status, gender, disability, age (within and the successful applicants is more or less equal within the the fixed age limits to apply for a loan), and race. two construed income-(sub-)groups (yearly salary over and Some of these, such as gender, race, and disability, are under 50.000€). Considering only the attribute (sex), the characteristics protected by national and international result might be that the percentage of unsuccessful female anti-discrimination acts such as the European anti-dis- applicants is higher than the percentage of successful female crimination law or the German Equal Treatment Act applicants (demographic disparity). Comparing the two (AGG). Others, even if not protected by anti-discrim- results, it is possible to make the assumption that income ination laws, might facilitate the deduction of one or was crucial for the decision whether a credit is granted or more protected attributes. not. If one considers the orientation toward the income as (3) Provide transparency for credit applicants and other an indicator for the financial capability to be necessary and actors involved. The following shall be made transpar- appropriate, the particular disadvantage might be justified. ent for the applicants: While generally achieving “conditional demographic parity” seems unlikely given the variety of choice for A, o Which data are processed (no personal data is pro- calculating the “conditional demographic parity” metric cessed without informed consent of the applicant). for different configurations of A can still provide valuable p Why an application is eventually rejected and what evidence to assist in detecting the relevant “particular disad- applicant features should be improved to obtain the vantage” (pertaining to the first step in determining an indi- loan. Therefore, the algorithmic decision must be rect discrimination). Especially, being more fine-granular counterfactually explainable, e.g., if the applicant than the non-conditional metrics which measure bias only had a higher income or if she/he was not unem- in the model’s overall results, their extensions can be used to ployed, she/he would have received the loan. explain bias by analyzing additional attributes which might provide further relevant information. This does not mean disclosing the entire computing process—which might be protected by trade secrets— 3.4 Ethical minimal requirements but guaranteeing transparency regarding the criteria applicants need to fulfill. We claim that the following minimal requirements must be standardized to address discrimination and procedural fair- ness issues concerning the application of ML systems in 4 Standardizing minimal ethical credit scoring for small personal loans: requirements to evaluate fairness (1) Regular check of the algorithmic outcome through a Standardization can connect different perspectives of “fair - fairness metric. We follow Wachter, Mittelstadt, and ness” and can establish a universal understanding in the con- Russel in suggesting that the conditional demographic text of AI. It can erase trade barriers and support interoper- parity fairness metric should be used to detect unfair ability as well as foster the trust in a system or application. outcome [98]. For our simple case study, the condition- Within the realm of standardization, existing definitions for als will be income and employment status. If the bank fairness are rather generic and currently not tailored for AI requires an external credit rating, then other parameters systems and applications; however, this may change soon that influence the rating such as past loan defaults or with the development of new AI-dedicated standards. number of credit cards must also be considered. How- Several documents are pushing in that direction: ever, in our case, comprehensive information on living ISO/IEC TR 24028:2020, Information technol- For instance, considering the “German Credit Dataset” (https:// ogy — Artificial intelligence — Overview of trustworthi- archi ve. ics. uci. edu/ ml/ datas ets/ Statl og+% 28Ger man+ Credit+ Data% ness in artificial intelligence as it also lists fairness as an 29) used by Verma and Rubin (2018) for their fairness metrics’ analy- essential part for ensuring trustworthiness in AI [55]. sis, it is possible to flag the following parameters according to this requirement: personal status and sex (attribute 9); age in years (attrib- ISO/IEC TR 24027:2021, Information technol- ute 13); foreign worker (attribute 20). ogy — Artificial intelligence (AI) — Bias in AI systems 1 3 AI and Ethics and AI aided decision-making addresses bias in relation a fair consensus process. The outcome of this process is a to AI systems [54]. recognized standard, enabling mutual understanding based ISO/IEC TR 24368:2022, Information technol- on agreed requirements, thus fostering trade and the new ogy — Artificial intelligence — Overview of ethical and development of quality AI products and services—either societal concerns aims to provide an overview of AI ethi- nationally, in Europe, or internationally depending on the cal and societal concerns, as well as International Stand- used standardization platform. ards that address issues arising from those concerns [56]. A standard can be used for a quality assessment in order to promote a product’s or service’s quality, trustworthiness, In addition, there are many other AI-specific projects pub- and user acceptability. In the assessment process of an AI lished or under development within the ISO and the IEC system or application, the related standardized fairness met- on the topics of ML, AI system life cycle processes, func- ric can be used to attest the system’s or application’s ability tional safety, quality evaluation guidelines, explainability, to execute fair decisions. Consequently, a fairness-related data life cycle frameworks, concepts and terminology, risk attestation based on corresponding standards (e.g., certifi- management, bias in AI systems, AI aided decision-making, cation) can increase the user acceptability and trustworthi- robustness assessment of neural networks, an overview of ness of the AI system or application, which can result in ethical and societal concerns, process management frame- increased sales figures. work for big data analytics, and other topics. The focus of these projects is to develop a framework of requirements for the development and operation of safe, robust, reliable, 5 Conclusion explainable, and trustworthy AI systems and applications. Following the establishment of the general AI requirement Evaluating the fairness of an AI system requires analyzing framework, the focus may likely shift to more use case- an algorithmic outcome and observing the consequences of specific standardization topics like “fairness,” which is the development and application of the system on individu- clearly needed in the standardization of AI, but cannot be als and society. Regarding the applied case of creditworthi- generalized. ness assessment for small personal loans, we highlighted Based on our interdisciplinary analysis, the standardiza- specific distributive and procedural fairness issues inherent tion of “fairness” in the context of AI with the aim to allow either to the computing process or to the system’s use in a an assessment requires multiple relevant measurable and real-world scenario: (1) the unjustified unequal distribution quantifiable parameters and/or attributes building state of of predictive outcome; (2) the perpetuation of existing bias the art use case-specific fairness metrics such as the above and discrimination practices; (3) the lack of transparency discussed conditional parity metric. Such fairness metrics concerning the processed data and of an explanation of the can be developed and standardized with an independent algorithmic outcome for credit applicants. We addressed consensus driven platform open to expertise from all use these issues proposing ethical minimal requirements for this case-related stakeholders, including views from the perspec- specific application field: (1) regularly checking algorithmic tives of philosophy, industry, research, and legislation. This outcome through the conditional demographic parity metric; platform can be either a national standards body, ISO, IEC, (2) excluding from the group of processed parameters those or the European Standardization Organization (CEN); where that could lead to discriminatory outcome; (3) guaranteeing the most appropriate option for this topic is the international transparency about the processed data, in addition to coun- joint committee between ISO and IEC, the ISO/IEC JTC 1/ terfactual explainability of algorithmic decisions. Defin - SC 42 “Artificial intelligence”. To begin a standardization ing these minimal ethical requirements represents a start- process for a use case-specific fairness metric, scope, out- ing point toward standards specifically addressing fairness line, and justification of the proposed standardization project issues in AI systems for creditworthiness assessments. These must be proposed to the respective standardization commit- requirements aim to prevent unfair algorithmic outcomes, as tee. To elevate the chances of approval, a first draft with the well as unfair practices related to the use of these systems. proposed fairness metric should also be included. The stand- ardization process within national standards bodies, ISO, IEC, and CEN provides all participating members an equal Funding Open Access funding enabled and organized by Projekt right to vote, comment and work on a standardization pro- DEAL. Our research is funded by the Ministerium für Wirtschaft, Industrie, Klimaschutz und Energie des Landes NRW (MWIDE NRW) ject. When working internationally or in the European field, in the framework of the project “Zertifizierte KI” (“Certified AI”); this means that all interested registered experts can work funding number 005–2011-0050. on the project; however, during mandatory voting (project proposal, drafts and finalization) each participating country (represented by delegated experts) has one vote to facilitate 1 3 AI and Ethics 14. Buolamwini, J., Gebru, T.: Gender shades: intersectional accu- Declarations racy disparities in commercial gender classic fi ation. Proceedings of the 1st Conference on Fairness, Accountability and Transpar- Conflict of interest The authors have no relevant financial or non-fi- ency, PMLR 81, 77–91 (2018) nancial interests to disclose. On behalf of all authors, the correspond- 15. Chakraborty, J., Majumder, S., Menzies, T.: Bias in machine ing author states that there is no conflict of interest. learning software: why? how? what to do? In: Proceedings of the 29th ACM joint meeting on European software engineer- Open Access This article is licensed under a Creative Commons Attri- ing Conference and Symposium on the. foundations of software bution 4.0 International License, which permits use, sharing, adapta- engineering, 429–440 (2021) tion, distribution and reproduction in any medium or format, as long 16. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: as you give appropriate credit to the original author(s) and the source, SMOTE: synthetic minority over–sampling technique. J. Artif. provide a link to the Creative Commons licence, and indicate if changes Intell. Res. 16, 321–357 (2002) were made. The images or other third party material in this article are 17. Çınar, Z.M., Abdussalam Nuhu, A., Zeeshan, Q., Korhan, O., included in the article's Creative Commons licence, unless indicated Asmael, M., Safaei, B.: Machine learning in predictive mainte- otherwise in a credit line to the material. If material is not included in nance toward sustainable smart manufacturing in industry 4.0. the article's Creative Commons licence and your intended use is not Sustainability. 12, 8211 (2020) permitted by statutory regulation or exceeds the permitted use, you will 18. Coeckelbergh, M.: AI for climate: freedom, justice, and other eth- need to obtain permission directly from the copyright holder. To view a ical and political challenges. AI Ethics 1, 67–72 (2021). https:// copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . doi. org/ 10. 1007/ s43681- 020- 00007-2 19. Cohen, G.A.: On the currency of egalitarian justice. Ethics 99, 906–944 (1989) 20. Colquitt, J.A.: On the dimensionality of organizational justice: a References construct validation of a measure. J. Appl. Psychol. 86, 386–400 (2001) 21. Colquitt, J.A., Rodell, J.B.: Justice, trust, and trustworthiness: a 1. Adams, J.S.: Inequity in social exchange. Adv. Exp. Soc. Psychol. longitudinal analysis integrating three theoretical perspectives. 2, 267–299 (1965) Acad. Manag. J. 54, 1183–1206 (2011) 2. Aggarwal, N.: Machine learning, big data and the regulation of 22. Colquitt, J.A., Rodell, J.B.: Measuring justice and fairness. In: consumer credit markets: the case of algorithmic credit scoring. Cropanzano, R.S., Ambrose, M.L. (eds.) The Oxford Handbook In: Aggarwal, N., Eidenmüller, H., Enriques, L., Payne, J., van of Justice in the Workplace, pp. 187–202. OUP, Oxford (2015) Zwieten, K. (eds.) Autonomous Systems and the Law, pp. 37–44. 23. Cremers, A.B. et al.. Trustworthy Use of Artificial Intelligence. Beck, München (2018) Priorities from a philosophical, ethical, legal, and technological 3. Ahn, Y., Lin, Y.R.: Fairsight: visual analytics for fairness in deci- viewpoint as a basis for certification of Artificial Intelligence, sion making. IEEE Trans. Vis. Comput. Graph. 26, 1086–1095 https:// www. ki. nrw/ wp- conte nt/ uploa ds/ 2020/ 03/ White paper_ (2020) Thrus twort hy_ AI. pdf (2019). Accessed 28 January 2022 4. Ariño, A., Ring, P.S.: The role of fairness in alliance formation. 24. Datenetikkommission der Bundesregierung. Gutachten der Strat. Manag. J. 31, 1054–1087 (2010) Datenethikkommission der Bundesregierung, https:// www. bmi. 5. BaFin, Big data and artificial intelligence: Principles for the use bund. de/ Share dDocs/ downl oads/ DE/ publi katio nen/ themen/ it- of algorithms in decision-making processes, https:// www. bafin. digit alpol itik/ gut ac hten- daten e t hik kommi ssion. pdf?__ blob= de/ Shar e dDocs/ Downl oads/ EN/ A uf si c htsr ec ht/ dl_ Pr inz ipien publi catio nFile &v=6 (2019). Accessed 28 January 2022 papier_ BDAI_ en. html (2021). Accessed 3 March 2023. 25. De Maio, F.G.: Income inequality measures. J. Epidemiol. Com- 6. Balayn, A., Gürses, S., Beyond Debiasing: Regulating AI and its munity Health. 61, 849–852 (2007) Inequalities. Accessed 16 December 2022 26. Druzhkov, P.N., Kustikova, V.D.: A survey of deep learning 7. Barocas, S., Hardt, M., Narayanan, A.: Fairness in machine learn- methods and software tools for image classification and object ing. Nips Tutorial. 1/2 (2017) detection. Pattern Recognit. Image Anal. 26, 9–15 (2016) 8. Barocas, S., Hardt, M., Narayanan, A. Fairness and machine 27. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fair- learning. https:// fairm lbook. org/ (2019). Accessed 30 August ness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, 214–226 (2012) 9. Bellamy, R.K.E., Dey, K., Hind, M., Hoffman, S.C., Houde, S., 28. Dworkin, R.: What is equality? Part 1: Equality of welfare. Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Philos. Publ. Aff. 2, 185–246 (1981) Nagar, S., Ramamurthy, K.N., Richards, J., Saha, D., Sattigeri, 29. Eidelson, B.: Discrimination and Disrespect. OUP, Oxford P., Singh, M., Varshney, K.R., Zhang, Y.: AI Fairness 360: an (2015) extensible toolkit for detecting and mitigating algorithmic bias. 30. European Banking Authority, Final Report – Guidelines on Loan IBM J. Res. Dev. 63, 1–15 (2019) Origination and Monitoring, https:// www. eba. europa. eu/ sites/ 10. Bies, R.J., Moag, J.F.: Interactional Justice: Communication Cri- defau lt/ docum ents/ files/ docum ent_ libra ry/ Publi catio ns/ Guide teria of Fairness Research on Negotiation in Organizations, 1st lines/ 2020/ Guide lines% 20on% 20loan% 20ori ginat ion% 20and% edn., pp. 43–55. JAI Press, Greenwich (1986) 20mon itori ng/ 884283/ EBA% 20GL% 202020% 2006% 20Fin al% 11. Binns, R.: Fairness in machine learning: lessons from political 20Repor t%20on% 20GL% 20on% 20loan% 20or iginat ion% 20and% philosophy. Proc. Mach. Learn. Res. 81, 1–11 (2018) 20mon itori ng. pdf (2020). Accessed 02 March 2023 12. Breck, E., Polyzotis, N., Roy, S., Whang, S., Zinkevich, M.: Data 31. European Commission. Proposal for A regulation of the Euro- validation for machine learning. Proceedings of the 2nd SysML pean Parliament and of the council laying down harmonized conference, Palo Alto, CA, USA (2019) rules on artificial intelligence (artificial intelligence act) and 13. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dha- amending certain union legislative acts, https:// eur- lex. europa. riwal, P., et al.: Language models are few–shot learners. Adv. eu/leg al-conte nt/ EN/ TXT/? ur i=cele x%3A520 21PC0 206 (2021). Neural Inf. Process. Syst. 33, 1877–1901 (2020) Accessed 18 March 2022 1 3 AI and Ethics 32. Ferguson, J.L., Ellen, P.S., Bearden, W.O.: Procedural and dis- Academic, Oxford (2021). https:// doi. or g/ 10. 1093/ oxfor dhb/ tributive fairness: Determinants of overall price fairness. J. Bus. 97801 98857 815. 013. 21 Ethics. 121, 217–231 (2014) 52. HLEG. HLEG on AI. Ethics guidelines for trustworthy AI. 33. Fisher, E., Fuchs, C.: Reconsidering Value and Labor in the Digi-https://digit al- s trategy .ec. eur opa.eu/ en/ libr ar y/e thics-guide lines- tal Age. Palgrave Macmillan, Basingstoke (2015)trust worthy- ai (2019). Accessed 10 May 2022 34. Folger, R., Konovsky, M.A.: Effects of procedural and distribu- 53. HLEG. Assessment List for Trustworthy AI (Altai) for self– tive justice on reactions to pay raise decisions. Acad. Manag. J. assessment. https:// digit al- s tr at egy . ec. eur opa. eu/ en/ libr a r y/ 32, 115–130 (1989)assessment- lis t-tr ustw orthy-ar ti ficial-intel lig ence- alt ai-self- asses 35. Friedler, S.A., Scheidegger, C., Venkatasubramanian, S., sment (2020). Accessed 01 December 2022 Choudhary, S., Hamilton, E.P., Roth, D.: A comparative study 54. ISO/IEC  TR  24027:2021, Information technology — Artifi- of fairness–enhancing interventions in machine learning. In: cial intelligence (AI) — Bias in AI systems and AI aided deci- Proceedings of the conference on fairness, accountability, and sion making. https:// www. iso. org/ stand ard/ 77607. html (2021). transparency, 329–338 (2019) Accessed 22 December 2022 36. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wal- 55. ISO/IEC TR. 24028:2020, Information technology — Artificial lach, H., Iii, I.: Datasheets for datasets. Commun. ACM. 64, intelligence — Overview of trustworthiness. https:// www . iso. or g/ 86–92 (2021)stand ard/ 77608. html (2020). Accessed 22 December 2022 37. Genovesi, S., Mönig, J.M.: Acknowledging sustainability in the 56. ISO/IEC TR 24368:2022, Information technology — Artificial framework of ethical certification for AI. Sustainability. 14, 4157 intelligence — Overview of ethical and societal concerns. https:// (2022)www.iso. or g/s tandar d/78507. html (2022). Accessed 22 Decem- 38. Gilliland, S.W.: The perceived fairness of selection systems – ber 2022 an organizational justice perspective. Acad. Manag. Rev. 18, 57. Kamiran, F., Calders, T.: Data pre–processing techniques for 694–734 (1993) classification without discrimination. Knowl. Inf. Syst. 33, 1–33 39. Giovanola, B., Tiribelli, S.: Beyond bias and discrimination: (2012) redefining the AI ethics principle of fairness in healthcare 58. Kamiran, F., Žliobaitė, I., Calders, T.: Quantifying explainable machine learning algorithms. AI & Soc, 1–15 (2022) discrimination and removing illegal discrimination in automated 40. Gomez, O., Holter, S., Yuan, J., Bertini, E.: ViCE: visual coun- decision making. Knowl. Inf. Syst. 35, 613–644 (2013) terfactual explanations for machine learning models. In: Pro- 59. Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness–aware ceedings of the 25th international conference on intelligent user classie fi r with prejudice remover regularizer. In: Flach, P.A., Bie, interfaces, 531–535 (2020) T., Cristianini, N. (eds.) Joint European Conference on Machine 41. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Learning and Knowledge Discovery in Databases, pp. 35–50. press, Cambridge (2016) Springer, Berlin (2012) 42. Gould, E., Schieder, J., Geier, K., What is the gender pay gap 60. Kasirzadeh, A. Algorithmic fairness and structural injustice. and is it real? Economic Policy Institute. https://files. epi. or g/pdf/ Insights from Feminist Political Philosophy. Proceedings of 112962. pdf (2016). Accessed 05 September 2022 the 2022 AAAI/ACM Conference on AI, Ethics, and Society 8, 43. Greenberg, J., Cropanzano, R.: The social side of fairness: inter- 349–356 (2022) personal and informational classes of organizational justice. In: 61. Kilbertus, N., Rojas, C.M., Parascandolo, G., Hardt, M., Janz- Cropanzano, R. (ed.) Justice in the Workplace: Approaching ing, D., Schölkopf, B.: Avoiding discrimination through causal Fairness in Human Resource Management, pp. 79–103. Law- reasoning. Adv. Neural Inf. Process. Syst. 30, 656–666 (2017) rence Erlbaum Associates, Hillsdale (1993) 62. Konovsky, M.A.: Understanding procedural justice and its impact 44. Grgić–Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: The on business organizations. J. Manag. 26, 489–511 (2000) case for process fairness in learning: feature selection for fair 63. Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual decision making. NIPS symposium on machine learning and the fairness. Adv. Neural Inf. Process. Syst. 30, 2 (2017) law (2016) 64. Lahoti, P., Gummadi, K.P., Weikum, G.: ifair: learning individu- 45. Grgić-Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: ally fair data representations for algorithmic decision making. Beyond distributive fairness in algorithmic decision making: fea- 35th international conference on data engineering (icde), IEEE ture selection for procedurally fair learning. AAAI. Proceedings Publications, 1334–1345 (2019) of the AAAI conference on artificial intelligence 32. 32 (2018) 65. Lee, M.S.A., Floridi, L.: Algorithmic fairness in mortgage lend- 46. Gudivada, V., Apon, A., Ding, J.: Data quality considerations for ing: from absolute conditions to relational trade–offs. Minds big data and machine learning: Going beyond data cleaning and Mach. 31, 165–191 (2021) transformations. Int. J. Adv. Softw. 10, 1–20 (2017) 66. Leventhal, G.S.: The distribution of rewards and resources in 47. Häberlein, L., Mönig, J.M., Hövel, P.: Mapping stakeholders and groups and organizations. Adv. Exp. Soc. Psychol. 9, 91–131 scoping involvement. A guide for HEFRCs, Deliverable 3.1 of (1976) the H2020–project ETHNA System. https://e thnasy stem.eu/ wp- 67. Leventhal, G.S.: What should be done with equity theory? content/ uploa ds/ 2021/ 10/ ETHN A_2021_ d3.1- s takeholde r mapp In: Gergen, K.J., Greenberg, M.S., Willis, R.H. (eds.) Social ing_ 21100 11. pdf (2021) Accessed 10 May 2021 exchange, pp 27–56, Springer, Boston (1980) 48. Hacker, P.: Teaching fairness to artificial intelligence: existing 68. Lippert-Rasmussen, K.: Born Free and Equal? A Philosophi- and novel strategies against algorithmic discrimination under EU cal Inquiry into the Nature of Discrimination. OUP, New York law (April 18, 2018). Common Mark. Law Rev. 55, 1143–1186 (2013) (2018) 69. Madaio, M.A., Stark, L., Wortman Vaughan, J.W., Wallach, H.: 49. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in super- Co– designing checklists to understand organizational challenges vised learning. Adv. Neural Inf. Process. Syst. 29, 2 (2016) and opportunities around fairness in AI. In: Proceedings of the 50. Haws, K.L., Bearden, W.O.: Dynamic pricing and consumer fair- 2020 CHI conference on human factors in computing systems, ness perceptions. J. Consum. Res. 33, 304–311 (2006) 1–14 (2020) 51. Herzog, L.: Algorithmic bias and access to opportunities. In: 70. Mathur, P., Sarin Jain, S.S.: Not all that glitters is golden: the Véliz, C. (ed.) The Oxford Handbook of Digital Ethics. Oxford impact of procedural fairness perceptions on firm evaluations 1 3 AI and Ethics and customer satisfaction with favorable outcomes. J. Bus. Res. quantifying algorithmic unfairness: measuring individual 117, 357–367 (2020) &group unfairness via inequality indices. In: Proceedings of the 71. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, 24th ACM SIGKDD international conference on knowledge dis- A.: A survey on bias and fairness in machine learning. ACM covery & data mining, 2239–2248 (2018) Comput. Surv. 54, 1–35 (2021) 90. Suresh, H., Guttag, J.: A framework for understanding sources of 72. Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: harm throughout the machine learning life cycle. Equity Access The ethics of algorithms: mapping the debate. Big Data Soc. Algor. Mech. Optim. 17, 1–9 (2021) (2016). https:// doi. org/ 10. 1177/ 20539 51716 679679 91. Taylor, L.: What is data justice? The case for connecting digital 73. Molnar, C. Interpretable machine learning. A guide for making rights and freedoms globally. Big Data Soc. (2017). https:// doi. black box models explainable, https://c hrist ophm.git hub.io/ inter org/ 10. 1177/ 20539 51717 736335 preta ble- ml- book/ (2022). Accessed 22 December 2022 92. Thibaut, J.W., Walker, L.: Procedural Justice: A Psychological 74. Mönig, J.M.: Privatheit als Luxusgut in der Demokratie? In: Analysis. Erlbaum Associates, Hillsdale (1975) Grimm, P., Zöllner, O. (eds.) Demokratie und Digitalisierung, 93. van Wynsberghe, A.: Sustainable AI: AI for sustainability and pp. 105–114. Steiner, Stuttgart (2020) the sustainability of AI. AI Ethics. 1, 213–218 (2021) 75. Myers West, S.: Redistribution and recognition. A feminist cri- 94. VDE SPEC 90012:2022 VCIO based description of systems for tique of algorithmic fairness. Catalyst. (2020). https://doi. or g/10. AI trustworthiness characterization. https://www .vde. com/ r esou 28968/ cftt. v6i2. 33043rce/ blob/ 21766 86/ a24b1 3db01 77374 7e6b7 bba4c e20ea 60/ vde- 76. Noble, S.U.: Algorithms of oppression: how search engines rein-spec- vcio- based- descr iption- of- syste ms- for- ai- trust worth iness- force racism. New York University Press, New York (2018)chara cteri sation- data. pdf (2022). Accessed 01 December 2022 77. Pillai, R., Schriesheim, C.A., Williams, E.S.: Fairness percep- 95. Véliz, C.: Privacy is Power: Why and How You Should Take tions and trust as mediators for transformational and transactional Back Control of Your Data. Bantam Press, London (2020) leadership: A two–sample study. J. Manag. 25, 897–933 (1999) 96. Verma, S., Rubin, J.: Fairness definitions explained. Proceedings 78. Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., Weinberger, of the international workshop on software fairness, 1–7 (2018) K.Q.: On fairness and calibration. Adv. Neural Inf. Process. Syst. 97. Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explana- 30, 2 (2017) tions without opening the black box: Automated decisions and 79. Poppo, L., Zhou, K.Z.: Managing contracts for fairness in buyer– the GDPR. SSRN J. 31, 841–888 (2018) supplier exchanges. Strat. Manag. J. 35, 1508–1527 (2014) 98. Wachter, S., Mittelstadt, B., Russell, C.: Why fairness cannot 80. Poretschkin, M., Schmitz, A., Akila, M., Adilova, L., Becker, D., be automated: bridging the gap between EU non–discrimination Cremers, A.B., Hecker, D., Houben, S., Mock, M., Rosenzweig, law and AI. Comput. Law Secur. Rev. (2021). https://doi. or g/10. J. et al. Leitfaden zur Gestaltung vertrauenswürdiger Künstlicher 1016/j. clsr. 2021. 105567 Intelligenz. https:// www. iais. fraun hofer. de/ conte nt/ dam/ iais/ fb/ 99. Wahlster, W., Winterhalter, C.: Deutsche Normungsroadmap. Kuens tliche_ intel ligenz/ ki- pruef katal og/ 202107_ KI- Pruef katal Künstliche Intelligenz. DIN. https:// www. dke. de/ resou rce/ blob/ og. pdf (2021). Accessed 10 March 202120194 82/ 0c291 25fa9 9ac4c 897e2 809c8 ab343 ff/ nr- ki- deuts ch--- 81. Rajkomar, A., Hardt, M., Howell, M.D., Corrado, G., Chin, downl oad- data. pdf (2020). Accessed 21 December 2022 M.H.: Ensuring fairness in machine learning to advance health 100. Wand, Y., Wang, R.Y.: Anchoring data quality dimensions in equity. Ann. Intern. Med. 169, 866–872 (2018) ontological foundations. Commun. ACM. 39, 86–95 (1996) 82. Rawls, J.: A Theory of Justice. Harvard University Press, Cam- 101. Young, I.M.: Justice and the Politics of Difference. Princeton bridge MA (1999) University Press, Princeton (1990) 83. Rawls, J.: Justice as Fairness: A Restatement. Harvard University 102. Yu, K.H., Beam, A.L., Kohane, I.S.: Artificial intelligence in Press, Cambridge MA (2001) healthcare. Nat. Biomed. Eng. 2, 719–731 (2018) 84. Rohde, F. et al.: Nachhaltigkeitskriterien für künstliche Intelli- 103. Zafar, M.B., Valera, I., Gomez Rodriguez, M., Gummadi, K.P.: genz. Entwicklung eines Kriterien– und Indikatorensets für die Fairness beyond disparate treatment & disparate impact: learning Nachhaltigkeitsbewertung von KI–Systemen entlang des Leb- classification without disparate mistreatment. Proceedings of the enszyklus. Schriftenr. IÖW. 220/21. https:// www. ioew. de/ filea 26th international conference on world wide web, 1171–1180 dmin/ user_ upload/ BILDER_ und_ Downl oadda teien/ Publi katio (2017) nen/2021/ IOEW_ SR_ 220_ N achhaltig k eitskr iter ien_fuer_ K uens 104. Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning tliche_ Intel ligenz. pdf (2021). Accessed 18 January 2022 fair representations. International Conference on Machine Learn- 85. Schürnbrand, J.: Organschaft im Recht der privaten Verbändecht. ing, 325–333 (2013) Mohr Siebeck, Tübingen (2007) 105. Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted 86. Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., Hall, biases with adversarial learning. Proceedings of the 2018 AAAI/ P. Toward a standard for identifying and managing bias in artifi- ACM Conference on AI, Ethics, and Society, 335–340 (2018) cial intelligence, National Institute of Standards and Technology. https:// nvlpu bs. nist. gov/ nistp ubs/ Speci alPub licat ions/ NIST. SP. Publisher's Note Springer Nature remains neutral with regard to 1270. pdf (2022). Accessed 21 December 2022 jurisdictional claims in published maps and institutional affiliations. 87. Sen, A.: Inequality Reexamined. Clarendon Press, Oxford (1992) 88. Skarlicki, D.P., Folger, R., Tesluk, P.: Personality as a modera- tor in the relationship between fairness and retaliation. Acad. Manag. J. 42, 100–108 (1999) 89. Speicher, T., Heidari, H., Grgić–Hlača, N., Gummadi, K.P., Singla, A., Weller, A., Zafar, M.B.: A unified approach to 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png AI and Ethics Springer Journals

Standardizing fairness-evaluation procedures: interdisciplinary insights on machine learning algorithms in creditworthiness assessments for small personal loans

Download PDF
17 pages

Loading next page...
 
/lp/springer-journals/standardizing-fairness-evaluation-procedures-interdisciplinary-w09lVrAZpo

References (77)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2023
ISSN
2730-5953
eISSN
2730-5961
DOI
10.1007/s43681-023-00291-8
Publisher site
See Article on Publisher Site

Abstract

In the current European debate on the regulation of Artificial Intelligence there is a consensus that Artificial Intelligence (AI) systems should be fair. However, the multitude of existing indicators allowing an AI system to be labeled as “(un)fair” and the lack of standardized, application field specific criteria to choose among the various fairness-evaluation methods makes it difficult for potential auditors to arrive at a final, consistent judgment. Focusing on a concrete use case in the application field of finance, the main goal of this paper is to define standardizable minimal ethical requirements for AI fairness-evaluation. For the applied case of creditworthiness assessment for small personal loans, we highlighted specific distributive and procedural fairness issues inherent either to the computing process or to the system’s use in a real-world scenario: (1) the unjustified unequal distribution of predictive outcome; (2) the perpetuation of existing bias and discrimination practices; (3) the lack of transparency concerning the processed data and of an explanation of the algorithmic outcome for credit applicants. We addressed these issues proposing minimal ethical requirements for this specific application field: (1) regularly checking algorithmic outcome through the conditional demographic parity metric; (2) excluding from the group of processed param- eters those that could lead to discriminatory outcome; (3) guaranteeing transparency about the processed data, in addition to counterfactual explainability of algorithmic decisions. Defining these minimal ethical requirements represents the main contribution of this paper and a starting point toward standards specifically addressing fairness issues in AI systems for creditworthiness assessments aiming at preventing unfair algorithmic outcomes, in addition to unfair practices related to the use of these systems. As a final result, we indicate the next steps that can be taken to begin the standardization of the three use case-specific fairness requirements we propose. Keywords Artificial intelligence · Data science · Fairness · Fairness metric · Standardization * Sergio Genovesi Romina Kleiner genovesi@uni-bonn.de kleiner@time.rwth-aachen.de Julia Maria Mönig Lena Krieger moenig@uni-bonn.de Lena.Krieger@din.de Anna Schmitz Alexander Zimmermann anna.schmitz@iais.fraunhofer.de Alexander.Zimmermann@din.de Maximilian Poretschkin University of Bonn, Bonn, Germany maximilian.poretschkin@iais.fraunhofer.de Fraunhofer IAIS, Sankt Augustin, Germany Maram Akila maram.akila@iais.fraunhofer.de RWTH, University of Aachen, Aachen, Germany Manoj Kahdan DIN, Berlin, Germany kahdan@time.rwth-aachen.de Vol.:(0123456789) 1 3 AI and Ethics 1 Introduction2 Defining fairness In the current European debate on the regulation of Artificial 2.1 AI ethics Intelligence there is a consensus that Artificial Intelligence (AI) systems should be developed in a human centered way In the current AI ethics discussions, fairness is generally and should be “trustworthy” [23, 24, 31, 99]. According to framed in accounts of distributive justice and is broadly these documents, one value that constitutes “trustworthi- referred to as unbiased distribution of access to services ness” is fairness. Many current publications on AI fairness and goods—e.g., access to treatments in healthcare [39, 81] predominantly focus on avoiding or fixing algorithmic dis- or access to credit [65]—and as absence of discrimination, crimination of groups or individuals and on data-de-bias- understood as unjustified unequal treatment of groups or ing, offering different metrics as tools to evaluate whether individuals [72, 76]. groups or individuals are treated differently [8 , 71, 96]. Concerning distributive justice and non-discrimination Moreover, the International Standardization Organization as equal treatment, one of the primary contemporary phil- (ISO)/International Electrotechnical Commission (IEC) osophical references is the Rawlsian idea of equality of TR 24028:2020, Information technology—Artificial Intelli- opportunities. This idea requires that citizens having the gence—Overview of trustworthiness in Artificial Intelligence same talents and being equally motivated should receive lists fairness as an essential part for ensuring trustworthiness the same educational and economic opportunities regard- in AI (ISO/IEC 2020). However, the multitude of existing less of their wealth or social status [83] (p. 44). Since in the indicators allowing the labeling of an AI system as “(un) social praxis basic rights and liberties are neither accessible fair” and the lack of standardized, application field specific nor enjoyable in the same way for different citizens, soci- criteria to choose among the various fairness-evaluation ety should take adequate measures in order for all citizens methods makes it difficult for potential auditors to arrive to enjoy their rights and liberties. Rawls develops on this at a final, consistent judgment [24, 96, 98]. The increasing intuition stating that “the worth of liberty to persons and need for standardized methods to assess the potential risks of groups depends upon their capacity to advance their ends AI systems is also highlighted by the draft for an “Artificial within the framework the system defines. […] Some have Intelligence Act” suggested by the European Commission in greater authority and wealth, and therefore greater means to April 2021, which, in accordance with the so-called “New achieve their aims” [82] (p. 179). Consequently, to avoid the Legislative Framework,” ascribes a major role to “Stand- exaggeration of the unequal enjoyment of basic rights and ards, conformity assessment, certificates [and] registration” liberties, a fair society must enact compensation mechanisms (Chapter 5). to maximize their worth to the least advantaged [82] (ibid). Focusing on a concrete use case in the application field of It is essential to avoid the development of vicious circles of finance, the main goal of this paper is to define standardiz- (un)privilege-polarization in society due to the moral harm able minimal ethical requirements for AI fairness evaluation. they produce. Therefore, removing the opportunities of the In Sect. 2, we explore different understandings of fairness less privileged to truly benefit from their rights and liberties from three perspectives and address the different vantage results in a harmful form of negative discrimination that points of many stakeholders involved in the development, amplifies economic inequalities and undermines the chances commercialization, and use of AI systems. In Sect.  3, we for the less advantaged to live an autonomous life and set discuss the example of a risk scoring machine learning self-determined goals. This is a form of disrespect toward (ML) model for small personal loans. As a main contribu- their personhood [29, 68] and can amplify social resentment. tion of the paper, we suggest ethical minimal requirements The philosophical debate about Rawls’ theory and other that should be complied with when evaluating fairness and forms of “egalitarianism” could help clarify current emerg- highlight a preferred fairness metric for fairness-evaluation ing issues concerning algorithmic fairness [11]. Egalitari- purposes in this specific application field. In Sect.  4, we anism in this sense means that “human beings are in some investigate how to translate our research findings into stand- fundamental sense equal and that efforts should be made to ardization criteria to be used when assessing ML credit scor- avoid and correct certain forms of inequality” [11] (p. 2). ing systems for small personal loans. Many approaches try to determine the kind of equality that Noble highlights intersectional questions of fairness underlining the adverse effects that “algorithms of oppression” have on black women. In general, feminist scholars have stressed that unfairness and injus- tices usually go hand in hand with domination and oppression [75, 75]. 1 3 AI and Ethics should be sought and which inequalities should be avoided the UN’s Sustainable Development Goals (SDG)—also to in civil society to uphold the fundamental equality of human the future and not only limiting it to present generations. In beings: among others, equality of preference-satisfaction the practice, fairness toward future generations means act- [19], equality of welfare, income, and assets [28], and the ing sustainably. equality of capabilities to achieve their goals [87]. However, These considerations lead us to the following prelimi- regarding the application of these views on algorithmic deci- nary understanding of fairness in the context of an AI ethics sions, defending an equal opportunity approach rather than assessment. First, focusing on the unbiased distribution of an equal outcome is not always the most effective solution. access to services and goods and on the absence of discrimi- If, for candidate selection or calculation of insurance, focus- nation of groups or individuals, fairness means the equal ing on equal opportunity might lead to increase “economic treatment of people regardless of their sex, race, color, justice,” in other contexts, such as during airport security ethnic or social origin, genetic features, language, religion checks, equality of outcome in the form of “parity of impact” or belief, political or any other opinion, membership of a could help establish a sense of social solidarity avoiding national minority, property, birth, disability, age or sexual the over-examination of certain groups [11] (p. 7). Thus, orientation, when it comes to granting or denying access the choice of a specific approach to evaluate (in)equality to products, services, benefits, professional or educational depends on the specific application context. As Balayn and opportunities, and medical treatments based on an auto- Gürses claim, the regulation of AI must go “beyond de- mated evaluation and classification of individual or groups. biasing” [6]. Mere data-based or outcome-based solutions In addition, a fair system should not involve work exploita- trying to solve local distributive issues of a system, such as tion or the violation of human rights of any of the involved trying to solve a racial bias in image recognition software stakeholders during its life cycle. Moreover, the real-world by enlarging the data basis with pictures from people of all application of the system should not create or amplify power ethnic backgrounds, are not sufficient alone to address struc- unbalances between stakeholders, nor place specific stake- tural inequality issues at the root [60]. If decision-making holders’ groups in a disadvantaged position. processes that influence people’s access to opportunities are biased, the intersection between algorithmic fairness and 2.2 Data science structural (in)justice requires investigation [51]. In addition, other fairness issues that are indirectly related Fairness is discussed in the context of data and data-driven with the algorithmic outcome, but rather with the entire systems whose inherent patterns or statistical biases can be system design and application processes, as well as with interpreted as “unfair.” Here, it is important to emphasize their consequences for society, can occur. These aspects of that the evaluation of whether certain patterns are “fair” or fairness also overlap with other human rights and societal “unfair” transcends the specific expertise of data scientists values. For instance, a structural fairness issue of many ML and requires further legal, philosophical, political, and socio- systems is the phenomenon of “digital labor” [33], referring economic considerations. What is being explored in data (among other things) to the precarious work conditions and science under the term “fairness” are quantitative concepts the very low pay of many click workers generating train- to identify patterns or biases in data, in addition to technical ing data for ML systems. In addition, the commodification methods to mitigate them. of privacy related with the use of many digital services Data analysis and data-based modeling of real-world raises fairness issues since users are often kept unaware of relationships have progressed in recent years especially the exact use of their data, and they are not always in the through Machine Learning (ML). ML is a subdiscipline of position to defend their right to privacy [74, 91, 95]—being AI research in which statistical models are fitted to so-called so a disadvantaged stakeholder compared with the service training data, recognize patterns and correlations in this providers. Finally, addressing the sustainability concerns data, and generate predictions for new (input) data on this that emerged during the so-called “third wave” of AI ethics, basis. ML methods have become a particular focus of fair- global and intergenerational justice can be highlighted as ness research, as they provide everyday applications using fairness issues [37, 93]. Considering intergenerational jus- personal data, e.g., employment decisions, credit scoring, tice means to add a temporal, anticipatory dimension to our understanding of fairness and extend the claim for the equity of human living conditions—as, for instance, expressed in These are the protected attributes listed in Article 21 (Non-discrim- ination) of the EU Charter of Fundamental Rights. While the term bias is often connoted negatively in other disci- plines (with discriminatory effects, etc.), in this context (computer In addition, this cannot be the solution to this problem because it science) it merely means a statistical deviation from the standard [41] would feed even more data into the systems of the service providers [7]. Whether this constitutes a case of discrimination is another ques- and would therefore support their data hunger and business logic. tion. 1 3 AI and Ethics and facial recognition [71]. Furthermore, they pose the chal- from manual processes or, if possible, from the observation lenge that bias within the training data might lead to biased whether in the given examples the loans were repaid in full. model results. When building a ML model, the training data is used to adjust the internal model parameters which determine the 2.2.1 Short introduction to machine learning mapping through f. For instance, in a neural network, the weights assigned to the network’s edges are adjusted by the ML-based applications have enabled technological progress learning algorithm during model building, a phase which is which can particularly be attributed to the fact that their also called “training.” Overall, ML is an optimization pro- functionality is based on learning patterns from data. By this cedure that finds internal model parameters such that they means, ML methods provide approaches to solving tasks that optimize a defined performance metric on a training dataset. could not be effectively addressed by “traditional” software In this case, the performance metric specified as optimiza- fully specified by human rules. In particular, deep neural net- tion objective is referred to as “loss function.” For example, works, a type of ML method involving vast amounts of data, in a classification task, a quantitative measure of the distance have significantly advanced areas, such as image [26] and between ground-truth and model output could be used as a speech recognition [13], in addition to predicting complex loss function. Consequently, such a model would be gener- issues, for instance medical diagnostics [102] and predictive ated in training, which optimally approximates the relation- maintenance [17]. ships between x and y provided in the training data. ML methods are designed to learn from data to improve Underlying the ML approach of fitting a model to train- performance on a given task [41]. A task can be viewed as ing data is the idea that the model infers patterns which finding a mapping that, for an input x , assigns an output y help produce valuable outputs when applied to new data. which is useful for a defined purpose. One ML task that is The term “generalizability” is used to describe the aim that particularly relevant for fairness is classification. The pur - the model performs well on data not seen during training. pose of classification is to identify to which of a set of cat- Thus, for model evaluation, an additional test dataset differ - egories a given input belongs, for instance, whether a person ent from the training data is used. Given training and test is creditworthy or not. ML is about finding such a model data, according to Goodfellow et al. model quality is indi- f that solves a task effectively by y = f(x). To achieve this, cated by two quantities: (i) the training error measured by a learning algorithm adjusts parameters within the model. the loss function, and (ii) the difference between training and The fitness of the model for the given task can be evaluated test error [41]. A model with a large training error is called using quantitative measures. Such quantitative indicators of “underfitting," while one with a low training error but large model or data properties are generally referred to as “met- difference between training and test error is "overfitting" the rics.” For example, a typical performance metric for clas- training data. sification tasks is precision, which measures the proportion to which the classification to a certain category by the model 2.2.2 Meaning and challenges of “fairness” was correct. The data which ML methods use to build a model, called Data has a crucial impact on the quality of a ML model. In “training data,” is a collection of input examples that the computer science, data quality has already been researched model is expected to handle as part of the task. A single for “classical” information systems, where it is considered example in the data is called a datapoint. For classification especially regarding large amounts of stored operational and tasks, a datapoint in the training data of a ML model con- warehousing data, e.g., a company’s client database. Numer- tains, in addition to the example x, a “ground-truth” label ous criteria for data quality have been proposed, which can that specifies how the ML model should process the respec- be mapped within four dimensions: “completeness, unambi- tive input x. Following on the example of creditworthiness guity, meaningfulness, and correctness” [100]. Only recently classification, the training data for a ML model addressing has the operationalization of data quality specifically for ML this task may be drawn from previous credit applications, been explored [46]. The issue of data completeness, relating and the individual datapoints could include features, such to the training and test data sufficiently capturing the appli- as income, age, or category of work activity (e.g., self- cation domain, is particularly relevant in this context. There employed, employed). Moreover, each datapoint should is a high risk that an ML model has low statistical power on also contain a ground-truth label that could be derived data either not included or statistically insignificant in its training set. To prevent strong declines in a model’s produc- tive performance, measures are being researched for deal- ing with missing or underrepresented inputs [46] as well as 5 for detecting distribution skews, e.g., between training and Reinforcement learning methods that learn from interaction with production data [12]. systems or humans are not considered in this description. 1 3 AI and Ethics In certain tasks and application contexts, individuals are been presented [96], particularly in light of providing statis- affected by the outputs of a ML model. For example, ML tical evidence for unequal treatment in classification tasks. models are being used to support recruiting processes, deci- Corresponding to the approach of identifying and compar- sions on loan approval, and facial recognition [71]. Conse- ing groups for identifying bias, so-called “group fairness quently, it is essential that the model performs equally well metrics” constitute a large part of the fairness metrics pre- for all individuals. The research direction in data science sented to date. These metrics compare statistical quantities that addresses related issues from a technical perspective is regarding groups defined on the basis of certain attributes referred to under the term “fairness.” Clearly, the motivation in a dataset (e.g., a group could be defined by means of age, for “fairness” in data or ML models does not derive from a gender, or location if these attributes are provided in the technical perspective, nor does data science as a scientific data). Among the group fairness metrics, one can further discipline provide a sufficient basis for evaluating under distinguish between two types: i) metrics which compare the which circumstances these should be classified as “fair.” The distribution of outputs, and ii) metrics which compare the approaches and methods researched in this area are usually correctness of the outputs with respect to different groups. neutral, as they can be applied to structurally similar sce- An example of the first type is to measure the discrepancy narios that do not involve individuals. to which a certain output is distributed by percentage among Regarding model quality, the data are a central object of two different groups. This quantification approach is called study in fairness from two perspectives. First, aspects of data “statistical parity,” and Sect. 3.3. provides a detailed elabo- quality should not differ regarding particular groups of peo- ration. The second type of metrics focus on model quality ple. Regarding the dimension of completeness, for instance, and compare performance-related aspects with respect to certain population groups could be underrepresented in the different groups (e.g., specific error rates or calibration). For training data resulting in a lower performance of the ML instance, the metric “equal opportunity” [96] calculates the model with respect to these groups [14]. Another example difference between the true-positive rates of a model on the is that the ML model might infer biased patterns from the respective data subsets representing two different groups. training data if their representativeness is compromised by Such metrics can highlight model weaknesses by providing non-random sampling, for example, if predominantly posi- insight on where the model quality may be inconsistent. tive examples are selected from one population group but Besides group fairness metrics, further measures have negative examples are selected from another. Second, even been developed to disclose biases. Two examples are “indi- if data are of high quality from a technical perspective, they vidual fairness” [27] and “counterfactual fairness” [63]. may (correctly) reflect patterns that one would like to pre - “Individual fairness” is based on comparing individuals. vent from being reproduced by the ML model trained on Therefore, a distance metric is defined that quantifies the it. For instance, data might capture systemic bias rooted in similarity between two datapoints. The underlying idea of (institutional) procedures or practices favoring or disadvan- this approach is that similar model outputs should be gener- taging certain social groups [86]. The technical challenge ated for similar individuals. In addition, measurable indica- that arises here is fitting a model to the training data but tors for an entire data set have been derived using such a simultaneously preventing inferring certain undesirable pat- distance metric, for example, “consistency” [104]. Similarly, terns that are present. Overall, proceeding from the variety inequality indices from economics such as the generalized of biases identified to date, both measures that “detect” and entropy index have also been proposed as bias indicators for measures that “correct” (potentially unfair) patterns in data- datasets [89], which require a definition of individual prefer - sets and models are being explored [48] (p. 1175). ences. “Counterfactual fairness” considers individual data- points, similar to “individual fairness”; however, it examines 2.2.3 Measures that “detect” the effect of changing certain attribute values on model out- puts. This can be used to uncover if the model would have Aiming to “detect,” one research direction is concerned with generated a different output for an individual if they had a developing technical approaches to disclose and quantify different gender, age, or ethnicity, for example. Many of the biases in the first place. Numerous “fairness metrics” have presented bias quantification and detection approaches have been implemented in (partially) open-source packages and tools [3, 9, 40] and are likewise applicable to input–output- 6 mappings not based on ML. A variety of bias causes and types has been explored that cannot Different fairness metrics might pursue different target be fully mapped here. For a categorization, in line with the two view- points described, into computational as well as human and systemic states, e.g., balanced output rates between groups (statisti- bias, we refer to Schwartz et  al. [86]. For a categorization of biases cal parity) versus balanced error rates (equal opportunity). along the feedback cycle of data, algorithm and user interaction, see Therefore, they also differ greatly in their potential conflict [71], and for a mapping of biases to the life cycle of AI applications, with other performance goals. For instance, consider a see [90]. 1 3 AI and Ethics dataset in which Group A contains 30% positive ground- model output and certain attributes [59], or using optimiza- truth labels and Group B contains 60%. If the model is to tion constraints to align certain error rates among different reach a low value for a fairness metric that measures the dis- groups [103]. Another in-processing approach, which affects crepancy in the distribution of positive labels across groups, the entire model architecture, is to include an adversarial its outputs must deviate from ground-truth. In addition to network in the training that attempts to draw an inference sacrificing accuracy, this could also result in unbalanced about protected attributes from the model outputs [105]. The error rates. Thus, depending on the nature of the data, fair- model and its adversary are trained simultaneously, where ness metrics may be mutually exclusive [7]. the optimization goal of the original model is to keep the performance of the adversary as low as possible. In contrast, 2.2.4 Measures that “correct” post-processing refers to those measures that are applied to fully trained models. For example, corresponding methods Another direction of research is striving to develop tech- comprise calibration of outputs [78] and targeted threshold nical measures which can “correct” or mitigate detected setting (with thresholds per group, if applicable) to equalize bias. To this end, approaches along the different develop- error rates [49]. Many of the in- and post-processing meas- ment stages of ML models are being explored [35]. The ures are researched primarily for classification tasks, and underlying technical issue, especially when facing systemic the methods developed are typically tailored to one type of or historical bias, is to train a model by inferring correlations model for technical reasons. in data that performs well on a given task—but simultane- ously preventing learning of certain undesirable patterns 2.2.5 Outlook that are present in the data. An important starting point for addressing this apparent contradiction is the data itself. A In the fairness research field, a variety of approaches have basal pre-processing method that has been proposed is “Fair- been developed to better understand and control bias. ness through Unawareness,” meaning that those (protected) Beyond these achievements, still, open research questions attributes are removed from the data set for which correla- remain unaddressed from a technical and interdisciplinary tion with model output values is to be avoided [63], or whose perspective. Regarding the first, many metrics, in addition inclusion could be perceived as “procedurally unfair” [44]. to the methods that work toward their fulfillment, are appli- However, this method alone is not recognized as sufficiently cable to specific tasks only and impose strong assumptions. effective as correlated “proxies” might still be contained in Here, one challenge is to adapt specific measures from one the data [61], and many mitigation methods actively incor- use case to another. Regarding the latter, a central issue is porate the protected attributes to factor out bias [63]. Fur- that different fairness metrics pursue different target states ther examples of pre-processing methods range from tar- for an ML model (see “Measures to detect”), therefore a geted reweighing, duplication, or deletion of datapoints to choice must be made when assessing fairness in practice. modifying the ground-truth [57] or creating an entirely new Furthermore, the concrete configuration of specific metrics, (synthetic) data representation [104]. The latter are usually for example, how a meaningful similarity metric for assess- based on an optimization in which the original datapoints are ing “individual fairness” should be defined, remains unre - represented as a debiased combination of prototypes. While solved. The question of which fairness metrics and meas- these methods primarily aim at equalizing ground-truth val- ures are desirable or useful in practice must be addressed in ues among different groups, some optimization approaches interdisciplinary discussion. A concrete example is provided for generating data representations also include aspects of in Sect. 3.3. individual fairness [64]. Furthermore, to mitigate representa- tion or sampling bias, over-sampling measures to counteract 2.3 Management science class imbalance [16] are being researched [15]. In addition to algorithmic methods, documentation guidelines have been Management literature [4, 21, 38, 77, 88] divides fairness developed to support adherence to good standards, e.g., in into four types: distributive, procedural, interpersonal, and data selection [36]. informational fairness. While the data centrally influences the model results Distributive fairness refers to the evaluation of the out- and pre-processing methods offer the advantage that they come of an allocation decision [34]. Equity is inherent to can typically be selected independently of the model to distributive fairness [79]. Hence, to achieve distributive be trained, research is also being conducted on so-called fairness, participants must be convinced that the expected in- and post-processing measures. In-processing measures value created by the organization is proportionate to their are those that intervene in modeling. This can be realized, contributions [4]. Procedural fairness refers to the process for example, by supplementing the loss function with a of decision right allocation (i.e., how do the parties arrive at regularization term that reduces the correlation between a decision outcome? [62]). To achieve procedural fairness, 1 3 AI and Ethics Table 1 Procedural and Fairness Type Components Description distributive fairness measurement scales [22, 79] Procedural (Rules Process Control Procedures provide opportunities for voice taken from [67, 92]) Decision Control Procedures provide influence over outcomes Consistency Procedures are consistent across persons and time Bias Suppression Procedures are neutral and unbiased Accuracy Procedures are based on accurate information Correctability Procedures offer opportunities for appeals of outcomes Representativeness Procedures consider concerns of subgroups Ethicality Procedures uphold standards of morality Distributive (Rules Equity Outcomes are allocated according to contributions taken from [1, 66]) Equality Outcomes are allocated equally Need Outcomes are allocated according to need a fair assignment of decision rights is required [4, 70]. The discrimination, aiming to exploit the market potential. decision rights must ensure fair procedures and processes for Banks’ business model is to spread risks and price risks future decisions that influence value creation [4 , 70]. according to their default risk in order to achieve the best Interpersonal and informational fairness refer to interac- possible return on investment. For example, banks price tional justice, which is defined by the interpersonal treat- the default risk of loans variously and derive differentiated ment that people experience in decision-making processes prices. Here, procedural fairness is crucial in the overall [10]. Interpersonal fairness reflects the degree of respect fairness assessment, since procedurally unfair price settings and integrity shown by authority figures in the execution of lead to higher overall price unfairness [32]. Ferguson, Ellen, processes. Informational fairness is specified by the level of and Bearden highlight that random pricing is assessed to truthfulness and justification during the processes [20, 43]. be more unfair than possible cost-plus pricing (price is the To ensure a differentiated assessment of AI systems from sum of product costs and a profit margin) within the proce- a socioeconomic perspective, these four dimensions should dural fairness assessment [32]. Furthermore, they provide be included in the evaluation of fairness. In particular, pro- evidence that procedural and distributive fairness positively cedural and distributive fairness should be emphasized, as interact and thus, if implemented accordingly, can maximize the credit scoring assessment concentrates primarily on the the overall fairness. As described in Table 1, the presented credit-granting decision process. Table 1 provides an over- six procedural components and three distributional compo- view of fairness measurement scales in management science nents should be considered in the pricing process to achieve based on Colquitt and Rodell’s [22] and Poppo and Zhou’s strong overall fairness. From an organizational perspective, [79] work. financial institutions should ensure that their credit ratings Within the data science perspective, inequality indices are neutral and unbiased based on accurate information such as the generalized entropy (GE) index or the Gini [30]. Regarding erroneous data, customers should be able coefficient are widely accepted [25]. Both measures aim to to review and correct the data if necessary. Pricing should evaluate income inequality from an economic perspective. be consistent to avoid the impression of random pricing. However, they differ in their meaning, with the GE index Therefore, people with the same attribute characteristics providing more detailed insights by collecting information should always receive the exact credit pricing. Furthermore, on the impact of inequality across different income spec- the possible use of algorithms should not disadvantage cer- trums. The GE index is also used in an interdisciplinary tain marginalized groups. Moreover, to maximize the overall context. For example, in computer science, it is used to fairness, banks should include the distributive components measure redundancy in data, which is used to assess the in their fairness assessment. Haws and Bearden emphasize disparity within the data. In addition to the economic level, that customers assess high prices with unfairness and vice approaches to fairness measurement also exist at the corpo- versa [50]. Thus, Ferguson, Ellen, and Bearden argue that rate level. distributional fairness is given when customers receive an However, the operationalizability of the fairness types advantageous price [32]. Consequently, when pricing loans, described, especially procedural and distributive fairness, banks should always adhere to market conditions to achieve remains unclear. This paper aims to address this issue. a maximum overall fairness. A typical instrument in management practice is price 1 3 AI and Ethics are granted without a comprehensive check-up by the credit 2.4 Summary institute. During the credit application process, as a preliminary Fairness can be generally considered as the absence of unjustified unequal treatment. This broad understanding step, a bank requests customer information, such as address, income, employment status, and living situation, which it takes on different specific connotations that require consid- eration when evaluating AI systems. In our interdisciplinary feeds into its own (simple) credit scoring algorithm. As opposed to the application process for higher volume credits, overview, two main aspects of fairness were highlighted. Distributive fairness is one of these. It concerns how auto- extensive information on the overall assets and wealth of the applicant is not required. Regarding particularly small mated predictions impacting the access of individuals to products, services, benefits, and other opportunities are allo- lending, account statements might not even be necessary. In some cases, the authorization to conduct a solvency check cated. This algorithmic outcome can be analyzed through statistical tools to detect an eventual unequal distribution of through a credit check agency might be requested. If so, the credit check agency will process additional informa- certain predictions and to assess whether this is justified by the individual features of group members, or whether this is tion concerning, among other things, the credit history of the applicant and other personal information to produce a due to biases or other factors. Considering procedural fair- ness is also fundamental for the evaluation of AI systems. credit rating. Finally, based on the creditworthiness assess- ment, a bank clerk decides whether the small personal loan Therefore, it should be considered how a decision is reached for different stakeholders’ groups and how members of these is granted. In some instances, the rates might be raised in order to compensate the credit institutes for the potential groups are treated in the different stages of the product life cycle. illiquidity of individual customers. In the European framework, guidelines to improve institu- tions’ practices in relation to the use of automated models for credit-granting purposes have been produced [2, 5, 30]. 3 Evaluating fairness In the report Guidelines on loan origination and monitor- ing, the European Banking Authority (EBA) recommends 3.1 T he use case: creditworthiness assessment scoring for small personal loans that credit institutions should “understand the quality of data and inputs to the model and detect and prevent bias in the Based on the empirical evidence of perpetuation of preex- credit decision-making process, ensuring that appropriate isting discriminatory bias and of the discrimination risks for specific demographic groups, recent literature on credit scoring algorithms investigated gender-related [96] and In the report Guidelines on loan origination and monitoring, the race-related [65] fairness issues of ML systems, looking for European Banking Authority lists a set of customer information that suitable tools to detect and correct discriminatory biases are admissible and, where applicable, recommended to collect for and unfair prediction outcomes. Here we consider the case credit institutions [30]. These are: purpose of the loan, when relevant to the type of product; employment; source of repayment capac- of small personal loans. These are small volume credits to ity; composition of a household and dependents; financial commit- finance, for instance, the purchase of a vehicle or pieces of ments and expenses for their servicing; regular expenses; collateral furniture, or to cover the costs of expenses such a wedding (for secured lending); other risk mitigants, such as guarantees, when or a holiday. They typically range from 1.000 EUR to 80.000 available (85.a — 85.h). A more extensive list of possible private cus- tomer information that might be asked in different loan application EUR—in some cases they can be up to 100.000€, which scenarios is included in the Annex 2 of the same publication. The col- 7 lection of all the listed information point is not mandatory. Different banks set different limits to the maximum amount of a small personal loan. For the purpose of this paper, we considered the For instance, the Commerzbank don’t require these for credit lend- five German top banks for number of customers (https:// www. mobil ing up to 15.000 EUR (https:// www. comme rzbank. de/ kredit- finan ebank ing. de/ mag az in/ bank en- r anki ng- die- g r oes s ten- bank en- deuts zieru ng/ produ kte/ raten kredi te/ klein kredit/). chlan ds. html) and found the following ranges: Sparkasse, 1.000– 80.000 EUR (https:// www. skpk. de/ kredit/ priva tkred it. html); Volks- On its website, the German credit rating agency Schufa lists the bank, 1.000–50.000 EUR (https:// www. vr. de/ priva tkund en/ unsere- following applicant features as impact factors for credit score: number produ kte/ kredi te/ priva tkred it. html); ING, 5.000–75.000 EUR (https:// and dates of relocations; number of credit cards and opening date of www. ing. de/ kredit/ raten kredit/); Postbank 3.000–100.000 EUR the credit cards’ accounts; number and dates of online purchases on (https:// www. postb ank. de/ priva tkund en/ produ kte/ kredi te/ priva tkred it- account; payment defaults; existing loans; number and opening dates direkt. html); Deutsche Bank 1.000–80.000 EUR (https:// www. deuts of checking accounts; existing mortgage loans (https:// www. schufa. che- bank. de/ opra4x/ public/ pfb/ priva tkred it/#/ page-2-0). Using online de/ score check tools/ pt- einfl ussfa ktoren. html). platforms to compare different lenders such as check24 (https:// www. For instance, in the credit application form of the Deutsche Bank check 24. de/) or verivox (https:// www. veriv ox. de/ kredit/ klein kredit/), is stated that the interest rate depends on the credit rating of the credit we found out that no credit lenders in Germany offers more than check agency (https:// www. deuts che- bank. de/ opra4x/ public/ pfb/ priva 100.000 EUR. tkred it/#/ page-2-0). 1 3 AI and Ethics safeguards are in place to provide confidentiality, integrity it represents a violation of basic human rights, for struc- and availability of information and systems have in place” tural reasons, many individuals belonging to disadvantaged (53.e, see also 54.a and 55.a), take “measures to ensure the groups still struggle to access credit. The gender pay gap is traceability, auditability, and robustness and resilience of a concrete example: a woman working full-time in the same the inputs and outputs” (54.b, see also 53.c), and have in position as a male colleague might earn less [42], and be less place “internal policies and procedures ensuring that the creditworthy from the bank’s perspective. quality of the model output is regularly assessed, using Since ethics is supposed to influence shaping a fairer soci- measures appropriate to the model’s use, including back- ety, the definition of minimal ethical requirements for a fair testing the performance of the model” (54.c, see also 53.f ML system in a specific application field should consider and 55.b) [30]. In the white paper Big data and artificial whether and how technologies can help prevent unfairness, intelligence, the German Federal Financial Supervisory and aim at assisting disadvantaged groups. Considering our Authority (BaFin) also recommends principles for the use fairness understanding as the absence of unjustified unequal of algorithms in decision-making processes. These include: treatment of individuals or groups, the first question leading preventing bias; ruling out types of differentiation that are to a definition of minimal ethical requirements is the follow - prohibited by law; compliance with data protection require- ing: Are there individuals belonging to certain groups that ments; ensuring accurate, robust and reproducible results; are not granted loans although they share the same relevant producing documentation to ensure clarity for both internal parameters with other successful applicants belonging to and external parties; using relevant data for calibration and other groups? This should not be mistaken with the claim validation purposes; putting the human in the loop; hav- that every group should have the same share of members ing ongoing validation, overall evaluation and appropriate being granted a loan (group parity), since it is not in the adjustments [5]. interest of the person applying for a loan to be granted one if The present work follows on these recommendations and they are unable to pay it back. This would also be unethical contributes to the regulatory discussion by highlighting use because it would further compromise the financial stability case-specific operationalizable requirements that address and creditworthiness of the person, causing legal trouble and the issues emphasized by European financial institutions. moral harm. To answer this question, fairness metrics can be We focus specifically on small volume credit for two main a useful tool to detect disparities among groups. reasons. First, the pool of potential applicants is significantly larger than the one for higher volume credits such as mort- 3.2.1 Which metric(s) to choose? gage lending. While high volume credits usually require the borrower to pledge one or more assets as a collateral and The choice of one metric in particular is not value-neutral. to be able to make a down payment to cover a portion of Several factors should be considered when investigating fair- the total purchase price of an expensive good, these condi- ness metrics. Among others, there is an ethical multi-stake- tions do not apply to small personal loans, making these also holder consideration to be performed [47]. Certain metrics accessible for citizens without consistent savings or other can better accommodate the businesses’ needs and goals, assets. Since the overall personal wealth should not inu fl ence while others will better safeguard the rights of those being the decision outcome in small personal loans, this makes it ranked or scored by a software. For instance, while evaluat- a particularly interesting scenario to evaluate potential dis- ing the accuracy of a credit scoring system among protected crimination of individual belonging to disadvantaged groups groups, financial institutes will be primarily interested in that are not eligible for higher volume credits, but could optimizing (to a minimal rate) the number of loans granted be granted a small personal loan. Second, the amount of to people who will not repay the debt (false positive rate) applicant information processed for creditworthiness assess- for all groups. However, it is in the interest of solvent credit ment is signic fi antly lower than in the case of higher volume applicants to optimize to a minimal rate the number of credit credits, allowing a clearer analysis of the relevant parameters applicants to whom credit is denied although they could have and their interplay. repaid the debt (false negative rate) for all groups. Therefore, if asked to choose a metric to evaluate fairness, the former 3.2 Preliminary ethical analysis could opt for a predictive equality fairness metric, measur- ing the probability of a subject in the negative class to have Regarding credit access, structural injustice severely afflicts a positive predictive value [96]. However, someone repre- women and demographic minorities. Although in contempo- senting the latter could rather choose the equal opportunity rary liberal democracies explicitly preventing credit access based on gender, race, disability or religion is illegal because See, e.g., the EU Charter of Fundamental Right, §21, non-discrim- ination, and §23, equality between women and men. 1 3 AI and Ethics metric, the probability of a subject in a positive class to The VDE SPEC 90012 (2022) “VCIO-based description have a negative predictive value [96]. Therefore, the choice of systems for AI trustworthiness characterization” recom- of a specific metric is not value-neutral since it could bet- mends to audit working and supply chain conditions, data ter serve the interest of certain stakeholder groups. Among processing procedures, ecological sustainability, adequacy other things, the role of AI ethics and AI regulation should of the systems outcome’s explanation to inform the affected be to prevent a minority of advantaged stakeholders from persons [94]. NIST Special Publication “Towards a Standard receiving the majority of advantages at the expense of those for Identifying and Managing Bias in Artificial Intelligence” who are less advantaged. In this specific use case, this goal recommends considering “human factors, including soci- should be reached by considering the equal treatment of all etal and historic biases within individuals and organizations, applicants as the actual chance of getting a loan when the participatory approaches such as human centered design, financial requirements are met irrespectively of the appli- and human-in-the-loop practices” when addressing bias cant’s demographic group. in AI [86]. Madaio et al. designed a check-list intended to In their paper, “Why fairness cannot be automated,” San- guide the design of fair AI systems including “solicit input dra Wachter, Brent Mittelstadt, and Chris Russel, highlight on definitions and potential fairness-related harms from dif- the “conditional demographic parity” as a standard baseline ferent perspectives,” “undertake user testing with diverse statistical measurement that aligns with the European Court stakeholders,” and “establish processes for deciding whether of Justice “gold standard” for assessment of prima facie dis- unanticipated uses or applications should be prohibited” crimination. Wachter, Mittelstadt, and Russel argue that, if among the to-dos [69]. adopted as an evidential standard, conditional demographic In the credit lending scenario, certain applicants’ groups parity will help answer two key questions concerning fair- have been structurally disadvantaged in their history of ness in automated systems: access to credit and could still experience obstacles in suc- cessfully participating in the application process. Conse- 1. Across the entire affected population, which protected quently, it should be ensured that only parameters which groups could I compare to identify potential discrimina- are relevant to assess the applicant’s ability to repay the tion? loan are processed  —  e.g., bank statements or monthly 2. How do these protected groups compare to one another income — and that parameters that may lead to direct or in terms of disparity of outcomes? [98] indirect discrimination and bias perpetuation — e.g., postal code, gender, or nationality — are excluded. On this point, Here we follow their general proposal and suggest to use we follow the privacy preserving principle of “data mini- this specific metric as an evaluation tool for the specific case mization” as expressed in Art. 5.1.(c) and 25.1. GDPR. of creditworthiness assessment for small personal loans. In Assuming that there are different computing methods to Sect. 3.3., we show how this metric can be used to evaluate optimize the algorithmic outcome in order to avoid unjusti- the algorithmic outcome in our application field. fied unequal treatment of credit applicants, those methods processing less data should be preferred over those requiring 3.2.2 De‑biasing is not enough a larger dataset containing more information on additional applicant’s attributes. A fairness metric alone is insufficient to address fairness Moreover, to empower credit applicants from all groups, issues. If it becomes clear that group inequality is moti- the decision process should be made explainable so that vated by a structural reason, both the algorithmic outcome rejected applicants can understand why they were unsuc- and the parameters and steps behind the decision process, cessful. This would prevent applicants from facing black box this process requires questioning [45]. Different examples decisions that cannot be contested, therefore diminishing the of checklists addressing procedural aspects of AI systems’ bargaining power unbalance between applicants and credit design, development and application can be found in recent institutes. The decision process can be questioned through reports, white papers, and standard proposals. The Assess- “counterfactual” explanations stating how the world would ment List for Trustworthy AI (ALTAI) for self-assessment have to be different for a desirable outcome to occur [73, 97]. by the High Level Expert Group for Artificial Intelligence As remarked by Wachter et al., in certain cases, knowing of the EU includes “mechanisms to inform users about the what is “the smallest change to the world that can be made purpose, criteria and limitations of the decisions generated to obtain a desirable outcome,” is crucial for the discus- by the AI system,” “educational and awareness initiatives,” sion of counterfactuals and can help understand the logic “a mechanism that allows for the flagging of issues related to of certain decisions [97] (p. 845). In our specific case, to bias discrimination or poor performance of the AI system,” provide applicants with this knowledge, the decisive parame- and an assessment taking “the impact of the AI system on ters or parameter combination (e.g., insufficient income and/ the potential end-users and/or subjects into account” [53]. or being unemployed) that led to credit denial should be 1 3 AI and Ethics made transparent, and the counterfactual explanation should parity” metrics [96] in addition to “demographic parity” explain how these parameters should have differed in order [98], which are equivalent under certain circumstances, for the credit application to be approved. This would provide constitute basic representatives of group fairness metrics the applicant the opportunity to contest the algorithmic deci- that compare the (distribution of) outputs. These metrics sion, to provide supplementary information relevant to sup- are best applied to scenarios where there is a commonly port their application or, eventually, to successfully reapply preferred output from the perspective of the affected indi- for a smaller loan. viduals (e.g., “credit granted” in case of credit scoring or This transparency requirement also relates to the issue “applicant accepted” in case of automated processing of job concerning processing data that could result in direct or indi- or university applications), and they compare how this out- rect discrimination since credit scoring might be performed put is distributed. “Statistical parity” serves to compare the based on data belonging to credit check agencies which are proportions to which different groups, defined by a sensitive/ not made available for private citizens. protected attribute, are assigned a (preferred) output. Let us illustrate this on the example of credit scoring: Denote c = 1 3.3 T he “conditional demographic parity” metric the prediction/outcome that a credit is granted (c = 0 if the credit is not granted), and S the sensitive attribute sex with In order to conduct the statistical calculation concerning the S = m denoting a male applicant and S = f a female applicant potential existence of indirect discrimination, in the interdis- (for now, we reduce this example to the binary case both for ciplinary literature, the so-called “conditional demographic the output and the sensitive attribute). The “statistical par- parity” metric has been proposed [98] (p. 54 ff.). This met- ity” metric (with respect to the groups of female and male ric mirrors a statistical approach, which can be applied to applicants) is defined as the difference between the propor - examine potential discrimination in the context of the Euro- tion to which male applicants are granted a loan and the pean anti-discrimination laws [98]. This technique should proportion to which female applicants are granted a loan. not be confused with the second step, meaning the ques- As a formula: tion of justification of a particular disadvantage. Instead, applicants with c = 1 and S = m it only concerns the first step, which deals with the ques- applicants with S = m tion whether a particular disadvantage within the meaning (1) applicants with c = 1 and S = f of the definition of indirect discrimination is present. From applicants with S = f a computer science perspective, numerous approaches for measuring fairness have been presented which fit in the For instance, if 80% of male applicants and 60% of female fundamental conceptions of “individual fairness” or “group applicants are granted a loan, the statistical (dis-)parity is fairness.” Individual fairness relates to the idea of comparing |80%–60%|= 20%. two persons, which can be classified as similar apart from Other than contrasting the protected group (here mean- the sensitive attribute; it is infringed if these two persons ing the people falling under the sensitive feature in question are not treated correspondingly [48] (p. 1175). However, and being examined in the specific case) with the non-pro- group fairness statistically compares two groups of persons tected group (what the “statistical parity” metric does), one [48] (p. 1175). A case of direct discrimination constitutes a could also consider solely the protected group and compare breach of individual fairness; a case of indirect discrimina- group proportions along preferred and non-preferred out- tion contravenes group fairness [48] (p. 1175). puts. “Demographic parity” as described by [98] follows Group fairness metrics generally compare statistical the latter approach. This metric compares to what propor- quantities regarding defined groups in a dataset, e.g., the tion the protected group is represented among those who set of data samples with annual income over 50.000€ and received the preferred output and among those who received the group with income less or equal to 50.000€. “Statistical the non-preferred output. According to the description in [98], demographic disparity exists if a protected group is to Remark: Under the assumption that both the output and the sensi- a larger extent represented among those with non-preferred tive attribute are binary, one can show that the concepts of “statistical output than among those with preferred output. parity” and “demographic parity” are equivalent, meaning that statis- Returning to our credit scoring example, “demographic tical parity is satisfied if and only if equality in demographic parity holds (for a proof under the above assumptions, see annex 1 in [98]). parity” here measures the difference between the proportion However, each sensitive attribute can be simplified to the binary case of females within the group of persons whom a credit is by considering the protected group (e.g., “female applicants”) on the granted, and the proportion of females within the group of one hand and “all others” on the other (i.e., the male and all third persons whom a credit is not granted. As a formula, demo- genders are thrown together to form the second group). Although not comparing two specific groups anymore but the protected group with graphic disparity exists if “all others,” for this simplification, “statistical parity” and “demo- graphic parity” are equivalent though. 1 3 AI and Ethics configuration of A , c, and S, “conditional demographic par- applicants with c = 0 and S = f ity” compares the proportion to which successful applicants applicants with c = 0 (2) satisfying A are female with the proportion to which unsuc- applicants with c = 1 and S = f cessful applicants satisfying A are female. As a formula: applicants with c = 1 applicants with c = 0, S = f and A For instance, if 36% of the applicants being granted a applicants with c = 0 and A loan are female, but 51% of those not being granted a loan (3) applicants with c = 1, S = f and A are female, demographic disparity exists with a discrepancy applicants with c = 1 and A of |36%–51%|= 15%. Both the “statistical parity” and “demographic parity” Let us assume that female loan applicants have an income metrics provide a first indication of the “particular disad- under 50.000€ statistically more often than non-female vantage” within the definition of indirect discrimination applicants. Using fictitious numbers, let 90% of the female presented above. Moreover, they can be easily calculated applicants satisfy A but only 70% of the non-female appli- independent of the (potentially biased) ground-truth data. cants. “Conditional demographic parity” can now help us However, groups defined by only one sensitive attribute better understand whether this gender pay gap provides an (e.g., sex) can be large. Thus, the metrics presented might explanation why female applicants are being granted a loan be coarse and unable to capture potential disparity within less frequently, or whether there is additional discrimina- a group. For example, within the group of females, single tion not resulting from unequally distributed income. Using applicants from the countryside might have a far lower fictitious numbers again, let us assume that 60% of the approval rate than the average female applicant, while mar- applicants who are being granted a loan and satisfy A are ried applicants from the city might be granted credits almost female and 62% of the unsuccessful applicants satisfying as often as men. A are female. Thus, analyzing small income only (where Considerations such as the previous, which aim to under- female applicants represent a higher percentage), female stand given statistical or demographic disparity more deeply and non-female applicants are not treated significantly (e.g., by finding correlated attributes to explain the exist- differently. Complementary, one could examine whether ing bias), should be informed by statistical evidence. One there is unequal treatment in the high-income group. Let approach to provide more granular information on poten- B = (“annual income” > = 50.000€). Following our exam- tial biases is to include (a set of) additional attributes A, ple, 30% of the non-female applicants fall in category B but which do not necessarily need to be sensitive/protected. In only 10% of the female applicants. Let us now apply con- particular, the statistical quantities which are subject to the ditional demographic parity with respect to “high-income.” metrics presented can be calculated on subgroups which Using fictitious numbers, let 27% of the applicants who are are characterized by attributes A (additional to the sensitive being granted a loan and satisfy B be female and 25% of attribute). This enables a comparison of (more homogene- the unsuccessful applicants satisfying B be female. Again, ous) subgroups. Following this approach, an extension of the the representation of females in the high-income group is demographic parity metric has been presented : fairly equal among successful and unsuccessful applicants. “Conditional demographic parity” [98] is defined in the Overall, analyzing the subsets of applicants with small same way as “demographic parity” but restricted to a data income and high-income separately, female and non-female subset characterized by attributes A. In other words, “condi- applicants seem to be treated equally among these groups. tional demographic parity” is violated if, for a (set of) attrib- Thus, our fictitious numbers indicate that the bias in overall utes A, the protected group is to a larger extent represented acceptance rates results from female applicants being to a among those with non-preferred output and attributes A than larger extent represented in the small income group than among those with preferred output and attributes A. non-female applicants. Returning to the credit scoring example, let A = (“annual Regarding the examination under the European non-dis- income” < 50.000€) the attribute characterizing a person crimination laws, the following remarks can be made with as having an annual income lower than 50.000€. For this regard to the example above: In principle, trying to avoid the There is also an extension of the statistical parity metric called “conditional (non-)discrimination” [58], also referred to as “con- These numbers are inspired by Einkommen von Frauen und Män- ditional statistical parity” [96], which is defined in the same way as nern in Deutschland 2021 | Statista (https:// de. stati sta. com/ stati stik/ “statistical parity” but is supposed to be calculated on a subset of da ten / s t udie / 2 9039 9/ u mfr a g e / u mfr a g e - in - de uts c hla nd- zu m- e inko the data characterized by attributes A. Due to the fact that this paper mmen- von- frauen- und- maenn ern/). According to this source, in 2021 focuses on conditional demographic parity, this metric is not sup- in Germany, 91,1% of the women had a monthly net income between posed to be further discussed in the following. 0 and 2.500€ while this holds for 72,1% of the men. 1 3 AI and Ethics non-repayment of a credit constitutes a legitimate aim of the costs and total wealth and assets is not required. These banking institute [85] (p. 310). As to this, the financial capa- can therefore not be considered as conditionals. bility of the applicant in question is decisive. In this respect, (2) Ensure the relevance of the chosen indicators. Param- it is conceivable to consider the applicants’ income — mak- eters which are not directly relevant to assess the appli- ing it a suitable means to foster the legitimate aim. With cant’s ability to repay the loan shall not be processed. this in mind one can see that when considering the criterion These include attributes, such as postal code, nation- “income,” the percentage of women among the unsuccessful ality, marital status, gender, disability, age (within and the successful applicants is more or less equal within the the fixed age limits to apply for a loan), and race. two construed income-(sub-)groups (yearly salary over and Some of these, such as gender, race, and disability, are under 50.000€). Considering only the attribute (sex), the characteristics protected by national and international result might be that the percentage of unsuccessful female anti-discrimination acts such as the European anti-dis- applicants is higher than the percentage of successful female crimination law or the German Equal Treatment Act applicants (demographic disparity). Comparing the two (AGG). Others, even if not protected by anti-discrim- results, it is possible to make the assumption that income ination laws, might facilitate the deduction of one or was crucial for the decision whether a credit is granted or more protected attributes. not. If one considers the orientation toward the income as (3) Provide transparency for credit applicants and other an indicator for the financial capability to be necessary and actors involved. The following shall be made transpar- appropriate, the particular disadvantage might be justified. ent for the applicants: While generally achieving “conditional demographic parity” seems unlikely given the variety of choice for A, o Which data are processed (no personal data is pro- calculating the “conditional demographic parity” metric cessed without informed consent of the applicant). for different configurations of A can still provide valuable p Why an application is eventually rejected and what evidence to assist in detecting the relevant “particular disad- applicant features should be improved to obtain the vantage” (pertaining to the first step in determining an indi- loan. Therefore, the algorithmic decision must be rect discrimination). Especially, being more fine-granular counterfactually explainable, e.g., if the applicant than the non-conditional metrics which measure bias only had a higher income or if she/he was not unem- in the model’s overall results, their extensions can be used to ployed, she/he would have received the loan. explain bias by analyzing additional attributes which might provide further relevant information. This does not mean disclosing the entire computing process—which might be protected by trade secrets— 3.4 Ethical minimal requirements but guaranteeing transparency regarding the criteria applicants need to fulfill. We claim that the following minimal requirements must be standardized to address discrimination and procedural fair- ness issues concerning the application of ML systems in 4 Standardizing minimal ethical credit scoring for small personal loans: requirements to evaluate fairness (1) Regular check of the algorithmic outcome through a Standardization can connect different perspectives of “fair - fairness metric. We follow Wachter, Mittelstadt, and ness” and can establish a universal understanding in the con- Russel in suggesting that the conditional demographic text of AI. It can erase trade barriers and support interoper- parity fairness metric should be used to detect unfair ability as well as foster the trust in a system or application. outcome [98]. For our simple case study, the condition- Within the realm of standardization, existing definitions for als will be income and employment status. If the bank fairness are rather generic and currently not tailored for AI requires an external credit rating, then other parameters systems and applications; however, this may change soon that influence the rating such as past loan defaults or with the development of new AI-dedicated standards. number of credit cards must also be considered. How- Several documents are pushing in that direction: ever, in our case, comprehensive information on living ISO/IEC TR 24028:2020, Information technol- For instance, considering the “German Credit Dataset” (https:// ogy — Artificial intelligence — Overview of trustworthi- archi ve. ics. uci. edu/ ml/ datas ets/ Statl og+% 28Ger man+ Credit+ Data% ness in artificial intelligence as it also lists fairness as an 29) used by Verma and Rubin (2018) for their fairness metrics’ analy- essential part for ensuring trustworthiness in AI [55]. sis, it is possible to flag the following parameters according to this requirement: personal status and sex (attribute 9); age in years (attrib- ISO/IEC TR 24027:2021, Information technol- ute 13); foreign worker (attribute 20). ogy — Artificial intelligence (AI) — Bias in AI systems 1 3 AI and Ethics and AI aided decision-making addresses bias in relation a fair consensus process. The outcome of this process is a to AI systems [54]. recognized standard, enabling mutual understanding based ISO/IEC TR 24368:2022, Information technol- on agreed requirements, thus fostering trade and the new ogy — Artificial intelligence — Overview of ethical and development of quality AI products and services—either societal concerns aims to provide an overview of AI ethi- nationally, in Europe, or internationally depending on the cal and societal concerns, as well as International Stand- used standardization platform. ards that address issues arising from those concerns [56]. A standard can be used for a quality assessment in order to promote a product’s or service’s quality, trustworthiness, In addition, there are many other AI-specific projects pub- and user acceptability. In the assessment process of an AI lished or under development within the ISO and the IEC system or application, the related standardized fairness met- on the topics of ML, AI system life cycle processes, func- ric can be used to attest the system’s or application’s ability tional safety, quality evaluation guidelines, explainability, to execute fair decisions. Consequently, a fairness-related data life cycle frameworks, concepts and terminology, risk attestation based on corresponding standards (e.g., certifi- management, bias in AI systems, AI aided decision-making, cation) can increase the user acceptability and trustworthi- robustness assessment of neural networks, an overview of ness of the AI system or application, which can result in ethical and societal concerns, process management frame- increased sales figures. work for big data analytics, and other topics. The focus of these projects is to develop a framework of requirements for the development and operation of safe, robust, reliable, 5 Conclusion explainable, and trustworthy AI systems and applications. Following the establishment of the general AI requirement Evaluating the fairness of an AI system requires analyzing framework, the focus may likely shift to more use case- an algorithmic outcome and observing the consequences of specific standardization topics like “fairness,” which is the development and application of the system on individu- clearly needed in the standardization of AI, but cannot be als and society. Regarding the applied case of creditworthi- generalized. ness assessment for small personal loans, we highlighted Based on our interdisciplinary analysis, the standardiza- specific distributive and procedural fairness issues inherent tion of “fairness” in the context of AI with the aim to allow either to the computing process or to the system’s use in a an assessment requires multiple relevant measurable and real-world scenario: (1) the unjustified unequal distribution quantifiable parameters and/or attributes building state of of predictive outcome; (2) the perpetuation of existing bias the art use case-specific fairness metrics such as the above and discrimination practices; (3) the lack of transparency discussed conditional parity metric. Such fairness metrics concerning the processed data and of an explanation of the can be developed and standardized with an independent algorithmic outcome for credit applicants. We addressed consensus driven platform open to expertise from all use these issues proposing ethical minimal requirements for this case-related stakeholders, including views from the perspec- specific application field: (1) regularly checking algorithmic tives of philosophy, industry, research, and legislation. This outcome through the conditional demographic parity metric; platform can be either a national standards body, ISO, IEC, (2) excluding from the group of processed parameters those or the European Standardization Organization (CEN); where that could lead to discriminatory outcome; (3) guaranteeing the most appropriate option for this topic is the international transparency about the processed data, in addition to coun- joint committee between ISO and IEC, the ISO/IEC JTC 1/ terfactual explainability of algorithmic decisions. Defin - SC 42 “Artificial intelligence”. To begin a standardization ing these minimal ethical requirements represents a start- process for a use case-specific fairness metric, scope, out- ing point toward standards specifically addressing fairness line, and justification of the proposed standardization project issues in AI systems for creditworthiness assessments. These must be proposed to the respective standardization commit- requirements aim to prevent unfair algorithmic outcomes, as tee. To elevate the chances of approval, a first draft with the well as unfair practices related to the use of these systems. proposed fairness metric should also be included. The stand- ardization process within national standards bodies, ISO, IEC, and CEN provides all participating members an equal Funding Open Access funding enabled and organized by Projekt right to vote, comment and work on a standardization pro- DEAL. Our research is funded by the Ministerium für Wirtschaft, Industrie, Klimaschutz und Energie des Landes NRW (MWIDE NRW) ject. When working internationally or in the European field, in the framework of the project “Zertifizierte KI” (“Certified AI”); this means that all interested registered experts can work funding number 005–2011-0050. on the project; however, during mandatory voting (project proposal, drafts and finalization) each participating country (represented by delegated experts) has one vote to facilitate 1 3 AI and Ethics 14. Buolamwini, J., Gebru, T.: Gender shades: intersectional accu- Declarations racy disparities in commercial gender classic fi ation. Proceedings of the 1st Conference on Fairness, Accountability and Transpar- Conflict of interest The authors have no relevant financial or non-fi- ency, PMLR 81, 77–91 (2018) nancial interests to disclose. On behalf of all authors, the correspond- 15. Chakraborty, J., Majumder, S., Menzies, T.: Bias in machine ing author states that there is no conflict of interest. learning software: why? how? what to do? In: Proceedings of the 29th ACM joint meeting on European software engineer- Open Access This article is licensed under a Creative Commons Attri- ing Conference and Symposium on the. foundations of software bution 4.0 International License, which permits use, sharing, adapta- engineering, 429–440 (2021) tion, distribution and reproduction in any medium or format, as long 16. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: as you give appropriate credit to the original author(s) and the source, SMOTE: synthetic minority over–sampling technique. J. Artif. provide a link to the Creative Commons licence, and indicate if changes Intell. Res. 16, 321–357 (2002) were made. The images or other third party material in this article are 17. Çınar, Z.M., Abdussalam Nuhu, A., Zeeshan, Q., Korhan, O., included in the article's Creative Commons licence, unless indicated Asmael, M., Safaei, B.: Machine learning in predictive mainte- otherwise in a credit line to the material. If material is not included in nance toward sustainable smart manufacturing in industry 4.0. the article's Creative Commons licence and your intended use is not Sustainability. 12, 8211 (2020) permitted by statutory regulation or exceeds the permitted use, you will 18. Coeckelbergh, M.: AI for climate: freedom, justice, and other eth- need to obtain permission directly from the copyright holder. To view a ical and political challenges. AI Ethics 1, 67–72 (2021). https:// copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . doi. org/ 10. 1007/ s43681- 020- 00007-2 19. Cohen, G.A.: On the currency of egalitarian justice. Ethics 99, 906–944 (1989) 20. Colquitt, J.A.: On the dimensionality of organizational justice: a References construct validation of a measure. J. Appl. Psychol. 86, 386–400 (2001) 21. Colquitt, J.A., Rodell, J.B.: Justice, trust, and trustworthiness: a 1. Adams, J.S.: Inequity in social exchange. Adv. Exp. Soc. Psychol. longitudinal analysis integrating three theoretical perspectives. 2, 267–299 (1965) Acad. Manag. J. 54, 1183–1206 (2011) 2. Aggarwal, N.: Machine learning, big data and the regulation of 22. Colquitt, J.A., Rodell, J.B.: Measuring justice and fairness. In: consumer credit markets: the case of algorithmic credit scoring. Cropanzano, R.S., Ambrose, M.L. (eds.) The Oxford Handbook In: Aggarwal, N., Eidenmüller, H., Enriques, L., Payne, J., van of Justice in the Workplace, pp. 187–202. OUP, Oxford (2015) Zwieten, K. (eds.) Autonomous Systems and the Law, pp. 37–44. 23. Cremers, A.B. et al.. Trustworthy Use of Artificial Intelligence. Beck, München (2018) Priorities from a philosophical, ethical, legal, and technological 3. Ahn, Y., Lin, Y.R.: Fairsight: visual analytics for fairness in deci- viewpoint as a basis for certification of Artificial Intelligence, sion making. IEEE Trans. Vis. Comput. Graph. 26, 1086–1095 https:// www. ki. nrw/ wp- conte nt/ uploa ds/ 2020/ 03/ White paper_ (2020) Thrus twort hy_ AI. pdf (2019). Accessed 28 January 2022 4. Ariño, A., Ring, P.S.: The role of fairness in alliance formation. 24. Datenetikkommission der Bundesregierung. Gutachten der Strat. Manag. J. 31, 1054–1087 (2010) Datenethikkommission der Bundesregierung, https:// www. bmi. 5. BaFin, Big data and artificial intelligence: Principles for the use bund. de/ Share dDocs/ downl oads/ DE/ publi katio nen/ themen/ it- of algorithms in decision-making processes, https:// www. bafin. digit alpol itik/ gut ac hten- daten e t hik kommi ssion. pdf?__ blob= de/ Shar e dDocs/ Downl oads/ EN/ A uf si c htsr ec ht/ dl_ Pr inz ipien publi catio nFile &v=6 (2019). Accessed 28 January 2022 papier_ BDAI_ en. html (2021). Accessed 3 March 2023. 25. De Maio, F.G.: Income inequality measures. J. Epidemiol. Com- 6. Balayn, A., Gürses, S., Beyond Debiasing: Regulating AI and its munity Health. 61, 849–852 (2007) Inequalities. Accessed 16 December 2022 26. Druzhkov, P.N., Kustikova, V.D.: A survey of deep learning 7. Barocas, S., Hardt, M., Narayanan, A.: Fairness in machine learn- methods and software tools for image classification and object ing. Nips Tutorial. 1/2 (2017) detection. Pattern Recognit. Image Anal. 26, 9–15 (2016) 8. Barocas, S., Hardt, M., Narayanan, A. Fairness and machine 27. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fair- learning. https:// fairm lbook. org/ (2019). Accessed 30 August ness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, 214–226 (2012) 9. Bellamy, R.K.E., Dey, K., Hind, M., Hoffman, S.C., Houde, S., 28. Dworkin, R.: What is equality? Part 1: Equality of welfare. Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Philos. Publ. Aff. 2, 185–246 (1981) Nagar, S., Ramamurthy, K.N., Richards, J., Saha, D., Sattigeri, 29. Eidelson, B.: Discrimination and Disrespect. OUP, Oxford P., Singh, M., Varshney, K.R., Zhang, Y.: AI Fairness 360: an (2015) extensible toolkit for detecting and mitigating algorithmic bias. 30. European Banking Authority, Final Report – Guidelines on Loan IBM J. Res. Dev. 63, 1–15 (2019) Origination and Monitoring, https:// www. eba. europa. eu/ sites/ 10. Bies, R.J., Moag, J.F.: Interactional Justice: Communication Cri- defau lt/ docum ents/ files/ docum ent_ libra ry/ Publi catio ns/ Guide teria of Fairness Research on Negotiation in Organizations, 1st lines/ 2020/ Guide lines% 20on% 20loan% 20ori ginat ion% 20and% edn., pp. 43–55. JAI Press, Greenwich (1986) 20mon itori ng/ 884283/ EBA% 20GL% 202020% 2006% 20Fin al% 11. Binns, R.: Fairness in machine learning: lessons from political 20Repor t%20on% 20GL% 20on% 20loan% 20or iginat ion% 20and% philosophy. Proc. Mach. Learn. Res. 81, 1–11 (2018) 20mon itori ng. pdf (2020). Accessed 02 March 2023 12. Breck, E., Polyzotis, N., Roy, S., Whang, S., Zinkevich, M.: Data 31. European Commission. Proposal for A regulation of the Euro- validation for machine learning. Proceedings of the 2nd SysML pean Parliament and of the council laying down harmonized conference, Palo Alto, CA, USA (2019) rules on artificial intelligence (artificial intelligence act) and 13. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dha- amending certain union legislative acts, https:// eur- lex. europa. riwal, P., et al.: Language models are few–shot learners. Adv. eu/leg al-conte nt/ EN/ TXT/? ur i=cele x%3A520 21PC0 206 (2021). Neural Inf. Process. Syst. 33, 1877–1901 (2020) Accessed 18 March 2022 1 3 AI and Ethics 32. Ferguson, J.L., Ellen, P.S., Bearden, W.O.: Procedural and dis- Academic, Oxford (2021). https:// doi. or g/ 10. 1093/ oxfor dhb/ tributive fairness: Determinants of overall price fairness. J. Bus. 97801 98857 815. 013. 21 Ethics. 121, 217–231 (2014) 52. HLEG. HLEG on AI. Ethics guidelines for trustworthy AI. 33. Fisher, E., Fuchs, C.: Reconsidering Value and Labor in the Digi-https://digit al- s trategy .ec. eur opa.eu/ en/ libr ar y/e thics-guide lines- tal Age. Palgrave Macmillan, Basingstoke (2015)trust worthy- ai (2019). Accessed 10 May 2022 34. Folger, R., Konovsky, M.A.: Effects of procedural and distribu- 53. HLEG. Assessment List for Trustworthy AI (Altai) for self– tive justice on reactions to pay raise decisions. Acad. Manag. J. assessment. https:// digit al- s tr at egy . ec. eur opa. eu/ en/ libr a r y/ 32, 115–130 (1989)assessment- lis t-tr ustw orthy-ar ti ficial-intel lig ence- alt ai-self- asses 35. Friedler, S.A., Scheidegger, C., Venkatasubramanian, S., sment (2020). Accessed 01 December 2022 Choudhary, S., Hamilton, E.P., Roth, D.: A comparative study 54. ISO/IEC  TR  24027:2021, Information technology — Artifi- of fairness–enhancing interventions in machine learning. In: cial intelligence (AI) — Bias in AI systems and AI aided deci- Proceedings of the conference on fairness, accountability, and sion making. https:// www. iso. org/ stand ard/ 77607. html (2021). transparency, 329–338 (2019) Accessed 22 December 2022 36. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wal- 55. ISO/IEC TR. 24028:2020, Information technology — Artificial lach, H., Iii, I.: Datasheets for datasets. Commun. ACM. 64, intelligence — Overview of trustworthiness. https:// www . iso. or g/ 86–92 (2021)stand ard/ 77608. html (2020). Accessed 22 December 2022 37. Genovesi, S., Mönig, J.M.: Acknowledging sustainability in the 56. ISO/IEC TR 24368:2022, Information technology — Artificial framework of ethical certification for AI. Sustainability. 14, 4157 intelligence — Overview of ethical and societal concerns. https:// (2022)www.iso. or g/s tandar d/78507. html (2022). Accessed 22 Decem- 38. Gilliland, S.W.: The perceived fairness of selection systems – ber 2022 an organizational justice perspective. Acad. Manag. Rev. 18, 57. Kamiran, F., Calders, T.: Data pre–processing techniques for 694–734 (1993) classification without discrimination. Knowl. Inf. Syst. 33, 1–33 39. Giovanola, B., Tiribelli, S.: Beyond bias and discrimination: (2012) redefining the AI ethics principle of fairness in healthcare 58. Kamiran, F., Žliobaitė, I., Calders, T.: Quantifying explainable machine learning algorithms. AI & Soc, 1–15 (2022) discrimination and removing illegal discrimination in automated 40. Gomez, O., Holter, S., Yuan, J., Bertini, E.: ViCE: visual coun- decision making. Knowl. Inf. Syst. 35, 613–644 (2013) terfactual explanations for machine learning models. In: Pro- 59. Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness–aware ceedings of the 25th international conference on intelligent user classie fi r with prejudice remover regularizer. In: Flach, P.A., Bie, interfaces, 531–535 (2020) T., Cristianini, N. (eds.) Joint European Conference on Machine 41. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Learning and Knowledge Discovery in Databases, pp. 35–50. press, Cambridge (2016) Springer, Berlin (2012) 42. Gould, E., Schieder, J., Geier, K., What is the gender pay gap 60. Kasirzadeh, A. Algorithmic fairness and structural injustice. and is it real? Economic Policy Institute. https://files. epi. or g/pdf/ Insights from Feminist Political Philosophy. Proceedings of 112962. pdf (2016). Accessed 05 September 2022 the 2022 AAAI/ACM Conference on AI, Ethics, and Society 8, 43. Greenberg, J., Cropanzano, R.: The social side of fairness: inter- 349–356 (2022) personal and informational classes of organizational justice. In: 61. Kilbertus, N., Rojas, C.M., Parascandolo, G., Hardt, M., Janz- Cropanzano, R. (ed.) Justice in the Workplace: Approaching ing, D., Schölkopf, B.: Avoiding discrimination through causal Fairness in Human Resource Management, pp. 79–103. Law- reasoning. Adv. Neural Inf. Process. Syst. 30, 656–666 (2017) rence Erlbaum Associates, Hillsdale (1993) 62. Konovsky, M.A.: Understanding procedural justice and its impact 44. Grgić–Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: The on business organizations. J. Manag. 26, 489–511 (2000) case for process fairness in learning: feature selection for fair 63. Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual decision making. NIPS symposium on machine learning and the fairness. Adv. Neural Inf. Process. Syst. 30, 2 (2017) law (2016) 64. Lahoti, P., Gummadi, K.P., Weikum, G.: ifair: learning individu- 45. Grgić-Hlača, N., Zafar, M.B., Gummadi, K.P., Weller, A.: ally fair data representations for algorithmic decision making. Beyond distributive fairness in algorithmic decision making: fea- 35th international conference on data engineering (icde), IEEE ture selection for procedurally fair learning. AAAI. Proceedings Publications, 1334–1345 (2019) of the AAAI conference on artificial intelligence 32. 32 (2018) 65. Lee, M.S.A., Floridi, L.: Algorithmic fairness in mortgage lend- 46. Gudivada, V., Apon, A., Ding, J.: Data quality considerations for ing: from absolute conditions to relational trade–offs. Minds big data and machine learning: Going beyond data cleaning and Mach. 31, 165–191 (2021) transformations. Int. J. Adv. Softw. 10, 1–20 (2017) 66. Leventhal, G.S.: The distribution of rewards and resources in 47. Häberlein, L., Mönig, J.M., Hövel, P.: Mapping stakeholders and groups and organizations. Adv. Exp. Soc. Psychol. 9, 91–131 scoping involvement. A guide for HEFRCs, Deliverable 3.1 of (1976) the H2020–project ETHNA System. https://e thnasy stem.eu/ wp- 67. Leventhal, G.S.: What should be done with equity theory? content/ uploa ds/ 2021/ 10/ ETHN A_2021_ d3.1- s takeholde r mapp In: Gergen, K.J., Greenberg, M.S., Willis, R.H. (eds.) Social ing_ 21100 11. pdf (2021) Accessed 10 May 2021 exchange, pp 27–56, Springer, Boston (1980) 48. Hacker, P.: Teaching fairness to artificial intelligence: existing 68. Lippert-Rasmussen, K.: Born Free and Equal? A Philosophi- and novel strategies against algorithmic discrimination under EU cal Inquiry into the Nature of Discrimination. OUP, New York law (April 18, 2018). Common Mark. Law Rev. 55, 1143–1186 (2013) (2018) 69. Madaio, M.A., Stark, L., Wortman Vaughan, J.W., Wallach, H.: 49. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in super- Co– designing checklists to understand organizational challenges vised learning. Adv. Neural Inf. Process. Syst. 29, 2 (2016) and opportunities around fairness in AI. In: Proceedings of the 50. Haws, K.L., Bearden, W.O.: Dynamic pricing and consumer fair- 2020 CHI conference on human factors in computing systems, ness perceptions. J. Consum. Res. 33, 304–311 (2006) 1–14 (2020) 51. Herzog, L.: Algorithmic bias and access to opportunities. In: 70. Mathur, P., Sarin Jain, S.S.: Not all that glitters is golden: the Véliz, C. (ed.) The Oxford Handbook of Digital Ethics. Oxford impact of procedural fairness perceptions on firm evaluations 1 3 AI and Ethics and customer satisfaction with favorable outcomes. J. Bus. Res. quantifying algorithmic unfairness: measuring individual 117, 357–367 (2020) &group unfairness via inequality indices. In: Proceedings of the 71. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, 24th ACM SIGKDD international conference on knowledge dis- A.: A survey on bias and fairness in machine learning. ACM covery & data mining, 2239–2248 (2018) Comput. Surv. 54, 1–35 (2021) 90. Suresh, H., Guttag, J.: A framework for understanding sources of 72. Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: harm throughout the machine learning life cycle. Equity Access The ethics of algorithms: mapping the debate. Big Data Soc. Algor. Mech. Optim. 17, 1–9 (2021) (2016). https:// doi. org/ 10. 1177/ 20539 51716 679679 91. Taylor, L.: What is data justice? The case for connecting digital 73. Molnar, C. Interpretable machine learning. A guide for making rights and freedoms globally. Big Data Soc. (2017). https:// doi. black box models explainable, https://c hrist ophm.git hub.io/ inter org/ 10. 1177/ 20539 51717 736335 preta ble- ml- book/ (2022). Accessed 22 December 2022 92. Thibaut, J.W., Walker, L.: Procedural Justice: A Psychological 74. Mönig, J.M.: Privatheit als Luxusgut in der Demokratie? In: Analysis. Erlbaum Associates, Hillsdale (1975) Grimm, P., Zöllner, O. (eds.) Demokratie und Digitalisierung, 93. van Wynsberghe, A.: Sustainable AI: AI for sustainability and pp. 105–114. Steiner, Stuttgart (2020) the sustainability of AI. AI Ethics. 1, 213–218 (2021) 75. Myers West, S.: Redistribution and recognition. A feminist cri- 94. VDE SPEC 90012:2022 VCIO based description of systems for tique of algorithmic fairness. Catalyst. (2020). https://doi. or g/10. AI trustworthiness characterization. https://www .vde. com/ r esou 28968/ cftt. v6i2. 33043rce/ blob/ 21766 86/ a24b1 3db01 77374 7e6b7 bba4c e20ea 60/ vde- 76. Noble, S.U.: Algorithms of oppression: how search engines rein-spec- vcio- based- descr iption- of- syste ms- for- ai- trust worth iness- force racism. New York University Press, New York (2018)chara cteri sation- data. pdf (2022). Accessed 01 December 2022 77. Pillai, R., Schriesheim, C.A., Williams, E.S.: Fairness percep- 95. Véliz, C.: Privacy is Power: Why and How You Should Take tions and trust as mediators for transformational and transactional Back Control of Your Data. Bantam Press, London (2020) leadership: A two–sample study. J. Manag. 25, 897–933 (1999) 96. Verma, S., Rubin, J.: Fairness definitions explained. Proceedings 78. Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., Weinberger, of the international workshop on software fairness, 1–7 (2018) K.Q.: On fairness and calibration. Adv. Neural Inf. Process. Syst. 97. Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explana- 30, 2 (2017) tions without opening the black box: Automated decisions and 79. Poppo, L., Zhou, K.Z.: Managing contracts for fairness in buyer– the GDPR. SSRN J. 31, 841–888 (2018) supplier exchanges. Strat. Manag. J. 35, 1508–1527 (2014) 98. Wachter, S., Mittelstadt, B., Russell, C.: Why fairness cannot 80. Poretschkin, M., Schmitz, A., Akila, M., Adilova, L., Becker, D., be automated: bridging the gap between EU non–discrimination Cremers, A.B., Hecker, D., Houben, S., Mock, M., Rosenzweig, law and AI. Comput. Law Secur. Rev. (2021). https://doi. or g/10. J. et al. Leitfaden zur Gestaltung vertrauenswürdiger Künstlicher 1016/j. clsr. 2021. 105567 Intelligenz. https:// www. iais. fraun hofer. de/ conte nt/ dam/ iais/ fb/ 99. Wahlster, W., Winterhalter, C.: Deutsche Normungsroadmap. Kuens tliche_ intel ligenz/ ki- pruef katal og/ 202107_ KI- Pruef katal Künstliche Intelligenz. DIN. https:// www. dke. de/ resou rce/ blob/ og. pdf (2021). Accessed 10 March 202120194 82/ 0c291 25fa9 9ac4c 897e2 809c8 ab343 ff/ nr- ki- deuts ch--- 81. Rajkomar, A., Hardt, M., Howell, M.D., Corrado, G., Chin, downl oad- data. pdf (2020). Accessed 21 December 2022 M.H.: Ensuring fairness in machine learning to advance health 100. Wand, Y., Wang, R.Y.: Anchoring data quality dimensions in equity. Ann. Intern. Med. 169, 866–872 (2018) ontological foundations. Commun. ACM. 39, 86–95 (1996) 82. Rawls, J.: A Theory of Justice. Harvard University Press, Cam- 101. Young, I.M.: Justice and the Politics of Difference. Princeton bridge MA (1999) University Press, Princeton (1990) 83. Rawls, J.: Justice as Fairness: A Restatement. Harvard University 102. Yu, K.H., Beam, A.L., Kohane, I.S.: Artificial intelligence in Press, Cambridge MA (2001) healthcare. Nat. Biomed. Eng. 2, 719–731 (2018) 84. Rohde, F. et al.: Nachhaltigkeitskriterien für künstliche Intelli- 103. Zafar, M.B., Valera, I., Gomez Rodriguez, M., Gummadi, K.P.: genz. Entwicklung eines Kriterien– und Indikatorensets für die Fairness beyond disparate treatment & disparate impact: learning Nachhaltigkeitsbewertung von KI–Systemen entlang des Leb- classification without disparate mistreatment. Proceedings of the enszyklus. Schriftenr. IÖW. 220/21. https:// www. ioew. de/ filea 26th international conference on world wide web, 1171–1180 dmin/ user_ upload/ BILDER_ und_ Downl oadda teien/ Publi katio (2017) nen/2021/ IOEW_ SR_ 220_ N achhaltig k eitskr iter ien_fuer_ K uens 104. Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning tliche_ Intel ligenz. pdf (2021). Accessed 18 January 2022 fair representations. International Conference on Machine Learn- 85. Schürnbrand, J.: Organschaft im Recht der privaten Verbändecht. ing, 325–333 (2013) Mohr Siebeck, Tübingen (2007) 105. Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted 86. Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., Hall, biases with adversarial learning. Proceedings of the 2018 AAAI/ P. Toward a standard for identifying and managing bias in artifi- ACM Conference on AI, Ethics, and Society, 335–340 (2018) cial intelligence, National Institute of Standards and Technology. https:// nvlpu bs. nist. gov/ nistp ubs/ Speci alPub licat ions/ NIST. SP. Publisher's Note Springer Nature remains neutral with regard to 1270. pdf (2022). Accessed 21 December 2022 jurisdictional claims in published maps and institutional affiliations. 87. Sen, A.: Inequality Reexamined. Clarendon Press, Oxford (1992) 88. Skarlicki, D.P., Folger, R., Tesluk, P.: Personality as a modera- tor in the relationship between fairness and retaliation. Acad. Manag. J. 42, 100–108 (1999) 89. Speicher, T., Heidari, H., Grgić–Hlača, N., Gummadi, K.P., Singla, A., Weller, A., Zafar, M.B.: A unified approach to 1 3

Journal

AI and Ethics – Springer Journals

Published: May 8, 2023

Keywords: Artificial intelligence; Data science; Fairness; Fairness metric; Standardization

There are no references for this article.