We performed a thorough survey of the recent literature, which we classified depending upon which area of the pipeline they analyze. We gathered papers from NEURIPS, ICML, ICLR, FAccT, AIES, EAAMO, CHI, and CSCW for the past five years, i.e. 2018-2022, that contain any of the following terms in their title, abstract, or keywords: "fairness", "fair", "discrimination", "disparity", "equity". In addition, we performed a series of Google Scholar searches to ensure our survey did not miss high-impact work published in other venues: One search used the keywords listed above and included papers published in any venue in the past five years with over 50 citations, through the top 50 results returned by this Google Scholar search. Additional Google Scholar searches used keywords from each step in the pipeline individually, to attempt to find papers targeted at each stage: for example, "data collection" and "fairness" and "machine learning". This results in ~1000 papers overall which fit our search criteria.
We then manually inspect each paper to understand whether and how the reported research is an instance of a pipeline-aware approach to fairness. That is, does it identify, measure, or mitigate a concrete cause of unfairness due to choices made in a specific stage of the ML pipeline. If so, we categorize the paper along two axes: what part of the pipeline it corresponds to (problem formulation, data choice, feature engineering, statistical modeling, testing and validation, or organizational realities), and whether it identifies, measures, mitigates, or provides a case study of a pipeline-based fairness problem. By identifying a pipeline fairness problem, we refer to papers that point to a previously unobserved source of bias on the machine learning pipeline either through theory or through experimentation with training pipelines on common machine learning datasets; by measuring a pipeline fairness problem, we refer to papers that provide a generalized technique for how to identify or gauge the magnitude of a specific source of unfairness along the machine learning pipeline; by mitigating, we mean paper which develop a technique for addressing a source of bias along the AI pipeline when it arises; and by a case study we refer to an example of how a choice on the machine learning pipeline lead to unfairness in a specific application, often on an already deployed model. Of our approximately 1000 papers, ~300 satisfied our criteria of being ``pipeline-aware'' approaches to fairness.
We present our full categorization of all the papers that we found related to pipeline-aware fairness, broken down into what stage of the pipeline they were most related to, across the stages presented inthis website.