Comments Received on A Proposal for Identifying and Managing Bias in Artificial Intelligence (SP 1270)

1

President of The Foundation for Sustainable Communities & Sr. Adjunct Professor

Deborah Hagar, MBA

319

A key link to identifying and establishing accountability for bias is in the decision making process - human interventions.  The accountability link for results is the key.  Ex. Sarbannes-Oaxley idientified responsibility.

President of The Foundation for Sustainable Communities & Sr. Adjunct Professor

Deborah Hagar, MBA

397

Ensuring the identification and full engagement across domains is critical to both accountability and rationale for proposed decisions (i.e., transparency)

President of The Foundation for Sustainable Communities & Sr. Adjunct Professor

Deborah Hagar, MBA

410

This is the key!  The full cycle of implementation is the true test that the outputs result in the intended AND desired outcomes for the identified Stakeholders.

President of The Foundation for Sustainable Communities & Sr. Adjunct Professor

Deborah Hagar, MBA

595

Again, the clear accountability that the initial problems are resolved, without a zero-sum game of negative impact on select stakeholders, will result in desired outcomes AND effective Stakeholder Capitalism.

2

Bayana Corporation

Joseph S. Bayana

198

to cultivate trust --- There is an assumption that artificial intelligence has big data, and these big data has equal representation. AI can be "trusted" only if the big data is actual and real-world based.

Bayana Corporation

Joseph S. Bayana

199

accuracy, "reliability" -- once again, there is an assumption that big data for AI has already been accumulated and substantiated. There are no actual, real-world sources for many data used currently for AI.

Bayana Corporation

Joseph S. Bayana

203

to understand and reduce harmful forms of AI bias -- there must be a minimum amount of big data with ample or sufficient representation from various sources (e.g. male, female, young, old, race, etc.) before AI can be used or professed as AI.

Bayana Corporation

Joseph S. Bayana

222

The presumption is that bias is present throughout AI systems, the challenge is identifying, measuring, and managing it.  --- once again, there is an assumption that AI has reliable big data.

Bayana Corporation

Joseph S. Bayana

223

There is also the assumption that methodology, "type," and "industrial sector," to name a few, are sufficient to proceed with AI, when in reality no big data is available.

Bayana Corporation

Joseph S. Bayana

234

proliferation of modeling and predictive approaches based on data-driven and machine learning techniques has helped to expose various social biases baked into real-world systems --- there are no reliable big data sources for "modeling," "predictive approaches," "techniques."

Bayana Corporation

Joseph S. Bayana

233

Just like a minimum viable product or prototype in manufacturing and/or industry, there must be a minimum amount of viable big data that can substantiate AI models, approaches, techniques, et al.

Bayana Corporation

Joseph S. Bayana

247

higher ethical standard -- an AI testing standard or measure should be done to check the minimum amount of viable big data, and to determine if the AI model, approach, and/or technique is based on factual, real-world data rather than a model or simulation.

Bayana Corporation

Joseph S. Bayana

251

Our group's research has data to substantiate that healthcare currently does not have enough big data to substantiate healthcare as we know it, much less substantiate healthcare AI. Healthcare industry refers to evidence-based practices but healthcare AI does not have real-world data nor does it have person-centered big data.

Bayana Corporation

Joseph S. Bayana

251

Healthcare big data, as it exists today, is not fact-based coming from direct, actual, real-world sources. Healthcare big data's methodology is plain and simple word search,thereafter wrapped up and presented as big data.

Bayana Corporation

Joseph S. Bayana

267

specific conditional traits associated with automation that exacerbate distrust in AI tools --- there are 1.5 million words in the English language, with limitless combinations and permutations of nouns, pronouns, verbs, adjectives, adverbs, prepositions, articles, determiners, etc. Buttressed with academic prestige coming from an author or writer, a write-up can be passed off as an AI article.

Bayana Corporation

Joseph S. Bayana

280

What's stopping the NIST and other groups, agencies, etc. from setting up "direct measures" that gauge, rate, and/or measure AI and its big data sources?

Bayana Corporation

Joseph S. Bayana

324

describe technology that is based on questionable concepts, deceptive or unproven practices, or lacking theoretical underpinnings --- Our group has the same findings but focusing more on healthcare big data and AI in healthcare.

Bayana Corporation

Joseph S. Bayana

335

Bias in healthcare AI can come from lack of primary sources, and real-world, fact- and evidence-based data, including the absence of actual sources from physical or chemical data.

Bayana Corporation

Joseph S. Bayana

410

1. Pre-design --- Assumes data is reliable, substantiated, and without bias.

Bayana Corporation

Joseph S. Bayana

411

2. DESIGN AND DEVELOPMENT: where the technology is constructed  --- data is already biased.

Bayana Corporation

Joseph S. Bayana

412

3. DEPLOYMENT: where technology is used by, or applied to, various individuals or groups. == Data is already biased.

Bayana Corporation

Joseph S. Bayana

424

Pre-design stage --- there should be a minimum viable data amount for all AI at the pre-design stage to ensure that the data is not biased nor are the data not just made up.

Bayana Corporation

Joseph S. Bayana

440

If at the pre-design stage there are no reliable big data sources, the AI bias is already present and all other stages after this will have bias.

Bayana Corporation

Joseph S. Bayana

506

Improving pre-design practices --- once again, there should be minimum amount of viable data from actual, real-world, fact-based primary sources.

Bayana Corporation

Joseph S. Bayana

615

 small number of participants --- most, if not all, AI sources rely on synonymous descriptions to substantiate their claims. In other words, there are really no standards or measurements for minimum viable data used for most, if not, all AI literature and/or research.

Bayana Corporation

Joseph S. Bayana

683

development of standards and a risk-based framework. --- once again, there should be minimum viable data amount as the "standard" in order to create the lowest benchmark for any and all "frameworks."

Bayana Corporation

Joseph S. Bayana

707

NIST will collaboratively develop additional guidance --- our group would like to work with NIST to develop a "standard" and/or "framework" to avoid AI bias in healthcare.

Bayana Corporation

Joseph S. Bayana

730

The biases we've identified in healthcare can be categorized as content production bias, data generation bias, exclusion bias, historical bias, inherited bias, institutional bias, error propagation, systemic bias, Loss of situational awareness bias, et al.

3

Comment out of scope

5

IEEE Life Fellow

Doug Verret PhD

106

This describes the desired attributes of an AI system, but in your reductionist, life-cycle approach you have no way to detect biases that arise at the interstices between stages or may not be able to be localized.  You need some form of traceability or schema that comprehends the interfaces.

IEEE Life Fellow

Doug Verret PhD

424

In software engineering circles the terminology "V and V" is used routinely.  You use "validation" terminology in your deployment stage, but you do not use "verification" with its specific technical meaning in the Pre-Design stage. The verification process targets software architecture, design, databases, input translation etc. and uses reviews, walk-throughs, inspections and desk-checking, but not coding.  Why not use the vocabulary that professional engineers recognize?

IEEE Life Fellow

Doug Verret PhD

462

A major flaw in the way problems are conceptualized here is the lack of recognition that in machine learning approaches there is no theoretical foundation for how neural nets work.  That is, machine-learning AI tools produce "solutions" or "decisions" or "recommendations" that are unpredictable in many cases and not obvious or explainable by even the developer.  On the one hand if it were obvious you wouldn't need the tool, but on the other hand it is not uncommon for the developer to be unable to explain how an output was arrived at other than mechanistically.  How then do you propose to discover bias in the pre-design stage when the code is not even written and there is no theory that is not inscrutable?

IEEE Life Fellow

Doug Verret PhD

638

This seems to suggest that systems should be designed with unintended uses in mind.  Engineers design systems to a spec and in a sense guarantee the spec.  It is utter folly to design a system for "off-road" use.  The possibilities are infinite and such a design is impossible.  Engineers can design robust systems whereby the system is protected from misuse but cannot design system that protect users from abuse and from harm.  This may not be what is intended by this sentence, but it should be clarified.

IEEE Life Fellow

Doug Verret PhD

702

As you have aptly pointed out, determining the qualities of and biases in AI tools is often context-dependent. If you include "off-road" use cases the possibilities are endless.  What I am missing in this document is a clear set of principles that teaches me how to generalize without having to examine each use case one at a time.  You want to be flexible and context-aware and at the same time generalize above all of these cases.  Other than "be aware of this" and "be aware of that" I am not getting how you want me to do that.

6

Rapid7

STUART MILLAR

Hello, thank you for reading my comments.  I appreciate the first draft is broad. I feel as a community we have the change to get into the low-level detail of how to mitigate bias, with a framework and also detailed process and where needed algorithm technical solutions to deal with bias.

Rapid7

STUART MILLAR

424

This pre-design stage also needs to consider the dataset itself being used for the use-case.  Is it being made from scratch vs. being inherited, is it labelled, if so who labelled it.  How accurate are those labels?  What does the class balance look like?  The lineage of the data should be tracked, consistently captured and have an audit trail.  A broader question is for the given dataset in question, figure out how we define, measure and deal with bias in that data. 

Rapid7

STUART MILLAR

424

At this stage one can also consider any legal requirements already existing to mitigate against bias, depending on the use-case.  If there is a legal requirement for the algorithm to be explainable/interpretable (in case of court case triggered by bias), that law is likely there for a reason.  If this is the scenario, consider very carefully whether this is the right-use case.  Be aware early on of any and all legal requirements across geographical regions that will affect your algorithm.

Rapid7

STUART MILLAR

426

Have a detailed and documented (in case of audit) discussion with stakeholders on:

What is the main intended objective?
What are any other intended objectives?
What possible learning outcomes do we need to mitigate against?
What is the specification of the algorithm?

Rapid7

STUART MILLAR

426

Decide what would a human review entail exactly to check for bias (at any point in the project) if artefacts (such as the data, or a trained model) were available for the closest of inspections.

Rapid7

STUART MILLAR

510

I feel like we need to recognise that algorithms are software.  We are building models and in essence writing software.  Best-practice for software is that is needs to go through rigorous, documented QA, meaning:

1. The algorithm does what it is meant to.
2. The algorithm learns what it is meant to.
3. The algorithm does not do what it is not meant to do.
4. The algorithm does not learn what it is not meant to learn.

I think point 4 is particularly poignant and we should be cognizant of that.

Rapid7

STUART MILLAR

574

In the deployment stage, most of what I have written in the previous comments still holds true, and extra tooling will be needed in dashboards etc to measure selected metrics that are identified as being indicators of bias, from class balance in datasets through to the distribution of predicts, data/concept and everything in-between.

Rapid7

STUART MILLAR

I feel a future framework should have technical algorithmic suggestions for dealing with bias.  For example, if an image classifier was learning from pictures of people, and we only wanted to predict their age, there are other learnings we may want to negate.  For example gender, and race.  One approach could be to label the training data with this information and actively try to negate the learning of these traits during training, for example with multi-task learning negating the appropriate gradients.

Rapid7

STUART MILLAR

Two interesting papers that are relevant:

Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction
Christina Wadsworth, Francesca Vera, Chris Piech

Does Object Recognition Work for Everyone?
Terrance DeVries, Ishan Misra, Changhan Wang, Laurens van der Maaten

7

John F. Raffensperger

I congratulate the authors on an interesting and important draft. You have outlined some important ideas and called out the long list of biases possible in AI. I hope this work expands and strengthens.
I am an operations researcher with experience in public policy, including problems of the commons. I know much about AI but am not a recognized expert in it. Further, I have some ignorance about your role and responsibilities. Feel free to ignore (or forward to some more appropriate party) any of comments here, which are my own.
Apologies for this format. I discovered your comments template after I had nearly finished writing!

John F. Raffensperger

1 Private versus public risk.

Your stated main audience is researchers and practitioners. These two groups probably don’t belong together in this way. Researchers don’t injure others very much through their work and they usually have little incentive to allow bias in their code; they even have incentive to call it out. But practitioners can injure a great many people through the enormous scale of their deployments, and practitioners have profit-seeking incentives to allow bias in code.
“Practitioners will benefit by gaining an understanding about bias in the use of AI systems.” I believe practitioners already have a very good understanding of the bias, as observed by the energetic discussion in the news media and scientific literature! But practitioners are not well motivated to eliminate bias. So I am skeptical (even cynical) of the benefit in explaining AI bias to practitioners.
“Risk-based framework…” Your paper describes the risk of bias in AI. Please expand on the distinct risks to each stakeholder group. (As I will explain below, your paper should name the various stakeholders clearly and consistently.) Most importantly, please distinguish private business risk from public risk.
Private firms engages in profit-seeking behavior which injures consumers. Thus, bias in AI is a problem of the commons analogous to a polluting firm or a restaurant selling bad meat. In doing so, the firm raises small risk to their own reputation while enjoying higher profits. But the injury to consumers can be large. Further, this injury is insidious, hidden, and hard to measure, like a toxin released to groundwater or like late-night stomach upset. Your paper mentions risks to firms and risks to users (“consumers” here), but the difference between them is much more important than your paper conveys.
Specifically, how the presence of bias in automated systems can contribute to harmful outcomes and a public lack of trust.
…How bias and trust interrelate is a key societal question…
The regulator need not care whether the public trusts an AI run by a business; the business should care because the business wants the consumers’ trust. The regulator should care only whether the automated business system harms the public; interaction of trust and bias is irrelevant to the regulator. In fact, your efforts to improve trust in AI could backfire by getting consumers to believe the AI is fine when in fact the AI is bad. Rather than encouraging the consumer to accept the validity of AI uncritically (“building trust”), perhaps the regulator’s job should include teaching the consumer to be skeptical of AI, to help them question the business about how their software works.
“Bias reduction techniques are needed that are flexible…” Again, the private company is responsible for finding their own bias reduction techniques. You are kind to try to give them techniques, but helping developers write software is not the regulator’s main job. The regulator should penalize bias. Knowles and Richards (your reference) have it exactly right, but your paper doesn’t follow through on their point. You need not suggest “resilience,” but rather please talk about regulating.
Here’s an interesting edge case: “interviews with admissions officials suggest that ‘they didn’t believe in the validity of the risk scores…’” So the school knew their AI was biased and should have fixed it. If they fixed it quickly, the regulator probably could stand idle. If they didn’t fix it, the regulator should hammer them for knowingly making biased admissions decisions.
“Contextual gaps lead to performance gaps… intention…” The regulator should not have to worry about intention until the court proceedings; the regulator should strenuously worry about performance.
“One challenge rests on the reality that decisions about which data to use for these indices are often made based on what is available or accessible,…” Again, you are generous to help the developers think about their data inputs. This is like telling Big Boy’s Burgers to monitor their refrigerator temperature. It’s helpful, but not the regulator’s main concern. More on Big Boy’s shortly.
You list core building blocks of accuracy, explainability and interpretability, privacy, reliability, robustness, safety, and security (resilience), and mitigation of harmful bias. But the costs in these blocks to business differ from their costs to the public.
• Accuracy, reliability, and robustness can cost the company and the public. The business will have incentive to improve these. That is, they will make sure the system works the way they want it to. These blocks could benefit or injure the public, but they are essentially a private problem.
• Explainability and interpretability cost the company but benefit the public. A business may have huge incentive to avoid explaining their system to the public, and even their managers and board members! The regulator could choose to require a firm to explain and interpret their AI to the firm’s customers as some kind of formal disclosure. I expect the firms would find such disclosure to be onerous, fraught with complications, and expensive. Can you write an example disclosure, say, from a hiring firm to its applicants? How could the regulator assess these disclosures?
• Privacy protects mainly the public. It is a high cost to the business and not in their interests. Easily avoided with a long EULA. The firm’s loss of privacy can benefit the public!
• Whose security and safety do you mean? These sound positive but seem vague. “Security” sounds like mainly a private benefit while “safety” sounds like a public benefit.
These “blocks” look like a great list but scattershot with their costs and benefits labeled as private or public.
“The intention is to leverage key locations within stages of the AI lifecycle for optimally identifying and managing bias.” Your AI lifecycle omits the most important part: when the AI is deployed. And this is why your paper doesn’t seem to really dig into the regulatory problem.
I have in mind a simple Big Boy’s Bad Burgers model. When a restaurant wants to save some money, the manager can be tempted to sell meat they know is bad. If the city health inspectors discover this behavior, the inspectors can warn, fine, or shut down the store. That is regulation. Please work through some simple scenarios of bias. What should the regulator do on finding bias in mortgage lending or job hiring?
Your paper recognizes bad beef in only a couple places — “In extreme cases, with tools or apps that are fraudulent, pseudoscientific, prey on the user, or generally exaggerate claims, the goal should not be to ensure tools are bias-free, but to reject the development outright in order to prevent disappointment or harm to the user as well as to the reputation of the provider.” These folk won’t reject the development! They want to prey on the user. For firms who give a bit more than lip service to avoiding fraud, “[e]xpertise matters…”, but it’s also expensive, when the developers are on short deadline.
“Improving pre-design practices to ensure more inclusive representation” is akin to telling restaurants to keep their refrigerators sufficiently chilled, to train managers to identify bad meat, and to discard meat they believe to be bad. All good ideas and helpful in mitigating bias. But what should the regulator do when the restaurant keeps selling bad burgers? Your paper doesn’t address this at all.

John F. Raffensperger

2 Please declare the actors. 

I began my review as though I were refereeing a journal manuscript. In the first read, I simply look for clarity, improvable sentences, etc. In your generally well-written paper, I found some passive voice. At first, the passive voice seemed like issues of minor copy-editing. But as I went on, it became worrisome as I will explain. Bear with me.
The writing tics “There is” and “There are” get tiresome. Search & destroy as follows:
• “There are specific conditional traits associated with automation that exacerbate distrust in AI tools” would be better as “Specific conditional traits associated with automation exacerbate distrust in AI tools.”
• “There are many challenges that come with this common practice” would be better as “Many challenges come with this common practice”.
• “There are also examples from the literature which describe technology that is based on ….” would be better as “Examples from the literature describe technology that is based on …”.
• “…there is increasing evidence that…” would be better as “…evidence is increasing that…”.
Passivizing perfectly good verbs:
• “bias is often connected” would be better as “bias often connects”.
• “… what is required for building systems…” would be better as “what building systems require” or “what system builders require” or “what developers require”.
• “Even if datasets are reflective of the real world…” would be better as “Even if datasets reflect the real world…”.
• “Improving trust in AI systems can be advanced…” would be better as “Improving trust in AI systems can advance…”.
Backwards passive sentences:
• “…many people are affected or used as inputs by AI technologies and systems” would be better as “… AI technologies and systems affect or use as inputs many people”, which now reveals the silliness of the sentence.
• “Historical, training data, and measurement biases are ‘baked-in’ to the data [sic]…” would be better as “… the data ‘bakes in’ historical training data and measurement biases…”, revealing the silly “data bakes in data”.
• “Data representing certain societal groups may be excluded in the training datasets used by machine learning applications” would be better as “Machine learning applications may use training datasets which exclude data representing certain societal groups”.
• “Indeed, there are many instances in which the deployment of AI technologies have been accompanied by concerns of whether and how societal biases are being perpetuated or amplified” would be better as “Indeed, concerns of whether and how societal biases are being perpetuated or amplified have accompanied many deployments of AI technologies”. Better yet: “Indeed, concerns of whether and how AI technologies perpetuate or amplify societal biases have often accompanied deployments”.
Omitting an important actor. Like the suggestions above, this suggestion may seem mere copy-editing, yet it takes us down a rabbit hole to major policy implications: please distinguish the various actors explicitly, e.g., owners, users, programmers, and the general public. Doing so may lend an accusatory tone in some places, but at least get each sentence to a point of clarity before toning it down. Below, I address a few phrases with passive voice and then I discuss some implications of choosing the actor.
• “… that harmful biases are mitigated.” Perhaps this example of passive voice is most important. Please state who is responsible for mitigating bias. Surely it’s not the job applicant, the Uber rider, nor the general public. Name all the actors and activate them consistently in now-passive sentences. Search for and clarify “user” everywhere, because your paper in different places uses that word to mean different folk. Here is a list of possible actors.
The Uber driver is not a “user” of the AI; Uber management is the user. Maybe you could call people like the Uber rider “consumers” in the market economy sense.
Some consumers are actually victims, not merely disadvantaged people. “Certain social groups” – many of us – have little ability to fight a large corporation imposing its EULAs and AIs on us. “This kind of systemic discriminatory pricing is perpetuated on the citizens
….”? The regulator should be busting heads! We need the regulator to shut down Big Boy, not simply suggest a “3 stage approach” to the chefs.
As your paper implies, developers have responsibility in this problem. You are talking to and about developers, but you do not emphasize their heavy responsibility. For example, NIST could certify AI programmers just as government agencies certify airline pilots and civil engineers. But this makes sense only if you identify them as having responsibility, like chefs who might cook bad meat. (Are programmers the same as developers? I think of a “programmer” as the person with hands on keyboard, while “developer” could refer to a company.)
Business executives, who manage AI implementation and control their features, should ultimately bear great responsibility. They can tolerate or perpetrate bias in AI for profit; they are often perpetrators.
“In such cases the technology can be taken out of production.” I’m sure you mean “In such cases, the business managers should take the technology out of production,” just like saying “the restaurant should stop selling bad meat.” Put this way, it’s simply naive, even with the rest of that paragraph.
You don’t discuss liability, but liability is a great motivator to executives. Liability enters the discussion as soon as we mention the business actor explicitly. Is liability sufficient to regulate bias in AI? That may be the laissez faire approach. I beg you to propose more energetic regulation! I thought about liability only as I thought about this actor, because I tried to re-word a sentence in active voice. Switching to active voice has policy implications.
What role has government? Surely regulators have a role! “Federal laws and regulations have been established to prohibit discrimination based on grounds such as gender, age, and religion.” Yet your paper omits “regulator” as well. Who regulates bias in AI? Who should regulate it? Should the regulator be NIST, another federal agency, a state agency, perhaps county or city governments?
What if the AI results in business action which violates the law? I hope someone would warn, fine, or shut down the business. Your paper has none of this.
“Federal laws and regulations have been established to prohibit discrimination based on grounds such as gender, age, and religion.” You might reword as “Federal law prohibits discrimination based on gender, age, and religion.” Why regulate bias? Because it breaks the law! Doesn’t matter whether they used “AI”.
Explain this to the developers in the introduction: “If your AI is biased, you violate Federal law. That’s a felony. We’re trying to help you stay out of jail.” They may take your “3 stage approach” more seriously. Bring the regulator and the law into your paper.
• Your paper mentions a few examples of government AI deployments. Please separate the discussion of private AI from government AI. These are different problems with different actors, costs, and risks. (Also, “expansion of AI into many aspects of public life” should probably be “expansion of AI into many aspects of commerce” or “expansion of AI into many functions of society”, because “public life” could mean only government.)
In contrast to for-profit AI, public trust matters for public AI. Even if the crime sentence calculator is unbiased and correct, our democracy requires public trust in that calculator.
Society must address bias in government AI with different tools than we use to address bias in private AI systems. I expect the tools will be court proceedings, inspectors general, and whistle-blowing researchers. How can society address intentional bias in government-operated AI systems? Please consider a full workup on the problem, a separate major section at a minimum. How can we regulate military AI?
• “Another cause for distrust may be due to an entire class of untested and/or unreliable algorithms deployed in decision-based settings. Often a technology is not tested – or not tested extensively – before deployment, and instead deployment may be used as testing for the technology.”
I’ll try to add actors: “Consumer distrust may rise from an entire class of untested and/or unreliable algorithms which businesses deploy in decision-based settings. Often the programmer did not test a technology – or did not test it extensively – before management deployed it, and instead management used deployment as testing for the technology.” This new wording raises questions of responsibility which you may want to avoid but should face.
Throughout this note, I’ve made a case for a stronger discussion about regulation, but I don’t know your role or responsibilities. If you’re going to ignore regulation, your paper should explain why you’re talking about regulation but ignoring regulation. You might change the title to “A Proposal for Developers to Identify and Manage Bias within Artificial Intelligence”. But developers share responsibility for AI bias with their handlers, bosses, and investors, so restricting your Proposal to developers won’t be the last word on this problem. At some point, perhaps in a year or so, you’ll have to address all the actors.
• “As these tools proliferate across our social systems, there has been increased interest in identifying and mitigating their harmful impacts.” The first clause has active voice, but makes AI seem to grow autonomously. Not yet, Skynet! The second clause is passive; can you document or reference this interest? Perhaps “As businesses and government agencies have deployed these tools across our social systems, the media [refs] and lawmakers [refs] have increased interest in identifying and mitigating the harmful impacts of these tools.”
• “Improving trust in AI systems can be advanced by putting mechanisms in place to reduce harmful bias in both deployed systems and in-production technology.” Who is in charge of this!?
Here is a possible rewording: “Businesses can advance consumers’ trust in AI systems by installing mechanisms to reduce harmful bias in both deployed systems and in-production technology.”
But you could instead mean “Developers can advance regulators’ trust by installing mechanisms to reduce harmful bias in their in-production technology and deployed systems.” This version has a stronger bite with this selection of actors, but this version may imply regulatory policy that you haven’t thought through or that you wish to avoid.
• “PRE-DESIGN: where the technology is devised, defined and elaborated DESIGN AND DEVELOPMENT: where the technology is constructed DEPLOYMENT: where technology is used by, or applied to, various individuals or groups.” Who does all this? Only developers?
Figure 1 vaguely mentions “stakeholders”. The section “Practices” mentions expert stakeholders. “DESIGN AND DEVELOPMENT STAGE” mentions software designers, engineers, and data scientists. “DEPLOYMENT STAGE” mentions end users, operators, subject matter experts, humans-in-the-loop, and decision-makers. Where do investors and Congress fit in?
Build a complete list of actors and include them all in your Proposal, even if your audience is only developers. Developers and their businesses have incentive to bia$ their decisions; they will be the perpetrators. Don’t be shy to say so.

John F. Raffensperger

3.1 Computational law and constrained AI

Consider the paperclip-making AI that took over the world. It had a fine objective function – to make paperclips. But it had too few constraints in its optimization. Most importantly, it had no constraint to follow the law.
Society has to require that AIs follow the law, just as society requires its member to follow the law. Computational law will prevent world domination by paperclip AIs and Skynets. We should start our implementation of computational law ASAP! Teslas should stop at stop signs. Military drones shouldn’t shoot civilians. And banking AIs shouldn’t discriminate on the basis of gender, age, or religion.
Once we think about bias in AI in terms of the law, as constraining the AI to follow the law, a lot of the process falls naturally into place.
(Technical note: as an operations researcher, I view AI as heuristic optimization over huge datasets. Among other shortcomings, modern AI as I understand it does not include the rich constraints of classical linear integer programming which can calculate propositional logic. I expect the need for full binary propositional logic in AI to implement computational law.)

John F. Raffensperger

3.2 Certify developers

Regulation of AI could eventually include certification of developers, as the FAA certifies pilots and state governments certify civil engineers. The certification could test developers for their knowledge of AI, bias, legal definitions of discrimination, and regulations. The regulator could require developers to sign their names on software specs to certify that the spec follows the law, just as a professional engineer must sign the design for a bridge. The developer could lose their coding license if the regulator finds bias in the software. This would make the developer directly accountable for bias. Businesses could then announce “Our developers are NIST-certified for bias.”

John F. Raffensperger

3.3 Certify code 

Similarly, the regulator could certify some AIs as the FAA certifies airplanes and as inspectors certify finished bridges. Because AI is deployed so widely, the regulator could require only selected “Bad Burger” businesses submit to certification should they fail to respond to warnings.
To get code certified, the business would submit their software to the regulator’s testing. The regulator could test an AI and inform the company of the results, or the regulator could provide test cases to the company and the company could return authenticated results.
The regulator could obtain test cases in several ways. The regulator could write tests. The regulator could require businesses to submit their tests. And the regulator could hire researchers to create the tests.
The testing could be voluntary, at least at the start for many applications. For some deployments, e.g., job hiring at previously naughty firms, the regulator could require validation.
Companies could then announce their success in passing these tests: “Our credit authorization software is certified free of bias.”
To implement this testing regime, your “blocks” need simplification.
(1) A business should be ready to show the specifications of their software.
(2) Those specifications must show respect for the law.
(3) The business must prepare for the regulator’s testing their code.
(4) The regulator’s tests must be able to check authoritatively whether their code follows the law.
This specification-and-test approach fits naturally in developer workflows. Developers just need the test cases from the regulator and an authentication mechanism to return the results. Further, this approach works for all code, not just “AI”, whatever that is. And then business could announce “Our hiring software is NIST certified for bias.”
The regulator could invite firms to submit edge cases automatically to the regulator’s databases for discussion and review (e.g., the Tesla may creep through the stop sign to allow passage of an ambulance). With this submission process, the business is transparent with the regulator about a case which the regulations address only vaguely.
A business’ willingness to submit these edge cases would play a role in court cases. For example, if the regulator finds bias in a company’s code, and the company had never submitted an edge case, the regulator could argue that the company was doing business in bad faith. If the company had submitted a great many edge cases, the company may be able to argue that the regulator’s tests were incomplete.

John F. Raffensperger

3.4 This testing mechanism enables the regulator can measure and manage consumer risk. 

Using the test mechanism, the regulator can manage risk explicitly, e.g., setting probability limits on bias and distinguishing risk to consumers from risk to business. Using a statistical approach, the regulator’s test cases could assess the probability of benefit to the public versus probability of harm to the public.
The regulator should carefully distinguish Type 1 error from Type 2 error, because they have asymmetric costs. For example, if a business loans too much money (false positive), the consumer benefits while the business loses. The regulator need do nothing in this case. If a business loans too little money (false negative), the consumer probably loses more, especially with bias. Require the probability of Type 1 (in this example) to be lower than the probability of Type 2. That is, the regulator should ignore Type 1 error and put all effort into preventing the Type 2 biased error.

John F. Raffensperger

3.5 The regulator must choose their battles. 

Because “AI” is vague and widely deployed, the regulator must choose business sectors (e.g., mortgage lending) to regulate in this way.
The regulator will need a process to call out specific businesses for scrutiny, the way the Justice Department calls out jurisdictions for scrutiny of voting procedures.
“This proliferation of AI bias into an ever-increasing list of settings makes it especially difficult to develop overarching guidance or mitigation techniques.” Given the regulator’s budget, the regulator has to choose their battles. Maybe the regulator could develop an AI to figure out which business sectors and applications will do greatest public damage. The risk assessment describe above improves if the regulator can use explicit costs to the consumer. The regulator could prioritize prosecution of bias with high expected cost to consumers over bias with low expected cost to consumers.
But “settings already known to be discriminatory” are easy to find. Follow the money.

John F. Raffensperger

4 Appendix A is excellent, a great contribution. 

Appendix A is a great contribution to the discussion. I urge you to tend it and build on it, so it can help set standards.
For each type of bias, is it likely to cost mostly business or mostly consumers? Which types cost consumers the most? How would you test a developer to be able to certify their knowledge of that type of bias? How would you test their code for that type of bias? I imagine a database of tests for each type of bias. How would a developer identify an edge case worthy of discussion?

John F. Raffensperger

5 Conclusion 

How would the regulator identify bias? How would the regulator assign responsibility for bias? What consequences should the regulator impose on the developer and the business for bias? How would the regulator decide the degree of penalty for a given instance of bias? I hope you would spend some time thinking about this problem.
“Participants also referred to the long-term nature of this challenge.” Yes. We’re in the first generation. Plan for the third generation. What should the regulator do with fabulous bias-detection software and an enormous database of bias-checking tests?
Congratulations again on an important paper! Thanks very much for attending to this important problem of AI bias. I hope you find some use in my comments here. Given my effort in writing these comments, I would be grateful for an acknowledgement from one of the authors. I submit this respectfully

8

American Statistical Association

Steve Pierson, Ph.D.

I write to pass along the recommendation of a member of the American Statistical Association. He recommends you reach out to the professional associations with experience in psychometrics and even the broader testing and measurement expertise community. These associations would include the following:
Associaton for Psychological Science (APS),
American Educational Research Association (AERA)
and National Council on Measurement in Educa5on (NCME)
University of Maryland Education and Measurement
His recommendation is based on the fact that, because AI is measuring and evaluating a variety of people on a large scale, one needs to establish reliability and validity and minimize adverse impacts against different demographic groups. Psychometricians do this with testing measurement invariance techniques for evaluating equal measurement item and test parameters via Structural Equation/ Confirmatory Factor Analysis techniques and differential item functioning parameters. Older techniques include those used by the Guilty Knowledge Test.

Mark Y. Czarnolewski, Ph.D., LLC

The American Statistical Association suggested that I contact you to add the American Psychological Association, particularly Division 5, to the list of organizations that have psychometric expertise.
Thank you for considering this list to contact in your important efforts to calibrate the use of AI for identification.

9

Salinger Privacy

Anna Johnston,

 Dear NIST,

I wish to offer some brief feedback on your recently released first draft document of “A Proposal for
Identifying and Managing Bias in Artificial Intelligence" (Special Publication 1270).
My viewpoint is that of a specialist advisor to clients – both government and private sector organisations – developing or procuring AI systems. I am the founder and Principal of Salinger Privacy, which is a specialist privacy consultancy firm based in Sydney and Melbourne, Australia.
In order to assist our client base, which is primarily the privacy officers within an organisation, or their legal / risk / compliance advisors, we recently published our own guide, “Algorithms, AI, And Automated Decisions - A guide for privacy professionals”.

Interestingly, your paper has framed the issues in quite a similar ‘lifecycle’ way to where we landed. In our guide we propose a “4 D’s framework”, in which both legal and ethical issues must be examined across four stages: Design, Data, Development and Deployment.
I support the need for a framework for trustworthy and responsible AI that is flexible enough to apply across different sectors and applications.

My feedback on your proposal is that I do not believe that the issue of managing bias can or should be divorced from other legal and ethical issues in AI and other algorithmic systems. In other words, I believe a holistic approach is needed. Further, such a framework must include guidance for non-technical advisors (such as those who will need to conduct Algorithmic Impact Assessments), as well as technical standards for system developers.

In my view, for an assessment of an algorithmic system to be robust, it should encompass:
Legal compliance – ensure the algorithmic system is lawful, with particular focus on privacy, anti-
discrimination, and consumer protection laws Social impacts – consider the social, political, and economic context for a deeper appreciation of potential privacy-related harms, and
Technical considerations – integrate testing for accuracy, performance, fairness and bias.
I have attached a copy of our guide as an example of what we have developed as guidance for a non-technical
audience (specifically, privacy officers). In particular, in addition to identifying legal and ethical risks and challenges, we have attempted to articulate what ‘good’ could look like, when developing or procuring AI systems.  Please note that the attached guide is a commercial publication, and as such is not for further distribution or publication without permission. However the text of this submission is not confidential.

Please do not hesitate to contact me if you have any questions arising.

10

OneSpan, Inc.

Michael Magrath

199

Stability over a long period of time of the systems should also be considered as an important characteristic of a trusted AI system. The high testing accuracy is not enough to ensure system performance over time, because assumptions made at time of training/creation may no longer hold after the system is deployed. 

OneSpan, Inc.

Michael Magrath

213

The ISO definition, taken out of its context, is not helpful: the term “truth” is highly loaded philosophically and would need an explanation in the context of this document. Will data scientists interpret “truth” the same way as regulators? Besides, we were unable to find this definition in the cited ISO standard. Instead, we found “Bias: expectation of error of estimation” which is much less ambiguous but is probably too technical for the current document.   https://www.iso.org/obp/ui/#iso:std:iso:3534:-1:ed-2:v2:en:term:1.33&nb…;

OneSpan, Inc.

Michael Magrath

219-231

 It would be beneficial if NIST were to include additional information on what is considered as harmful bias. Is there a categorization between non-negative and harmful biases or are they all considered equally harmful in the context of this report (link to terminology in the last section)?

OneSpan, Inc.

Michael Magrath

267

The specific conditional traits that exacerbate distrust in AI could be analysed in more details. It would be beneficial if NIST provided some examples. 

OneSpan, Inc.

Michael Magrath

389

The quoted formulation is too generic. Can “expert auditors” really provide the sufficient “sanction of authority” that “AI can be trusted” or just that a given AI system or class of AI system for some given applications can be trusted? 

OneSpan, Inc.

Michael Magrath

445

In several cases, the practitioners who need to use the AI tool, are also often those whose job might be at risk because of the AI tool itself. Their incentive is therefore often lower. This makes the point of 441-442 even stronger. 

OneSpan, Inc.

Michael Magrath

465

Be a bit more specific about rejecting flawed designs. How is such rejection process envisioned and who would be responsible for it (i.e., developers of AI, providers of AI, expert auditors, governing bodies)? 

OneSpan, Inc.

Michael Magrath

490

As written, “This is also a place where innovation in approaching bias can significantly contribute to positive outcomes.”. We suggest replacing the word “can” with “could”. 

OneSpan, Inc.

Michael Magrath

513

Subject matter experts are also crucial at that stage because for their deep knowledge of the domain in which data is collected. For instance, it is hard to believe that a medical AI tool could solely be designed and developed by “designers, engineers, and data scientists” without an ongoing close collaboration with medical doctors. 

OneSpan, Inc.

Michael Magrath

558

Include more information on what is considered as “modest approach” and how it would translate in an AI bias risk context. The cited report [111] does not provide a lot of information on it. 

OneSpan, Inc.

Michael Magrath

633

e ability of the user to provide feedback on the recommendation/decision of the AI system is another factor that impacts this gap. 

OneSpan, Inc.

Michael Magrath

703

We found the Glossary around bias types and their definitions very useful for setting a common language in this field. However, this glossary may benefit from some grouping of the biases, by similarity, or by relationship with the 3 stages defined in the documents. 

11

Kathy Rondon

As a data governance professional focusing on the public sector, I would like to submit comments in response to the request for public comments on SP 1270:
Proposal for Identifying and Managing Bias in Artificial Intelligence. I believe this is a critical topic for our society at this juncture in the development of AI and machine learning technology. My comments are from the perspective of a data
professional, rather than an AI developer, and are attached to this email, using the suggested template.

Kathy Rondon

212

In data ethics terms, in seeking to minimize harmful impacts of bias in AI, the ISO definition that casts bias in terms of "deviation from the truth" may not be optimal. If the truth is actively discriminatory, having AI perpetuate that "truth" may not be an ethical approach. Suggest using an alternate bias definition that reflects the need to identify more than just the data's deviation from the as-is state. Data can be accurate and precise but still not be appropriate for use in a specific or general use case. The appropriateness of data for the specific AI utility should also be considered in any definition of bias.

Kathy Rondon

292

Using what data is available, rather than allowing the research design to dictate data collection and use is indeed a problem. Data science that does not follow rigorous research design is not science--it's alchemy. NIST standards should make it clear that letting available data drive the development and training of AI technology yields products that cannot be  deemed reliable under NIST standards. Other scientific standards would not allow this "backing into" results.

Kathy Rondon

335

Suggest consideration of another "bullet point" to the list of reasons for distrust: The application of technology or data that were developed for collected for a specific purpose to a different use case (particularly without public  transparency regarding this "re-application") which may decrease the reliability of results. You mention this later in the publication, but emphasizing it with a bullet here would seem to be appropriate.

Kathy Rondon

521

Accuracy as the sole or primary characteristic of quality is a concept that has been challenged in academic literature as it applies specifically to data quality. Suggest the incorporation of different quality frameworks, such as the one based upon the work of MIT professors Richard Wang and Diane Strong in their article "Beyond Accuracy: What Data Quality Means to Data Consumers."

Kathy Rondon

650

The risk of individuals "offloading" decisions to an automated tool suggests a workforce or general public without sufficient education or training in data and AI literacy to do otherwise. Suggest NIST consider a standard that new AI applications include documentation and "user guides" that specifically address how the AI should be incorporated, used, and communicated to user groups. Also, more rigorous data ethics and data quality training for AI developers, whose academic work and training are typically more programming based and not sufficiently focused on data quality or ethics.

Kathy Rondon

707

As NIST works collaboratively to develop guidance on this topic, suggest an effort to recast the AI bias issue entirely to focus less on technology development and more on the identification, preparation, and validation of data used to train AI.  It's not that data bias isn't discussed in this publication, but rather that it appears to a subsidiary of the AI technology development cycle, rather than as the primary issue. Data--articulating what the right data is and then collecting it and making it transparent to the public--should be the primary focus of standards seeking to minimize harmful effect of bias in AI.

12

University of South Alabama

Aviv Segev

729

Add new definition in Table 1. "Bias bias - importance of the bias component of prediction error is inflated, and the variance component of prediction error, which reflects an oversensitivity of a model to different samples from the same population, is neglected." "Seagroatt V, Stratton I. Bias in meta-analysis detected by a simple, graphical test. Test had 10% false positive rate. BMJ. 1998;316(7129):470-471.
Henry Brighton, Gerd Gigerenzer, The bias bias, Journal of Business Research, Volume 68, Issue 8, 2015, Pages 1772-1784, ISSN 0148-2963, https://doi.org/10.1016/j.jbusres.2015.01.061."

13

Arthur AI

Lizzie Kumar

Arthur appreciates the opportunity to provide feedback on NIST’s proposal for identifying and managing bias in artificial intelligence (AI). As a company building bias detection solutions, we commend NIST's efforts to further research on this important issue. We are especially happy to see that the document draws attention to two important conclusions from their literature review. First, technology must be considered within the social system in which it is developed and deployed. Second, and relatedly, bias detection and mitigation efforts need to span the full development cycle of the technology at hand. These ideas lead the authors to point to "contextual gaps" between the initial design stages and the eventual deployment stage as a source of unexpected problems. The authors suggest that this makes it necessary to closely monitor how AI performs in the wild, a problem which Arthur's team has been tackling.

In addition to our more specific feedback (submitted through the provided form), we would like to point out that an AI project lifecycle does not "end" at the deployment stage, as is suggested by the framework presented in this proposal: changes within the deployment setting also need to be monitored. In practice, AI models developed with machine learning are rarely static, and continuous improvement efforts are often
ongoing. A model may be manually retrained as more data is collected, or may be updating in real-time as it encounters new scenarios, as in the case of reinforcement learning; bias can be introduced through these changes even if it did not exist at the initial point of deployment. 1 The context of the deployment setting itself may also change over time, even if the model remains static—due to either a feedback loop from the model itself,2 or processes external to it, causing a problem commonly known as data drift. For any of these reasons, at some point during the deployment stage, a developer may need to circle back to the beginning of the design process and craft a whole new solution to the problem at hand. We therefore caution NIST against unintentionally framing deployment as a static stage. Instead, like the document's Figure 1 indicates,
the deployment phase can evolve in a way that leads the product back into design or pre-design. If standards are developed with the aim to identify and manage bias in this phase, they should be implemented in a way that is mindful of this dynamic. Sincerely, Arthur AI
1 https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist
2 http://proceedings.mlr.press/v81/ensign18a.html

Arthur AI

Lizzie Kumar

525

Not sure ecological fallacy is the best way to describe the way machine learning uses group membership to make inferences about individuals— many such inferences are not fallacies

Arthur AI

Lizzie Kumar

655

Unclear why counterfactual fairness is highlighted here as a natural way to address "contextual gaps." While it is true CF addresses weaknesses of other fairness metrics, it is difficult to implement in practice and requires very specific models of the world to identify bias.

Arthur AI

Lizzie Kumar

673

This figure suggests a project can move from Deployment back to Pre-design. However, we did not see this notion addressed in the text.

• Where bias stems from is also an important consideration. Bias can be introduced through experimental bias or inadvertent bias. The existing framework provides numerous examples of these types, and would benefit from calling out these categories so that readers can separate process from type.
  • Experimental bias: Can stem from data selection or sampling that may not be representative, or data that is sensitive in nature (e.g., yelp effect).
  • Inadvertent bias: Can occur in the construction of algorithms (e.g., making assumptions in heuristics based on world experience)

AI Blindspot

Americans for Financial Reform Education Fund

California Reinvestment Coalition

Center for Community Progress

Center for Responsible Lending

Consumer Action

Consumer Federation of America

Fair Housing Advocates Association

Fair Housing Advocates of Northern California

Fair Housing Center of Central Indiana

Fair Housing Center of Northern Alabama

Fair Housing Center of Southwest Michigan

Fair Housing Center of West Michigan

Fair Housing Council of Greater San Antonio

Integrated Community Solutions, Inc.

The Leadership Conference on Civil and Human

Rights

Long Island Housing Services, Inc.

Louisiana Fair Housing Action Center

Miami Valley Fair Housing Center, Inc.

MICAH- Metropolitan Interfaith Council on

Affordable Housing

Mountain State Justice

NAACP Legal Defense and Educational Fund, Inc.

(LDF)

National CAPACD

National Coalition For The Homeless

National Community Reinvestment Coalition

National Consumer Law Center (on behalf of its

low-income clients)

National Fair Housing Alliance

North Texas Fair Housing Center

NYU Center on Race, Inequality, and the Law

SolasAI

South Suburban Housing Center

Southern Poverty Law Center Action Fund

Woodstock Institute