	A	B	C	D	E	F	G
1
2		All comments will be made public as-is, with no edits or redactions. Please be careful to not include confidential business or personal information, otherwise sensitive or protected information, or any information you do not wish to be posted.
3
4				Comment Template for Responses to NIST Artifical Intelligence Risk Management Framework Request for Infromation (RFI)			Submit comments by August 19, 2021:
5
6
7
8
9	General RFI Topics (Use as many lines as you like)	Response #	Responding organization	Responder's name	Paper Section (if applicable)	Response/Comment (Include rationale)	Suggested change
10
11
12
13
14
15	Responses to Specific Request for information (pages 11,12, 13 and 14 of the RFI)
16	1. The greatest challenges in improving how AI actors manage AI-related risks – where “manage” means identify, assess, prioritize, respond to, or communicate those risks;		Corner Alliance, Inc.		CVPR 2021 Workshop on Autonomous Vehicles (https://www.youtube.com/watch?v=YZTlaiu_vWE)	In the rapidly changing world of AI, there are a number of challenges in improving how AI actors manage AI-related risks. As with other rapidly changing fields, there are known knowns; known unknowns; and unknown unknowns of AI failure modes. Currently, the greatest challenge in AI-risk management policy is the inability to anticipate model failure. Until the AI community defines a comprehensive list of model susceptibilities, from gender and racial bias to adversarial examples and manipulation to bad actors, this weakness is impossible to measure and improve upon. Therefore, it is imperative to identify a comprehensive framework or schema of what could go wrong in a model's performance, and then test rigorously against these factors. This need to clearly define categories and examples of failure modes and weaknesses is challenging for industry to implement without guidance because, if one were to take this task on their own, the undertaking would be insufficient and incomplete without reference to a central document. Additionally, it might include tasks that are industry-specific, rather than applying to the AI community as a whole. Whether doing semantic search, skill matching, or other tasks, the necessity of producing results that include acceptable levels of risk is imperative to ensure user trust in a system. In addition to the difficulty of anticipating model vulnerabilities, there is the another challenge of knowing whether or not a practitioner's data is biased. Relying on large language models, such as Transformers, can be risky. Even if one has state of the art performance on natural language processing benchmarks, one is often trained on datasets scraped from the edges of the internet. For example, the T5 dataset includes representations from Reddit and other websites that often contain incendiary viewpoints that poison model performance. Therefore, having definitive guidance for dataset creation to support the development of risk-free data would ensure broader deployment of off-the-shelf datasets that practitioners and users can have solidified confidence in. In order to solve these challenges, a number of avenues can be pursued. Some apparent examples include: the development of use cases, unit tests, red team exercises, and prize challenges to identify and document the full gambit of strategies to mitigate AI model risk. Providing the user community with the opportunity to respond to all the scenarios where AI could go wrong could be a useful method to motivate experts to add their voice and solve the problem. However, as it stands, there is nothing central to test against to objectively evaluate model robustness for bias, fairness, explainability, efficiency, safety, and more. NIST is well positioned to develop an authoritative and comprehensive Framework to identify and test against vulnerabilities before model deployment and provide a forum to brainstorm the full spectrum of where risk can be introduced in a model. Only then can risk mitigation strategies be developed to ensure blind spots are addressed in the AI model development pipeline. For an example of how industry identifies corner cases and employs rigorous review processes in their pipeline, see the CVPR 2021 Workshop on Autonomous Vehicles. The key is to continually update a library of test cases that a model is being proved on. The chances of success are increased when this information can be crowdsourced from experts around the world to identify vulnerabilities. Only with the development of a comprehensive AI Risk Framework can companies across industries identify and define their policy goals and see how they measure in their field.
17
18	2. How organizations currently define and manage characteristics of AI trustworthiness and whether there are important characteristics which should be considered in the Framework besides: accuracy, explainability and interpretability, reliability, privacy, robustness, safety, security (resilience), and mitigation of harmful bias, or harmful outcomes from misuse of the AI;		Corner Alliance, Inc.		Explaining and Harnessing Adversarial Examples (https://arxiv.org/pdf/1412.6572.pdf) Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3213954)	The “Explaining and Harnessing Adversarial Examples” paper provides a number of applicable definitions, both new and included in the list above. The terms and definitions are as follows. Confidence: When the deep neural networks (DNNs) decision is congruent with the thought process of a user. Deterministic computation (not logical rationale). The user has to be able to view both correct and incorrect decisions. Trust: When the decision-making process does not have to be validated. Is model prediction safe to be adopted? Adversarial studies diminish the validity of this trait. Developed through satisfactory testing and experience. Safety: Consistently operate as expected. Given cues from its input, guard against choices that can negatively impact the user or society. Exhibit high reliability under both standard and exceptional operating conditions. Provide feedback to a user about how operating conditions influence its decisions Ethics: Model does not violate a code of moral principles defined by the user. In addition to these definitions on explainability, the "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security" paper also provides a useful framework for risks related to national security implications of deep fakes powered by machine learning (ML). Deep Fakes definition: A digital impersonation that leverages machine learning algorithms to insert faces and voices into video and audio recordings of actual people and enables the creation of realistic impersonations out of digital whole cloth. ML makes deep fakes and other forgeries more sophisticated. Amplifies cognitive bias and truth decay. Enables enhanced exploitation, intimidation, and personal sabotage. There are multiple causes and consequences of this disruptive technological change. Examples of ML-based Deep Fakes: GANS (thispersondoesnotexist.com); DeepMind’s Wavenet model; Baidu’s DeepVoice; Plus commercial applications like Lyrebird. What are the risks of deep fakes? Harm to Individuals or Organization (Exploitation; Sabotage); Harm to Society (Distortion of democratic discourse; Manipulations of elections; Eroding trust in institutions; Exacerbating social divisions; Undermining public safety; Undermining diplomacy; Jeopardizing national security; Undermining journalism).
19
20	3. How organizations currently define and manage principles of AI trustworthiness and whether there are important principles which should be considered in the Framework besides: transparency, fairness, and accountability;		Corner Alliance, Inc.		On the Dangers of Stochastic Parrots (https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)	In Timnit Gebru’s “On the Dangers of Stochastic Parrots,” the authors explore at length the considerations that researchers should be aware of thoughtful future AI and ML development. The authors' main argument is that large AI models are not showing better understanding, but rather, showing better manipulation of natural language. Therefore, the authors ask the question: is it worth it to pursue large models, considering the expenses? The authors then identify main drawbacks and risks associated with large models, namely environmental and financial costs, training data challenges, and propagation of hegemonic worldviews. In terms of environmental and financial costs, the section underscores the need for energy efficient model architectures and training paradigms, known as Green AI. Efficiency should be promoted as an evaluation metric in order to benchmark energy usage. The amount of compute used to train the largest deep learning models has increased 300,000x in six (6) years, increasing at a far higher pace than Moore’s Law, according to Gebru et al. However, standards can shape a future where data driven systems have minimal negative impact on the environment. Currently, a Transformer with neural architecture search during training emits 284 tons of carbon dioxide. Furthermore, BERT on GPUs requires as much energy as a trans-American flight. The authors implore researchers to analyze the cost versus accuracy gain, pointing to the example that for an increase in 0.1 BLEU score, it costs $150,000 for computer cost, plus carbon emissions. To encourage more equitable access to NLP research and reduce carbon footprint, the authors recommend to make reporting training time a requirement. Additionally, they urge governments to invest in compute clouds to provide equitable access to researchers. Regarding training data challenges, most language technology is built to serve the needs of those who already have the most privilege in society, according to the authors. Due to training data available online, representative values are often not being encoded in AI systems. Rather, stereotypical and derogatory associations are encoded along gender, race, ethnicity, and disability status. Large, uncurated Internet-based datasets encode the dominant/hegemonic view which further harms people at the margins. The authors recommend resource allocation towards dataset curation and documentation practices. They argue that size of the dataset does not guarantee data diversity. Without a critical examination of dataset content, researchers risk perpetuating dominant viewpoints, increasing power imbalances, further reifying inequality. Furthermore, static datasets do not reflect changing social views. A central aspect of social movement formation involves using language strategically to destabilize dominant narratives and call attention to underrepresented social perspectives. Therefore, thoughtful curation practices to capture reframing and techniques should be more frequently implemented in dataset development. Additionally, models easily encode bias if trained on datasets with stereotypical associations. This can include labeling males as doctors and females as nurses, or having negative sentiments towards specific ethnic groups. The authors recommend building out systems that verify the “safety” of a language model for a protected class. Finally, curation and documentation are key to accountability in dataset development. In order to curate and document better language model training data, a justice-oriented data collection methodology should be developed. Without documentation, practitioners cannot try to understand training data characteristics to fix issues. Therefore, researchers and facilities should budget for documentation as part of the costs of project development, according to the authors. Finally, in terms of perpetuating hegemonic worldviews, English-based language models can be exclusionary and therefore risk degrading culture. In order to participate, users must subscribe to the global hegemonic order upon which these models are based. While utterances are reflected in worldviews, people with privilege are overrepresented in the training data. This can lead to framing bias, dehumanizing verbiage, and microaggressions that become automated. In order to reduce the risk of producing and amplifying automated biases, a technological ecosystem should be developed in a way that evenly represents more marginalized communities worldwide. The authors conclude by posing the problem through the lens of compound interest: the actions we choose today become amplified over time, so it is in our best interest long term to make choices that favor the environment, cost, marginalized communities, and proper documentation/accountability. Due to the fact that these will be the benefits or disadvantages that become amplified over time, it is important to make the right choices now in order to enjoy compounded benefits later.
21
22	4. The extent to which AI risks are incorporated into different organizations' overarching enterprise risk management – including, but not limited to, the management of risks related to cybersecurity, privacy, and safety;		Corner Alliance, Inc.		Explaining and Harnessing Adversarial Examples (https://arxiv.org/pdf/1412.6572.pdf) Switching Gradient Directions for Query-Efficient Black-Box Adversarial Attacks (https://arxiv.org/pdf/2009.07191.pdf) Switching Gradient Directions for Query-Efficient Black-Box Adversarial Attacks (https://arxiv.org/pdf/2009.07191.pdf) Adversarial Examples in Deep Learning Multivariate Time Series Regression (https://arxiv.org/abs/2009.11911)	Many organizations are ill-suited at incorporating AI risk considerations into their overarching enterprise risk management. More training is required on how to develop and test models with safety in mind. In particular, checking for robustness against adversarial attacks should become more standard practice. Adversarial attacks are the major security threat to deep neural networks (DNNs) that add human-imperceptible perturbations to benign images for the misclassification of DNNs. The following three papers provide valuable insights about adversarial attacks that could be incorporated into organizations’ risk management strategies. The “Explaining and Harnessing Adversarial Examples” paper provides an overview of the adversarial training technique and describes how a single-pixel change in an image can lead to catastrophic misclassification. The paper defines an adversarial example as a perturbation to an image that results in an incorrect answer with high confidence. For example, this vulnerability can lead to a picture of a panda getting classified as a gibbon after adding a small layer of noise to the data. It suggests that neural networks are susceptible to adversarial examples due to their linear nature. To address the fundamental blind spot in training algorithms, this paper introduces a fast method of computing adversarial examples using a heavy weight optimization technique. Alternatively, it suggests practitioners can generate adversarial examples as they train and evaluate on that. While this does not solve the problem of vulnerability to adversarial examples, it does substantially reduce the problem through additional regularization benefits. The “Switching Gradient Directions for Query-Efficient Black-Box Adversarial Attacks” paper describes how practitioners can use adversarial training to strengthen models against adversarial examples. It provides further details on how to use a model to generate adversarial examples and then include them in the training dataset. Black-box attacks are more realistic in real world systems because they do not require the parameters and gradients of the target model. The paper describes two methods of black-box adversarial attacks, including transfer attacks and query-based attacks. The authors propose a simple and highly query-efficient black-box adversarial attack named SWITCH, which at the time of publishing, had state-of-the-art performance under L2 and L infinity norms. Transfer-based attacks generate adversarial examples by attacking a pre-trained surrogate model to fool the target model. They do not require querying the target model. Query-based attacks require oracle access to the target model. Score-based attack setting requires accessing the loss function value. Random-search-based attacks use sampling of a random perturbation at each iteration. Then, the modified image is fed to the target model to compute a loss value. The model introduced in the paper, called SWITCH, bridges the gap between transfer-based attacks and random-search-based ones to improve the query efficiency. The paper’s main contribution is that the proposed approach switches the gradient to avoid following the wrong direction and consequently bypass the potential obstacle in optimization, which keeps the loss function increasing as much as possible. Finally, the “Adversarial Examples in Deep Learning Multivariate Time Series Regression” provides an example of how adversarial examples can also be implemented in non-image data. Multivariate Time Series (MTS) regression is common in data mining applications including finance, cybersecurity, energy, healthcare, prognostics, and other areas. Now that deep learning is being used for solving MTS data mining problems, it is safety-critical and cost-critical to ensure these models do not have significant security vulnerabilities. Since deep learning algorithms are known for their susceptibility to adversarial examples, this work leverages existing adversarial attack generation techniques to create adversarial MTS examples for CNNs, LSTMs, and GRUs. This method was evaluated on Google stock and household power consumption dataset. The paper’s main finding is that the models are all vulnerable to adversarial attacks, which can lead to catastrophic consequences. Since adversarial attacks in the non-image domain is vastly under-explored, this paper’s main contribution is the confirmation of model fragility across domains in addition to discussing potential defense techniques.
23
24	5. Standards, frameworks, models, methodologies, tools, guidelines and best practices, and principles to identify, assess, prioritize, mitigate, or communicate AI risk and whether any currently meet the minimum attributes described above;		Corner Alliance, Inc.		TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (https://arxiv.org/pdf/2005.05909.pdf) Lessons learned in designing TextAttack (https://textattack.readthedocs.io/en/latest/1start/api-design-tips.html) Explainable Deep Learning: A Field Guide for the Uninitiated (https://arxiv.org/abs/2004.14545) CAM Paper (https://arxiv.org/pdf/1512.04150.pdf) Grad-CAM Paper (https://arxiv.org/pdf/1610.02391.pdf) Lottery Ticket Hypothesis (https://arxiv.org/pdf/1803.03635.pdf)	When it comes to NLP model robustness, adversarial attacks can impact classification, entailment, and translation. “TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP” provides a framework of how to develop attacks and how to utilize them to improve model robustness. By allowing researchers to test and study the effects of adversarial attacks, it will hopefully allow for the creation of user-friendly, open-source NLP libraries that are more robust against adversarial attacks, and therefore ready for wider distribution. The paper has a python library for adversarial attacks and data augmentation, all of which are run via the command line. It provides pre-trained models for more than 80 common datasets. It introduces four components of NLP Attacks, namely: creating adversarial attacks through goal function; applying constraints from adversarial attacks literature; including transformations from adversarial attacks literature; and using search method. These options can be combined to create new adversarial attacks, called “attack recipes.” This paper unites 15-plus papers from the NLP adversarial attack literature into a single shared framework, therefore allowing researchers to test and study the weaknesses of their NLP models. TextAttack provides dozens of pre-trained models (LSTM, CNN, and various transformer-based models) and supports tasks like summarization, machine translation, and all nine tasks from the GLUE benchmark. Alternatively, in terms of frameworks specifically for explainability, “Explainable Deep Learning: A Field Guide for the Uninitiated” provides a thorough discussion on methods for explaining deep neural networks. The main methods it lists include: visualization methods; model distillation; and intrinsic methods. Visualization methods highlight the characteristics that influence the output of DNNs. It answers the question: to what degree does a specific input feature contribute to a decision? It describes how to use back propagation to visualize feature relevance based on volume of gradient passed through network layers during network training. Some examples of how this is done in practice include deconvolution, CAM, Grad-CAM, layer-wise relevance propagation, and perturbation. With deconvolution, it is possible to visualize higher layer features in the input space using rectified linear function and by running a CNN in reverse. CAM and Grad-CAM use class activation maps using global average pooling in CNNs to indicate the image regions that were most important to a network prediction for that class. CAM is an approach to localization which uses global average pooling on the last convolutional layer, resulting in the soft max layer producing a heatmap to indicate areas of importance for the network prediction. However, because a vanilla CAM is unable to go through fully connected layers, Grad-CAM is applicable on a broader range of CNNS, only requiring that the final activation function used for the network prediction to be a differentiable function. Grad-CAM linearly combines the importance scores of each feature map and passes them through a ReLU to obtain a relevance score map. The relevance score map is then up-sampled to be of the same dimension as the input image to produce the class activation map. The next example of visualization for explainability is called layer-wise relevant propagation, which creates a saliency map to represent the relevance of each input feature to the output of the network. The sensitivity metric measures the change in response to the network’s output as a result of changing attributes in the input. The relevance metric measures the strength of the connection between the input or pixel to the specific network output. One example of layer-wise relevance propagation is DeepLIFT, which assigns relevance scores to input features based on the difference between an input x and a reference input x prime. Finally, perturbations visualize feature relevance by comparing network output between an input and a modified copy of the input. It compues input feature relevance by altering or removing the input feature and comparing the difference in network output between the original and altered one. It measures occlusion sensitivity by covering parts of the image and seeing how the model reacts. Alternatively, in NLP it uses representation erasure where words are deleted to see how their absence impracts prediction. These meaningful perturbations act as meta-predictors for a black box’s output, given certain inputs. These methods of visualization use back propagation to visualize feature relevance based on the volume of gradient passed through network layers during network training. It is specifically useful for image and text data types. The downside of this method is the extra human effort needed for verification, which is potentially costly and undesirable if an automatic instant explanation is needed. These visualization metrics could be very useful in identifying rationale behind many vision-based DNNs. The “Explainable Deep Learning” paper also describes methods for model distillation, which add a “white-box” machine that identifies the decision rules of input features. This provides a post-training explanation methods where knowledge is distilled into a representation amenable for explanation by a user. Distilled models are like a hypothesis as to why a DNN has assigned some class label to an input. One way to achieve this is through local approximation, in which the practitioner learns a simple model whose input/output behavior mimics that of a DNN for a small subset of input data. For examples of local approximation, researchers can look towards LIME (Local Interpretable Model-Agnostic Explanations), as well as Shapley values for sensitivity analysis. Another way to achieve a distilled model is through model translation, in which the practitioner trains an alternative smaller model that mimics the input/output behavior of a DNN. Distillation can be done to decision trees, finite state automata, graphs, and into causal and rule-based models. One example of successful mode distillation is the Lottery Ticket Hypothesis to find sparse, trainable neural networks. The final method for explainability proposed by the “Explainable Deep Learning” paper is intrinsic methods, which render an explanation along with the output. This can be done through attention mechanisms, or through joint training. Attention mechanisms leverage the key, value, query system to learn conditional distributions over given input units, composing a weighted contextual vector for downstream processing. The attention visualization reveals inherent explainability. Joint training adds an additional explanation “task” to the original model tasks and jointly trains the explanation task along with the original task. For example, this could mean a model uses natural language processing to explain how it arrived at a certain decision. All of these methods, while still nascent, are promising avenues to encourage explainability at the forefront of model development. In addition to the methods listed above, the “Explainable Deep Learning” paper also provides details about additional topics related to explainability. While not methods or frameworks per se, understanding these topics relative to explainability could help practitioners better implement these concepts in their own models. Therefore, these topics could be stressed at eventual educational opportunities related to explainability. These topics include the learning mechanism, model debugging, adversarial attack and defense, and fairness and bias in DNNs. The “Explainable Deep Learning” paper discusses the topic of learning mechanisms as a way to explain the evolution of a model’s parameters through back propagation. Through this process, semantic meaning is assigned to weights and activations. Statistical patterns indicate convergence to a learned state. This observation is helpful to learn more about how layers evolve during training, how different layers converge, and if there are properties of generalization and memorization with the DNNs. The “Explainable Deep Learning” paper also describes model debugging as something like a neural stethoscope, quantifying the importance of specific influential factors in the DNN’s learning process, in which some information is actively promoted or suppressed. Boolean functions are used to verify the state of the model during training. The same paper also identifies adversarial attacks and defenses as critical for explainability. These attacks intentionally disturb the judgement of a DNN. A black-box attack occurs when the attacker has no access to the model parameters. A white-box attack occurs when the parameters are accessible. To avoid infiltration by these attacks, adversarial defense should be a critical component of model development and testing before deploying to wider audiences. Finally, “Explainable Deep Learning” devotes a section to fairness and bias in DNNs, discussing differences in group fairness, individual fairness, demographic parity, and statistical parity. Emphasizing the importance of avoiding disparate mistreatment, the paper identifies three stages in which analysis for bias should be conducted. First, during pre-processing, learn alternative representation of the input data that removes information correlated to the sensitive attributes while maintaining model performance. Second, in the process of training, the practitioner should directly introduce fairness learning constraints to the model in order to punish unfair decisions and act as a fairness regularizer. Finally, during post-process, add ad hoc fairness procedures to a trained model to ensure deployability.
25
26	6. How current regulatory or regulatory reporting requirements (e.g., local, state, national, international) relate to the use of AI standards, frameworks, models, methodologies, tools, guidelines and best practices, and principles;				N/A	N/A
27
28	7. AI risk management standards, frameworks, models, methodologies, tools, guidelines and best practices, principles, and practices which NIST should consider to ensure that the AI RMF aligns with and supports other efforts;		Corner Alliance, Inc.		Explainable Deep Learning (https://arxiv.org/abs/2004.14545)	The “Explainable Deep Learning” paper describes a number of user design explanations for practitioners developing models. User experience matters! Practitioners should always think: Who is the end user? How practically impactful are the decisions of the DNN? How “extendable” is an explanation? Understanding the End User: Low level technical explanations (input feature analytics; hidden state interactions); High level explanations for everyday people (functionality of the model; logical reasoning) The Impact of DNN Decisions: Time-critical (self driving cars; military operations); Decision-critical (medical diagnosis) Design Extendability: How far reaching can the implications of these model adjustments be? How will this improve the lifecycle of a model? Will implementing explainability mechanisms more frequently in models lead them to become more reusable? Improve ROI of a model?
29
30	8. How organizations take into account benefits and issues related to inclusiveness in AI design, development, use and evaluation – and how AI design and development may be carried out in a way that reduces or manages the risk of potential negative impact on individuals, groups, and society.		Corner Alliance, Inc.		OECD: Principles on AI (https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449)	The OECD: Principles on AI provides a useful framework on how benefits of AI can be carried out in a way that reduces negative impacts on society. Key points are as follows: Inclusive growth, sustainable development, and well-being: Beneficial outcomes for people and the planet (such as augmenting human capabilities and enhancing creativity, advancing inclusion of underrepresented populations, reducing inequalities, and protecting natural environments. Human-centered values and fairness: Respect the rule of law, human rights, and democratic values (including freedom, dignity, autonomy, privacy, and data protections, equality, fairness, social justice, and labor rights); Safeguards should be implemented in certain situations, such as including a human in the loop. Transparency and Explainability: Actors must disclose meaningful information to help stakeholders understand their interactions with AI and how outcomes were decided. Robustness, Security, and Safety: AI shouldn't pose unnecessary risk; Systems should be traceable, regarding datasets, processes, how it came to prior decisions so its outcomes can be analyzed; Actors should apply systematic risk management approaches for privacy, digital security, strategy, bias, etc. Accountability: Actors should be accountable for proper functioning of AI systems.
31
32	9. The appropriateness of the attributes NIST has developed for the AI Risk Management Framework. (See above, “AI RMF Development and Attributes”);				N/A	N/A
33
34	10. Effective ways to structure the Framework to achieve the desired goals, including, but not limited to, integrating AI risk management processes with organizational processes for developing products and services for better outcomes in terms of trustworthiness and management of AI risks. Respondents are asked to identify any current models which would be effective. These could include – but are not limited to – the NIST Cybersecurity Framework or Privacy Framework, which focus on outcomes, functions, categories and subcategories and also offer options for developing profiles reflecting current and desired approaches as well as tiers to describe degree of framework implementation; and				N/A	N/A
35
36	11. How the Framework could be developed to advance the recruitment, hiring, development, and retention of a knowledgeable and skilled workforce necessary to perform AI-related functions within organizations.				N/A	N/A
37
38	12. The extent to which the Framework should include governance issues, including but not limited to make up of design and development teams, monitoring and evaluation, and grievance and redress.		Corner Alliance, Inc.		Common Code (https://www.cnas.org/publications/reports/common-code)	Governance should be a key consideration to ensure unified policies for risk management. Establishing consistent processes in the design and development of teams, monitoring and evaluation, and redress can ensure a minimum set of requirements are met in terms of impact, scope, and resources. The Center for a New American Security’s Common Code report provides a number of recommendations to consider when designing governance considerations for technology policy. The recommendations of interest for the development of an AI Risk Management Framework include: Collaborating With Other Countries Adopting A Voting System Engaging With Other Stakeholders Establishing Meeting Structure And Frequency Craft Standards And Norms For A Beneficial Technology Future These recommendations, and others listed in the report could be useful in determining the structure, organization, and methods employed by the Framework to design and develop teams.
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979