Key Concepts in AI Saftey: Robustness and Adversarial Examples

Introduction As machine learning becomes more widely used and applied to areas where safety and reliability are critical, the risk of system failures causing significant harm rises. To avoid such failures, machine learning systems will need to be much more reliable than they currently are, operating safely under a wide range of conditions.  1  In this paper, we introduce adversarial examples-a particularly challenging type of input to machine learning systems-and describe an artificial intelligence (AI) safety approach for preventing system failures caused by such inputs. Machine learning systems are designed to learn patterns and associations from data. Typically, a machine learning method consists of a statistical model of the relationship between inputs and outputs, as well as a learning algorithm. The algorithm specifies how the model should change as it receives more information (in the form of data) about the input-output relationship it is meant to represent. This process of updating the model with more data is called "training." Once a machine learning model has been trained, it can make predictions (such as whether an image depicts an object or a human), perform actions (such as autonomous navigation), or generate synthetic data (such as images, videos, speech, and text). An important trait in any machine learning system is its ability to work well, not only on the specific inputs it was shown in training, but also on other inputs. For example, many image classification models are trained using a dataset of millions of images called ImageNet; these models are only useful if they also work well on real-life images outside of the training dataset. Modern machine learning systems using deep neural networks-a prevalent type of statistical model-are much better in this regard than many other approaches. For example, a deep neural network trained to classify images of cats and dogs in black and white is likely to succeed at classifying similar images of cats and dogs in color. However, even the most sophisticated machine learning systems will fail when given inputs that are meaningfully different from the inputs they were trained on. A cat-and-dog classifier, for example, will not be able to classify a fish as such if it has never encountered an image of a fish during training. Furthermore, as the next section explores in detail, humans cannot always intuit which kinds of inputs will appear meaningfully different to the model.
 Adversarial Examples One of the most significant current challenges in AI safety is creating machine learning systems that are robust to adversarial examples. Adversarial examples are model inputs (for example, images) designed to trick machine learning systems into incorrect predictions. In the case of a machine learning system designed to distinguish between cats and dogs, an adversarial example could be an image of a cat modified to appear to the model as a dog. Since machine learning systems process data differently from humans, the cat image could be altered in ways imperceptible to humans but meaningfully different to the machine learning system. The modified image may still resemble a cat to humans, but to a machine learning system, it "looks like" a dog. Adversarial examples can be generated systematically, either by digitally altering the input to a system or by directly altering the appearance of objects in the physical world. Unlike other adversarial attacks, such as "data poisoning," which seeks to attack the algorithm used to train a machine learning model, adversarial examples are designed to attack already trained models.   Although modern machine learning systems usually generalize remarkably well to data similar to the data used for training, adversarial examples can be created from surprisingly simple modifications to model inputs. Changes such as blurring or cropping images, or altering the appearance of the physical-world objects shown in an image, can fool an otherwise reliable system. In Figure  3b , an adversarial example is constructed by reducing the resolution of the original image, thereby changing the model's prediction from correct to incorrect. Unlike the adversarial examples in Figures  1 and 2 , the adversarial example in Figure  3  was created via a black-box attack-that is, created without access to the trained classification model. It is not as subtle as the alteration in Figure  1  and not as targeted as the alteration in Figure  2 . However, it demonstrates that modern machine learning systems can be fooled with little effort and no knowledge of the prediction model.
 Robustness to Adversarial Examples Robust machine learning systems need to be able to identify data that is meaningfully different from training data and provide a defense against adversarial examples. There are a wide range of different research areas attempting to make progress in this direction. One such research direction aims to incorporate predictive uncertainty estimates into machine learning systems. This way, any prediction from the system would come with an estimate of certainty. If the machine learning system indicates uncertainty about the correctness of its prediction, a human operator can be alerted. To understand predictive uncertainty estimates and how they can make machine learning systems more robust to adversarial examples, consider the classification "probability scores" given in the descriptions of Figures  1 and 3 . In reality, these scores, which express the probability of an input belonging to a certain class (e.g., the class "cat" or "dog"), are misleading. While they do express a probability, they do not actually express the model's level of certainty about the correctness of the predictions. To fully understand this point, consider a machine learning system trained to distinguish between two classes: cats and dogs. Such a system will by design have two outputs: one for the class "cat" and one for the class "dog." If the model is given an image of a dog, it will output values between zero and one for each class-for instance, 90 percent and 10 percent for the classes "dog" and "cat," respectively, so that the values sum up to 100 percent. However, if given an image of a fish, the model will still make predictions for the two classes on which it was trained, unaware that it is being asked to identify an object it was not trained to recognize. In a best-case scenario, it would give outputs of 50 percent for each class, indicating that the input is equally likely to be either a cat or a dog. In a worst-case scenario, it would give a high probability score for one class, providing a false sense of certainty. But the way most machine learning systems are designed, they cannot give a low score to both the "cat" and "dog" labels. As such, these outputs should not be read as the machine learning system's "confidence" in the correctness of its classification. Predictive uncertainty estimates can fill this gray spot. They complement the regular model outputs by expressing the model's uncertainty about the correctness of its predictions. If a machine learning system has good predictive uncertainty estimates, then the probability scores in Figure  3  would be accompanied by a high uncertainty score, indicating that the model is highly uncertain about the correctness of the predictions. Such uncertainty estimates can help a human operator avoid wrong predictions in safety-critical settings and ensure the system's reliability and safety, as demonstrated in Figure  4 . Unfortunately, obtaining reliable predictive uncertainty estimates for modern machine learning systems remains an unsolved problem. While several existing methods can generate uncertainty estimates, there are no mathematical guarantees that these uncertainty estimates are actually accurate. Furthermore, while empirical studies demonstrate that certain methods produce good predictive uncertainty estimates in some settings, those results cannot be generalized to any setting. Like other areas of robustness research, developing methods that yield reliably wellcalibrated uncertainty estimates for modern machine learning systems is an active and ongoing area of research.
 Outlook While modern machine learning systems often perform well on narrowly defined tasks, they can fail when presented with tasks meaningfully different from those seen during training. Adversarial attacks exploit this vulnerability by presenting inputs to machine learning systems specifically designed to elicit poor predictions. Adversarially robust machine learning systems seek to fix this vulnerability through mechanisms allowing the system to recognize when an input is meaningfully different from data seen during training, making the system more reliable in practice. Unfortunately, while an active area of research, existing approaches to detecting and defending against adversarial attacks do not yet provide satisfactory solutions, and the timeline to develop and deploy truly robust modern machine learning systems remains uncertain. For now, anyone considering deploying modern machine learning systems in safety-critical settings must therefore grapple with the fact that in doing so, they are introducing safety risks that we do not yet know how to mitigate effectively. 2 Figures 1 1 Figures 1 and 2 show systematically generated adversarial examples. Specifically, the adversarial example in Figure 1c digitally modifies the original image by an imperceptibly small amount, whereas the adversarial example in Figure 2b is created by adding patches to the image designed to mimic irregularities found in the physical world (such as graffiti or stickers). Both adversarial examples are generated via so-called white-box attacks, which assume the attacker knows how the trained classification model works and can exploit this knowledge to create adversarial examples that trick the model into making incorrect predictions.
 Figure 1 . 1 Figure 1. An example of a "white-box" adversarial example from Goodfellow et al.(2015). The original image (a) is classified as "panda" with 57.7 percent probability. After being overlaid with a minimal amount of noise-the adversarial alteration (b) multiplied by a factor of 0.007-the resulting image (c) is classified as "gibbon" with 99.3 percent probability. The difference between (a) and (c) is imperceptible to the human eye.
 Figure 2 . 2 Figure 2. An example of a white-box adversarial example designed to generate physical alterations for physical-world objects. The adversarial alteration (b), which is designed to mimic the appearance of graffiti (a), tricks an image classifier into not seeing a stop sign.
 Figure 3 . 3 Figure 3. An example of a black-box adversarial example. The original image (a)is classified as "washer" with 53 percent probability. The image is altered by reducing its resolution to create an adversarial example (b), which is classified as "safe" with 37 percent and as "loudspeaker" with 24 percent probability.
 Figure 4 . 4 Figure 4. An example of predictive uncertainty estimates for autonomous vehicles.The first column shows the image fed into the system, the second column shows the ground truth classification of objects in the image (buildings, sky, street, sidewalk, etc.), the third column shows the model's classification, and the rightmost column shows the system's uncertainty about its classification. As can be seen from the image on the bottom right, the system is uncertain about its classification of parts of the sidewalk and could alert the human operator to take over the steering wheel.

			 Andrew Lohn, "Hacking AI" (Center for Security and Emerging Technology, December 2020), https://cset.georgetown.edu/research/hacking-ai/.

Abstract: This paper is the second installment in a series on "AI safety," an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. The first paper in the series, "Key Concepts in AI Safety: An Overview," described three categories of AI safety issues: problems of robustness, assurance, and specification. This paper introduces adversarial examples, a major challenge to robustness in modern machine learning systems.


Diana Gehlhaus
Luke Koslosky
Kayla Goode
Claire Perkins

U.S. AI Workforce: Policy Recommendations

Executive Summary The U.S. artificial intelligence workforce, which stood at 14 million people in 2019, or 9 percent of total U.S. employment, has grown rapidly in recent years. This trend is likely to continue, as AI occupational employment over the next decade is projected to grow twice as fast as employment in all occupations. Such an important and increasing component of the U.S. workforce demands dedicated education and workforce policy. Yet one does not exist. To date, U.S. policy has been a piecemeal approach based on inconsistent definitions of the AI workforce. For some, current policy is focused on top-tier doctorates and immigration reform. For others, the conversation quickly reverts to STEM education. This report addresses the need for a clearly defined AI education and workforce policy by providing recommendations designed to grow, sustain, and diversify the domestic AI workforce. We use a comprehensive definition of the AI workforce--technical and nontechnical occupations--and provide data-driven policy goals. Our policy goals and recommendations build off of previous CSET research along with new research findings presented here. Previous research in this series defined the AI workforce, described and characterized these workers, and assessed the relevant labor market dynamics. For example, we found that the demand for computer and information research scientists appears to be higher than the current supply, while for software developers and data scientists, evidence of a supply-demand gap is mixed. To understand the current state of U.S. AI education for this report, we manually compiled an "AI Education Catalog" of curriculum offerings, summer camps, after-school programs, contests and challenges, scholarships, and related federal initiatives. To assess the current landscape of employer demand and hiring experiences, we also interviewed select companies engaged in AI activities. Our research implies that U.S. AI education and workforce policy should have three goals:  (1)  increase the supply of domestic AI doctorates, (2) sustain and diversify technical talent pipelines, and (3) facilitate general AI literacy through K-12 AI education. To achieve these goals, we propose a set of recommendations designed to leverage federal resources within the realities of the U.S. education and training system. Our first recommendation sets the foundation for facilitating these goals by creating a federal coordination function. We believe such a function is critical given ongoing fragmented AI education initiatives, and would harness the potential of the newly established National Artificial Intelligence Initiative Office for Education and Training within the White House Office of Science and Technology Policy. We recommend this office coordinate federal and state initiatives, convene key stakeholders to share lessons learned and best practices of statelevel AI education initiatives, and compile and publish information on AI education and careers on a publicly available "AI dashboard." The remaining recommendations advocate for a multipronged approach to implement policies across goals, including: Importantly, our recommendations prioritize creating multiple viable pathways into AI jobs to diversify the AI workforce and leverage all U.S. talent. Our research shows the dominant pathway to enter the AI workforce remains having a four-year college degree. However, this may be restricting the amount of talent entering the AI workforce, unnecessarily limiting opportunity for those who are otherwise qualified and able. Our recommendations therefore prioritize harnessing the potential of community and technical colleges, minority-serving institutions, and historically Black colleges and universities in training tomorrow's U.S. AI workforce. In addition, to promote alternative pathways into AI jobs, we propose that the National Institute of Standards and Technology work with industry to establish industry-accepted standards for AI and AI-related certifications to enhance their legitimacy. And as a top employer of technical talent, the federal government could modify its hiring criteria to lead by example. We hope that this report and recommendations advance the discourse on AI education and workforce policy. Now is a critical time to invest in training and equipping a globally competitive AI workforce for tomorrow. With concerted and targeted efforts, it is possible to lead the world in AI talent. Ultimately, an AI workforce policy inclusive of all of our report's elements is more likely to be the most effective. However, we also present our recommendations as a road map to guide U.S. policymakers in crafting an AI education and workforce agenda.
 Table of Contents Executive
 Introduction The acceleration of artificial intelligence adoption has profoundly changed how businesses, individuals, and governments conduct their day-to-day activities. As this trend continues, having an education and workforce policy specific to AI is essential to future U.S. competitiveness. In effect, AI education and workforce policy is now a national security priority. Such a workforce policy includes both adequately training the U.S. workforce for AI jobs and preparing the country's youth and adults for the changing nature of jobs resulting from AI and other emerging technologies. This report pulls together previous CSET research on the U.S. AI workforce to offer data-driven policy goals and recommendations designed to grow, sustain, and diversify the domestic AI workforce. This is the third and final paper in a three-part series on the U.S. AI workforce. The first defined, described, and characterized the supply of U.S. AI workers while the second evaluated the associated labor market dynamics. Although not in the three-part series, we also published several papers as part of our body of research on the U.S. AI workforce. This includes published papers on the prevalence of AI and AIrelated certifications, a comparative assessment of AI education in the United States and China, an overview of U.S. AI summer camps, the U.S. Department of Defense's (DOD) AI workforce, and a forthcoming examination of the latent potential of community colleges in training tomorrow's AI workforce.
 Current U.S. AI Education and Workforce Policy To date, most discussions on broader AI education and workforce policy have focused on STEM or on top-tier technical talent. For example, both the Global AI Talent Report  1  and Stanford University's annual Artificial Intelligence Index Report 2 break down the status of the AI workforce through a detailed analysis of PhDlevel computer research scientists. In its Final Report, the National Security Commission on Artificial Intelligence similarly emphasizes technical talent, before focusing its broader U.S. AI workforce discussion on K-12 STEM education.  3  Equating STEM policy to AI education and workforce policy is also apparent in the executive branch. The education and training arm of the new National Artificial Intelligence Initiative, housed in the White House Office of Science and Technology Policy (OSTP), is tasked with focusing on AI, but in practice, emphasizes STEM.  4  Similarly, other recent White House science and technology (S&T) education initiatives focus on STEM over AI or other emerging technologies. For example, in 2018 the White House issued a STEM education policy that included a five-year STEM education strategic plan. 5 AI education and workforce policy is, by our definition, more expansive than top-tier PhD-level computer science talent and STEM education. It includes policies for technical and nontechnical talent involved in safe AI design, development, and deployment.  6  The recommendations in this report are designed with that in mind. In this regard, AI education and workforce policy is more aligned with broader S&T education and workforce policy. Although much of the discussion surrounding STEM focuses on K-12 and four-year colleges, a large component of S&T education and workforce policy is focused on postsecondary education and training outside of the traditional four-year degrees, including associate's degrees, certificates, microcredentials, certifications, and apprenticeships. These pathways are vital for youth that do not attend four-year college, as well as for adult learners upskilling for S&T jobs. This means greater alignment with AI education and workforce policy; however, these alternative pathways generally receive little attention.
 A Unique Need for AI Education and Workforce Policy Five factors make the AI workforce distinct from the STEM and S&T workforce, and from the workforce of other emerging technologies. Such uniqueness creates a need for a dedicated AI education and training policy to adequately grow and cultivate a globally competitive AI workforce. Effective AI education and workforce policy must therefore understand the long road already traveled on S&T--particularly STEM--education and workforce policy. This includes the maze of federal programs, congressional proposals, and body of related research. Effective AI education and workforce policy must acknowledge the reasons why S&T workforce policy continues to be a challenging area for meaningful progress and change.
 Current S&T Education and Workforce Policy Challenges As a large part of AI education and workforce policy, it is important to understand why S&T--particularly STEM--education and workforce policy continues to be a major challenge even after decades of national prioritization. Since many of these challenges will also affect AI education and training policy, they must be considered within this context. There are three main challenges affecting U.S. S&T education and workforce policy: (1) the decentralized and fragmented structure and design of the U.S. education system; (2) lack of coordination and evaluation of federal and state-level initiatives for success, information sharing, and scaling, including public-private partnerships; (3) disagreement on the nature and extent of S&T workforce supply gaps and the need for policy interventions. First, the U.S. education system is a patchwork of public and private schools largely administered by individual states. The federal government provides relatively minor education oversight through the compilation and reporting of education statistics, along with promoting equitable access to education and enforcing a prohibition on institutional discrimination. However, similar to S&T education programs, these programs also suffer from a lack of coordination.  13  For example, the Workforce Innovation and Opportunity Act (WIOA), the primary law governing federal workforce development programs, spans 19 programs administered by three federal agencies. Each has its own data systems and reporting requirements. 14 There is also complexity in administration, as programs often require collaboration between state agencies, local industry associations, and employers, in addition to braided funding across public and private sources that creates yet more coordination challenges. Third, there are marked differences in opinions about the appropriate policy goal and federal role in growing and cultivating a S&T workforce. For example, a 2017 Congressional Research Service report noted the following viewpoints: "There is a shortage;" "There is not a shortage;" "More scientists and engineers are needed regardless of the existence of a shortage;" "There may be shortages [but only] in certain industries, occupations, or fields."
 AI Education and Workforce Policy Coordination A key lesson learned from the experience of S&T education and workforce policy is the large potential value of coordination. Facilitating equity in access to AI educational resources will be critical, and the S&T experience at the federal and state level shows this does not happen without concerted effort. S&T education and workforce policy highlights how important coordination is to maximize the impact and reach of federal and state programs. Coordination has a clear impact on how effective policy initiatives are; previously, no one federal entity had ownership over the entire federal STEM enterprise: ". . . too little attention has been paid to replication and scale-up to disseminate proven programs widely, and too little capacity at key agencies has been devoted to strategy and coordination."  18  Because of this, Congress elevated federal coordination of STEM initiatives to the executive branch.   25  To enhance our assessment for this paper, we supplemented this work with the creation of an AI Education Catalog and interviews with employers of AI talent.
 Size, Supply, and Labor Market Dynamics The U.S. AI workforce is sizable. In 2019, it consisted of 14 million workers, or about 9 percent of total U.S. employed. It is also growing rapidly relative to the total workforce. Over 2015-2019, the U.S. AI workforce grew 21 percent compared to 6 percent for total U.S. employed. Demand for AI occupations will likely be strong over the next decade, projected to grow twice as fast as for all U.S. occupations.  26  The U.S. AI workforce is also fairly concentrated in major metropolitan hubs, especially for technical talent. In absolute terms, large concentrations of workers were found in major hubs on the West Coast (see Figure  2 ). As a share of county employment, we also found large concentrations of technical workers in Seattle and San Francisco, as well as the Washington, D.C. metropolitan area. Product Team workers are more geographically distributed, although still concentrated in urban areas. Importantly, our analysis shows the technical AI workforce has a lack of racial and gender diversity. For Technical Team occupations, most workers are male, with few identifying as Black or Hispanic.  27  Figure  2  summarizes some of these key findings. Note: People who identify as Hispanic may be of any race. Our previous research also shows that the dominant pathway to enter the AI workforce is a four-year college degree. However, this may be unnecessarily limiting, potentially leaving qualified and able AI talent behind. While the majority of AI workers have at least a bachelor's degree, a sizable share of the AI workforce does not: we found 44 percent of those employed in Product Team jobs and one-third of those employed in Commercial Team jobs have less than a bachelor's degree.  28  The challenge is that many employers do not yet view alternative pathways such as certifications and other sub-baccalaureate credentials as viable market signals of qualification.  29  In terms of four-year degrees, we do find evidence of talent supply catching up to demand through a shift toward AI-related majors. Computer science and engineering were the two fastest growing fields of study over 2015-2018. Anecdotally, however, the surge in demand for computer science courses at many U.S. universities appears in some cases to be greater than their ability to supply them.  30  While we do not make claims of a blanket AI talent shortage, our assessment of AI labor market dynamics found that there are likely current gaps in supply relative to demand in the AI workforce which vary by occupation. For example, for computer and information research scientists there appears to be more demand than supply, while for software developers and data scientists evidence of supply-demand gaps is mixed. In contrast, we found no evidence of supply-demand gaps for user experience designers and project management specialists. 31
 AI Education in the United States In addition to labor market data, crafting effective AI workforce policies also requires knowing how and where to target AI education and training investments to prepare youth for AI careers. To do this, we need to understand:  (1)  what states are doing in AI education;  (2)  what types of AI curricula, programs and scholarships are available; and (3) where the accessibility and opportunity gaps are. This is not a simple task given the fragmented design of the U.S. education system, but it is essential for good policy. To understand the current U.S. AI education landscape, we created an AI Education Catalog to see what AI-related efforts are underway across the country. We collected data on curricula, afterschool programs, summer camps, challenges, contests, conferences, federal initiatives, and scholarships. 32 A large share of AI educational programming currently available is online, and comes from a mix of both private and nonprofit organizations. Investors in these programs and course offerings ranged from school districts and local universities to big tech companies like Amazon, Google, and Microsoft. 33 Importantly, the cost of these offerings varies widely, ranging from being free to requesting a quote (e.g., the cost varies on factors like the number of students in the classroom or the size of the school district). Summaries for each category, including descriptive statistics, are provided in Appendix B. By design, the catalog also provides insights on equity in access gaps. Overall, we find gaps in access and opportunity are more acute in rural and low-income school districts that lack access to AI education resources and opportunities. The catalog also shows that while there are free and accessible programs for students to learn about computer science and AI, the majority are online-requiring the student to have high-speed broadband and access to a computer. Previous CSET research found AI education in the United States is being provided in a piecemeal way that varies by state and places a heavier emphasis on computer science education. The decentralized structure of the U.S. education system creates both opportunities and challenges; schools have the flexibility to integrate innovative AI curricula designs and experiment with new approaches in pedagogy, but this could also exacerbate existing educational disparities. Many school districts lack access to adequate resources to support AI education or its evaluation, and there is no consensus or consistency in AI curricula. A detailed discussion of current efforts to integrate AI education at the K-12 and postsecondary levels is provided in the CSET report "AI Education in China and the United States: A Comparative Assessment." 34
 Perspectives from AI Employers In addition to these quantitative assessments, we also wanted to hear about labor market dynamics directly from AI employers. We conducted 10 interviews with companies engaged in AI activities ranging from AI research, software development, hardware design and production, and provision of AI-enabled service or capabilities. We include several of these insights in our proposed recommendations. While not a generalizable sample, we believe these discussions provide key insights that further validated our quantitative assessment. For example, our discussions provided insights on current AI labor market dynamics that could not be gleaned from federal survey data. This includes hiring conditions for technical talent: top-tier PhD-level talent was much harder to recruit than other technical talent. Moreover, while employers did not report difficulty in recruiting software developers and data scientists, they did report difficulty in finding "exceptional" software developers.  35  Moreover, consistent with our research on AI job postings data, all employers emphasized the importance of four-year college degrees as a minimum requirement for AI and AI-related positions. When asked about alternative credentials such as certifications, many acknowledged their relevance but none reported using them to hire AI talent. Several also noted that there were no clear standards for which certifications were at a quality equivalent to their hiring needs. Finally, while a few employers noted providing internal training and upskilling incentives for AI and AI-related skills, the majority did not.
 AI Workforce Policy Goals and Frameworks
 AI Workforce Policy Goals Although we use four categories to define the AI workforce, our research found the need for three targeted AI education and workforce policy goals: Goal 1: Grow, cultivate, and attract PhD-level technical AI talent while also improving diversity in the field. Goal 2: Ensure a diverse and sustainable supply of non-doctorate technical talent. Goal 3: Promote and provide AI literacy education for everyone. 36
 Education and Workforce Policy Frameworks To capture each part of the career and education pipeline, we believe AI education workforce policy should be considered through two frameworks: (1) Short-term (employer) This framework covers the talent management life cycle, from "hiring to retiring," as a way to depict the AI workforce challenges in the current recruiting and retention environment. The focus of this framework as shown in Figure  3  is on leveraging AI expertise in the current workforce. This includes good talent management in the form of advancement, continuous education and upskilling, promotion, and other professional development opportunities, as well as discouraging AI talent from leaving the field. Each phase represents a possible intervention point--mostly for employers as they assess their AI workforce needs. Center for Security and Emerging Technology | 20  Many stakeholders are involved in career and educational decision-making and workforce development. As such, it is important both frameworks also consider the entire stakeholder ecosystem. This means identifying all of the players that have roles and responsibilities in implementing AI education and workforce policy. Figure  5  shows each tier of stakeholder in a series of concentric circles, to demonstrate the many levels of influence from ego (self) to federal authorities and institutions. Each tier interacts with and on each other--as part of the larger society--to influence individuals' career and educational decisions.
 AI Workforce Policy Recommendations We consider our policy recommendations through the lens of our frameworks as they relate to our defined goals for AI workforce policy. That is, to grow, cultivate, and sustain a globally competitive cadre of technical and nontechnical AI talent. We use the evidence presented above to design policies that we believe are most likely to be effective in achieving the stated goals; however, we also strongly advocate where possible for rigorous program evaluation and iteration in design and implementation as appropriate. Our recommendations accept the legacy and challenges of previous efforts to design and effectively implement U.S. STEM education and training policies, as a part of AI education policy. We work within the current system--noting there are advantages and disadvantages to its structure--to provide actionable ideas that we believe can advance the goal of growing and cultivating U.S. AI talent. We also focus heavily on the role of the federal government, although much of this relies on coordination and cooperation with states, industry, academia, and nonprofit stakeholders. It also relies on buy-in from teachers, adult learners, and students, to integrate AI in the classroom and for youth to aspire to AI careers. We cannot generally mandate what nonfederal stakeholders do. However, we can propose a clear role for the federal government, and ways the government can best leverage its resources to engage stakeholders to promote AI workforce goals. That means making resources related to AI education easily and readily available, meeting stakeholders where they are, and providing avenues to coordinate, cultivate, create, collect, convene, and disseminate such information.  38  In addition, as a top employer of AI talent, the federal government can lead by example for creating alternative pathways into AI jobs. Finally, in encouraging youth, parents, and educators to embrace and aspire to AI careers, we propose recommendations specific to leveraging federal resources to create AI career and educational information. We designed these recommendations with EAST principles in mind--make it Easy, Attractive, Social, and Timely.  39  We present our recommendations in groupings by the nature of the proposed action. We provide a summary of the rationale for each type of action, followed by the specific recommendation(s). For all recommendations except the first, we also provide a parenthetical tag in the following structure:
 Empowering the National Artificial Intelligence Initiative Office for Education and Training The most successful AI workforce policy will include a coordinated effort between federal and state governments. Convene: Through these networks, provide states with support to design and integrate AI/computer science curricula into core subjects, and general AI literacy curricula, by hosting conferences, professional development sessions, and workshops that share best practices. Issue an annual report recognizing K-12 school districts and community and technical colleges that have made a major advancement in AI education as exemplars.
 Creating and Disseminating AI Educational and Career Information A critical part of the coordination function of the National Artificial Intelligence Initiative Office for Education and Training includes being a one-stop resource for AI career and educational information. Research shows that youth start planning for college and careers early, continuing and iterating throughout high school and complete the process at different times in their lives. For example, one study found "three main considerations have a sizeable impact and act together on youths' career and educational decisions: (1) the use of and reliance on influencers, (2) engagement in information-seeking activities during high school, and (3) social-emotional well-being, degree of focus, and selfmotivation."  46  While federal policy cannot directly engage all the above factors, it can make encouraging AI education and careers easy, attractive, social, and timely (EAST). We propose leveraging federal resources to create AI career and educational resources to raise awareness of AI career pathways. This is a critical step to ensuring youth, parents, educators, and counselors have access to the right resources at the right time as youth approach their educational and career decision-making processes. We propose this happen in two steps: (1) oversee the creation of better AI education and workforce data 47 and AI education and career materials, funded by Congress, and (2) oversee the dissemination of these materials.
 Rec
 Facilitating Alternative Pathways into AI Jobs As noted earlier, previous CSET research shows that a four-year college degree remains the dominant pathway into AI jobs. Moreover, employers are largely not using the vast array of AI and AI-related certifications available as a way to qualify applicants for AI jobs. 60 However, this limits access and opportunity. Our research found that this approach leaves talent behind that could thrive in AI jobs and reduce the potential for bias in AI through greater diversity. To
 Integrating K-12 AI Curricula and Course Design The U.S. K-12 education system has important advantages along with limitations when it comes to integrating new curricula, as noted in the STEM challenges section and in other CSET research.  74  Perhaps the biggest advantage is the ability for school districts to experiment with new curricula and pedagogical approaches, and establish partnerships to address the specific needs of that school or community. However, serious challenges also exist. Initiatives and partnerships will have limited effectiveness and reach if they are not consistently funded, tracked, evaluated, and scaled.  75  For example, even after more than a decade of concerted efforts by states, industry, and advocacy groups, just half of all U.S. high schools offer computer science education.  76  In the case of AI education, there is the added challenge that curricula are neither consistently defined nor agreed upon in importance. Still, to adequately train and equip tomorrow's AI workforce, school systems must start somewhere. Our AI Education Catalog shows there is a large range of online AI education programs and resources already available for teachers and students (see Appendix B). Moreover, our research found an impressive range of innovative pilot initiatives currently underway across the country to bring AI in the classroom. We provide two such examples in the text boxes below, based on interviews with project leads and supplemented by publicly available information.
 Example 1: Gwinnett County Public Schools (Georgia) Gwinnett County Public Schools (GCPS) is instituting a comprehensive K-12 AI-centric education curriculum spanning three elementary schools, one middle school, and one high school. Each grade from Pre-K to high school has a dedicated AI curriculum that will be integrated into core subjects. Unlike other AI education programs in the United States that depend on special academic placement, GCPS is creating these schools as part of its effort to meet a growing population, and students living in the district will be automatically assigned to these schools. The program aims to increase AI readiness for every student, regardless of background or aptitude, through six components: coding, applications, data science, computational thinking, problemsolving, and ethics. Each component of "AI Readiness" strives to align learning goals with the relevant skills and knowledge graduates need for postsecondary education or the workforce.
 AI Workforce Policy Recommendations: Summary Each recommendation aside from the first recommendation is tagged with the department and agencies ("agencies") involved, which framework this falls under (short-term or medium-/longterm), and which AI workforce policy goal it addresses (1, 2, 3).
 Empowering the National
 Creating and Disseminating AI Educational and Career Information Rec: Congress should appropriate funding for the BLS, U.S. Census Bureau, and NCSES to design, collect, compile, and publish occupation or skills-based data on the U.S. AI workforce along with education statistics from NCES.  Rec: Congress should fund targeted undergraduate scholarships and research fellowships that cover tuition, cost of living, and incentive pay for AI and AI-related expertise.
 Investing in PreK-12 AI Education and Experiences Rec: Congress should appropriate funding for federal grants to states for public K-12 schools to equip facilities with "AI labs" that enable hands-on learning along with virtual learning platforms for anytime remote or individualized learning.  Rec: Congress should authorize FCC funding to secure access to high-speed internet and high-speed internet devices, with eligibility for any K-12 youth participating in the free lunch program. (Agencies: FCC and USDA; Framework: Long-term; Policy goals: 1+2+3)
 Integrating K-12 AI Curriculum and Course Design Rec: Congress should appropriate funding for federal grants to rural and low-income school districts to integrate K-12 AI education designed using promising practices and proven education models. (Agencies: USDA, DOE, NSF, and Ed; Framework: Long-term; Policy goals: 1+2+3)
 Cultivating and Supporting K-12 AI Educators Rec: Congress should fund and task Ed to create a national repository of peer-reviewed AI teaching materials, including offthe-shelf AI-enabled products, experiment kits, and in-class labs. (Agencies: Ed and NSF; Framework: Medium-term; Policy goals: 1+2+3)
 Conclusion The technical and nontechnical workers that comprise the U.S. AI workforce have experienced rapid growth in recent years. Strong demand for workers with AI and AI-adjacent expertise will likely continue, as AI-enabled applications rapidly proliferate across the economy. Such an important and rapidly growing component of the U.S. workforce demands dedicated education and workforce policy. Yet one does not exist. Current AI education and workforce policy either focuses narrowly on top-tier PhD-level talent in computer science and engineering, or broadly on STEM education. This paper addresses the need for clearly defined and targeted policies to grow and cultivate the domestic AI workforce. Our approach to AI education and workforce policy is data-driven, clearly defining this segment of the workforce and analyzing the demand for and supply of AI talent across the United States. We also manually compiled an AI Education Catalog to understand the landscape of AI educational programming in the United States, and spoke with employers engaged in AI activities about their experience recruiting and retaining AI talent. Using this information, we argue AI education and workforce policy should consist of three goals. First, to grow, cultivate, and attract PhD-level technical AI talent. Second, to ensure a diverse and sustainable supply of non-doctorate technical talent. And third, to promote and provide AI literacy education for everyone. We designed our policy recommendations with these goals in mind. Our recommendations also appreciate the reality and challenges inherent in the design of the U.S. education system, the authorities of the federal government, and the long and persistent challenges of U.S. STEM education and workforce policy. Although AI education and workforce policy is bigger than STEM, we acknowledge the overlap and must consider the challenges accordingly. Given the complexities of federal education and workforce policy, our first recommendation calls for federal coordination through leveraging the new National Artificial Intelligence Initiative Office for Education and Training within OSTP. Our remaining recommendations emphasize investment in schools, students, and teachers; in better AI workforce data and AI education and careers information; and in the research that will better equip classrooms across all subjects to teach AI literacy. We also prioritize creating and cultivating multiple pathways into the AI workforce that includes leveraging the potential of community colleges, MSIs, and HBCUs, along with establishing national industry-accepted standards for AI certifications. We believe the most effective AI workforce policy will include all of the various elements outlined in this report. However, our recommendations can also be considered a road map for policymakers interested in understanding each segment of the AI workforce. This is the third paper in a three-part series on the U.S. AI workforce. We hope this report and recommendations advance the discourse on AI education and workforce policy. The United States is at a critical moment to invest in training and equipping a globally competitive AI workforce, and with dedicated effort it is possible for the country to lead in AI talent.
 Authors Diana Gehlhaus is a research fellow at CSET, where Luke Koslosky and Kayla Goode are research analysts. Claire Perkins is a former CSET student research analyst. the exception of the 4th level (blue) on the map used to represent nonprofit initiatives). • Programs and initiatives must be statewide and not confined to a specific region of the state • Program must be standalone and not just a tab on the respective state's department of education website highlighting STEM education For the state search, key terms were used such as "(insert state here) STEM education", "(insert state here) STEM Education initiatives", and "(insert state here) STEM programs." Programs and initiatives were evaluated using the criteria previously laid out. Once the STEM offices/programs and initiatives were evaluated, they were sorted into four groups as shown in the map. If a state was found to have both an office and an initiative, then the one with the highest-ranking program score (from 1-4 as outlined below) was used for the map classification. If a state also had a non-state-affiliated program, it was only colored blue if there was not a state-affiliated program for the state. This was done because there are a lot of similarities between initiatives and the STEM offices/programs creating a lot of crossover between the two. The criteria for each group are laid out below. 1. Level 1 (Yellow): The first level is most commonly used to classify programs that function as a working group or advisory council. These are the programs that are placing an emphasis on existing efforts rather than establishing new ones. In addition, they have very few if any sponsored STEM programs, activities, extracurriculars, and events.
 Level 2 (Orange): The second level is for programs that are more active than the first level and have sponsored STEM programs, activities, extracurriculars, and events. Their programs might not be the most active in all of the schools and might not reach every student but nevertheless are still being impactful.
 Level 3 (Purple): The third level is for programs that have proven metrics for their success. This could be through an increase in performance on standardized testing or increased rankings by a national party. Their sponsored STEM programs, activities, extracurriculars, and events are successful and are impacting the greatest number of students. 4. Level 4 (Blue): The fourth level is for programs that operate on the state level but do not have state involvement. This includes programs that are nonprofits that have a focus on K-12 STEM education in that particular state.
 Limitations When finding STEM programs the search was designed to be as broad as possible in an effort to try and identify the state programs given that it would be presumed they are one of the larger STEM programs in that given state. Even with a criteria for categorizing the programs listed, there was still a degree of subjectivity used in the process. Because there were limited strict boundaries used for the groups, some could potentially fit into multiple groups and a judgment call was made to place the program into one group over the other. as a barrier to access. The programs were generally inclusive of all K-12 educators rather than only offering a curriculum that is grade specific. One of the more unique examples of curricula for educators is the NVIDIA teaching kits, which provide free resources and materials necessary for educators to teach from AI curricula. In addition, some of the nonprofit curriculum programs have a mentorship program associated with them to help educators learn the material like Microsoft TEALS and Exploring Computer Science. Other programs just include online lesson plans or modules that educators can use. The MIT Media Lab offers a free example of the online curriculum, while Project Lead the Way provides a curriculum that requires educators to request a quote in order to access information on the price of the program. The AI Education Project designed a program that can teach AI literacy to middle and high school students even if they do not have access to computer science courses.
 After-School Programs The after-school programs found take place predominantly online and are run by private organizations. Their activity type ranges from curricula that children are able to engage with on their own time, to live after-school classes, and seasonal camps. After-school programs are predominantly operating online and accessible nationwide as a result. These programs serve as a curricular supplement rather than acting as an additional club or organization. Many schools also have their own sponsored coding clubs that are unique to the school. However, those were not documented for this catalog as that information is generally not publicly available. Most after-school programs are run by private organizations and are available for K-12 students. Their duration can be anywhere from one week of lessons, to year-round content. Funding and investors were difficult to find for these programs given that they are privately owned and self-sufficient. Their cost varies by the program as many of the at-your-own-pace modules are free, the live lessons range anywhere from $50-$450 a month, and the fiveday courses are around $450.
 Key Numbers • 91 programs identified or a graduated high school senior. However, the main target audience for the camps are high school students. We found a mix of virtual, day, and overnight camps, with the majority of them being hosted on a college campus (whether it is run by the university itself or a private organization). The majority of the camps are run by private organizations with iD Tech being the largest but there are also camps that are run by college programs and nonprofit organizations as well. The cost of the camps range from being free for every student to $7,375 with the bulk of the camps costing around $500-$1,000.
 Key Numbers • 447 AI and AI-adjacent summer camps identified in the United States • 47 programs are free for students • 148 programs are overseen by iD Tech Location and cost are the largest barriers to a child's access to AI summer camps. The camps are heavily concentrated in three states: California, New York, and Texas. This uneven distribution is further demonstrated by the fact that eight states alone host more camps than the other 42 states. In addition, most camps take place on a university campus, but the overwhelming majority of them are run by private organizations. Even though cost is often dependent on factors such as duration, host organization type, and whether or not it is overnight, most camps still have a price tag greater than $750, and this can be a deterrent for some families.
 Conferences/Challenges While there were less than 20 AI conferences identified, most of them were under one week in length, open to all educators and some to the public as well and took place in a virtual format. The conferences were hosted by a wide variety of organizations including universities, computer science associations, AI4All, government organizations like the DOD and the Defense Advanced Research Projects Agency (DARPA), and big tech companies like Google and Microsoft. The conferences are more geared towards educators to help provide them with the tools to implement AI and computer science in their classrooms. The challenges portion of our catalog was segmented to include challenges, competitions, hackathons, and robotics competitions. Challenges and competitions were programs that included computer science pitch competitions, cybersecurity competitions, computer science test competitions, and collaborative challenges. Because the focus of the catalog is for K-12 students rather than undergraduate students, most of the hackathons identified for the catalog were catered towards high school students with very few aimed at middle and elementary school students. The robotics competitions were not as explicit in citing age or grade eligibility as they only put "students" down for eligible participants but many were for middle and high school students. The challenges and competitions were more broad in eligibility with many being open to anyone but otherwise catered towards both high school and university students. Most of these challenges (challenges, competitions, hackathons, and robotics) were sponsored by a wide array of large companies, college universities, and government organizations like the DOD, NASA, DARPA, DOE, etc. Many of the challenges had more than one sponsor with many having more than 10 representing a large array of companies. California was found to have the most hackathons but due to the COVID-19 pandemic, most of the hackathons and other challenges were operating in a virtual format. Some competitions were as short as one day to as long as a yearlong competition process.
 Key Numbers • 100 programs identified • 14 programs targeted towards underrepresented groups • Price point not really as relevant since competitions free to access • 30 challenges identified • 14 conferences identified • 50 hackathons identified (although there are many more not identified in the catalog) • 8 robotics competitions identified The competitions (challenges, hackathons, and robotics) are all accessible in the fact that there does not appear to be a financial barrier in terms of entry. However, participation in many of these competitions requires school support. Whether school support be a faculty advisor, funds for building supplies, or access to curricula to learn the skills necessary to compete, schools with a lack of funding and resources could be at a severe disadvantage. Hackathons are an interesting program in the fact that many of the ones listed such as AIHacks, a student ran hackathon for female and gender nonconforming students, has a mentorship component embedded into it. Hackathons are the largest competition-based programs in the catalog with at least 50 documented. There were more hackathons that were not reported in the catalog because they either changed each year in terms of the topic or were more local competitions. Students are not required to have a background in coding to compete in one of these competitions and there is a lot of learning that takes place at these events. BEST Robotics is also an example of a robotics program that reaches 18,000 middle and high school students each year that is free to enter and has a mentorship aspect built in with the volunteers for the program. All of the AI challenges (including robotics and hackathons) incorporate AI and AI-adjacent skills but tend to be relatively unique. The University of Alabama has a cybersecurity capture the flag competition that operates in a similar fashion to Jeopardy!, the trivia quiz game show. For the conferences, they are more aimed towards educators and the greater community but a large one that takes place each year is the AI4ED conference.
 Federal Initiatives Federal initiatives include a range of programs that different federal departments and agencies have set up in order to support, recruit, or work with AI and AI-adjacent talent. These initiatives we found were primarily targeted towards undergraduate and graduate students, although a few were focused on K-12 outreach. The different program types include apprenticeships, challenges, fellowships, and internships. There was no dominant government agency hosting these programs, with representation from most intelligence, defense, and research-oriented agencies. The programs were predominantly targeted towards U.S. citizens and permanent residents. The duration of these programs vary given that this section includes a wide array of different programs with durations ranging from one summer to several years.
 Key Numbers • 70 programs identified Internships and fellowships made up the majority of the federal initiatives in effort to boost their workforces. The fellowships mostly covered the cost of students' undergraduate or graduate education in addition to providing them with a stipend in exchange for their participation for a set period of time to work for the respective agency. Because of the position tied to the internship or fellowship, almost all of the programs required citizenship or permanent residency status in addition to the student's ability to pass a federal background check of varying levels of clearances. The Graduate Research Fellowship program through the NSF is an example of one of the fully funded programs that is aimed towards funding graduate students' education with a set stipend and education allowance each year.
 Scholarships Given the breadth of scholarship offerings in the United States, it is likely that our catalog is not as comprehensive as the other sections presented in this report. Moreover, we found the purposes of each scholarship sponsoring organizations differs. The selection of scholarship programs was more sparse and difficult to find niche programs. Many of the programs identified were more general STEM scholarships that had the space for people with AI and AI-adjacent skills to apply.  32 The catalog will be fully available and searchable as an online interactive tool, to be released alongside this report and created in partnership with the AI Education Project. 33 Noting information is limited for many programs that are behind a paywall.  34  Peterson, Goode, and Gehlhaus, "AI Education in China and the United
 States." 35 This gets into quality differences within occupations, a separate topic not addressed in this report. 36 Having some AI literacy will facilitate career awareness and preparedness for responsible use of AI, especially for those in nontechnical AI careers.  37  For more on youth career and educational decision making, see Diana Gehlhaus, "Youth Information Networks and Propensity to Serve in the Military" (Pardee RAND Graduate School Dissertation, May 2020, published September 2021). Behavioural Insights Team, 2015. grants to improve teacher education in certain specific areas (e.g. instructing students with disabilities or leveraging new technology for instruction)" but it has never been funded. Finally, this could also be explicitly added to the NSF's Advanced Technological Education (ATE) program.  84  For many U.S. STEM college graduates, doctorates in current form are not a marketable value proposition in the same way as for international students. Pursuing a postdoc can have a negative financial impact for years. On average, ex-postdocs, "give up about one-fifth of their earning potential in the first 15 years after finishing their doctorates -which, for those who end up in industry, amounts to $239,970." See Devin Powell, "The price of doing a postdoc." • Creating and Disseminating AI Educational and Career Information • Establishing AI Education and Training Tax Credits • Investing in Postsecondary AI Education and Scholarships • Facilitating Alternative Pathways into AI Jobs • Investing in PreK-12 AI Education and Experiences • Integrating K-12 AI Curriculum and Course Design • Cultivating and Supporting K-12 AI Educators • Funding AI Education and Careers Research
 Summary .....................................................................................................
 Figure 1 . 1 Figure 1. CSET Research Series on the U.S. AI Workforce
 Figure 2 . 2 Figure 2. Summary of the U.S. AI Workforce
 Figure 3 . 3 Figure 3. Short-Term Policy Framework (Employer)
 Figure 4 . 4 Figure 4. Medium-and Long-term Framework (Life cycle)
 Figure 5 . 5 Figure 5. Stakeholders in AI Education and Workforce Policy
 ( Departments or agencies ["agencies"] involved [e.g., DOD]; Framework [short or medium/long]; Policy goal(s) being addressed [1 -Increase U.S. AI doctorates, 2 -Sustain and diversify nondoctorate technical talent, 3 -Promote AI literacy education]).
 : Congress should appropriate new funding for: -The U.S. Bureau of Labor Statistics (BLS), U.S. Census Bureau, and National Center for Science and Engineering Statistics (NCSES) to design, collect, compile, and publish occupation or skills-based data on the AI workforce along with education statistics from the National Center for Education Statistics (NCES). 48 (Agencies: NSF, U.S. Department of Commerce, Ed, and DOL; Framework: Medium-term; Policy goals: 1+2+3) -The BLS and Employment and Training Administration (ETA) to create school/career counselor and student/parent resources for AI technical and nontechnical careers, similar to an Occupational Outlook Handbook, along with a short video training for counselors. (Agencies: DOL; Framework: Medium-term; Policy goals: 1+2+3) -The National Artificial Intelligence Initiative Office of Education and Training to work with NSF, industry, and other stakeholders to provide guidance on approaching the online education offerings in the above dashboard to assist youth, parents, teachers, and counselors.
 ( Agencies: Ed; Framework: Long-term; Policy goals: 1+2+3) Rec: The NSF should allocate a portion of its ITEST program and Discovery Research preK-12 program (DRK-12) research dollars for AI education research. (Agencies: NSF; Framework: Long-term; Policy goals: 1+2+3) Rec: Ed should require all Regional Education Labs (RELs) to include AI education research as at least 15 percent of their research portfolios.
 Figure 6 . 6 Figure 6. Summary of AI Workforce Policy Recommendations
 ( Agencies: NSF, DOD, and DOE; Framework: Medium-term; Policy goal: 1)
 ( Agencies: NSF, DOE, DOD, and HHS; Framework: Long-term; Policy goals: 1+2)Rec: Congress should appropriate funding for federal grants to states for K-12 AI experiential learning opportunities.
 ( Agencies: NSF, DOE, DOD, and HHS, USDA; Framework: Long-term; Policy goals: 1+2)
 31 Gehlhaus and Rahkovsky, "U.S. AI Workforce: Labor Market Dynamics."

 Still, this report acknowledges the importance of STEM education policy as both a core part of S&T workforce policy and in AI education and workforce policy. That makes it equally as important to acknowledge STEM education and training policy is not a new phenomenon, but a very complex one that has gone on for decades. Despite showing mixed progress, recommendations from a 2010 report on K-12 education from the President's Council of Advisors on Science and Technology still ring true.7     1. Effective, safe, and trustworthy AI deployment and adoption is critical to future U.S. economic competitiveness and national security. AI is unique among emerging technologies, and rapidly evolving in terms of application and adoption. Without equitably providing quality AI education, we head toward greater economic and social disparities as demonstrated by the digital divide and accelerated by the COVID-19 pandemic. 5. To date there is no consistent definition of "AI education" or uniformly accepted curricula, as schools are still working to integrate computer science into classrooms. 2. The AI workforce is broader than the STEM and S&T workforce, including technical and nontechnical talent.3. Similarly, AI education is broader than STEM education, including concepts related to bias, governance, and ethics. This means STEM policy does not adequately address AI education and workforce policy needs.4. In a world characterized by ubiquity in AI, AI literacy of the population will be as important to national economic competitiveness as basic literacy and digital literacy. Without national elevation, U.S. states may have less urgency to make AI education and workforce training a priority.
 At the federal level, for example, we counted 16 agencies involved in STEM education programming. A 2012 study by the Congressional Research Service identified between 105 and 254 federal STEM education programs, depending on how the scope was defined. 10  State and local governments, as well as private industry and nonprofit organizations, also have their own STEM policies and educational agendas. 11  For example, the Utah STEM Action Center coordinates and communicates policies, resources, and events. Not surprisingly, there is limited coordination across federal and state governments. While efforts exist to coordinate initiatives and programs within the federal government (i.e., within OSTP) there is no obvious coordination with states or between states. As such, some companies and nonprofit entities are filling this gap. Organizations like Code.org and Microsoft's TEALS, for example, are working to provide equitable access to computer science education. However, without clear standards, the effectiveness of some of these programs is less clear, particularly in the case of forprofit coding bootcamps. 8  (We explain the design of the U.S. education system in detail in the CSET report "Education in China and the United States: A Comparative System Overview." 9 ) Second, federal funding for education comes from an assortment of government departments and agencies. 12 Many other states similarly have their own STEM offices or initiatives; a map of these by state is provided in Appendix A.In terms of federal workforce development programs, few are specifically designed for technical career fields such as AI. Most employment and training programs are intended for specific populations such as dislocated workers, low-income adults, and other underrepresented and underserved communities. One notable exception is the Registered Apprenticeship program through the U.S. Department of Labor (DOL), but the impact and reach of this program for AI-related careers is fairly minimal.
 15  Such a range of views held by economists and policymakers creates additional confusion in the scope and scale of federal and state S&T education initiatives and programs. This makes S&T education and workforce policy a very important, but extremely difficult challenge to effectively tackle without coordination. That sentiment has been articulated by both Democratic and Republican administrations: "Over the past few decades, a diversity of Federal projects and approaches to K-12 STEM education across multiple agencies appears to have emerged largely without a coherent vision and without careful oversight of goals and outcomes." 16 (2010) "More than a decade of studies by the National Academies of Sciences, Engineering, and Medicine document the need to prepare learners for the jobs of the future and identify a host of challenges and opportunities within the U.S. STEM education ecosystem." 17 (2018)
 Rec: The National Artificial Intelligence Initiative Office for Education and Training should be fully leveraged to coordinate federal and state U.S. AI education and training policy. We further recommend the National Artificial Intelligence Initiative Office for Education and Training submit, and Congress fund, a five-year funding proposal that enables all activities of the office. 41 There is a new opportunity for the federal government to coordinate AI education, training and workforce development policy, with the 2021 establishment of the Education and Training Office within the National Artificial Intelligence Initiative Office at the White House. 40 We recommend this office work under three guiding principles to facilitate equitable access and opportunity to AI education, training, and workforce development across the United Manage and oversee all AI federal education and training funding streams and initiatives. 43 Report annually on these efforts and relevant new legislation, working with NSTC, CoSTEM, the National Science Foundation (NSF), the U.S. Departments of Agriculture (USDA), Defense, Commerce, Education (Ed), Energy (DOE), and Labor (DOL), as well as the National Institute of Standards and Technology (NIST), and other relevant agencies. Coordinate these initiatives with states and relevant industry stakeholders. 44Compile: Track and report annually on federally funded AI and AIrelated initiatives, contests, grant challenges, scholarships, and AI education research in coordination with relevant federal agencies. Publish and maintain an AI/computer science education dashboard searchable by state for available after-school programs, contests, challenges, and conferences, scholarships, and online curricula.45     States: -Coordinate 42 -Compile -Convene Coordinate:
 Department of Education (Ed) to build in AI program information to its College Scorecard. (Agencies: Ed; Framework: Medium-term; Policy goals: 1+2+3) -A multi-platform, multi-year national AI careers public service announcement (PSA) campaign for youth and parents. 49 (Agencies: OSTP; Framework: Medium-term; Policy goals: 1+2+3) (Agencies: DOL, NSF, and Ed; Framework: Medium-term; Policy goals: 1+2+3) -The U.S.-Free virtual-
 chat career guidance and AI career assessments in conjunction with the AI careers landing page at BLS. (Agencies: DOL and Ed; Framework: Medium-term; Policy goals: 1+2+3)Establishing AI Education and Training Tax CreditsIt is important that employers participate in upskilling workers for AI jobs. However, our research finds only sporadic evidence this is currently happening. Across the companies we interviewed, few provided for or rewarded AI upskilling. For the companies that were embarking on related initiatives, they were generally for small cadres of AI talent or for broader categories of technical talent. This suggests there are opportunities for the federal government to help the private sector grow and cultivate AI talent internally by creating appropriate incentives.Rec:
 Congress should establish employer tax credits for employer-provided AI training, partnerships with community and technical colleges, and adult education programs that result in AI hires, 50 including from nondegree AI apprenticeships and other promising nondegree programs. 51 (Agencies: Internal Revenue Service; Framework: Short-term; Policy goals: 1+2+3)
 Investing in Postsecondary AI Education and Scholarships 52 Our research documents the clear need to increase the pipeline of U.S.-citizen doctorates, 53 as well as cultivate diverse and sustainable pathways for AI and AI-adjacent technical talent. This includes all postsecondary pathways, from undergraduates at twoyear and four-year institutions, to career and technical education programs, online boot camps, and adult learning centers.First, while there are many reasons for the lack of U.S.-citizen AI doctorates, research shows at least some of this is financially motivated. 54  Increasing the U.S. pipeline of PhD-level talent will address not only acute gaps in the supply of top-tier AI talent, but mitigate the challenge of AI and computer science faculty shortages and promote long-term U.S. AI competitiveness. Rec: Congress
 should fund NSF, DOE, and DOD graduate and postgraduate scholarships and fellowships for U.S. students pursuing AI and AI-related studies that are competitive with commensurate/peer private sector salaries. Moreover, NSF, DOE, and DOD should report annually on the number and demographic composition of applicants, awardees, and reviewers, along with the selection process. 55  For scholarships and fellowships funded by DOE and DOD, recipients should work collaboratively with a federally funded research lab during the funding period that is organized by either the university or scholarship administrator. (Agencies: NSF, DOD, and DOE; Framework: Medium-term; Policy goal: 1)Second, the U.S. government should provide greater access and opportunity for nontraditional students to pursue postsecondary coursework and majors in AI and AI-related fields of study.Rec:
 Congress should fund undergraduate scholarships and research fellowships related to AI and AI-related fields of study that cover tuition, cost of living, and incentive pay for acquiring AI and AI-related expertise .56  We further recommend the research fellowships be available to two-year and four-year college students, during the academic year and summers, and are coordinated by NSF AI Centers of Excellence in partnership with universities and federally funded research labs.57   Third, the U.S. government should increase the capacity of all postsecondary institutions to provide AI and AI-related courses and degree programs. This includes AI educational opportunities for students of all majors. AI education at the postsecondary level--particularly AI literacy, ethics, governance, and other introductory principles--should be integrated in all majors, not siloed into AI, engineering, or computer science departments with limited course capacity. Rec: Congress
 should appropriate funding for NSF to award grants to accredited two-or four-year postsecondary institutions for faculty and experienced AI and AI-related industry professionals to (1) complete professional development related to AI education, and (2) teach courses that integrate AI literacy, ethics, governance, and other introductory technical principles into the curricula. 58 (Agencies: NSF; Framework: Medium-term; Policy goals: 1+2)Rec: Congress
 should appropriate funding for NSF to award grants to sub-baccalaureate institutions, minority-serving institutions (MSI), tribal colleges and universities, and historically Black colleges and universities (HBCU) to equip facilities with "AI labs" for hands-on learning along with virtual learning platforms for any time remote or individualized learning. 59 A share of this funding should also support the hiring and training of laboratory support staff as well faculty to safely and effectively demonstrate, operate, and maintain all equipment and tools.(Agencies: NSF; Framework: Medium-term; Policy goals: 1+2)
 Congress should fund and task NIST, in coordination with industry and relevant trade associations, with establishing national, industry-recognized standards for AI certifications, stackable credential pathways, and sub-baccalaureate nondegree programs that can be program-accredited to enhance facilitate additional alternative pathways into AI careers, the U.S. government must better leverage the latent potential of community and technical colleges, MSIs, HBCUs, and other nontraditional pathways in training tomorrow's AI workforce.61     Rec: Congress should establish and fund a joint Ed, DOL, and NSF working group that oversees and administers a new AI workforce development grant program. 62 These grants will establish, track, and evaluate AI-specific public-private partnerships in a way that leverages the existing federal employment and training funding infrastructure. 63 We further propose prioritizing partnerships that create AI and AI-adjacent credential programs with guided pathways, stackable credentials, student advising, and other wraparound supports to increase opportunity and diversity in the AI workforce. Eligible participants should include state workforce boards, community and technical colleges, MSIs, HBCUs, and other underrepresented community groups and organizations. employer marketability. Employers lack clear, standard market signals from the value of certifications to use as a substitute for four-year degrees. If AI certification programs were accredited in the way colleges and universities are, it could legitimize certifications as a true pathway into the AI workforce. (Agencies: Ed, DOL, and U.S. Department of Commerce; Framework: Medium-term; Policy goals: 1+2)Regarding industry recognition of AI and related certifications, more coordinated effort is required at the national level to create industry-accepted national standards related to the quality of such certifications. To advocate that everyone needs a four-year degree misses the mark; four-year degrees are expensive64  and not all AI jobs need one. 65Rec:(Agencies: NIST; Framework: Short-term; Policy goals: 1+2) Rec:
 Congress should fund and task NCSES (within NSF) to issue a report on the new National Training, Education, and Workforce Survey with recommendations for enabling alternative pathways. 66 (Agencies: NSF; Framework: Medium-term; Policy goals: 1+2) Finally, our research found the federal government is a top employer of AI talent, particularly technical talent. 67 Redesigning eligibility criteria for AI and AI-adjacent jobs at the federal level could set an example for other employers to accept credentials outside of four-year college degrees. 68 Rec: Congress
 should task the Office of Personnel Management (OPM) to establish federal government hiring criteria and pathways for AI and AI-adjacent jobs that are based on portfolios of work and certifications. (Agencies: OPM; Framework: Short-term; Policy goals: 1+2)Investing in PreK-
 12 AI Education and Experiences 69 Much of the above extends to PreK-12 education, and we propose analogous recommendations to ensure equity in access and opportunity for exposure to AI and AI-related education. We further emphasize the importance of access to experiential learning opportunities such as site visits, demonstrations, internships and externships, contests, challenges, and hackathons, and in-person after-school programs. For students with limited access to in-person opportunities, online education is critical but current offerings are wide-ranging and have unknown effectiveness in motivating or preparing youth for AI careers. 70 Our earlier recommendations for the National Artificial Intelligence Initiative Office for Education and Training were designed to address this uncertainty about what is effective. We also recommend: Rec: Congress
 should appropriate funding for federal grants to states for public K-12 schools to equip facilities with "AI labs" that enable hands-on learning along with virtual learning platforms for any time remote or individualized learning. (Agencies: NSF, DOE, DOD, and U.S. Department of Health and Human Services (HHS); Framework: Long-term; Policy goals: 1+2) Rec: Congress should appropriate funding for federal grants to states for K-12 AI experiential learning opportunities. For example, field trips to robotics engineering facilities, alternative or virtual reality simulations, unmanned aerial vehicle or drone demonstrations, scholarships to participate in AI or cyber hackathons, summer programs, conferences, and competitions, AI- related youth apprenticeships, and financial support for extracurricular activities, clubs, and other after school programs. 71 (Agencies: NSF, DOE, DOD, HHS, and USDA; Framework: Long-term; Policy goals: 1+2) Rec: Congress should authorize the Federal Communications Commission (FCC) or USDA funding to secure access to highspeed internet and connected devices, 72 with eligibility for any K-12 youth participating in the free lunch program. 73 (Agencies: FCC and USDA; Framework: Long-term; Policy goals: 1+2+3)
 Congress should appropriate funding for federal grants to rural and low-income school districts to integrate K-12 AI education designed using promising practices 79 and proven education models In a partnership with Denver Public Schools, researchers at the University of Colorado Boulder received a five-year, $20 million NSF grant to lead a research collaboration examining the role of AI in education and workforce development. The project will focus on how middle and high school students and teachers can leverage AI to augment learning and collaboration inperson and virtually. Through rigorous program evaluation, the study hopes to reveal insights on how to leverage AI to advance educational outcomes through human-machine collaboration and understand how teachers, students, and AI interact. The research team will also codesign middle and high school AI curricula that will be integrated into core courses, with the help of education stakeholders and the local community. Such efforts aim to prepare teachers and students to understand, critique, and design new uses of AI. .80  While a stand-alone course is better than no course, ideally, funded proposals would integrate AI education within core subjects. AI education could include using AI as a learning tool; building AI technical skills, statistics competency, and critical thinking; AI literacy; and AI career education. Example 2: Denver Public Schools (Colorado) As noted above, our AI Education Catalog shows accessibility gaps for AI education are more acute in rural and low-income school districts that lack access to AI education resources and opportunities. 77 In the long term such inequities could result in additional disparities in workforce opportunities and outcomes across segments of the population, meaning these districts should be prioritized. We believe with dedicated resourcing and using evidence-based best practices in learning and course design, it is possible to ensure equitable access to AI education across the United States. 78 goals: 1+2+3) Cultivating and Supporting K-12 AI Educators Successfully integrating AI education in the classroom requires having qualified educators. Instead of setting a target benchmark for recruiting new STEM educators, which many initiatives are currently doing, we propose empowering all teachers with the materials and resources they need to teach AI education in the Rec: (Agencies: USDA, DOE, NSF, and Ed; Framework: Long-term, Policy classroom. This includes technical curricula and general AI literacy. To provide this curriculum, GCPS is certifying teachers through internal training programs, partnering with local businesses, and procuring technology and infrastructure needed for instruction. Rec:
 Congress should fund and task Ed to create a national repository of peer-reviewed AI teaching materials, including off- the-shelf AI-enabled products, experiment kits, and in-class labs. 81(Agencies: Ed and NSF; Framework: Medium-term; Policy goals: 1+2+3) Rec:
 Congress should fund and task Ed to create a complementary national clearinghouse for AI/computer science and related education programs within the What Works Clearinghouse. 82 (Agencies: Ed and NSF; Framework: Medium-term; Policy goals: 1+2+3) Rec:
 Congress should create a national fund through the NSF for K-12 teachers to pursue AI training via AI certifications, AI conference attendance, and hosting AI curriculum, pedagogy, and course design professional development (PD) sessions. 83 effectively, (3) understanding and addressing AI career attrition, and (4) how to reimagine AI and AI-related doctoral programs to facilitate and encourage more U.S.-citizen applicants and graduates. Rec: Congress (Agencies: NSF; Framework: Medium-term; Policy goals: 1+2+3) Funding AI Education and Careers Research Concurrent to efforts that elevate access to AI education and training is a need to conduct AI education research at all education levels. This includes: (1) defining technical and general AI education, (2) integrating AI education into the classroom
 should commission a National Academies of Sciences (NAS) study on reimagining the design of STEM doctoral programs .84  The study should describe and characterize the applicant pool for AI and AI-related doctoral programs, and, if more U.S. students are applying than there are spots available, understand the barriers to increasing U.S. enrollment (e.g., competitiveness in preparation vs. faculty and funding constraints). (Agencies: NAS; Framework: Medium-term; Policy goal: 1) Rec:
 Congress should fund Ed to issue two funding challenges: (1) for K-12 integration of AI curricula into core course offerings 85 and (2) for public K-12 schools to have a career counseling and exploration course that include (a) module(s) on AI careers. Course offerings should consider: AI as a learning tool; building AI technical skills, statistics competency, and critical thinking; AI literacy; and AI career pathways.
 Artificial Intelligence Initiative Office for Education and Training Rec: The National Artificial Intelligence Initiative Office for Education and Training should be fully leveraged to coordinate federal and state U.S. AI education and training policies, and Congress should authorize five years of funding.
 Congress should fund the U.S. Department of Education (Ed) to build in AI program information to its College Scorecard. Congress should fund more NSF, DOE, and DOD graduate and postgraduate scholarships and fellowships for U.S. students pursuing AI and AI-related studies that are competitive with commensurate/peer private sector salaries. Agencies should also report annually on the composition of applicants, awardees, and application reviewers. (Agencies: Ed; Framework: Medium-term; Policy goals: 1+2+3) Rec: Congress should fund a multi-platform multi-year national AI careers PSA campaign for youth and parents. (Agencies: OSTP; Framework: Medium-term; Policy goals: 1+2+3) Rec: Congress should fund free virtual chat career guidance and AI career assessments in conjunction with the AI careers landing page at BLS. (Agencies: DOL, NSF, Commerce, and Ed; Framework: Medium-term; Policy goals: 1+2+3) Rec: Congress should fund the BLS and ETA to create school/career counselor and student/parent resources for AI technical and nontechnical careers, similar to an Occupational Outlook Handbook, along with a short video training for counselors. (Agencies: DOL; Framework: Medium-term; Policy goals: 1+2+3) Rec: Congress should fund the National Artificial Intelligence Initiative Office for Education and Training, NSF, industry, and other stakeholders to provide guidance on approaching the online education offerings in the above dashboard to assist youth, parents, teachers, and counselors. (Agencies: OSTP, Framework: NSF; Medium-term; Policy goals: 1+2+3)Rec:(Agencies: DOL and Ed; Framework: Medium-term; Policy goals: 1+2+3)Establishing AI Education and Training Tax CreditsRec: Congress should establish employer tax credits for employerprovided AI training, partnerships with community and technical colleges, and adult education programs that result in AI hires, including from nondegree AI apprenticeships and other promising nondegree programs.(Agencies: IRS; Framework: Short-term; Policy goals: 1+2+3)Investing in Postsecondary AI Education and ScholarshipsRec:
 The Strengthening Career and Technical Education for the 21st Century Act): For example, NVIDIA offers a research grant for AI related research, the National Oceanic and Atmospheric Administration offers the Ernest F. Hollings Undergraduate Scholarship that provides undergraduate students with two years of academic assistance and a full-time summer internship, and the CTSA provides scholarships to educators to use for professional development opportunities focused on addressing inequity in computer science education. Programs are free to apply Scholarship programs' duration ranges from a one-time scholarship award amount to a set fellowship during the duration of a degree. We found that they are funded by a mix of government organizations, nonprofits, and private for-profit organizations. With the diverse section of programs in this category it is difficult to make generalizations about the data for the section since they all operate differently. provide financial assistance to underrepresented and underserved students. Provides grants to develop and support career technical education programs at secondary and postsecondary institutions. Programs funded by the legislation are administered by the U.S. Department of Education and funds are allocated to states by formula. Funds are used to help recipients attain an industryrecognized credential, a certificate, or a postsecondary degree.• Registered Apprenticeship system: A registry of apprenticeship programs, maintained by states and administered by the U.S. Department of Labor.Apprenticeship sponsors (e.g., employers, unions, industry groups, etc.) within these registered programs receive preferential treatment in federal systems, making them eligible for Workforce Innovation and Opportunity Act funding and other federal programs, and apprentices may receive a nationally-recognized credential. for STEM Education (Washington, DC: National Science & Technology Council, December 2018), https://trumpwhitehouse.archives.gov/wpcontent/uploads/2018/12/STEM-Education-Strategic-Plan-2018.pdf. The strategy states: "this report sets out a Federal strategy for the next five years based on a Vision for a future where all Americans will have lifelong access to high-quality STEM education and the United States will be the global leader in STEM literacy, innovation, and employment." Note it is unclear if this is still in effect or if it was de facto cancelled with the new administration, which happens and is an example of the challenges with such policy efforts. Here, technical talent includes those with knowledge, skills, and abilities to engage in the design, development and deployment of AI or AI-enabled capabilities. Nontechnical talent includes those in roles that complement technical talent such as user experience designers, compliance officers, and program managers. Nontechnical talent should have AI literacy. President's Council of Advisors on Science and Technology, Prepare and Inspire: K-12 Education in Science, Technology, Engineering, and Math (STEM) for America's Future (Washington, DC: Executive Office of the President, September 2010), https://nsf.gov/attachments/117803/public/2a--there is variation by geographic region or institutional tier or type. Reasons could include departmental funding challenges, faculty recruitment challenges, and the incentive for tenure-track faculty to conduct research and publish over teaching courses (which also has a significant impact on STEM doctorate experiences and attrition). Key Numbers • 29 programs identified • 14 programs overseen by the government • 17 programs targeted towards postsecondary students • 9 programs targeted towards professionals • Strategy • Perkins V (6 7
 40 "William M. (Mac) Thornberry National Defense Authorization Act for Fiscal Year 2021," H.R. 6395, 116 th Cong. (2020), https://www.congress.gov/bill/116th-congress/house-bill/6395.41  The legislation establishing this office also stipulates for funding as is submitted in the President's annual budget through the Office of Management and Budget.42 Even more essential if current proposed legislation passes which calls for a massive increase in K-16 STEM education programming and related grants; we propose below similar initiatives that are AI-specific and can be leveraged from this funding stream. See the House of Representatives' "National Science Foundation for the Future Act" (H.R. 2225, 117 th Cong. (2021), https://www.congress.gov/117/bills/hr2225/BILLS-117hr2225eh.pdf) and the Senate version "United States Innovation and Competition Act of 2021" (S. 1260, 117 th Cong. (2021), https://www.congress.gov/117/bills/s1260/BILLS-117s1260es.pdf). 43 We note this office is already supposed to be coordinating federal efforts across agencies. See "Summary of AI Provisions from the National Defense Authorization Act 2021" (Stanford Human-Centered Artificial Intelligence [HAI]), accessed July 26, 2021, https://hai.stanford.edu/policy/policyresources/summary-ai-provisions-national-defense-authorization-act-2021. See the one-stop AI education and training marketplace for Singapore, "AI Singapore," accessed July 26, 2021, https://aisingapore.org/. Meanwhile, OSTP's new STEM4ALL federal programs portal is more limited; see Networking and Information Technology Research and Development (NITRD), "R&D Workforce Training: Federal Agencies' STEM Internships, Scholarships, and Training Opportunities," accessed July 26, 2021, https://www.nitrd.gov/STEM4ALL/. 46 Gehlhaus, "Youth Information Networks and Propensity to Serve in the Military." 47 For challenges with existing AI workforce data collected and published by the federal government, see Gehlhaus and Mutis, "The U.S. AI Workforce: Understanding the Supply of AI Talent." could include creating new occupational categories, fielding a household and/or establishment survey supplement, and engaging in other representative data capture that accounts for declining survey participation rates. 44  For example, the National Governors Association, states' departments of education, state legislatures, and state workforce boards.45
			 (Agencies: NSF, DOE, and DOD; Framework: Medium-term; Policy goal: 2)
			 (Agencies: Ed; Framework: Long-term; Policy goals: 1+2+3)
			 © 2021 by the Center for Security and Emerging Technology. This work is licensed under a Creative Commons Attribution-Non Commercial 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/4.0/. Document Identifier: doi: 10.51593/20200087
			 National Security Commission on Artificial Intelligence, Final Report (Washington, DC: NSCAI, March 2021), https://www.nscai.gov/wpcontent/uploads/2021/03/Full-Report-Digital-1.pdf.4 See OSTP's National AI Initiative Office, "Education and Training," https://www.ai.gov/strategic-pillars/education-and-training/. To date, the office created a portal for federal STEM initiatives and scholarship opportunities; is overseeing a study on the Future of Work; and is advocating for school access to computing resources.5 Committee on STEM Education, Charting a Course for Success: America's
			 In addition to challenges in program effectiveness for improving workforce outcomes.14 See Appendix C for more information on WIOA and other relevant laws governing federal education and workforce policy.
			 We note our policy recommendations are complementary to NSCAI's NDEA II recommendation.
			 This could be similar to military recruiting, except directing people to AI

Abstract: 61   A forthcoming CSET brief on the latent potential of community colleges will explore this in greater detail. 62  This working group would then be part of the federal efforts that the National Artificial Intelligence Initiative Office for Education and Training coordinates. 63 For example, options to leverage existing funding streams include WIOA, Perkins Act, H1B revenue, and other related NSF programs. For more on how WIOA funds are allocated, see: (1) "Federal Sources of Workforce Funding," Urban Institute, accessed July 26, 2021, https://workforce.urban.org/strategy/federal-sources-workforce-funding; (2) Donna Counts, "Federal Funding for State Employment and Training Programs Covered by the WIOA," CSG Knowledge Center, accessed July 26, 2021, http://knowledgecenter.csg.org/kc/content/federal-funding-state-employmentand-training-programs-covered-wioa; and (3) U.S. Employment and Training Administration, "Current Grant Funding Opportunities," U.S. Department of Labor, accessed July 26, 2021, https://www.dol.gov/agencies/eta/grants/apply/find-opportunities. 64 For example, tuition inflation, rising student debt burdens, a 40 percent underemployment rate for recent college graduates, and rapidly increasing master's degree enrollments. 65 What credential employers demand is not necessarily what is actually needed to perform the associated tasks. Bachelor's degrees are a market signal to Center for Security and Emerging Technology | 66 employers; instead of proposing degree creep or degree inflation we propose training more talent up to the required level needed to be proficient. This requires making those options viable and credible signals to employers.