Within government, there are existing projects that are more mature than AI chatbots and are immediately ready to deliver more efficient government operations. Through a partnership between three offices, the Department of State is seeking to automate the cumbersome process of document declassification and prepare for the large volume of electronic records that will need to be reviewed in the next several years. The Bureau of Administration’s Office of Global Information Services (A/GIS), the Office of Management Strategy and Solutions’ Center for Analytics (M/SS CfA), and the Bureau of Information Resource Management’s (IRM) Messaging Systems Office have piloted and are now moving toward production-scale deployment of AI to augment an intensive, manual review process that normally necessitates a page-by-page human review of 25-year-old classified electronic records. The pilot focused mainly on cable messages which are communications between Washington and the department’s overseas posts. 

The 25-year declassification review process entails a manual review of electronic, classified records at the confidential and secret levels in the year that their protection period elapses; in many cases, 25 years after original classification. Manual review has historically been the only way to determine if information can be declassified for eventual public release, or exempt from declassification to protect information critical to our nation’s security.

However, manual review is a time-intensive process. A team of about six reviewers works year-round to review classified cables and must use a triage method to prioritize reviewing the cables most likely to require exemption from automatic declassification. In most years, they are unable to review every one of the between 112,000 and 133,000 electronic cables under review from 1995-1997. The risk of not being able to review each document for any sensitive material is exacerbated by the increasing volume of documents. 

This manual review strategy is quickly becoming unsustainable. Around 100,000 classified cables were created each year between 1995 and 2003. The number of cables created in 2006 that will require review grew to over 650,000 and remains at that volume for the following years. While emails are currently an insignificant portion of 25-year declassification reviews, the number of classified emails doubles every two years after 2001, rising to over 12 million emails in 2018. To get ahead of this challenge, we have turned to artificial intelligence. 

Considering AI is still a cutting-edge innovation with uncertainty and risk, our approach started with a pilot to test the impact of the process on a small scale. We trained a model, using human declassification decisions made in 2020 and 2021 on cables classified confidential and secret in 1995 and 1996, to recreate those decisions on cables classified in 1997. Over 300,000 classified cables were used for training and testing during the pilot. The pilot took three months and five dedicated data scientists to develop and train a model that matches previous human declassification review decisions at a rate of over 97 percent and with the potential to reduce over 65 percent of the existing manual workload. The pilot approach allowed us to consider and plan for three AI risks: lack of human oversight of automated decision-making, the ethics of AI, and overinvestment of time and money on products that aren’t usable.

The new declassification tool will not replace jobs. The AI-assisted declassification review process requires human reviewers to remain part of the decision-making process. During the pilot and the subsequent weeks of work to put the model into production, reviewers were consistently consulted and their feedback integrated into the automated decision process. This combination of technological review with human review and insight is critical to the success of the model. The model cannot make a decision with confidence on every cable, necessitating that human reviewers make a decision as they normally would on a portion of all cables. Reviewers also conduct quality control. A small, yet significant, percentage of cables with automated confident decisions are given to reviewers for confirmation. If enough of the AI-generated decisions are contradicted during the quality control check, the model can be re-trained to consider the information that it missed and integrate reviewer feedback. This feedback is critical to sustaining the model in the long term and for considering evolving geopolitical contexts. During the pilot, we determined that additional input from the Department’s Office of the Historian (FSI/OH) could help strengthen future declassification review models by providing input about world events during the years of records being reviewed.

There are ethical concerns that innovating with AI will lead to governing by algorithm. Although the descriptive AI used in our pilot does not construct narrative conversations like large language models (LLMs) and ChatGPT, it is designed to make decisions by learning previous human inputs. The approximation of human thought raises concerns of ethical government when it replaces what is considered sensitive and specialized experience. In our implementation, AI is a tool that works in concert with humans for validation, oversight, and process refinement. Incorporating AI tools into our workflows requires continually addressing the ethical dimensions of automated decision-making. 

This project also saves money — potentially millions of dollars’ worth of personnel hours. Innovation for the sake of being innovative can result in overinvestment in dedicated staff and technology, which is unable to sustain itself or end up in long-term cost savings. Because we tested our short-term pilot within the confines of existing technology, when we forecast the workload reduction across the next ten years of reviews, we anticipate an almost $8 million savings on labor costs. Those savings can be applied to piloting AI solutions for other governmental programs managing increased volumes of data and records with finite resources, such as information access requests for electronic records and Freedom of Information Act requests.

Rarely in government do we prioritize the time to try, and potentially fail, in the interest of innovation and efficiency. The small-scale declassification pilot allowed for a proof of concept before committing to sweeping changes. In our next phase, the Department is bringing the pilot to scale so that the AI technology is integrated with existing Department technology as part of the routine declassification process.

Federal interest in AI use cases has exploded in only the last few months, with many big and bold ideas being debated. While positive, these debates should not detract from use cases like this, which can rapidly improve government efficiency and transparency through the release of information to the public. Furthermore, the lessons learned from this use case – having clear metrics of success upfront, investing in data quality and structure, starting with a small-scale pilot — can also be applied to future generative AI use cases as well. AI’s general-purpose capabilities mean that it will eventually be a part of almost all aspects of how the government operates, from budget and HR to strategy and policy making. We have an opportunity to help shape how the government modernizes its programs and services within and across federal agencies to improve services for the public in ways previously unimagined or possible.  

Matthew Graviss is chief data and AI officer at the Department of State, and director of the agency’s Center for Analytics. Eric Stein is the deputy assistant secretary for the office of Global Information Services at State’s Bureau of Administration. Samuel Stehle is a data scientist within the Center for Analytics.

The post The Department of State’s pilot project approach to AI adoption appeared first on FedScoop.

Secretary Antony Blinken’s modernization agenda and the department’s first-ever Enterprise Data Strategy — signed one year ago this week — are changing the game. The Enterprise Data Strategy (EDS) helped the Department surge data policy, data engineering, and data science resources to high-priority, high-visibility mission and management topics in successive, six-month “data campaigns.” Already, data is informing decisions across management issues like anomalous health incidents and cybersecurity, crisis operations like the Afghanistan retrograde, and foreign policy issues including strategic competition with China and U.S. engagement in multilateral organizations. We see our unique “campaign” approach to delivering on the data strategy as a key reason the State Department is currently seeing so much success.

What makes our campaigns different? Oftentimes in government, a strategy is blessed by senior leaders and organizations commit to impossible goals, even over the long term. And organizational and cultural change in large government organizations is famously hard, even with senior leader buy-in.

Instead, we recognize that like any technical project, priorities change quickly. No proposed implementation plan would survive first contact with mission realities. The EDS intentionally promotes an abstract three to five-year “implementation roadmap” centered on delivering decision advantage through analytics, effectively governing data as a strategic asset, building a culture of data literacy, and developing the needed tech backbone. 

The data campaigns are vehicles for applying the EDS to targeted priorities with which the department workforce is already intimately engaged. For example, the first two data campaigns were strategic competition with the People’s Republic of China (PRC), and workforce diversity, equity, inclusion, and accessibility (DEIA). Our bet was that if we could prove data and analytics — and a cultural mindset supportive of them — could deliver real, tangible results on strategic competition and workforce DEIA, people throughout the organization would start to trust that data and analytics could help with their mission too. Change by showing, not telling. 

Animating philosophy and campaign structure

The animating philosophy for our data campaigns is “12-8-4”: We plan to accomplish 12 things, we’ll get eight of them done, and that will still be four more than anyone thought possible. We back up this philosophy with an aggressive surge of resources to apply analytics to a topic with a go-fast, sprint-to-the-finish mentality. We believe the core benefit of a campaign-based approach to digital transformation is that high-priority mission and management topics motivate people to partner, get to yes, and deliver results more quickly and robustly than abstract strategic goals. No one gets excited about showing up to a “data quality working group,” but a working group on how to unlock HR data to dismantle structural barriers to racial equity in our diplomatic workforce? That’s an effort worth getting behind. 

Functionally, a campaign is all about integrated, cross-functional delivery across our own teams. Each campaign is assigned not just dedicated analytics teams, but also communications staff, data policy and data-sharing agreement specialists, and full-stack engineers. Cross-functional delivery ensures we are bringing the right tools to the problem. If what we need is a new data policy, not machine learning, we shouldn’t be technological determinists just because it sounds more innovative. Often what our customers need first and foremost is systematized, sustainable data management and information synthesis, not predictive algorithms. Our cross-functional teams ensure we have and deliver the right tools to the problem. 

Crucially, the campaign construct attracts executive attention. Working on the highest-priority issues means analytics teams get the attention and support needed from senior leaders to actually implement the strategy in the face of inevitable technical and bureaucratic blockers. In exchange, campaign teams are accountable for actually delivering value — not producing “shelfware” strategy-implementation status reports. Fortunately, this visibility has helped deliver results. And we’ve been able to build trust among data skeptics and help leaders and staff alike understand the value of data to their own goals – why analytics is not just a “nice to have,” but a “must-have” on foreign assistance, competition with China, diplomatic engagement, or crisis operations.

Data campaigns and beyond

To help bring better data to the massive challenge of diversity, equity, inclusion, and accessibility (DEIA), we took a collaborative approach with Ambassador Gina Abercrombie-Winstanley and her new Office of Diversity and Inclusion, the Bureau of Global Talent Management, the Office of Civil Rights, and the Office of the Legal Advisor. After a six-month DEIA data campaign, we produced a baseline assessment available to the entire department to bring hard numbers to the challenge of DEIA. One career ambassador told us this effort was “the most transparent and actionable information on DEIA” they had seen in their 30 years of service. To build this report, our campaign team worked with partners throughout the agency to develop the first DEIA data policy in the history of the U.S. government. The policy has made HR information more transparent and accessible while protecting individual privacy and meeting all legal requirements. 

Our China work has also been essential to the Department’s growing focus on strategic competition with the PRC. First, we developed an analytics platform tracking PRC activities around the world, which is regularly used to inform our foreign policy, strategic planning, global presence, and resource allocations. We also took a hybrid subject matter expertise survey and algorithmic approach to recommend foreign assistance projects under the Countering PRC Influence Fund, aligning foreign assistance to strategic priorities. We have also built a prototype platform to derive insights from millions of diplomatic cables at scale using machine learning, which helps the State Department make fuller use of our most important novel data asset: on-the-ground reporting from our worldwide network of diplomatic posts. 

The value of data to diplomacy that these campaigns and other efforts have shown has begun to pay dividends elsewhere. We’ve seen record enrollment in the Foreign Service Institute’s training courses on data literacy and analytics and the inclusion of data literacy in promotion precepts for the Foreign Service. We are also successfully competing for the top data science talent in the industry. Over the past year, dozens of new data scientists joined the State Department across an array of mission areas and bureaus – not just the Office of Management Strategy and Solutions’ Center for Analytics. And State’s current initiative to hire at least 50 data scientists for positions across the department received over 400 applications in only a few days. As Deputy Secretary for Management and Resources Brian McKeon said to Congress, it may surprise you to learn that top data scientists want to serve their country at the State Department, and are leaving top jobs in industry and academia to do so. With the unique opportunity afforded by the EDS and our data campaigns, this does not surprise us. 

When British Prime Minister Lord Palmerston received his first telegraph message in the 1860s, he exclaimed, “My God, this is the end of diplomacy!” Yet here we are, and needless to say, technology will never be the “end” of diplomacy. Rather, by infusing the art of diplomacy with modern technology and the science of data, we are strengthening the digital backbone of America’s diplomatic corps and ensuring the country’s oldest cabinet agency delivers results for the American people far into the future — one campaign at a time. 

Matthew Graviss is the chief data officer at the Department of State. Garrett Berntsen is his deputy.

The post Opinion: How do you make the State Department data-driven? One campaign at a time appeared first on FedScoop.