The Art of the Possible: How the U.S. Army Test and Evaluation Command Is Embracing Innovation and Data Transformation in a Changing World
By Jennifer Monahan
In a time of rapid change, innovation and transformation are the lifeblood of any organization. For the U.S. Army, the need to stay at the cutting edge of technology and innovation is crucial.
“Any organization that's standing still is going to be stagnant, and our adversaries aren't standing still,” explained Maj. Gen. Patrick Gaydon, commanding general of the U.S. Army Test and Evaluation Command (ATEC).
That imperative to innovate is at the heart of the Army’s partnership with Carnegie Mellon University (CMU) to train senior military leaders through the Data Driven Leadership (DDL) program at CMU’s Heinz College of Information Systems and Public Policy.
The Partnership
The Army Futures Command (AFC) is tasked with delivering the Army of 2030 and designing the Army of 2040. One way they’re pursuing this goal is by establishing the Artificial Intelligence Integration Center (AI2C), located in Pittsburgh, Pa.
“The Army's decision to establish a presence in Pittsburgh was deliberate, driven by Carnegie Mellon's reputation as a global leader in artificial intelligence (AI) and robotics,” explained LTC Thomas Dirienzo, senior operations research analyst at AI2C. “This initiative is part of the Army’s effort to create a pipeline of AI-trained professionals to integrate cutting-edge AI and machine learning (ML) capabilities across its functions.”
AI2C works closely with academia, industry, and other defense partners. They’ve developed a comprehensive suite of educational programs to support the Army's modernization efforts. The partnership with CMU allows the Army to train personnel through programs like master's degrees, AI technician training, and executive leadership programs like the DDL program.
“The DDL program has significantly contributed to the data maturity of participating organizations,” Dirienzo said. “It’s creating a ripple effect as attendees share their knowledge and recruit others to join.” With more than 850 students from across Army organizations having engaged in the DDL program during the past three years, AFC is seeing results.
Setting the Tone at the Top
ATEC reports directly to the Vice Chief of Staff of the Army, but it also provides direct support to AFC. ATEC is charged with ensuring that soldiers – today and in the future – have the right equipment to protect and defend their nation. They test and evaluate everything from vehicles and weapons to software and AI systems to ensure they meet Army standards for performance, safety, and reliability before being widely used by soldiers.
We must become data-centric if we want to be part of the national security solution for the future.U.S. Army. Maj. Gen. James Gallivan (Ret.)
In 2022, a team of approximately 25 senior leaders from across ATEC joined the fifth DDL cohort and spent five days learning key strategies for data management and analytics, data science, data-driven decision making, emerging technologies like AI and ML, change management, and data security to support the mission of the U.S. Army. Maj. Gen. James Gallivan (Ret.), who served as commanding general of ATEC from 2020 – 2024, was part of that cohort, along with James Amato, executive technical director of ATEC, and Christopher Barrett, executive director of the U.S. Army Evaluation Center (AEC) within ATEC.
Barrett remembers Gallivan challenging the ATEC team at the end of each session. What are we going to do with what we just learned? How are we going to apply this?
“Learning together in DDL was vital and transformative,” Gallivan said. “You get buy-in, and you realize that everyone can enter the conversation. It was a game changer.”
“We took a quantum leap in that week,” Barrett said. “Going through the program as a team helped everyone understand how to collect and structure data so that we could do a better job of analysis and storytelling.”
That ability to analyze and explain data is critical.
“We must become data-centric if we want to be part of the national security solution for the future,” Gallivan said.
Leveraging Data for Impact and Efficiency
ATEC’s role is similar in some ways to Consumer Reports, explained Amato. They ask questions like, If we give this weapons system to a unit, can they more effectively complete their mission? Are they safer? Are they more likely to survive a cyber-attack? Can they keep it in the fight for the duration? Do they have the right spare parts and tools they’ll need if it breaks? ATEC provides an unbiased assessment of everything they know about the system, including what it does well and where it struggles.
Using the data management and analysis strategies they learned in DDL has transformed ATEC’s ability to do this work efficiently and effectively.
“It used to take us a month to just give a subjective assessment back to the decision-makers of how the new Army jeep did. We now do that in seven days,” Gallivan said. “It used to take us three to six months to do a quantitative report about how a system performed. We now do that in 30 days.”
Streamlined reporting times are one positive outcome. Increased capacity and efficiency are two others.
“Now we can do thousands of runs with a given weapons system in the virtual space and figure out what are the few cases that really matter. Those are the ones that we can go out and test in the live dirt to verify,” Amato said. “Alternatively, we can identify where we have least confidence in our digital tools and use live testing to be able to get the data we need to make our tools better.”
A test of the Army's new Precision Strike Missile at U.S. Army White Sands Missile Range. Photo courtesy of ATEC.
Night testing of the Stryker Infantry Carrier and Common Remote Weapon Station - Javelin (CROWS-J) at U.S. Army Aberdeen Test Center. The CROWS-J allows the crew to engage threats from a distance and stay under cover when firing. Photo courtesy of ATEC.
Implementing Change: Strategy and Infrastructure
Gallivan spearheaded ATEC's shift toward data-centricity by fostering a shared understanding across the organization. Through town hall meetings at all 21 ATEC locations, he emphasized the critical need for the command to evolve or risk losing resources and relevance. The transformation is ongoing, with benefits expected to fully materialize by 2028.
“We did a lot of listening to our leaders and to the workforce to hear their excitement and also their fears about these changes,” Gallivan said.
The initial stages involve patience and investment. Early hires, focused on digital engineering, spent a year laying the groundwork. They conducted pilot projects to get grassroots ownership and score some early wins. They leveraged resources like Power BI and Microsoft Cloud tools to create data lakes.
“We mapped the entire organization, who we connect to, what data we collect, and what data we use,” Gallivan said.
Their efforts are now driving tangible progress. ATEC added a Chief Data and Analytics Officer (CDAO) position at the headquarters level.
“All of our test centers are now connected and sharing data. We now have trained CDOs [chief data officers] in every single command,” Gallivan said.
ATEC inventoried all its modeling and simulation tools. The ongoing effort allows ATEC to identify gaps and communicate to the Army and industry what is needed for testing new systems, demonstrating real-world impact.
The Analytics, Artificial Intelligence, and Digital Engineering Directorate (AAIDED) was created three years ago to establish a group of people who would serve as change agents and demonstrate the value of data analytics. AAIDED Director Elizabeth Jones said that as ATEC continues its transformation, their mission continues to evolve as well.
We’re building the infrastructure and the architecture to support analytics, AI, and digital engineering.AAIDED Director Elizabeth Jones
“We’re building the infrastructure and the architecture to support analytics, AI, and digital engineering,” Jones said. Those changes are evident in some measurable outcomes.
Concrete Impact: Business Processes and Staffing Needs
“Our staff has transitioned away from PowerPoint briefings to Power BI dashboards so that we can see data in real time and make decisions,” Amato said. The change has allowed their team to evaluate better whether financial resources are being used effectively. It has also enabled them to anticipate hiring needs for 2030 and to recruit or upskill the staff who will be testing new technologies in AI, robotics, synthetic biology, and quantum computing.
Gaydon, who assumed command of ATEC in July, highlighted G1 (personnel office) work in this area.
“I'm extremely impressed with what they’re doing in talent management, recruiting, retention, and understanding the demographics of our workforce,” Gaydon said.
Concrete Impact: The AI Challenge
In 2023, ATEC initiated its first AI Challenge.
“Our AI team within the AEC saw an opportunity to supplement our academic learning with some on-the-job training,” Jones said. “The AI challenge is an opportunity to learn and build solutions to real ATEC problems.”
The first challenge focused on automating a manual process where acoustic sensors are used to triangulate mortar impact points. Previously, analysts manually searched through sine wave data to determine impact points. The challenge invited six teams to develop AI algorithms to automate this process. Amato said that the winning solution, now in daily use, has reduced the analysis time by 90%.
Testing Body Armor to Protect Soliders
Body armor plate of the type used in the 2024 ATEC AI Challenge, along with X-rays of body armor used in the Challenge. All of the X-ray images show defects. Photos courtesy of ATEC.
2024 AI Challenge Winner
Young Bang (right), principal deputy assistant secretary of the Army for Acquisition, Logistics and Technology, presents the award for winning the 2024 ATEC AI Challenge to Fikru Gennese (center) of Team "No to Destructive Testing," along with James Amato (left), ATEC executive technical director. Photo courtesy of ATEC.
Barrett estimated that each of these AI tools are already saving or will save approximately 1,000 manual work hours each year.
Concrete Impact: Data Summit
ATEC recently hosted its third annual Data Summit. Brian Kelly, CDAO for ATEC, said the summit highlights internal accomplishments, introduces new solutions, and outlines future goals. It features live demonstrations of ATEC capabilities, offering a hands-on view of how things are working, rather than relying solely on reports or presentations.
“We are knocking down targets, knocking down problems that are solving things for the entire testing and evaluation community,” Kelly said. “We're driving the standards that way by knowing that we're solving problems that can be leveraged across the entire enterprise.”
Concrete Impact: More Efficient Testing
“Before DDL, we were data rich. But now we’re data-centric,” Gallivan said. “Our reputation in the Army was that ATEC costs too much and takes too long to test. We’ve reduced inefficiencies. There has been a shift to more streamlined, data-centric testing methods.”
Improved data collection tools now allow ATEC to combine multiple tests into one, reducing redundant efforts. For instance, testing a jeep on both rough and paved terrain can be done simultaneously, saving time and resources. Remote and distributed testing also enhance efficiency by enabling evaluators to analyze data from different locations without being on-site.
Concrete Impact: Mindset
“One of the biggest differences after DDL is the mindset across the organization,” Jones said. “The desire to know what's in the realm of the possible, the desire to be a part of that, the desire to do it.”
David Ulicne, executive director of Heinz College’s executive education programs and the DDL program director, agreed.
“What we've seen across the 850 Army leaders who have gone through the program is that there is momentum for transformation,” Ulicne said. “With the pace of disruptive technology today, especially with the data-centric technologies of AI/ML, the technology is so far outpacing the people in the process. Driving data transformation is important for any organization today, and especially for the Army. ATEC is a great example of what can be accomplished by setting a clear tone from top leaders in an organization.”
Driving the Art of the Possible into the Future
The Army’s transformation efforts demonstrate a commitment to staying ahead in a rapidly evolving technological landscape. By embracing data-centric strategies, leveraging cutting-edge tools, and fostering a culture of innovation, ATEC is not only improving testing efficiency but also ensuring that its workforce is equipped to meet future challenges. As the Army continues to prioritize innovation and transformation, its partnership with CMU will play a critical role in shaping the Army of 2030 and beyond.
“It’s such a great time to be inside the Army where we're all rowing together to solve hard problems collectively,” Amato said. “The Army leadership has given us a clear set of priorities that we're all focused on, and they’ve given us the latitude to be creative and innovative with our ideas and initiatives.”