New AI will teach soldiers to dress wounds, fly helicopters in AR

DARPA has demo’d new AI assistants to help soldiers with complex tasks.

DARPA (the Defense Advanced Research Projects Agency) is developing AI assistants to help military personnel perform complex tasks — by showing them step-by-step instructions in augmented reality (AR).

AI assistants: Depending on the day, military personnel might be tasked with bandaging battle wounds, repairing fighter jets, or flying them. To make it easier for them to perform such complex duties quickly and correctly, DARPA established the Perceptually-enabled Task Guidance (PTG) program in 2021.

“What we need in the middle is an assistant that can … provide step-by-step guidance.”

Bruce Draper

The idea behind the program is that future soldiers will wear sensors, such as microphones and head-mounted cameras, that collect data on what they’re hearing, seeing, and saying. Instructions from an AI assistant on how to complete tasks could then be displayed to them through AR goggles.

“What we need in the middle is an assistant that can recognize what you are doing as you start a task, has the prerequisite know-how to accomplish that task, can provide step-by-step guidance, and can alert you to any mistakes you’re making,” said Bruce Draper, manager of the PTG program, at the time it was announced.

“The idea is partly for us to see their progress, but also for all of them to work together.”

Bruce Draper

What’s new? Since then, DARPA has awarded PTG contracts and grants to researchers from NYU, Northrop Grumman, and more. Recently, the groups came together at MIT’s Hacker Reactor to demonstrate the first versions of their AI assistants. 

“The idea is partly for us to see their progress, but also for all of them to work together so we get this sort of cross-stimulation of groups working with each other,” said Draper.

 “Cooking is a really good practice domain for all kinds of other highly skilled tasks.”

Bruce Draper

Order up: The first three types of tasks DARPA wants PTG systems to assist with are mechanical repair, battlefield medicine, and pilot guidance.

For this first demo, though, the teams weren’t expected to show how their AI assistants could help fly helicopters — instead, they demonstrated how the systems could help someone follow a cooking recipe.

“[Cooking is] visually quite complex,” said Draper. “There’s specialized terminology. There are specialized devices, and there’s a lot of different ways it can be accomplished, so it’s a really good practice domain for all kinds of other highly skilled tasks.”

“I’m really impressed at how quickly people are making progress toward the goals,” he added.

In addition to showing DARPA their in-development systems, the teams also had a chance to demo them for some of the military members likely to actually use the tech — this allowed the researchers to get feedback on what is or isn’t working from the target end user.

“At the same time, [the military members] hopefully get an idea of where the technology is going so they can bring that back to their leadership with regard to what future possibilities are,” said Draper.

“We believe that our system would not only be useful to DARPA and the military, but also to regular users.”

Ehsan Elhamifar

Over-achieving AIs: It’s not clear when DARPA expects the military to be able to deploy PTG systems, but some of the teams are ahead of schedule, according to Draper, already demonstrating how their AIs could do battlefield medicine tasks.

It’s possible civilians might get a chance to use the AI assistants, too — other tech we use everyday, such as GPS and the internet, was developed for the military before it made its way to the consumer market.

“We believe that our system would not only be useful to DARPA and the military, but also to regular users and people who want to perform new or complex tasks,” said Ehsan Elhamifar, a professor at Northeastern University and one of the PTG grant recipients.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at tips@freethink.com.

Related
Farmers can fight invasive insects with AI and a robotic arm
As the invasive spotted lanternfly threatens to expand its range, Carnegie Mellon researchers are developing a robot to fight back.
Google unveils AI try-on feature for shopping
Google’s AI-powered virtual try-on feature lets shoppers see what an article of clothing would look like on a wide range of models.
GitHub CEO says Copilot will write 80% of code “sooner than later”
GitHub CEO Thomas Dohmke goes in depth to answer questions about how AI-powered development will change the future of innovation itself.
No, AI probably won’t kill us all – and there’s more to this fear campaign than meets the eye
A dose of scepticism is warranted when considering the AI doomsayer narrative — there are commercial incentives to manufacture fear of AI.
AI is riding to the rescue on wildfires
AI-powered systems designed to detect, confirm, and detail wildfires at the earliest possible time may help firefighters tame infernos in the West.
Up Next
Subscribe to Freethink for more great stories