The marvels of AI change the face of film

Researchers from the University of Adelaide’s Australian Institute for Machine Learning (AIML) joined forces with Rising Sun Pictures (RSP) to create visual effects using artificial intelligence for Marvel Studios’ latest blockbuster, Shang-Chi and the Legend of the Ten Rings.

A team of technologists from AIML came together with RSP’s creative team on the film to develop a novel technique using artificial intelligence (AI) for replacing the face of stunt performers with those of key actors in combat scenes.

“We’re creating AI solutions for a range of startups and established businesses, as well as government and defence. AIML has its own engineering team, so we’re well placed to incorporate the latest research developments into software products for our clients and partners."Professor Simon Lucey, Director of the University of Adelaide's AIML.

Together the team applied machine learning tools to train machine models to learn the faces of key actors, and applied them to the faces of stunt doubles engaged in martial arts fight scenes.

Dr Ben Ward and Dr John Bastian from the University of Adelaide’s AIML worked on the film with RSP. In recognition of their talent in AI, and the rising role of this technology in the film industry, they have both been snapped up by RSP and are now part of their visual effects team.

Dr Ward said: “Rather than the traditional 2D and 3D face replacements typically used in high-intensity action scenes, the team used an AI deep fake method.

“Deep fake is the process by which a person in an existing image is replaced with the likeness of someone else.”

An 'identikit’ of both the stunt doubles’ and actors’ faces was learnt against a shared 'dictionary' of facial features. The faces of the stunt doubles were then detected in the film plates, and the machine then rebuilt the image using the identikit of the intended actor.”

The team used approximately 30,000 face images across five characters, training five principal machine models in over 4 million training iterations in total. These models were used for 51 face replacements over six key scenes.

Using AI technology in this way has transformed what a few years earlier would have been a more costly and labour-intensive process.

University of Adelaide AIML Director Professor Simon Lucey said the collaboration with RSP is another example of AI being used to expand capability across a number of industries.

“We’re creating AI solutions for a range of startups and established businesses, as well as government and defence. AIML has its own engineering team, so we’re well placed to incorporate the latest research developments into software products for our clients and partners,” Professor Lucey said.

“Our work is global, of course, but being based at Lot Fourteen puts us at the centre of South Australia’s growing innovation space.”