Growing up in Paris, I pretty much learned English by doing two simple things: binge-watching Netflix and listening to music. At the time, picking translated movies or shows over their original language resulted in slight shame—make an effort Alma if you want to move to London—as well as a considerable amount of irritation throughout the selected entertainment. You know what I’m talking about here; no matter how good a translation is, actors’ lips are usually out of sync and it just isn’t the same.
Enters London-based AI startup Flawless, which claims its deepfake technology could make dubbed movies (and shows, I assume) look way more natural. How? According to the company, its technology fixes the out-of-sync translation by creating mouth movements that match the spoken translation, then slapping them over the original image.
Named ‘TrueSync’, the deepfake tech is the world’s first system that uses AI to create perfectly lip-synced visualisations in multiple languages. “At the heart of the system is a performance preservation engine which captures all the nuance and emotions of the original material,” reads Flawless’ website.
The startup’s co-founder Nick Lynes tells The Verge that this process retains the movie’s original style and performance. Although the end result isn’t 100 per cent perfect, it’s pretty close. And Flawless says it can offer it quickly, cheaply, and in any language.
It’s also easier than a complete do-over, like Metástasis, the Colombian telenovela-style remake of Breaking Bad that doesn’t exactly replicate the performance that won Bryan Cranston four Emmys…
Now, I can already hear some of you saying they prefer the authenticity of subtitles—I’m looking at you, movie snobs. But look at it this way; while subtitles help those who are deaf or hard of hearing, dubbing helps those who are blind or have low vision. Still ready to stand by your pretentious argument now?
In fact, most people prefer the ‘lazy’ way out. In 2018, the streaming giant Netflix found that people were more likely to watch a dubbed show than one with subtitles, which is why it’s made the dubbed version the default. The company is now working with over 170 studios worldwide that offer dubs in more than 35 languages, according to Bloomberg. In fact, Lupin, its number one show this quarter, is a French-dubbed work.
Flawless’ deepfake tech could reshape the movie industry, in both alluring and troubling ways. It promises to allow directors to effectively reshoot movies in different languages, making foreign versions less jarring for audiences and more faithful to the original. But the power to automatically alter an actor’s face so easily might also prove controversial if not used carefully.
Soon enough, the AI dubbing technology will be invisible. People will be watching something and they won’t realise it was originally shot in another language. While this sounds pretty exciting for movie snobs, it highlights the augmented risk deepfakes could represent in the near future.
After all, the same technology has been used to create fake celebrity porn (also known as deepnudes) and damaging revenge porn clips targeting women. Experts worry that deepfakes showing a famous person in a compromising situation might help spread misinformation and even sway an election. Is all of this really worth it in exchange for near-perfect dubbing?
We’ve previously witnessed the appearance of AI-generated deepnudes, which, by the way, were later on revealed as being made using real images of sexual abuse—how awful. Around the same time, Kanye West thought it would be a great idea to buy his then-wife Kim Kardashian a talking hologram of her late father, Robert Kardashian. No need to ponder why they’re currently getting divorced. Even deepfake memes became popular!
All in all, it’s safe to say that deepfakes have comfortably infiltrated our lives, just like the rest of the, somewhat surprisingly, recent technologies that we take for granted in our daily life. Just because one trend is never enough for gen Zers—I should know, I am one myself—deepfakes now play a part in yet another trend of the moment: nostalgia.
From Y2k fashion to the viral retro music genre vaporwave, it’s obvious that we have a thing for reminiscence and sentimentality, even for eras we weren’t born in time for. This explains why we’re now all going berserk for MyHeritage’s new free feature called ‘deep nostalgia’, which allows users to upload pictures of their late relatives (or anyone else too, someone uploaded a photograph of the legendary Rosalind Franklin, ‘just because’) and have them come to life, eyes swivelling, faces tilting, and all that jazz.
The Black Mirror-esque technology has already taken TikTok by storm, with users sharing videos of them showing their parents AI-generated animations of their great great grandfather, grandmother, and other relatives, inevitably leading to emotional reactions and sometimes tears.
The creepy yet fascinating tool comes from MyHeritage, the Israeli online genealogy platform mostly known for its DNA test kits which provide customers with DNA matching and ethnicity estimates. But MyHeritage’s AI-powered viral deepfakery isn’t as complicated as it seems: the company is simply going straight for tugging on your heartstrings to grab data that can then be used to drive sign-ups for its other (paid) services. In other words, selling DNA tests is its main business, not ‘making it’ on TikTok, although that’s always a plus for any company.
As TechCrunch explains, last year for example, “the Norwegian Consumer Council reported MyHeritage to the national consumer protection and data authorities after a legal assessment of the T&Cs found the contract it asks customers to sign to be ‘incomprehensible’.”
Back in 2018, MyHeritage also suffered a major data breach. The data from that breach was later found for sale on the dark web, among a wider cache of hacked account info pertaining to several other services.
That being said, if you’re able to set aside the ethics of encouraging people to drag their long-lost relatives into the dark hole that is MyHeritage’s cross-sell DNA testing, then yes, the deepfake tool is pretty impressive.
But MyHeritage is not the only company to be praised (or condemned) for the deep nostalgia trend. Another Israeli company, D-ID, helped power it. As a TechCrunch Disrupt Battlefield alumni, D-ID started out building tech to digitally de-identify faces with an eye on protecting images and video from being identifiable by facial recognition algorithms. Oh, the irony!
The company released a demo video of the newer, photo-animating technology last year. The tech uses a driver video to animate the photo, mapping facial features from the photo onto that base driver to create a “live portrait.”
“The Live Portrait solution brings still photos to life. The photo is mapped and then animated by a driver video, causing the subject to move its head and facial features, mimicking the motions of the driver video,” D-ID said in a press release. “This technology can be implemented by historical organizations, museums, and educational programs to animate well-known figures.” So, not really your great great uncle.
Like all good things in life, MyHeritage’s deep nostalgia feature is not completely free—after the first few free nostalgia hits, users are asked to pay a monthly fee. I would be lying if I said I’m not going to be one of the many to have a fiddle with the tool, however, knowing that a paywall is bound to cut me short in my nostalgia mania is a welcomed thought.