in Machine Learning, Tutorial

An Introduction to DeepFakes

This tutorial will cover the theory and practice of creating deepfakes: videos in which faces have been swapped using Machine Learning and Deep Neural Networks. If you are interested in learning more about this novel technique, this is the course for you.

After a theoretical introduction, this course will focus on how to make the most out of the popular applications FakeApp and faceswap; most of the deepfakes you will find online (such as the ones featuring Nicolas Cage) have been created using them.

You can read all the posts in this series here:

If you are interested in reading more about AI Art (Stable Diffusion, Midjourney, etc) you can check this article instead: The Rise of AI Art.

Introduction

Face detection has been a major research subject in the early 2000s. Almost twenty years later, this problem is basically solved and face detection is available as a library in most programming languages. Even face-swap technology is nothing new, and has been around for a few years.

In his article from 2016, Face Swap using OpenCV, author Satya Mallick showed how to swap faces programmatically, warping and colour correcting Ted Cruz’s face to fit Donald Trump (below).

When applied correctly, this technique is uncannily good at swapping faces. But it has a major disadvantage: it only works on pre-existing pictures. It cannot, for instance, morph Donald Trump’s face to match the expression of Ted Cruz.

That has changed in late 2017, when a new approach to face-swap has appeared on Reddit. Such a breakthrough relies on neural networks, computational models that are loosely inspired by the way real brains process information. This novel technique allows generating so-called deepfakes, which actually morph a person’s face to mimic someone else’s features, although preserving the original facial expression.

When used properly, this technique allows the creation of photorealistic videos at an incredibly low cost. The finale of Rogue One, for instance, featured a digital version of Princess Leia; a very expensive scene which required the expertise of many people. Below, you can see a comparison between the original scene and another one recreated using Deep Learning.

Creating Deepfakes

At the moment there are two main applications used to create deepfakes: FakeApp and faceswap. Regardless of which one you will use, the process is mostly the same, and requires three steps: extraction, training and creation.

Extraction

The deep- in deepfakes comes from the fact that this face-swap technology uses Deep Learning. If you are familiar with the concept, you should know that deep learning often requires large amounts of data. Without hundreds (if not thousands!) of face pictures, you will not be able to create a deepfake video.

A way to get around this is to collect a number of video clips which feature the people you want to face-swap. The extraction process refers to the process of extracting all frames from these video clips, identifying the faces and aligning them.

The alignment is critical, since the neural network that performs the face swap requires all faces to have the same size (usually 256×256 pixels) and features aligned. Detecting and aligning faces is a problem that is considered mostly solved, and is done by most applications very efficiently.

Training

Training is a technical term borrowed from Machine Learning. In this case, it refers to the process which allows a neural network to convert a face into another. Although it takes several hours, the training phase needs to be done only once. Once completed, it can convert a face from person A into person B.

This is the most obscure part of the entire process, and I have dedicated two posts to explain how it works from a technical point of view: An Introduction to Neural Networks and Autoencoders and Understanding the Technology Behind DeepFakes). If you really want to create photorealistic deepfakes, a basic understanding of the process that generates them is necessary.

Creation

Once the training is complete, it is finally time to create a deepfake. Starting from a video, all frames are extracted and all faces are aligned. Then, each one is converted using the trained neural network. The final step is to merge the converted face back into the original frame. While this sounds like an easy task, it is actually where most face-swap applications go wrong.

The creation process is the only one which does not use any Machine Learning. The algorithm to stitch a face back onto an image is hard-coded, and lacks the flexibility to detect mistakes.

Also, each frame is processed independently; there is no temporal correlation between them, meaning that the final video might have some flickering. This is the part where more research is needed. If you are using faceswap instead of FakeApp, have a look at df which tries to improve the creation process.

Conclusion

Deep Learning has made photorealistic face-swap not just possible, but also accessible. This technique is still in its infancy and many more improvements are expected to happen in the next few years.

In the meantime, places like the FakeApp forum or the fakeapp GitHub page are where most of the technical discussion around deepfakes is currently taking place. The community around deepfakes is constantly exploring new approaches, and developers are often very willing to share their creations. This is the case of user ZeroCool22 which created a deepfake video of Jimmy Fallon interviewing himself.

Another interesting reading on the subject is Exploring DeepFakes.

It cannot be denied that deepfakes have finally shown the world a practical application of Deep Learning. However, this very technique has often been used without the explicit consent of the people involved. While this is unlikely to be an issue with videos such as the ones shown in this article, the same cannot be said when it is used to create pornographic content. This is why, before showing how to create deepfakes, the next lecture in this online course will focus entirely on the legal and ethical issues of deepfakes.

You can read all the posts in this series here:

A special thanks goes to Christos Sfetsios and David King, who gave me access to the machine I have used to create the deepfakes used in this tutorial.

👾 This course was made possible thanks to the support of patrons on Patreon
🥇 $25+
🥈 $10+
🥉 $8+
🥉 $5+
💵 $1+
Dry Cactus
Nicolas Gascon
Aakash Sastry
Binary Soul
Antonio Cervantes
Ariel Ahrens
Sean
Magnopus
Skippy
Takeshi
Mike King
Hữu Lộc
John Faulkenbury
Jashan Chittesh
Piotr Podziemski
Dexter André Osiander
Perren Spence-Pearse
Roberta Mota
Arcadely
Keith Miller
Adam Boyne
Craig Bowler
Bryan Duggan
Shawn Asmussen
Dries De Smet
Druelbozo
TrojanFighter
Shohei Takei
1011101
Glenn Jones
Jean Georges
Adam Frisby
David Peterson
Daniel Farrow
Minseck Cho
Zac Dixon
Nathan Grotticelli
Jessie Thomson
Theo Lagendijk
Ben Beagley
Michael Cook
Anthony Duquette
Brad Hammond
Danny Huynh
zhouhua
SaharaX
TraceYang
Brandon Tay
Joakim Westgren
Rohit Gupta
David Erosa
Jonathan Hopkins
Michael Romaszewicz
Omar Espinosa
Kevin Comerford
Shimon Vainer
Fernando Monroy
benfei
Lawrence
Hammad Bashir
Al-Rashid Jamalul
Kieran Belkus
Yin Liu
A. Moore
fangeles
Pixeye
Juan Trelles Trabucco
JoneyZ
Kyle Phillips
SUPER315
Abel
Magnus Paulson Erga
HyperDigital
Joan Llobera
Alb Pérez-Bermejo
FruityFusion
Mike Wuetherick
Luke Kaalim
Thomas Ingram
unizen
rizal ardianto
William Wilson
Stefano Gandolfo
Stephen
Lukasz Pasek
fredqin
Arielle Kim
Jack Riddle
Yangtze River
培基 朱
Josquin Z
OccupiedVR
shangmingyou
Jordan Jackson
Rick Palmer
James Neal
Yaser Saqib
Apoc
Shane Nilsson
Jarrod Phillips
Julian Heinken
François Chéné
Moritz Voss
Jonathan Wright
Jason Adams
Jordan Carroll
Juwenw Sun
Aaron Eastburn
Tomáš Koza
Chris Sorrell
rustinlee
obscuresteel
TxCx
Ciro
Elan Feingold
Eric Cloutier
Wilco Weggeman
Mario Gutierrez
FELIX
Jaewon Jung
Anthony Rosenbaum
Justin Loudermilk
Ryan Schake
Mike Corsaro
Sebastian Kędra
Tom
Márton Helényi
CatchCo
Hans Palm
mars
Matei Giurgiu
Zephyr Zhang
Jules Stevenson
Edouard Philippe
Dave Cowling
Andy De Meyer
Phil Gosch
Dave Chenell
JeanD
Kit Matthews
Klemen Brajkovič
Liam Brady
Andrew Johnson
Gil Damoiseaux
Jungchul Ha
J
Jeremy Zeler
Elijah Cauley
Jonadab Oomon
Jake OBrien
tmabs
Gordon Chapman
Kevin Lee Jr.
Colin Schultz
Brad Weiers
Gabor Tornyos
Marcel Medak
Anders Ørum Pedersen
Dirk Van Welden
Thomas Key
Yotam Harris
John Bullock
Justin Lindsey
Nick Maurer
Ray
Markus Arning
Bruno Di Pentima
Tuboy Thers
Daniel Saulnier
Jacob Herold
Ashley James
Lukas Lundberg
Jeremy Griffith
Tom Campbell
Luca Prasso
Luis Jose Quintana
Manjit Bedi
Fernando Urquijo Sánchez
starportx@sohu.co
SpaceToad
Jose Miguel Casas Pagan
Edward del Villar
Garrett Stevens
Atle Mæland
Leonardo Marques
Maddox Max
Gopeekrishnan Sobhana Wariar
Devyn Cyphers
Martin Hedlund
Yaser S
Fox Buchele
Francis Ge
mscottmcbee
Gamoga Notyag
Rolly
Ted Milker
Michal Szczepaniak
Ian Thompson
Kevin Doyon
Ian Sterling
Miikka Harjuntausta
kc austin
Martin Leissler
Alex Chih Chung Lin
Diego Sarmentero
Andy Gittins
Neil Blakey-Milner
Victoria Mitchell
Louis Hong
mirko jugurdzija
Derek Reynolds
Mitu Khandaker
nathan witkowski
Daniel Plemmons
Nicoll Hunt
Kerry
Joe
Jonny Ree
Kevin Webb
Tim
Vladislav Chetrusca
Shaders Laboratory
Peter Cardwell-Gardner
Robin Baumgarten
Andreas Wilcox
Terkel Gjervig Nielsen
Stella Cannefax
Iruken
Martin Eigel
Ryan
Chris
Pierce McBride
Shadow Darkwell
Alo Mis
Jim
clownbaby
Christopher Pereira
Steve
Demi
Iain
Wonky Kim
Karl Goodloe
Daniel Cassidy
Alessandro Salani
Doug Snook
Marco Elizondo
Edd Toomey
Hotgates
Brian Nicolucci
Bryan Cinman
Jan Ambroziak
Alice Ruppert
Josh Leong
parag ponkshe
Martin Pražák
Clément
Clement
Gavin Rich
Kevin Gregory Agwaze
Håkon Nessjøen
James Id
Warren Moore
devMidgard
jp
Jason Burton
Ivan Herrera Maciel
Noaksey
Daniel Sanchez
Jon Kimbel
André Gröschel
friuns
Manish Gupta
David Proffer
Pavel
Marek Majcher
Arthur Cousseau
Orlando Batista
CptBonex
Jakub Barszczewski
Arlin
Matheus
Dom Camus
Aubrey Hesselgren
Kemp
Roger Fretwell
Cary Miller
Matthew Senne
Chris Butler
C Paterson
Nic DanielOthers:
NachoB
b
Stewart
Chris
Yaz Khabiri
Abisaid Fernandez de Lara
valryon
Thomas Schmall
Patrick Peluse
Goran Lalic
💖 Support this blog

This website exists thanks to the contribution of patrons on Patreon. If you think these posts have either helped or inspired you, please consider supporting this blog.

Patreon Patreon_button
Twitter_logo

YouTube_logo
📧 Stay updated

You will be notified when a new tutorial is released!

📝 Licensing

You are free to use, adapt and build upon this tutorial for your own projects (even commercially) as long as you credit me.

You are not allowed to redistribute the content of this tutorial on other platforms, especially the parts that are only available on Patreon.

If the knowledge you have gained had a significant impact on your project, a mention in the credit would be very appreciated. ❤️🧔🏻

Write a Comment

Comment

Webmentions

  • How To Install FakeApp - Alan Zucconi October 15, 2020

    […] Part 1. An Introduction to DeepFakes and Face-Swap Technology […]

  • The Rise of AI Art - Alan Zucconi October 15, 2020

    […] If you are interested in learning more about deepfakes—how they work, how to create one, and when not to do it—I highly recommend the series An Introduction to Deepfakes. […]

  • 5 Languages to for WordPress Developers to learn in 2020 October 15, 2020

    […] or Pytorch but Python is used extensively and in emerging fields for good or ill including DeepFakes.There are so many tools, libraries and frameworks for Python that I suggest just diving through the […]

  • An Introduction to Neural Networks and Autoencoders - Alan Zucconi October 15, 2020

    […] Part 1. An Introduction to DeepFakes and Face-Swap Technology […]

  • Deepfake (falsi realistici): cosa sono, come funzionano e come si fanno October 15, 2020

    […] con un’altra, come è possibile vedere in questa immagine (tratte dal blog del programmatore Alan Zucconi) in cui viene mostrata l’applicazione dell’algoritmo di face recognition su una foto di […]