Skip to content
naulon
Not checkedNot checked for notices yet.

Deepfake forensics analysis: An explainable hierarchical ensemble of weakly supervised models

Samuel Henrique Silva, Mazal Bethany, Alexis Megan Votto and 3 more

Forensic Science International: Synergy · 2022

doi:10.1016/j.fsisyn.2022.100217 (opens in a new tab)

We email you when a new notice or a question about this paper appears.

Abstract

Deepfakes have become exponentially more common and sophisticated in recent years, so much so that forensic specialists, policy makers, and the public alike are anxious about their role in spreading disinformation. Recently, the detection and creation of such forgery became a popular research topic, leading to significant growth in publications related to the creation of deepfakes, detection methods, and datasets containing the latest deepfake creation methods. The most successful approaches in identifying and preventing deepfakes are deep learning methods that rely on convolutional neural networks as the backbone for a binary classification task. A convolutional neural network extracts the underlying patterns from the input frames. It feeds these to a binary classification fully connected network, which classifies these patterns as trustworthy or untrustworthy. We claim that this method is not ideal in a scenario in which the generation algorithms constantly evolve since the detection algorithm is not robust enough to detect comparably minor artifacts introduced by the generation algorithms. This work proposes a hierarchical explainable forensics algorithm that incorporates humans in the detection loop. We curate the data through a deep learning detection algorithm and share an explainable decision to humans alongside a set of forensic analyses on the decision region. On the detection side, we propose an attention-based explainable deepfake detection algorithm. We address this generalization issue by implementing an ensemble of standard and attention-based data-augmented detection networks. We use the attention blocks to evaluate the face regions where the model focuses its decision. We simultaneously drop and enlarge the region to push the model to base its decision on more regions of the face, while maintaining a specific focal point for its decision. In this case, we use an ensemble of models to improve the generalization. We also evaluate the models’ decision using Grad-CAM explanation to focus on the attention maps. The region uncovered by the explanation layer is cropped and undergoes a series of frequency and statistical analyses that help humans decide if the frame is real or fake. We evaluate our model in one of the most challenging datasets, the DFDC, and achieve an accuracy of 92.4%. We successfully maintain this accuracy in datasets not used in the training process.

Abstract licence: CC BY-NC

References93

    40 more references, whose wording the paper’s licence keeps private.

    All 93 references

    Reads

    Read by people on this page

    None yet.

    Checked or read by AI through naulon

    None yet.

    A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

    Are you an author?

    Is this your paper?

    Claim it to add your note and be paid when agents cite it. Free.