Deepfake Blurring the Line Between Real and Fake... 'Completed with Just 1 Minute of Voice Data'

Deepfake Targeting Even Ordinary People
IT Industry Busy Developing Detection Technology

# A in their 20s living in Seoul, an office worker, was startled last week when they saw a photo sent from an unknown KakaoTalk account. The sender had combined A’s profile photos posted on Instagram and KakaoTalk with revealing photos of a female idol and sent it. At first, only the composite photos were sent, but then the sender threatened to distribute them to A’s social network service (SNS) acquaintances unless $300 was transferred.

# B, a university student, started phone consultation work after hearing that the hourly wage was higher than elsewhere. However, it turned out to be a scam where B’s voice was altered to sound like the consultation customer’s acquaintance. B is now facing police investigation as a perpetrator of deepfake voice phishing.

The shock from the Taylor Swift deepfake video is spreading even to ordinary people. As ‘deepfake,’ an AI-based image and voice synthesis technology, becomes increasingly sophisticated, the damage is growing. IT companies have begun developing watermarking and detection technologies to prevent deepfake damage.

According to the Korea Communications Standards Commission on the 7th, the number of cases requesting blocking or deletion of deepfake sexual false videos in South Korea reached 5,996 from January to November last year. This far exceeded the total of 3,574 cases in 2022. Earlier, there were only 473 cases in 2020, but it increased to 1,913 in 2021. It is not difficult to find cases complaining about deepfake damage in online communities.

Deepfake has rapidly spread since the emergence of generative AI. This is because even ordinary people can easily create deepfake voices or videos. In the past, creating deepfakes required training AI with vast amounts of data, but recently this is no longer necessary. With the development of methods to train AI with small amounts of data, it is now possible to produce highly realistic deepfakes using just a few photos or one minute of voice data. The quality has also improved. Previously, details such as unnatural hand shapes or blurred facial contours made detection possible upon close inspection, but now it is becoming increasingly difficult to distinguish.

Deepfake Completed for 6,500 Won

With technological advancement, the related market has also grown. Apps have appeared that not only simply composite photos but also naturally change lip movements to match voice. Searching ‘deepfake’ on Google or Apple App Store reveals numerous related apps. The ‘Reface’ app, used by over 100 million people worldwide, allows users to create watermark-free deepfake images for a weekly fee of 6,500 won. Uploading a photo of a specific person and another photo or video to be composited overlays the face.

Domestic companies providing generative AI services have taken measures. The Ministry of Science and ICT has recommended voluntary industry responses until watermarking of AI-generated content becomes mandatory. Naver and Kakao are researching AI content identification technologies to prevent their generative AI from being used to mass-produce deepfakes. Kakao is considering introducing invisible watermarks to ‘Karlo,’ Kakao Brain’s image generation model. Invisible watermarks are subtle pixel-level traces that are not visible to the eye. They are difficult to erase or edit, making them a technology to prevent deepfakes. Naver is researching various technologies, including embedding encryption algorithms like digital certificates into AI-generated content. Ha Jung-woo, head of Naver Cloud AI Innovation Center, said, “The industry agrees on the need for marks such as watermarks on AI-generated content,” adding, “We are preparing technology considering the possibility that marks may be erased or destroyed.”

AI Catching Deepfakes

Deepfake detection technology is also evolving. AI startup DeepBrain AI recently released a detection solution using AI. This technology trains AI models with both original and deepfake data to distinguish between them. It captures subtle differences such as specific noise between voices or pitch variations at certain sentence endings in altered data.

However, it is not easy to filter out all increasingly sophisticated deepfake technologies. Some methods contaminate deepfake images to make detection difficult, and new deepfake technologies continue to emerge. Lee Yoo-hyun, a researcher in DeepBrain AI’s deep learning team, explained, “Because demand for deepfakes is higher, more capital and technology are being invested. It is like giving a multiplication problem to an AI that has only learned addition and subtraction?it cannot understand it. Similarly, detecting new deepfake technologies is difficult.”

This content was produced with the assistance of AI translation services.