Lipsynthesis incorporating audio-visual synchronisation

Cong JIN; Jie WANG; Zichun GUO; Jing WANG

doi:10.11959/j.issn.2096-6652.202335

您当前的位置：

首页 >

文章列表页 >

Lipsynthesis incorporating audio-visual synchronisation

Special Topic: Diffusion Model and Artificial Intelligence Generated Content | 更新时间：2024-04-04

- Lipsynthesis incorporating audio-visual synchronisation
- Chinese Journal of Intelligent Science and Technology Vol. 5, Issue 3, Pages: 397-405(2023)
- 作者机构：
  
  1. 中国传媒大学信息与通信工程学院，北京 100024
  2. 北京化工大学，北京 100089
  3. 北京理工大学信息与电子学院，北京 100081
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202335
  CLC： TP391
- Revised：2023-08-01，
  
  Online First：2023-09，
  
  Published：15 September 2023
- 稿件说明：
移动端阅览
Cong JIN, Jie WANG, Zichun GUO, et al. Lipsynthesis incorporating audio-visual synchronisation[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(3): 397-405.
DOI：

Cong JIN, Jie WANG, Zichun GUO, et al. Lipsynthesis incorporating audio-visual synchronisation[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(3): 397-405. DOI： 10.11959/j.issn.2096-6652.202335.

摘要

以视频为载体的信息传播方式兴盛发展，视频的音画同步逐渐成为衡量视频质量的重要标准。深度合成技术在国际传播领域不断进入大众视野，融合音画同步的唇形合成技术吸引了越来越多的关注。现有唇形合成模型主要是基于静态图像的唇形合成，对于动态视频的合成效果不佳，且大多采用英文数据集进行训练，导致中文普通话合成效果较差。针对这些问题，基于对Wav2Lip唇形合成模型的研究，对其进行中文语境下的优化实验，通过多组实验测试了不同路线的训练模型效果，为后续Wav2Lip系列研究提供重要的参考价值。实现由语音驱动到文字驱动的唇形合成，并对唇形合成在虚拟数字人等多领域的应用展开讨论，为唇形合成技术更广阔的应用与发展奠定基础。

Abstract

With the flourishing development of video-based information dissemination

audio and video synchronization is gradually becoming an important standard for measuring video quality.Deep synthesis technology has been entering the public's view in the international communication field

and lip-sync technology integrating audio and video synchronization has attracted more and more attention.The existing lip-synthesis models are mainly based on lip-synthesis of static images

which are not effective for synthesis of dynamic videos

and most of them use English datasets for training which results in poor synthesis of Chinese Mandarin.To address these problems

this paper conducted optimization experiments on the Wav2Lip lip synthesis model in Chinese context based on its research foundation

and tested the effect of different routes of training models through multiple sets of experiments

which provided important reference values for the subsequent Wav2Lip series research.This study realized lip synthesis from speech-driven to text-driven

discussed the application of lip synthesis in multiple fields such as virtual digital human

and laid the foundation for the broader application and development of lip synthesis technology.

关键词

Keywords

references

PRAJWAL K R , MUKHOPADHYAY R , NAMBOODIRI V P , et al . A lip sync expert is all you need for speech to lip generation in the wild [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York:ACM , 2020 : 484 - 492 .

CHUNG J S , JAMALUDIN A , ZISSERMAN A , et al . You said that? [EB ] . arXiv preprint , 2017 ,arXiv:1705.02966.

KARPATHY A , TODERICI G , SHETTY S , et al . Large-scale video classification with convolutional neural networks [C ] // Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2014 : 1725 - 1732 .

SUWAJANAKORN S , SEITZ S M , KEMELMACHER-SHLIZERMAN I . Synthesizing Obama:learning lip sync from audio [J ] . ACM Transactions on Graphics , 2017 , 36 ( 4 ): 1 - 13 .

CHEN L L , LI Z H , MADDOX R K , et al . Lip movements generation at a glance [C ] // Proceedings of European Conference on Computer Vision . Cham:Springer , 2018 : 538 - 553 .

PRAJWAL K R , MUKHOPADHYAY R , PHILIP J , et al . Towards automatic face-to-face translation [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM , 2019 : 1428 - 1436 .

CHUNG J S , ZISSERMAN A . Out of time:automated lip sync in the wild [C ] // Proceedings of Computer Vision-ACCV 2016 Workshops . Cham:Springer , 2017 : 251 - 263 .

TOMAR S . Converting video formats with FFmpeg [J ] . Linux Journal , 2006 ( 146 ): 10 .

MCFEE B , RAFFEL C , LIANG D , et al . Librosa:audio and music signal analysis in python [C ] // Proceedings of the 14th Python in Science Conference .[S.l:s.n. ] , 2015 : 18 - 25 .

CRESWELL A , WHITE T , DUMOULIN V , et al . Generative adversarial networks:an overview [J ] . IEEE Signal Processing Magazine , 2018 , 35 ( 1 ): 53 - 65 .

Views

620

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Artificial intelligence and deep learning methods for solving differential equations: the state of the art and prospects

A review of prediction methods for moving target trajectories

Reinforcement learning:toward action-knowledge merged intelligent mechanisms and algorithms

Artificial intelligence is entering the post deep-learning era

On intelligence: the origin and the goal

Related Author

Jingwei LU

Xiang CHENG

Fei-Yue WANG

Wen LIU

Kunlin HU

Yan LI

Zhao LIU

Fei-Yue WANG

Related Institution

The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences

Qingdao Academy of Intelligent Industries

School of Artificial Intelligence, University of Chinese Academy of Sciences

School of Navigation, Wuhan University of Technology

National Engineering Research Center for Water Transportation Safety

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：162003 Visits Today：166

⁰