Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

Pal, Proyag; Thompson, Brian; Virkar, Yogesh; Mathur, Prashant; Chronopoulou, Alexandra und Federico, Marcello (2023): Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters. Interspeech Conference, Dublin, Ireland, 20. - 24. August 2023. International Speech Communication Association (Hrsg.), In: 24th Annual Conference of the International Speech Communication Association (INTERSPEECH 2023), Bd. 1 New York: Curran Associates. S. 37-41

Abstract

To translate speech for automatic dubbing, machine translation needs to be isochronous, i.e. translated speech needs to be aligned with the source in terms of speech durations. We introduce target factors in a transformer model to predict durations jointly with target language phoneme sequences. We also introduce auxiliary counters to help the decoder to keep track of the timing information while generating target phonemes. We show that our model improves translation quality and isochrony compared to previous work where the translation model is instead trained to predict interleaved sequences of phonemes and durations.

• Dokumententyp: Konferenzbeitrag (Paper)
• Fakultätsübergreifende Einrichtungen: Centrum für Informations- und Sprachverarbeitung (CIS)
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik ; 400 Sprache > 410 Linguistik
• ISBN: 978-1-7138-8880-2
• ISSN: 2958-1796
• Ort: New York
• Sprache: Englisch