proxectonos/Nos_TTS-paulo-vits-phonemes

Paulo: Nos Project's Galician TTS Model

Model description

Paulo is a Galician TTS model developed by the Nós project. It was trained from scratch using the Coqui TTS Python library on the Paulo corpus of the dataset CRPIH_UVigo-GL-Voices. This corpus comprises a total of 1,316 sentences recorded by an amateur voice talent.

The model was trained on phonemes input, so it needs a phonetic transcription of the input text to synthesize speech. The Cotovía tool is required to generate this transcription. The Cotovía linguistic module also performs text normalization as a preliminary step to phonetic transcription.

You can test the model in our live inference demo (Nós-TTS).

Intended uses and limitations

You can use this model to generate synthetic speech in Galician.

Installation

Cotovía

For phonetic transcription, the Cotovía front-end must be used. This software is available for download on the SourceForge website. The required Debian packages are cotovia_0.5_amd64.deb and cotovia-lang-gl_0.5_all.deb, which can be installed using the following commands:

sudo dpkg -i cotovia_0.5_amd64.deb
sudo dpkg -i cotovia-lang-gl_0.5_all.deb

TTS library

To synthesize speech, you need to install the Coqui TTS library:

pip install TTS

How to use

Python usage

Normalization and synthesis can also be performed within Python:

import argparse
import random
import re
import string
import subprocess
from TTS.utils.synthesizer import Synthesizer

PUNCLIST = [';', '?', '¿', ',', ':', '.', '!', '¡']

def sanitize_filename(filename):
    """Remove or replace any characters that are not allowed in file names."""
    return ''.join(c for c in filename if c.isalnum() or c in (' ', '_', '-')).rstrip()

def canBeNumber(n):
    try:
        int(n)
        return True
    except ValueError:
        # Not a number
        return False

def is_number(index, text):
    if index == 0:
        return False
    elif index == len(text) - 1:
        return False
    else:
        return canBeNumber(text[index - 1]) and canBeNumber(text[index + 1])

def split_punc(text):
    segments = []
    puncs = []
    curr_seg = ""
    previous_punc = False

    for i, c in enumerate(text):        
        if c in PUNCLIST and not previous_punc and not is_number(i, text):
            segments.append(curr_seg.strip())
            puncs.append(c)
            curr_seg = ""
            previous_punc = True
        elif c in PUNCLIST and previous_punc:
            puncs[-1] += c
        else:
            curr_seg += c
            previous_punc = False

    segments.append(curr_seg.strip())

    #Remove empty segments in the list
    segments = filter(None, segments)

    # store segments as a list
    segments = list(segments)

    return segments, puncs

def remove_tra3_tags(phontrans):
    s = re.sub(r'#(.+?)#', r'', phontrans)
    s = re.sub(r'%(.+?)%', r'', s)
    s = re.sub(' +',' ',s)
    s = re.sub('-','',s)
    return s.strip()

def to_cotovia(text_segments):
    # Input and output Cotovía files
    res = ''.join(random.choices(string.ascii_lowercase + string.digits, k=5))
    COTOVIA_IN_TXT_PATH = res + '.txt'
    COTOVIA_IN_TXT_PATH_ISO = 'iso8859-1' + res + '.txt'
    COTOVIA_OUT_PRE_PATH = 'iso8859-1' + res + '.tra'
    COTOVIA_OUT_PRE_PATH_UTF8 = 'utf8' + res + '.tra'
    
    with open(COTOVIA_IN_TXT_PATH, 'w') as f:
        for seg in text_segments:
            if seg:
                f.write(seg + '\n')
            else:
                f.write(',' + '\n')

    # utf-8 to iso8859-1
    subprocess.run(["iconv", "-f", "utf-8", "-t", "iso8859-1", COTOVIA_IN_TXT_PATH, "-o", COTOVIA_IN_TXT_PATH_ISO], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
    # call cotovia with -t3 option
    subprocess.run(["cotovia", "-i", COTOVIA_IN_TXT_PATH_ISO, "-t3", "-n"], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
    # iso8859-1 to utf-8
    subprocess.run(["iconv", "-f", "iso8859-1", "-t", "utf-8", COTOVIA_OUT_PRE_PATH, "-o", COTOVIA_OUT_PRE_PATH_UTF8], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

    segs = []
    try:
        with open(COTOVIA_OUT_PRE_PATH_UTF8, 'r') as f:
            segs = [line.rstrip() for line in f]
            segs = [remove_tra3_tags(line) for line in segs]
    except:
        print("ERROR: Couldn't read cotovia output")

    subprocess.run(["rm", COTOVIA_IN_TXT_PATH, COTOVIA_IN_TXT_PATH_ISO, COTOVIA_OUT_PRE_PATH, COTOVIA_OUT_PRE_PATH_UTF8], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

    return segs

def merge_punc(text_segs, puncs):
    merged_str = ""

    for i, seg in enumerate(text_segs):
        merged_str += seg + " "
        if i < len(puncs):
            merged_str += puncs[i] + " "

    # remove spaces before , . ! ? ; : ) ] of the merged string
    merged_str = re.sub(r"\s+([.,!?;:)\]])", r"\1", merged_str)

    # remove spaces after ( [ ¡ ¿ of the merged string
    merged_str = re.sub(r"([\(\[¡¿])\s+", r"\1", merged_str)

    return merged_str.strip()


def accent_convert(phontrans):
    transcript = re.sub('a\^','á',phontrans)
    transcript = re.sub('e\^','é',transcript)
    transcript = re.sub('i\^','í',transcript)
    transcript = re.sub('o\^','ó',transcript)
    transcript = re.sub('u\^','ú',transcript)
    transcript = re.sub('E\^','É',transcript)
    transcript = re.sub('O\^','Ó',transcript)
    return transcript


def text_preprocess(text):
    #Split from punc
    text_segments, puncs = split_punc(text)
    cotovia_phon_segs = to_cotovia(text_segments)
    cotovia_phon_str = merge_punc(cotovia_phon_segs, puncs)
    phon_str = accent_convert(cotovia_phon_str)
    return phon_str


def main():
    parser = argparse.ArgumentParser(description='Cotovia phoneme transcription.')
    parser.add_argument('text', type=str, help='Text to synthetize')
    parser.add_argument('model_path', type=str, help='Absolute path to the model checkpoint.pth')
    parser.add_argument('config_path', type=str, help='Absolute path to the model config.json')

    args = parser.parse_args()

    print("Text before preprocessing: ", args.text)
    text = text_preprocess(args.text)
    print("Text after preprocessing: ", text)
    synthesizer = Synthesizer(
        args.model_path, args.config_path, None, None, None, None,
    )

    # Step 1: Extract the first word from the text
    first_word = args.text.split()[0] if args.text.split() else "audio"
    first_word = sanitize_filename(first_word)  # Sanitize to make it a valid filename

    # Step 2: Use synthesizer's built-in function to synthesize and save the audio
    wavs = synthesizer.tts(text)
    filename = f"{first_word}.wav"
    synthesizer.save_wav(wavs, filename)

    print(f"Audio file saved as: {filename}")

if __name__ == "__main__":
    main()

This Python code takes a text input, preprocesses it using Cotovía's front-end, synthesizes speech from the preprocessed text, and saves the synthetic output speech as a .wav file.

A more advanced version, including additional text preprocessing, can be found in the script synthesize.py, avaliable in this repository. You can use this script to synthesise speech from an input text as follows:

python synthesize.py text model_path config_path

Training

Hyperparameter

The model is based on VITS proposed by Kim et al. The following hyperparameters were set in the coqui framework.

Hyperparameter	Value
Model	vits
Batch Size	16
Eval Batch Size	16
Mixed Precision	true
Window Length	1024
Hop Length	256
FTT size	1024
Num Mels	80
Phonemizer	null
Phoneme Language	null
Text Cleaners	null
Formatter	nos_fonemas
Optimizer	adam
Adam betas	(0.8, 0.99)
Adam eps	1e-09
Adam weight decay	0.01
Learning Rate Gen	0.0002
Lr. scheduler Gen	ExponentialLR
Lr. scheduler Gamma Gen	0.999875
Learning Rate Disc	0.0002
Lr. scheduler Disc	ExponentialLR
Lr. scheduler Gamma Disc	0.999875

The model was trained for 610,000 steps.

The nos_fonemas formatter is a modification of the LJSpeech formatter with one extra column for the phonemized input.

Additional information

Authors

Antonio Moscoso.

Contact information

For further information, send an email to proxecto.nos@usc.gal

Funding

This research was produced within the framework of the Proxecto Nós, funded by the Ministry for Digital Transformation and Public Administration and the Recovery, Transformation, and Resilience Plan – Funded by the European Union – NextGenerationEU, as part of the Ilenia Project with reference 2022/TL22/00215336, and previously “The Nós project: Galician in the society and economy of Artificial Intelligence”, resulting from the agreement 2021-CP080 between the Xunta de Galicia and the University of Santiago de Compostela, and thanks to the Investigo program, within the National Recovery, Transformation and Resilience Plan, within the framework of the European Recovery Fund (NextGenerationEU).

Citation information

If you use this model, please cite as follows:

Moscoso, Antonio. 2024. Nos_TTS-paulo-vits-phonemes. URL: https://huggingface.co/proxectonos/Nos_TTS-paulo-vits-phonemes/