Welcome to naturalspeech2-pytorch Discussions!

[lucidrains]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[Curlypla]

I hope the community will manage to re-create naturalspeech2, we need a reliable alternative to ElevenLabs. (To be able for example to fine tune it on the desired language!)

[lucidrains]

really the big breakthrough is with the invention of the residual vector quantization from the soundstream paper. the rest is just combining different techniques

[davidmartinrius]

i think audiolm will pan out too, but latent diffusion route is worth exploring in tandem

Hi @lucidrains , is the quality of audiolm results comparable to naturalspeech2 ?

[lucidrains]

yea i think so lucidrains/audiolm-pytorch#170 (simplified version of audiolm basically)

but i defer to an audio researcher who knows the current state of the art

[lucidrains]

besides, just looking at the paper, the authors are coming to the slow realization that attention is all you need. they added so much attention since v1

[lucidrains]

the benefit of this type of architecture would really be faster sampling, when used in conjunction with more recent techniques like consistency models

[manmay-nakhashi]

i think we need FiLM for noise reduction, it's there in soundstream paper also and also here , that is a important block for noise conditioning.

[manmay-nakhashi]

pitch can be extracted through https://github.com/r9y9/pyreaper and for duration predictor we can use aligner. and reduce mse loss
for phoneme encoder we can use espeak and do text to ids.

[lucidrains]

ok the film conditioning is done! will get the trainer done this weekend

[lucidrains]

pitch can be extracted through https://github.com/r9y9/pyreaper and for duration predictor we can use aligner. and reduce mse loss for phoneme encoder we can use espeak and do text to ids.

that's awesome, thanks for sharing this

what is aligner?

[manmay-nakhashi]

aligner is to align phoneme sequences to waveform durations
we can use https://github.com/coqui-ai/TTS/blob/dev/TTS/tts/layers/generic/aligner.py
https://arxiv.org/abs/2108.10447

[lucidrains]

thank you for sharing your expertise!

[manmay-nakhashi]

i have written this from the paper params , params are hard coded. just see if this makes sense.

import torch
import torch.nn as nn

class DurationPredictor(nn.Module):
    def __init__(self, n_layers=30, kernel_size=3, n_att_layers=10, n_heads=8, hidden_size=512, dropout=0.2):
        super(DurationPredictor, self).__init__()

        self.conv_layers = nn.ModuleList([
            nn.Conv1d(hidden_size, hidden_size, kernel_size, padding=kernel_size // 2)
            for _ in range(n_layers)
        ])

        self.attention_layers = nn.ModuleList([
            nn.MultiheadAttention(hidden_size, n_heads)
            for _ in range(n_att_layers)
        ])

        self.fc = nn.Linear(hidden_size, 1)

        self.dropout = nn.Dropout(dropout)

    def forward(self, x, prompt_encoder_output):
        # x shape: (batch_size, hidden_size, length)
        # prompt_encoder_output shape: (prompt_length, batch_size, hidden_size)

        # Transpose prompt_encoder_output to (batch_size, hidden_size, prompt_length)
        prompt_encoder_output = prompt_encoder_output.transpose(0, 1)

        # Apply Q-K-V attention layer
        for att_layer in self.attention_layers:
            x, _ = att_layer(x, prompt_encoder_output, prompt_encoder_output)

        # Apply conv layers
        for conv_layer in self.conv_layers:
            x = conv_layer(x)
            x = torch.relu(x)
            x = self.dropout(x)

        # Apply fully connected layer
        x = self.fc(x.transpose(1, 2)).squeeze(-1)

        # Make sure predicted durations are positive
        x = torch.relu(x)

        return x

[manmay-nakhashi]

if yes, i'll shoot a pr for pitch and Duratioin predictor.

[manmay-nakhashi]

x is phonemeEncoder output

[lucidrains]

nice! yea they are simply employing cross attention in the paper. i'm not sure why they named it q-k-v layer

[manmay-nakhashi]

i feel that means a normal attention layers because they have cited Attention is all you need. paper , maybe i am wrong.

[lucidrains]

yea, it is just attention, you aren't wrong

[hunglmht243]

paper refer to SDE diffusion formula than DDPM right ? correct if i wrong.

[lucidrains]

they are basically equivalent approaches

[hunglmht243]

how about speech prompting mechanism ?

willing to contribute this repo

[lucidrains]

yea, working on it

feel free to jump in / submit a PR!

[donghyeon4885]

Hi, I am studying artificial intelligence in Korea. I'm especially interested in NLP and TTS. I'm going to do a tts project using this github page so I'll simply learn Korean voice data into this model. For example, if I type 'hello' in text in input, I'm going to use this model because I think your natural speech is the best, but I don't know how to do it with README.md alone.
What I'm curious about is 1. How to do TTS work without creating a mel-spectrogram and a vocoder
2. I've read README.md, but I don't know how to proceed after looking at your github page
If you answer my question, it will be very helpful for me to study. Thank you in advance

[manmay-nakhashi]

@lucidrains

@staticmethod
def expand_encodings(phoneme_enc, attn, pitch):
    expanded_dur = torch.einsum("klmn, kjm -> kjn", [attn, phoneme_enc])
    expanded_pitch = torch.einsum("klmn, kjm -> kjn", [attn, pitch])
    # expanded_pitch = expanded_pitch.repeat(1,expanded_dur.shape[1],1)
    expanded_encodings = expanded_dur + expanded_pitch
    return expanded_encodings

so i am expanding phoneme_enc and pitch like this do you think this is a correct implementation according to the paper ?

[intern4654]

Hello, thank you for your work lucidrains, it helps me a lot to do tts.

I have one question concerning wavenet block :
In your code you say that you do not use Attention Block in a wavenet block contrary to the paper and use instead a perceiver resampler. Could you please explain this choice ?