which unit sequence to use for training and inference

[yunlong-hub]

scripts/diffusion/train.sh：

trianing en -> other direction with reduced unit

lang="fr"
data_dir=$cvss_dir/$lang-en/en2${lang}/orig_unit

scripts/diffusion/unit_gen.sh：

use to synthesize reduced es/fr speech-units based on diffusion model

lang="fr"
reduce_tsv=$cvss_dir/$lang-en/en2${lang}/reduce_unit
orig_tsv=$cvss_dir/$lang-en/en2${lang}/orig_unit

Your comment says you trained with the simplified unit sequence, but the actual path points to the orig directory.
During inference, the reasoning is done on the simplified sequence.
Could you tell me more about what you're working on? Knowing the context might help me provide a more specific and helpful explanation.

[steventan0110]

Hi,

Thanks for your interests of our work. As mentioned, for both training and inference, I used simplified unit sequence (which is provided by the reduce_tsv here in the script.

For diffusion training, I believe that data_dir ($cvss_dir/$lang-en/en2${lang}/orig_unit) is not used because for diffusion, it's using the speech features pre-extracted rather than discrete units. I'm sorry for the legacy code that makes things complicated. You can refer to the dataloader class to be precise about what kind of data input is used.

Let me know if you have more questions!

[yunlong-hub]

hello，thank you for your answer, but i still have question.

        #  DiffNorm/blob/main/fairseq/data/audio/repr_to_repr_unit_dataset.py 350-353line
        tgt_tokens = [int(x) for x in tgt_audio_token.split(" ")]
                if not (len(tgt_tokens) == int(tgt_feat_len)):
                    print(f"warning: mismatched feature and unit size. tgt_tokens: {len(tgt_tokens)}, tgt_feat_len: {tgt_feat_len}")
                    continue

when train defussion we use this.
we can see you make a test to ensure tgt_tokens ==tgt_feat_len.
but here tgt_feat_len which from hubert process is no reduce ,so make there tgt_tokes which load from tsv is no reduce length.
I finally use no reduce units‘tsv to train to fit this code “ (len(tgt_tokens) == int(tgt_feat_len))”.

and when inference , we use DiffNorm/research/TranSpeech/diff_norm_synthesis.py，

        for i, batch_data in tqdm(enumerate(data), total=len(data)):
            audio_ids, src_audios, src_n_frames, tgt_feat, ref_units, tgt_lengths = prepare_batch_data(batch_data)
            tgt_mask = lengths_to_mask(tgt_lengths)
            # prepare the initial input
            pred_units, _, _ = model.encoder.ddim_sample(
                tgt_feat,
                input_mask=tgt_mask,
                cond_scale=1.0, # no cond is used
                ref_units=ref_units,
                start_step=args.start_step
            )
            for j, diff_unit in enumerate(pred_units):
                tgt_n_frame = diff_unit.shape[0]
                diff_unit = diff_unit.cpu().numpy()
                diff_unit_str = diff_unit.tolist()
                # print(diff_unit_str)
                diff_unit_str, _, _ = reduce_token(diff_unit_str)
                audio_id = audio_ids[j]
                src_audio = src_audios[j]
                src_n_frame = src_n_frames[j]
                tgt_audio = " ".join([str(_) for _ in diff_unit_str])
                line_to_write = f"{audio_id}\t{src_audio}\t{src_n_frame}\t{tgt_audio}\t{tgt_n_frame}"
                print(line_to_write, file=write_handle)

you predict diff_unit_str and use reduce_token() to reduce. so I guess you probably used full-unit sequences during defussion training and the resulting simplified sequence tsv was used for subsequent S2ST training.

[steventan0110]

Ah thanks for the details. Your understanding is correct, it's been a while since I checked this repo.
Yes, I did diffusion over the original speech feature (not reduced) and predicted speech units. Then the predicted units are de-duplicated and used for NAT training.