NVIDIA Neural Modules 1.22.0

Highlights

Models

NeMo Parakeet

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet/

• https://huggingface.co/nvidia/parakeet-rnnt-1.1b
• https://huggingface.co/nvidia/parakeet-ctc-1.1b
• https://huggingface.co/nvidia/parakeet-rnnt-0.6b
• https://huggingface.co/nvidia/parakeet-ctc-0.6b

NeMo Parakeet-TDT

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet-tdt/

• https://huggingface.co/nvidia/parakeet-tdt-1.1b

ASR

• stt_en_fastconformer_transducer_large_ls #7641
• stt_en_fastconformer_ctc_larg_ls #7641
• stt_en_fastconformer_hybrid_large_streaming_multi
• stt_nl_fastconformer_hybrid_large_pc
• stt_fa_fastconformer_hybrid_large

NeMo ASR

• Multi-lookahead cache-aware streaming Conformer #6711
• Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) by @burchim #7330
• Speech ehancement tutorial #6492
• Support punctuation error rate #7538

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.10

Detailed Changelogs

ASR

Changelog

• Fix missing pip package 'einops' by @RobinDong :: PR: #7397
• Fix failure of installing pyaudio in Online_Offline_Speech_Commands_Demo.ipynb by @RobinDong :: PR: #7396
• [ASR] Confidence measure -> method renames by @GNroy :: PR: #7434
• RNN-T confidence and alignment bugfix by @GNroy :: PR: #7381
• Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) by @burchim :: PR: #7330
• [TTS] Read audio as int32 to avoid flac read errors by @rlangman :: PR: #7477
• Fix typos in confidence tutorial notebooks by @Kipok :: PR: #7581
• Safeguard nemo_text_processing installation on ARM by @blisc :: PR: #7485
• add fc large ls models by @nithinraok :: PR: #7641
• [ASR] RNN-T greedy decoding max_frames fix for alignment and confidence by @GNroy :: PR: #7635
• Create per.py by @ssh-meister :: PR: #7538
• Update docs: readme, getting started, ASR intro by @erastorgueva-nv :: PR: #7679
• [ASR] Multichannel mask estimator with flex number of channels by @anteju :: PR: #7317
• Fix code block typo in docs by @erastorgueva-nv :: PR: #7717
• Replace gpus with devices by @athitten :: PR: #7743
• docs: fix typos by @shuoer86 :: PR: #7758
• Snake act by @nithinraok :: PR: #7736
• fix(clustering_diarizer.py): fix typo by @jqueguiner :: PR: #7772
• Add some docs and update scripts for ASR by @titu1994 :: PR: #7790
• remove TN from ctc_segm tut by @ekmb :: PR: #7807
• Add support for finetuning with huggingface datasets by @stevehuang52 :: PR: #7834
• Adding long-form audio speaker diarization (clustering) class and functions by @tango4j :: PR: #7737
• Fix k2 installation: update for latest PyTorch, move script to dir by @artbataev :: PR: #7887
• [ASR] GSS-based mask estimator by @anteju :: PR: #7849
• add Dutch P&C FC model info by @zhehuaichen :: PR: #7892
• Add checks for unit tests that are looking for data from CI machine by @ericharper :: PR: #7943
• update branch name by @nithinraok :: PR: #7990
• fix librosa display issue by @nithinraok :: PR: #7991
• Fixes Notebooks for ASR by @titu1994 :: PR: #7994
• cherry pick bug 4405781 by @karpnv :: PR: #8044
• fix noise augmentation by @stevehuang52 :: PR: #8056
• Fix various issues with broken links and bugs by @titu1994 :: PR: #8064
• run with non-dev option by @nithinraok :: PR: #8077
• update broken links by @nithinraok :: PR: #8079
• langid bug fix by @karpnv :: PR: #8134

TTS

Changelog

• Add steps for document of getting dataset 'SF Bilingual Speech' by @RobinDong :: PR: #7378
• Fix checking of cuda/cpu device for inputs of Decoder by @RobinDong :: PR: #7444
• Fix failure of ljspeech's get_data.py by @RobinDong :: PR: #7430
• [TTS] Fix audio codec type checks by @rlangman :: PR: #7373
• [TTS] Add dataset to path of logged artifacts by @rlangman :: PR: #7462
• Fix adding positional embeddings in-place in FFTransformerDecoder by @The0nix :: PR: #7440
• Add dataset 'AISHELL-3' from OpenSLR for training mandarin TTS by @RobinDong :: PR: #7409
• [TTS] Fix FastPitch data prep tutorial by @rlangman :: PR: #7524
• add italian tokenization by @GiacomoLeoneMaria :: PR: #7486
• Remap speakers to continuous range of speaker_id for dataset AISHELL3 by @RobinDong :: PR: #7536
• add ItalianPhonemesTokenizer by @GiacomoLeoneMaria :: PR: #7587
• [TTS] Add STFT and SI-SDR loss to audio codec recipe by @rlangman :: PR: #7468
• Fix typo in audio codec config, encoder target by @anteju :: PR: #7697
• Group-residual vector quantizer by @anteju :: PR: #7643
• French g2p with pronunciation dictionary by @mgrafu :: PR: #7601
• add pleasefixme marker for potential failed nightly tests. by @XuesongYang :: PR: #7678
• Add new text segmentation library for better TTS quality by @RobinDong :: PR: #7645
• ConditionalInput: cat along the feature dim, not the batch dim by @anferico :: PR: #7785
• Add selection criteria for reference audios in the submodule by @anferico :: PR: #7788
• [Codec] Update codec checkpoint config by @anteju :: PR: #7835
• [Codec] Finite scalar quantizer by @anteju :: PR: #7886
• Tar codec by @nithinraok :: PR: #7867

LLM

Changelog

• Allow disabling sanity checking when num_sanity_val_steps=0 by @athitten :: PR: #7413
• Add comprehensive error messages by @PeganovAnton :: PR: #7261
• layer selection for ia3 by @arendu :: PR: #7417
• Add rope dynamic linear scaling by @hsiehjackson :: PR: #7437
• Fix sft dataset truncation by @hsiehjackson :: PR: #7464
• fix bug when loading dist ckpt in peft by @lhb8125 :: PR: #7452
• Fix sft chat dataset truncation by @hsiehjackson :: PR: #7478
• SFT model parallel fix for dist ckpt by @aklife97 :: PR: #7511
• remove auto generated examples by @arendu :: PR: #7510
• Add the argument to by @odelalleau :: PR: #7264
• PEFT GPT & T5 Refactor by @meatybobby :: PR: #7308
• fix a typo by @BestJuly :: PR: #7496
• StarCoder SFT test + bump PyT NGC image to 23.09 by @janekl :: PR: #7540
• fix llama2 70b lora tuning bug by @cuichenx :: PR: #7622
• generalized chat sft prompt by @yidong72 :: PR: #7655
• Set base frequency from config by @shan18 :: PR: #7734
• Megatron LLM documentation updates by @ssh-meister :: PR: #7400
• Remove incorrect extra argument of load_from_checkpoint_dir() by @RobinDong :: PR: #7500
• Add nemo to mcore GPT conversion script by @cuichenx :: PR: #7730
• set context for text memmap to fork by @arendu :: PR: #7784
• Support flash decoding by @hsiehjackson :: PR: #7744
• update text server to support compute logprobs by @Zhilin123 :: PR: #7733
• Revert PEFT eval fix by @ericharper :: PR: #7693
• Fix tn duplex by @ekmb :: PR: #7808
• Multimodal merge by @yaoyu-33 :: PR: #7728
• Fix flash decoding precision by @hsiehjackson :: PR: #7852
• Removing duplicate Megatron-LM installation by @Davood-M :: PR: #7864
• adding special_tokens from tokenizer config for transformer-lm model by @clumsy :: PR: #7613
• Add Adapter and IA3 support for MCore models by @cuichenx :: PR: #7750
• Add back import guard by @cuichenx :: PR: #7882
• Change FP8 Defaults by @cuichenx :: PR: #7894
• Added knob for ub_tp_comm_overlap for the MCORE pass by @sanandaraj5597 :: PR: #7902
• Upgrade NeMo to latest mcore and TE by @dimapihtar :: PR: #7862
• Pad sequences to multiples of 16 for GPTSFTDataset by @vysarge :: PR: #7904
• upgrade to latest mcore and TE by @dimapihtar :: PR: #7908
• added missing torch import by @Davood-M :: PR: #7913
• Fix CPU initialization of GPT models by @cuichenx :: PR: #7889
• Fix pinned triton version by @hsiehjackson :: PR: #7925
• fix tp_overlap config var name by @xrennvidia :: PR: #7928
• only enable query key scaling during fp16 by @gshennvm :: PR: #7946
• Fix for gpt3 eval hang with PP (a dtype issue) by @yaoyu-33 :: PR: #7927
• Pass in rotary_base to mcore and from HF by @Kipok :: PR: #7933
• Use NLPDDPStrategyNotebook in Multitask_Prompt_and_PTuning.ipynb by @athitten :: PR: #8061

General Improvements

Changelog

• Add fix for max time to quit trainer gracefully, without running validation by @SeanNaren :: PR: #7731
• SDE Tutorial minor fix by @Jorjeous :: PR: #7598
• Temporary pin Lightning-Utilities version due to broken NamedTuple by @artbataev :: PR: #8022
• Karpnv/issue 7320 by @karpnv :: PR: #7418
• Speech Simulator, update README.md: output_path --> output_manifest_filepath by @popcornell :: PR: #7442
• Fix None dataloader issue in PTL2.0 by @KunalDhawan :: PR: #7455
• HF StarCoder to NeMo conversion script by @janekl :: PR: #7421
• [doc] fix broken link by @stas00 :: PR: #7481
• dllogger - log on rank 0 only by @stas00 :: PR: #7513
• Add two youtube introductory videos to README and Docs. by @XuesongYang :: PR: #7570
• defaults changed by @arendu :: PR: #7600
• Bound transformers version in requirements by @athitten :: PR: #7620
• Fix import error no module name model_utils by @menon92 :: PR: #7629
• Fix in the confidence ensemble test by @Kipok :: PR: #7682
• move core install to /workspace by @aklife97 :: PR: #7706
• distributed checkpoint average script by @yidong72 :: PR: #7721
• fix hybrid eval by @karpnv :: PR: #7757
• fix(diarization-README): typo by @jqueguiner :: PR: #7771
• Configure MCore logger by @mikolajblaz :: PR: #7781
• Nemo to HF converter for LLaMA model by @uppalutkarsh :: PR: #7770
• [Fix] Save best NeMo model only when necessary by @anteju :: PR: #7836
• add guard if its a distributed checkpoint by @gshennvm :: PR: #7845
• Update transformers cache on Jenkins by @ericharper :: PR: #7854
• Update README.rst for container update by @fayejf :: PR: #7844
• Fix mcore conversion bug by @cuichenx :: PR: #7846
• add comment on script and fix target check by @gshennvm :: PR: #7881
• fix issues with convert_nemo_llama_to_hf.py by @Zhilin123 :: PR: #7922
• Instructions for running ci on pr template by @ericharper :: PR: #7944
• Distributed checkpoint averaging supports bf16 type by @yidong72 :: PR: #7888
• Fix tokenizer argparse in scripts by @titu1994 :: PR: #8012
• Check dependencies in installation script by @artbataev :: PR: #8019
• [SE Tutorial] USe GPU for inference, when available by @anteju :: PR: #8048
• update reqs by @ericharper :: PR: #8072
• Remove typo by @ericharper :: PR: #8146