[WIP] 🚀🚀🚀 Transformers.js V3 🚀🚀🚀

[xenova]

In preparation for Transformers.js v3, I'm compiling a list of issues/features which will be fixed/included in the release.

• WebGPU support (upgrade onnxruntime-web to 1.17.0).
  • closes WebGPU Support #20
  • closes [Feature request] WebGPU support #73
  • closes [Bug] image segmentation very slow #79
  • closes Whisper on webGPU? #100
  • closes [Question] A WebGPU-accelerated ONNX inference run-time #119
  • closes [Feature request] Update ONNX runtime to version 1.15.1 #298
  • closes GPU Acceleration to increase performance #377
  • closes [Feature request] Real time whisper transcription #405
  • closes How do I use WebGL as executionProvider? #505
  • closes YOLOS model extremely slow #533
• Fix logging/disable warnings (onnxruntime-web → 1.17.0). Closes:
  • closes [Question] How to stop warning log #270
  • closes env.backends.onnx.logLevel = "fatal"; does not work #529
• Fix WASM backend for large models (onnxruntime-web → 1.17.0). Closes:
  • closes An error occurred during model execution: "RangeError: offset is out of bounds". #499
• Deno support (upgrade sharp.js to 0.33.x). Closes:
  • closes [Feature request] Deno Support #78
  • closes Sharpe Linux-x86 #541
• CommonJS compatibility. Closes [Feature request] CommonJS compatability #152
• Skip local model check when running in-browser, unless explicitly set by user. This is an issue experienced by many beginners, where requests made to localhost redirect to error page, but dev server incorrectly returns status code 200. Closes
  • closes [Question] React.js serve sentence bert in browser keep reporting models not found. #328
  • closes getModelJSON fails with a bundler (see reproduction) #366
• Improve unit test suite and allow local testing. Closes Running tests locally fail #491
• Create model conversion space. Closes
  • closes [Visual Model Conversion] Do-It-Yourself: A visual tool that allows users to convert their own models without sweats and tears #222
  • closes [Question] How to convert to ONNX a fine-tuned model #274
• Upgrade conversion script dependency versions (+fixes sentence-transformers conversions). Closes
  • closes [Question] distiluse-base-multilingual-cased-v2 - wrong vector dimension (768 vs 512) in onnx version? #230
• Versioned documentation, so that users still on v2 will be able to access the correct documentation.
• Improve PretrainedModel, PretrainedTokenizer, and Processor types. In a similar way to how the pipeline API has conditional types, we'll add the same for the other classes accessible by users.
• Consistency issues:
  • topk -> top_k parameter.
  • Tensor transpose -> permute
  • Like the python library, encapsulate multimodal preprocessing in the AutoProcessor class, which uses image processor and tokenizer
• Improved model caching of revisions, and purge cache if model files are out-of-date. Ideally, this should be enabled by a setting to avoid unnecessary network calls to check versions if the files are present in cache.
• Improve pipeline fallback errors
  • closes Unsupported model type: whisper and CORS error #314
• WebNN support
  • reference: [DO NOT MERGE] Code example for testing ORT-Web WebNN EP #608

How to use WebGPU

First, install the development branch

npm install xenova/transformers.js#v3

Then specify the device parameter when loading the model. Here's example code to get started. Please note that this is still a WORK IN PROGRESS, so the following usage may change before release.

import { pipeline } from '@xenova/transformers';

// Create feature extraction pipeline
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
    device: 'webgpu',
    dtype: 'fp32', // or 'fp16'
});

// Generate embeddings
const sentences = ['That is a happy person', 'That is a very happy person'];
const output = await extractor(sentences, { pooling: 'mean', normalize: true });
console.log(output.tolist());

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Huguet57]

Hey! This is great. Is this already in alpha?

[beaufortfrancois]

@xenova For some models, the performance may be a blocker. Since model downloads can be quite large, I wonder if there should be a way for web developers to know their machine performance class for running a model without downloading it completely first.

I believe this would involve running the model code with zeroed-out weights, which would still require buffer allocations but would allow the web app to catch out-of-memory errors or such. The model architecture would still needed to generate shaders, but this be much smaller than model weights.

Essentially, knowing the model architecture and testing with empty weights would allow for assessing performance capability without downloading the full model.

I thought I could use from_config for that but I wonder now if this should be a built-in V3 feature. What are your thoughts?