Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @jaybdub
I am introducing this new QAT workflow which is compatible with TensorRT 8.
TRT introduced IQuantize and IDequantize Layers which are to be manually placed in the network based on the guidelines mentioned in Q/DQ placement.
I have added support to quantize
nn.Conv2d
,nn.MaxPool2d
andnn.AdaptiveAvfPool2d
- layers that are necessary to quantize Resnet(s). I have also added aQuantGenericTensor
which can be used to add QDQ layer anywhere in the model based on Nvidia's guidelines.This PR also introduces the option to choose between per tensor quantization and per channel quantization. All quant layers are scriptable with
torch.jit.script
Most of the files that I have modified / changed are under
contrib
folders, so it doesn't affect the main torch2trt library.I will continue to add support for more layers but I believe this PR is big enough to land and then I can put up smaller PRs to add more functionalities.
Entire workflow is tested with Pytorch NGC Container
22.04-py3
Thanks.