Add Attention Fuse pass #809
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiangjiajun
requested changes
Jul 16, 2022
paddle2onnx/mapper/exporter.cc
Outdated
| #include "paddle2onnx/optimizer/fuse_constant_cast.h" | ||
| #include "paddle2onnx/optimizer/fuse_constant_reshape.h" | ||
| #include "paddle2onnx/optimizer/fuse_constant_unsqueeze.h" | ||
| #include "paddle2onnx/optimizer/fuse_paddle_conv_bias.h" | ||
| #include "paddle2onnx/optimizer/fuse_unsqueeze_conv2d_squeeze.h" | ||
|
|
||
| namespace paddle2onnx { | ||
| MapperHelper* MapperHelper::helper = nullptr; | ||
| MapperHelper *MapperHelper::helper = nullptr; |
|
|
||
| std::string getPassName() const override { return "fuse_attention"; } | ||
|
|
||
| bool patternMatchPredicate(Node *node) override { |
There was a problem hiding this comment.
在原始代码中都使用指针符号靠左对齐，新添加的代码需对齐
| // +------|---+ | ||
| // | | | ||
| // Add | ||
|
|
There was a problem hiding this comment.
这里有一幅整图的fuse前后对比
在下面的代码中也对应写出fuse中每一步的几个点节（而不单纯只用QKV WEIGHT来描述）
| @@ -35,6 +35,7 @@ struct OptimizerOption { | |||
| passes.push_back("fuse_matmul_add_bias_into_gemm"); | |||
| passes.push_back("eliminate_identity"); | |||
| passes.push_back("eliminate_deadend"); | |||
| passes.push_back("fuse_attention"); | |||
There was a problem hiding this comment.
这不是一个必选的fuse，所以需要额外的开关来控制
There was a problem hiding this comment.
Done.添加enable_extra_ort_opt开关，默认为False
|
这个PR，需给出最终的实验数据和效果 |
在描述中给出相关实验数据，CPU上增加repeat次数，解决方差较大问题 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Do follow contributes
1、添加Attention fuse pass，仅支持ORT
2、添加enable_extra_ort_opt开关，含义为是否开启额外只针对ORT的优化，默认为false
若使用，使用如下命令
相关性能测试
1、非裁剪模型
添加paddle2onnx Attention fuse pass，导出ONNX模型自带Attention node
GPU：
100次warmup，1000次预测取平均，预测时间仅包含run+数据拷贝，表示为（mean），单位为ms，w/o和w代表是否命中attention pass：
CPU:
取100次warmup以及2000次预测取平均，线程数为1，预测时间包含run+数据拷贝，表示为（mean），单位为ms，w/o和w代表是否命中attention pass：
2、裁剪模型
添加paddle2onnx Attention fuse pass，导出ONNX模型自带Attention node
GPU：
100次warmup，1000次预测取平均，预测时间仅包含run+数据拷贝，表示为（mean），单位为ms，w/o和w代表是否命中attention pass：
CPU:
取100次warmup以及2000次预测取平均，线程数为1，预测时间包含run+数据拷贝，表示为（mean），单位为ms，w/o和w代表是否命中attention pass：