使用 activations.py 中的激活函数

[JunnYu]

背景

如图所示，PaddleNLP中没有复用激活函数，在许多模型文件中都自己定义了一个激活函数。

问题

激活函数是可以复用的，而当前没有统一使用激活函数，这个 PR 新增了与HF等价的激活函数activations.py，希望未来新增的模型 以及 旧的模型都替换使用最新的激活函数。

import paddlenlp.transformers.activations as pdact
import transformers.activations as ptact
import torch
for actstring in ptact.ACT2FN.keys():
    a1 = pdact.get_activation(actstring)
    a2 = ptact.get_activation(actstring)
    pdx = paddle.randn((4,100))
    ptx = torch.tensor(pdx.numpy()).cuda()
    diff = abs(a1(pdx).numpy() - a2(ptx).cpu().numpy())
    minmax = diff.mean(), diff.max()
    print(f"{actstring} activation diff: {minmax[0]}, {minmax[1]}")

# gelu activation diff: 1.0716030729440718e-08, 1.1920928955078125e-07
# gelu_10 activation diff: 1.0223593527314279e-08, 2.384185791015625e-07
# gelu_fast activation diff: 0.0, 0.0
# gelu_new activation diff: 2.840533919190591e-10, 6.891787052154541e-08
# gelu_python activation diff: 1.6588019846608404e-08, 4.76837158203125e-07
# linear activation diff: 0.0, 0.0
# mish activation diff: 1.0873191236271396e-08, 1.341104507446289e-07
# quick_gelu activation diff: 0.0, 0.0
# relu activation diff: 0.0, 0.0
# relu6 activation diff: 0.0, 0.0
# sigmoid activation diff: 0.0, 0.0
# silu activation diff: 0.0, 0.0
# swish activation diff: 0.0, 0.0
# tanh activation diff: 0.0, 0.0

[sijunhe]

挺好。我们后续单测环境里会有torch和transformers的依赖，上面着一些代码可以写成一个单测

[JunnYu]

OK，其实还有很多代码可能要统一一下，比如 scheduler、initializer 等

[wj-Mcat]

关于这块的统一都可以以 issue 或 discussion 的形式留存下来讨论，这样后续也方便跟进。 @JunnYu