Skip to content

xiaoyu2018/TextSum

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TextSum

0 使用说明

  1. 项目相关依赖已写入requirements.txt文件 pip install -r requirements.txt
  2. 项目使用了transformers提供的预训练模型,相关模型、配置文件、词典文件等于此处下载
  3. 运行项目前,于/source/settings.py中修改路径设置为本地实际绝对路径
  4. 项目结构:
    TextSum
    --dataset(数据集、词典、词频表)
    --params(预训练模型、模型参数保存文件)
    --source(源代码)
    ----go.py(主控函数)
    ----pretrained_models.py(预训练模型)
    ----models.py(自定义模型)
    ----settings.py(项目设置)
    ----utils.py(工具函数)
  5. python go.py 运行项目,可选命令行参数如下:
    -h, --help            show this help message and exit
    -p, --preprocess      预处理数据
    -b, --build           建立词频表
    -m, --make            建立词典
    -t 模型名, --train           训练
                            
    -f 模型名, --fine_tune       微调
                            
    -g 模型名 参数路径, --gen             生成submission
                            
    

1 数据处理

本项目数据处理共分为部分:数据清洗与划分、词典生成、张量转换

  • 数据清洗与划分
    • 使用正则表达式清洗原始数据,去除文本中与任务无关的信息
    • 从原始训练集中划分出验证集
    • 将原始CSV文件转换为逐条文本的JSON文件
  • 词典生成
    统计数据集中出现过的所有单词的词频,取一定数目的高频词生成字典
  • 张量转换
    读取预处理完毕的json文件,进一步处理后将文本数据集转换为成batch的Tensor

2 模型结构

本项目使用pytorch实现了模型基础结构、自定义损失函数、优化器以及模型训练、验证过程;
本项目还使用transformers提供的预训练模型(bart、t5、pegasus)及函数接口实现了模型的微调与推断
以下给出部分模型的网络结构

  1. GRU编码器-解码器架构网络结构如下:
    EncoderDecoder(
    (encoder): GruEncoder(
        (embdding): Embedding(10004, 512)
        (rnn): GRU(512, 256, num_layers=2)
    )
    (decoder): GruDecoder(
        (embdding): Embedding(10004, 512)
        (rnn): GRU(768, 256, num_layers=2)
        (dense): Linear(in_features=256, out_features=10004, bias=True)
    )
    )
  2. t5(small)
    T5ForConditionalGeneration(
        (shared): Embedding(32128, 512)
        (encoder): T5Stack(
            (embed_tokens): Embedding(32128, 512)
            (block): ModuleList(
            (0): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    (relative_attention_bias): Embedding(32, 8)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (1): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (2): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (3): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (4): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (5): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            )
            (final_layer_norm): T5LayerNorm()
            (dropout): Dropout(p=0.1, inplace=False)
        )
        (decoder): T5Stack(
            (embed_tokens): Embedding(32128, 512)
            (block): ModuleList(
            (0): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    (relative_attention_bias): Embedding(32, 8)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (1): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (2): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (3): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (4): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            (5): T5Block(
                (layer): ModuleList(
                (0): T5LayerSelfAttention(
                    (SelfAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (1): T5LayerCrossAttention(
                    (EncDecAttention): T5Attention(
                    (q): Linear(in_features=512, out_features=512, bias=False)
                    (k): Linear(in_features=512, out_features=512, bias=False)
                    (v): Linear(in_features=512, out_features=512, bias=False)
                    (o): Linear(in_features=512, out_features=512, bias=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                (2): T5LayerFF(
                    (DenseReluDense): T5DenseReluDense(
                    (wi): Linear(in_features=512, out_features=2048, bias=False)
                    (wo): Linear(in_features=2048, out_features=512, bias=False)
                    (dropout): Dropout(p=0.1, inplace=False)
                    )
                    (layer_norm): T5LayerNorm()
                    (dropout): Dropout(p=0.1, inplace=False)
                )
                )
            )
            )
            (final_layer_norm): T5LayerNorm()
            (dropout): Dropout(p=0.1, inplace=False)
        )
        (lm_head): Linear(in_features=512, out_features=32128, bias=False)
        )

3 最终成绩

本项目最终成绩为0.32107609

参数设置如下:

  • 模型:bart-large-cnn
  • 搜索束个数:2
  • 最大序列长度:1024
  • 激活函数:gelu
  • 预测序列最短长度:30
  • 预测序列最长长度:590
  • 是否允许提前停止(预测出<EOS>即停止):是

Releases

No releases published

Packages

No packages published

Languages