Skip to content

Releases: shibing624/text2vec

1.2.9

20 Sep 03:14
@shibing624 shibing624
1.2.9
5cc36a9
Compare
Choose a tag to compare
1.2.9 Latest
Latest

1.2.9版本

  1. 支持多卡推理（多进程实现多GPU、多CPU推理），text2vec支持多卡推理(计算文本向量): examples/computing_embeddings_multi_gpu_demo.py

  2. 新增命令行工具（CLI），可以无需代码开发批量获取文本向量：

pip install text2vec>=1.2.9
text2vec --input_file input.txt --output_file out.csv --batch_size 128 --multi_gpu True

Full Changelog: 1.2.8...1.2.9

Assets 2

1.2.8

19 Sep 06:42
@shibing624 shibing624
1.2.8
7c56a08
Compare
Choose a tag to compare
1.2.8

1.2.8版本

  1. 支持多卡推理（多进程实现多GPU和多CPU推理），text2vec支持多卡推理(计算文本向量): examples/computing_embeddings_multi_gpu_demo.py

  2. 新增命令行工具（CLI），可以无需代码开发批量获取文本向量：

pip install text2vec -U
text2vec --input_file input.txt --output_file out.csv --batch_size 16

Full Changelog: 1.2.4...1.2.8

Assets 2

1.2.4

04 Sep 07:16
@shibing624 shibing624
1.2.4
e39924e
Compare
Choose a tag to compare
1.2.4

v1.2.4版本

  1. 实现了BGE微调训练方法 ，支持自定义样本集训练 https://github.com/shibing624/text2vec/blob/master/examples/training_bge_model_mydata.py ；支持构建训练样本集 https://github.com/shibing624/text2vec/blob/master/examples/data/build_zh_bge_dataset.py ；支持使用C-MTEB评估 https://github.com/shibing624/text2vec/blob/master/tests/eval_C-MTEB.py
  2. 发布了中文匹配模型shibing624/text2vec-bge-large-chinese，用CoSENT方法训练，基于BAAI/bge-large-zh-noinstruct用人工挑选后的中文STS数据集shibing624/nli-zh-all/text2vec-base-chinese-paraphrase-dataset训练得到，并在中文测试集评估相对于原模型效果有提升，相较于原模型在短文本区分度上提升明显。

Full Changelog: 1.2.3...1.2.4

Assets 2

v1.2.2

22 Jun 07:05
@shibing624 shibing624
1.2.2
a537947
Compare
Choose a tag to compare
v1.2.2

v1.2.2版本

英文匹配数据集的评测结果：

Arch BaseModel Model English-STS-B
GloVe glove Avg_word_embeddings_glove_6B_300d 61.77
BERT bert-base-uncased BERT-base-cls 20.29
BERT bert-base-uncased BERT-base-first_last_avg 59.04
BERT bert-base-uncased BERT-base-first_last_avg-whiten(NLI) 63.65
SBERT sentence-transformers/bert-base-nli-mean-tokens SBERT-base-nli-cls 73.65
SBERT sentence-transformers/bert-base-nli-mean-tokens SBERT-base-nli-first_last_avg 77.96
CoSENT bert-base-uncased CoSENT-base-first_last_avg 69.93
CoSENT sentence-transformers/bert-base-nli-mean-tokens CoSENT-base-nli-first_last_avg 79.68
CoSENT sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 shibing624/text2vec-base-multilingual 80.12
  • 本项目release模型的中文匹配评测结果：
Arch BaseModel Model ATEC BQ LCQMC PAWSX STS-B SOHU-dd SOHU-dc Avg QPS
Word2Vec word2vec w2v-light-tencent-chinese 20.00 31.49 59.46 2.57 55.78 55.04 20.70 35.03 23769
SBERT xlm-roberta-base sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 18.42 38.52 63.96 10.14 78.90 63.01 52.28 46.46 3138
Instructor hfl/chinese-roberta-wwm-ext moka-ai/m3e-base 41.27 63.81 74.87 12.20 76.96 75.83 60.55 57.93 2980
CoSENT hfl/chinese-macbert-base shibing624/text2vec-base-chinese 31.93 42.67 70.16 17.21 79.30 70.27 50.42 51.61 3008
CoSENT hfl/chinese-lert-large GanymedeNil/text2vec-large-chinese 32.61 44.59 69.30 14.51 79.44 73.01 59.04 53.12 2092
CoSENT nghuyong/ernie-3.0-base-zh shibing624/text2vec-base-chinese-sentence 43.37 61.43 73.48 38.90 78.25 70.60 53.08 59.87 3089
CoSENT nghuyong/ernie-3.0-base-zh shibing624/text2vec-base-chinese-paraphrase 44.89 63.58 74.24 40.90 78.93 76.70 63.30 63.08 3066
CoSENT sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 shibing624/text2vec-base-multilingual 32.39 50.33 65.64 32.56 74.45 68.88 51.17 53.67 4004

说明：

Full Changelog: 1.2.1...1.2.2

Assets 2

v1.2.1

19 Jun 14:39
@shibing624 shibing624
1.2.1
4e46834
Compare
Choose a tag to compare
v1.2.1

v1.2.1

Release Models

  • 本项目release模型的中文匹配评测结果：
Arch BaseModel Model ATEC BQ LCQMC PAWSX STS-B SOHU-dd SOHU-dc Avg QPS
Word2Vec word2vec w2v-light-tencent-chinese 20.00 31.49 59.46 2.57 55.78 55.04 20.70 35.03 23769
SBERT xlm-roberta-base sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 18.42 38.52 63.96 10.14 78.90 63.01 52.28 46.46 3138
Instructor hfl/chinese-roberta-wwm-ext moka-ai/m3e-base 41.27 63.81 74.87 12.20 76.96 75.83 60.55 57.93 2980
CoSENT hfl/chinese-macbert-base shibing624/text2vec-base-chinese 31.93 42.67 70.16 17.21 79.30 70.27 50.42 51.61 3008
CoSENT hfl/chinese-lert-large GanymedeNil/text2vec-large-chinese 32.61 44.59 69.30 14.51 79.44 73.01 59.04 53.12 2092
CoSENT nghuyong/ernie-3.0-base-zh shibing624/text2vec-base-chinese-sentence 43.37 61.43 73.48 38.90 78.25 70.60 53.08 59.87 3089
CoSENT nghuyong/ernie-3.0-base-zh shibing624/text2vec-base-chinese-paraphrase 44.89 63.58 74.24 40.90 78.93 76.70 63.30 63.08 3066

Full Changelog: 1.2.0...1.2.1

Assets 2

v1.2.0

16 Jun 05:42
@shibing624 shibing624
1.2.0
3d2fa90
Compare
Choose a tag to compare
v1.2.0

v1.2.0版本

  • 发布了中文匹配模型shibing624/text2vec-base-chinese-nli，基于ERNIE-3.0-base模型，使用了中文NLI数据集shibing624/nli_zh全部语料训练的CoSENT文本匹配模型，在各评估集表现提升明显。

  • 发布了2个中文NLI数据集：shibing624/snli-zh 和 shibing624/nli-zh-all

  • 本项目release模型的中文匹配评测结果：

Arch BaseModel Model ATEC BQ LCQMC PAWSX STS-B Avg QPS
Word2Vec word2vec w2v-light-tencent-chinese 20.00 31.49 59.46 2.57 55.78 33.86 23769
SBERT xlm-roberta-base sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 18.42 38.52 63.96 10.14 78.90 41.99 3138
CoSENT hfl/chinese-macbert-base shibing624/text2vec-base-chinese 31.93 42.67 70.16 17.21 79.30 48.25 3008
CoSENT hfl/chinese-lert-large GanymedeNil/text2vec-large-chinese 32.61 44.59 69.30 14.51 79.44 48.08 2092
CoSENT nghuyong/ernie-3.0-base-zh shibing624/text2vec-base-chinese-nli 51.26 68.72 79.13 34.28 80.70 62.81 3066
  • 本项目release的数据集：
Dataset Introduce Download Link
shibing624/nli-zh-all 中文语义匹配数据合集，整合了文本推理，相似，摘要，问答，指令微调等任务的820万高质量数据，并转化为匹配格式数据集 https://huggingface.co/datasets/shibing624/nli-zh-all
shibing624/snli-zh 中文SNLI和MultiNLI数据集，翻译自英文SNLI和MultiNLI https://huggingface.co/datasets/shibing624/snli-zh
shibing624/nli_zh 中文语义匹配数据集，整合了中文ATEC、BQ、LCQMC、PAWSX、STS-B共5个任务的数据集 https://huggingface.co/datasets/shibing624/nli_zh
or
百度网盘(提取码:qkt6)
or
github
  • 基于更大数据集shibing624/nli-zh-all的CoSENT匹配模型在训练中。

Full Changelog: 1.1.8...1.2.0

Assets 2

1.1.4

12 Mar 06:16
@shibing624 shibing624
1.1.4
00c3c2e
Compare
Choose a tag to compare
1.1.4

v1.1.4版本

发布了中文匹配模型shibing624/text2vec-base-chinese，基于中文STS训练集训练的CoSENT匹配模型。

  • 本项目release模型的中文匹配评测结果：
Arch BaseModel Model ATEC BQ LCQMC PAWSX STS-B Avg QPS
Word2Vec word2vec w2v-light-tencent-chinese 20.00 31.49 59.46 2.57 55.78 33.86 23769
SBERT xlm-roberta-base sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 18.42 38.52 63.96 10.14 78.90 41.99 3138
CoSENT hfl/chinese-macbert-base shibing624/text2vec-base-chinese 31.93 42.67 70.16 17.21 79.30 48.25 3008

Full Changelog: 1.1.3...1.1.4

add word2vec tencent light embeddings file: light_Tencent_AILab_ChineseEmbedding.bin

Assets 3

1.1.3

07 Mar 14:39
@shibing624 shibing624
1.1.3
0e19206
Compare
Choose a tag to compare
1.1.3

Full Changelog: 1.1.2...1.1.3

Assets 2

1.1.2

01 Mar 02:49
@shibing624 shibing624
1.1.2
5f7ee24
Compare
Choose a tag to compare
1.1.2

add dataset of nli_zh

Assets 3

1.1.0

26 Feb 06:48
@shibing624 shibing624
1.1.0
83bb3d5
Compare
Choose a tag to compare
1.1.0

重写了CoSENT, SentenceBERT模型的训练和预测代码：

  1. 句子匹配模型训练逻辑继承基类SentenceModel，
  2. 新增train_model, eval_model, 代码结构更清晰，
  3. 预测均使用基类的encode实现。
Assets 2