update big model ram #3062

psky1111 · 2023-12-16T15:12:46Z

ram and ram plus pretrain：
链接：https://pan.baidu.com/s/1VV9PvEnC1e6yFj8Ccwr7Dg
提取码：ifgr
--来自百度网盘超级会员V6的分享

paddle-bot · 2023-12-16T15:13:07Z

Thanks for your contribution!

cuicheng01

1.comment需要好好改一下，所有的py需要注意下代码规范
2.建议ram放到arch下，不要放在backbone下
3.所有的文件需要过pre-commit
4.需要增加详细的文档，包括训练，infer、推理，可以参考：https://github.com/PaddlePaddle/PaddleClas/blob/develop/ppcls/configs/MultiLabelCOCO/MLDecoder/README.md

cuicheng01 · 2023-12-17T00:05:44Z

deploy/python/postprocess.py

+
+class RamOutPut(object):
+
+ def __init__(self, language="cn",tag_list="", tag_list_chinese="", threshold=0.68, delete_tag_index=[], ram_class_threshold_path="ppcls/utils/RAM/ram_tag_list_threshold.txt"):


文件需要过一下pre-commit，ram_class_threshold_path在yaml中传入

cuicheng01 · 2023-12-17T00:06:40Z

deploy/python/predict_multimodal.py

@@ -0,0 +1,52 @@
+import os


需要增加版权信息

cuicheng01 · 2023-12-17T00:07:16Z

ppcls/arch/backbone/clip/clip.py

@@ -0,0 +1,513 @@
+import paddle


需要增加版权信息

cuicheng01 · 2023-12-17T00:09:32Z

ppcls/arch/backbone/clip/clip.py

+ )
+ return model, get_transforms(224)
+
+def clip_vit_b_16_224():


这里的命名最好和paddleclas的整体命名保持一致

cuicheng01 · 2023-12-17T00:10:03Z

ppcls/arch/backbone/clip/tokenizer.py

@@ -0,0 +1,140 @@
+import gzip


增加版权信息，增加代码引用信息

cuicheng01 · 2023-12-17T01:46:44Z

tools/infer_multimodal.py

@@ -0,0 +1,18 @@
+from __future__ import absolute_import


去掉，增加版权信息

cuicheng01 · 2023-12-17T01:46:56Z

tools/infer_multimodal.py

+from __future__ import print_function
+import os
+import sys
+__dir__ = os.path.dirname(os.path.abspath(__file__))


注意规范和顺序

cuicheng01 · 2023-12-17T01:47:03Z

tools/train_multimodal.py

@@ -0,0 +1,34 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


cuicheng01 · 2023-12-17T01:47:11Z

tools/train_multimodal.py

+from __future__ import print_function
+import os
+import sys
+__dir__ = os.path.dirname(os.path.abspath(__file__))


注意规范和顺序

cuicheng01 · 2023-12-17T01:47:31Z

tools/train_multimodal.py

+
+if __name__ == "__main__":
+ args = config.parse_args()
+ args.config = "./ppcls/configs/ram/config_train.yaml"


这不要写死

psky1111 · 2023-12-18T02:25:20Z

updated ram parameters：
链接：https://pan.baidu.com/s/1UKn5yfRsV6KIP4K5rgK8VA
提取码：dbvi
--来自百度网盘超级会员V6的分享

psky1111 · 2023-12-22T06:12:35Z

clip pretrain param:
链接：https://pan.baidu.com/s/1Nx0quYYkpNOlmkkv5mt8mg
提取码：lx88
--来自百度网盘超级会员V6的分享
clip pretrained param script:
"""python
map_key = {
"in_proj_": "in_proj."
}
def paddle_to_torch(torch_dict, paddle_dict):
new_paddle_dict = {}
for torch_key in torch_dict.keys():
paddle_key = torch_key
for mm_k in map_key.keys():
if mm_k in paddle_key:
paddle_key = paddle_key.replace(mm_k, map_key[mm_k])
if ('out_proj.weight' in paddle_key) or ("in_proj.weight" in paddle_key) or (("mlp" in paddle_key) and ("weight") in paddle_key):
paddle_tensor = paddle.to_tensor(torch_dict[torch_key].cpu().numpy().transpose())
shape_ori, shape_new = paddle_dict[paddle_key].shape, paddle_tensor.shape
if shape_ori == shape_new:
new_paddle_dict[paddle_key] = paddle_tensor
else:
new_paddle_dict[paddle_key] = paddle_tensor.T
else:
paddle_tensor = paddle.to_tensor(torch_dict[torch_key].cpu().numpy())
shape_ori, shape_new = paddle_dict[paddle_key].shape, paddle_tensor.shape
if shape_ori == shape_new:
new_paddle_dict[paddle_key] = paddle_tensor
else:
new_paddle_dict[paddle_key] = paddle_tensor.T
return new_paddle_dict
"""

psky1111 · 2023-12-22T06:24:01Z

ram code with readme
链接：https://pan.baidu.com/s/1X1qvQs8z3zOk8VTQI9g4Pw
提取码：bi58
--来自百度网盘超级会员V6的分享

cuicheng01 · 2023-12-22T08:24:35Z

.gitignore

@@ -15,3 +15,4 @@ nohup.out
 .idea
 inference/
 test.py
+clip_text.py


这个去掉吧

cuicheng01 · 2023-12-22T08:25:36Z

deploy/configs/inference_ram.yaml

+ size: 384
+ - NormalizeImage:
+ scale: 0.00392157
+ mean: [0.485, 0.456, 0.406]


官方的CLIP的mean和std不是这个值，这里需要确认下要不要改

cuicheng01 · 2023-12-22T08:26:20Z

deploy/python/postprocess.py

@@ -71,7 +71,7 @@ def __init__(self, func_list, main_indicator="Topk"):
 def __call__(self, x, image_file=None):
 rtn = None
 for func in self.func_list:
- tmp = func(x, image_file)
+ tmp = func(*x, image_file)


改这个的目的是？

cuicheng01 · 2023-12-22T08:34:40Z

deploy/python/postprocess.py

+ delete_tag_index=[],
+ ram_class_threshold_path=""):
+ self.language = language
+ assert tag_list, tag_list_chinese


这里需要给出报错信息

cuicheng01 · 2023-12-22T08:36:02Z

deploy/python/predict_multimodal.py

+import cv2
+import numpy as np
+
+from paddleclas.deploy.utils import logger, config


这里不需要从paddleclas下到入吧？

cuicheng01 · 2023-12-22T08:54:41Z

ppcls/configs/ram/README.md

+
+| Model | BackBone | Size | Inference Prompt | OpenImages-MAP |
+|-------|------------|--------|------------------|----------------|
+| RAM | Swin-large | 5.63GB | LLM Tag Dec | 82.2 |


这里需要指明CLIP用的是哪个模型

cuicheng01 · 2023-12-22T08:55:18Z

ppcls/configs/ram/README.md

+* 前往官方[repo](https://github.com/xinyu1205/recognize-anything/tree/main)下载对应数据集json文件。同时按照json文件目录格式，准备相应的数据。目录格式为：
+```json
+{
+ {


里边的每个字段需要解释下

cuicheng01 · 2023-12-22T08:58:32Z

ppcls/configs/ram/README.md

+```
+**注意:**
+1. 目前多标签分类的损失函数默认使用`AsymmetricLoss`。
+2. 目前多标签分类的评估指标默认使用`MAP(integral)`。


注意的这两个不太对吧？

cuicheng01 · 2023-12-22T08:59:29Z

ppcls/configs/ram/README.md

+
+得到类似下面的输出：
+```
+{'class_ids': [[0, 593], [0, 871], [0, 998], [0, 2071], [0, 3336], [0, 3862]], 'scores': [871], 'label_names': ['棕色 | 鸡 | 公鸡 | 母鸡 | 红色 | 站/矗立/摊位']}


这个scores是？

cuicheng01 · 2023-12-22T09:00:11Z

ppcls/configs/ram/README.md

+python3 -m paddle.distributed.launch \
+ --gpus="0,1,2,3" \
+ tools/train_multimodal.py \
+ -c ./ppcls/configs/ram/RAM.yaml


只有训练，没有finetune示例？

cuicheng01 · 2023-12-22T09:25:31Z

ppcls/configs/ram/RAM_plus.yaml

+
+ sampler:
+ name: DistributedBatchSampler
+ batch_size: 52


psky1111 · 2024-01-06T08:50:34Z

models--bert-base-uncased.zip

1. the behavior of static inference is different and thus need to reshape dimension. 2. training dataloader add collect_fn to replace original one. 3. fix the threshold.

1. fix ram training problem. 2. finish clip vetorlize inference

update big model ram

965ad40

paddle-bot bot added the contributor label Dec 16, 2023

psky1111 added 2 commits December 17, 2023 01:25

Update ramloss.py

e973e28

Update ram.py

feb4d25

cuicheng01 reviewed Dec 17, 2023

View reviewed changes

psky1111 added 3 commits December 17, 2023 14:14

pre-commit format fix

5deb703

update ram doc

e717628

update

fdbbca6

psky1111 added 6 commits December 20, 2023 17:31

fix format problem

79fe074

pre-commit fix

7cb7b5c

update

8321d03

pre-commit

a906776

update format

ac1eb00

Update __init__.py

345964a

update

38be455

cuicheng01 reviewed Dec 22, 2023

View reviewed changes

ppcls/configs/ram/RAM_plus.yaml

sampler:

name: DistributedBatchSampler

batch_size: 52

Copy link

Collaborator

cuicheng01 Dec 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

单卡bs

psky1111 added 2 commits December 23, 2023 13:31

update

ffb5d82

Update .gitignore

fed21b3

psky1111 added 6 commits January 7, 2024 12:09

fix bug

eed2bbf

1. the behavior of static inference is different and thus need to reshape dimension. 2. training dataloader add collect_fn to replace original one. 3. fix the threshold.

fix train gpu problem

b54d5de

compatiable for static graph

7433000

update format

0fe4d4e

fix format problem

54b4234

del the static support

265ee4a

psky1111 added 4 commits January 11, 2024 00:11

update

73661e6

1. fix ram training problem. 2. finish clip vetorlize inference

Update ram_dataset.py

131b42c

add print

5aad8fc

remove clip part

c5a7678

paddle-bot bot assigned cuicheng01 Feb 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update big model ram #3062

update big model ram #3062

psky1111 commented Dec 16, 2023

paddle-bot bot commented Dec 16, 2023

cuicheng01 left a comment

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

cuicheng01 Dec 17, 2023

psky1111 commented Dec 18, 2023

psky1111 commented Dec 22, 2023 •

edited

psky1111 commented Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

cuicheng01 Dec 22, 2023

psky1111 commented Jan 6, 2024


		class RamOutPut(object):

		def __init__(self, language="cn",tag_list="", tag_list_chinese="", threshold=0.68, delete_tag_index=[], ram_class_threshold_path="ppcls/utils/RAM/ram_tag_list_threshold.txt"):

		@@ -0,0 +1,34 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

update big model ram #3062

Are you sure you want to change the base?

update big model ram #3062

Conversation

psky1111 commented Dec 16, 2023

paddle-bot bot commented Dec 16, 2023

cuicheng01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

psky1111 commented Dec 18, 2023

psky1111 commented Dec 22, 2023 • edited

psky1111 commented Dec 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

psky1111 commented Jan 6, 2024

psky1111 commented Dec 22, 2023 •

edited