MUGE Text-to-Image Retrieval (Official Validation Set):
| Setup | Zero-shot | Finetune | ||||||
|---|---|---|---|---|---|---|---|---|
| Metric | R@1 | R@5 | R@10 | MR | R@1 | R@5 | R@10 | MR |
| CN-CLIPRN50 | 42.6 | 68.6 | 77.9 | 63.0 | 48.6 | 75.1 | 84.0 | 69.2 |
| CN-CLIPViT-B/16 | 52.1 | 76.7 | 84.4 | 71.1 | 58.4 | 83.6 | 90.0 | 77.4 |
| CN-CLIPViT-L/14 | 56.3 | 79.8 | 86.2 | 74.1 | 63.3 | 85.6 | 91.3 | 80.1 |
| CN-CLIPViT-L/14@336px | 59.0 | 81.4 | 87.8 | 76.1 | 65.3 | 86.7 | 92.1 | 81.3 |
| CN-CLIPViT-H/14 | 63.0 | 84.1 | 89.2 | 78.8 | 68.9 | 88.7 | 93.1 | 83.6 |
Flickr30K-CN Retrieval (Official Test Set):
| Task | Text-to-Image | Image-to-Text | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Setup | Zero-shot | Finetune | Zero-shot | Finetune | ||||||||
| Metric | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| CN-CLIPRN50 | 48.8 | 76.0 | 84.6 | 66.7 | 89.4 | 94.1 | 60.0 | 85.9 | 92.0 | 84.2 | 96.7 | 98.0 |
| CN-CLIPViT-B/16 | 62.7 | 86.9 | 92.8 | 79.1 | 94.8 | 97.4 | 74.6 | 93.5 | 97.1 | 93.5 | 99.0 | 99.5 |
| CN-CLIPViT-L/14 | 68.0 | 89.7 | 94.4 | 82.7 | 96.7 | 98.6 | 80.2 | 96.6 | 98.2 | 96.1 | 99.5 | 99.9 |
| CN-CLIPViT-L/14@336px | 69.0 | 90.7 | 95.4 | 84.4 | 97.1 | 98.7 | 83.3 | 97.2 | 98.5 | 96.6 | 99.8 | 100.0 |
| CN-CLIPViT-H/14 | 71.2 | 91.4 | 95.5 | 83.8 | 96.9 | 98.6 | 81.6 | 97.5 | 98.8 | 95.3 | 99.7 | 100.0 |
COCO-CN Retrieval (Official Test Set):
| Task | Text-to-Image | Image-to-Text | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Setup | Zero-shot | Finetune | Zero-shot | Finetune | ||||||||
| Metric | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| CN-CLIPRN50 | 48.1 | 81.3 | 90.5 | 66.8 | 91.1 | 97.0 | 51.6 | 81.2 | 90.5 | 68.4 | 93.3 | 97.8 |
| CN-CLIPViT-B/16 | 62.2 | 86.6 | 94.9 | 77.0 | 97.1 | 99.0 | 57.0 | 84.1 | 93.6 | 77.4 | 96.2 | 98.9 |
| CN-CLIPViT-L/14 | 64.0 | 89.2 | 94.4 | 78.9 | 96.3 | 99.0 | 60.4 | 84.2 | 92.9 | 80.2 | 96.7 | 99.2 |
| CN-CLIPViT-L/14@336px | 64.7 | 89.6 | 94.6 | 80.1 | 96.7 | 99.2 | 63.4 | 87.2 | 94.4 | 81.2 | 97.2 | 99.1 |
| CN-CLIPViT-H/14 | 69.2 | 89.9 | 96.1 | 81.5 | 96.9 | 99.1 | 63.0 | 86.6 | 92.9 | 83.5 | 97.3 | 99.2 |
Zero-shot Image Classification:
| Task | CIFAR10 | CIFAR100 | DTD | EuroSAT | FER | FGVC | KITTI | MNIST | PC | VOC | ImageNet |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CN-CLIPRN50 | 72.7 | 40.6 | 36.9 | 27.0 | 21.9 | 5.4 | 30.2 | 50.2 | 47.7 | 82.1 | 33.5 |
| CN-CLIPViT-B/16 | 92.0 | 64.4 | 43.6 | 46.9 | 47.2 | 12.8 | 33.5 | 67.6 | 54.0 | 83.3 | 48.3 |
| CN-CLIPViT-L/14 | 94.9 | 75.1 | 44.2 | 56.9 | 54.6 | 16.0 | 49.9 | 69.8 | 63.5 | 84.5 | 54.7 |
| CN-CLIPViT-L/14@336px | 94.1 | 73.5 | 43.8 | 50.7 | 55.1 | 17.1 | 49.8 | 65.0 | 62.9 | 84.9 | 56.7 |
| CN-CLIPViT-H/14 | 96.0 | 79.7 | 51.2 | 52.0 | 49.2 | 26.2 | 39.1 | 79.4 | 52.4 | 84.9 | 59.6 |