Insights from Images

This proof of concept focuses on helping the japanese casual wear designer, manufacturer and retailer Uniqlo understand consumer behavior. We would be going about this by being able to detect uniqlo products and people in images using object detection. After which we would be going on to captioning these images hoping to capture how the user is using the product or anything else that could serve as an actionable insight.

Final results can be found in the following links :

• First iteration results of object detection
• Higher precision object detection
• Image Captioning first iteration results
• Presentation on poc of insights from images

Data collected

**Last updated 06/05/2021

The data collected from the instagram account @uniqlousa can be found here . Some of the metadata in terms of json files are missing for which the scraping needs to be repeated. This will be updated again.

To uncompress the json files and view the metadata in a readable format you could use jq (for ubuntu) and the following command:

xzcat 2021-05-31_19-09-24_UTC.json.xz | jq .node

Public Datasets

• Zalando's dataset Fashion MNIST which covers products from their catalogue.
• A customized dataset for zero shot object detection based on Fashion-MNIST. (paper)
• DeepFashion is a large scale clothes database. Take a look at the benchmarks as well for possible ideas.

References

Instance Segmentation

1. Zero-Shot Instance Segmentation (CVPR 2021) [paper] [code]

Object Detection

1. GTNet: Generative Transfer Network for Zero-Shot Object Detection (AAAI 2020) [paper]
2. Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification (ECCV 2020) [paper] [code]
3. Synthesizing the Unseen for Zero-shot Object Detection (2020) [paper] [code].
4. Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts (2018) [paper] [code]

Image Captioning

1. Fast Parameter Adaptation for Few-shot Image Captioning and Visual Question Answering (ACM 2018) [paper]

For a more extensive list of resources and future references the following 'awesome'-github repositories may be useful.

• Zero-shot object detection
• Papers and implementations of image augmentation, image duplication and object detection
• Papers on zero shot object detection : older collection of zsd, no code.
• A collection of papers/code on few shot learning
• Papers and implementations of image captioning
• Datasets and popular implementations of image captioning

Useful tools

• Scraper used to download pictures and other metadata from instagram: Instaloader
• Data annotation tool for manually labelling products: Diffgram
• Object detection and instance segmentation components and modules: mmdetection
• An open source visual analysis toolbox which does fashion attribute prediction, in-shop clothes retrieval, fashion parsing and segmentation, fashion landmark (upper body , lower body clothes) detection, fashion compatibility and recommendation and virtual try ons. (This is an incredibly useful resource!) : mmfashion

Use cases

• How facebook uses computer vision to aid shopping?

Feedback and contribution