Hysia Video to Retail Platform

_{* This project is supported by Cloud Application and Platform Lab led by Prof. Yonggang Wen}

An intelligent multimodal-learning based system for video, product and ads analysis. You can build various downstream applications with the system, such as product recommendation, video retrieval. Several examples are provided.

The system is under active development currently. You are welcome to create a issue, pull request here. We will credit them into our next version.

Showcase • Features • Setup Environment • Configuration • Demo • Citation

News

• (2020-08) The work has been accepted as an open-source competation paper at ACMMM2020!
• (2020-05) The docker image has been updated
• (2020-05) You can easily bind your model to our system

Showcase

👉 Full list of showcase.

1. Upload video and process it by selecting different models

   

2. Display video processing result

   

3. Search scene by image and text

   

4. Insert product advertisement and display insertion result

   

Features

• Multimodal learning-based video analysis:
  • Scene / Object / Face detection and recognition
  • Multimodality data pre-processing
  • Results align and store
• Downstream applications:
  • Intelligent ads insertion
  • Content-product match
• Visualized testbed
  • Visualize multimodality results
  • Can be installed separately

Setup Environment

1. Download Data

👉 For ❌ no Google Drive access.

# Make sure this script is run from project root
bash scripts/download-data.sh

2. Installation

👉 Install with Docker 🐳

docker pull hysia/hysia:v2o

Configuration

Change decoder and model server running devices at device_placement.yml:

decoder: CPU
visual_model_server: CUDA:1
audio_model_server: CUDA:2
feature_model_server: CUDA:3
product_search_server: CUDA:2
scene_search_server: CUDA:3

Device value format: cpu, cuda or cuda:<int>.

Demo

Run with docker 🐳

docker run --rm \
  --gpus all -d -p 8000:8000 \
  -v ${PWD}/server/config/device_placement.yml:/content/server/config/device_placement.yml \
  hysia/hysia:v2o

Then you can go to http://localhost:8000. Use username: admin and password: admin to login.

Some Useful Tools

• Large dataset preprocessing
• Video/audio decoding
• Model profiling
• Multimodality data testbed

Contributing

You are welcome to contribute to Hysia! Please refer to here to get start.

Paper Citation

If you use Hysia in your work, we would be very grateful if you cite

@inproceedings{10.1145/3394171.3414536,
    author = {Zhang, Huaizheng and Li, Yuanming and Ai, Qiming and Luo, Yong and Wen, Yonggang and Jin, Yichao and Ta, Nguyen Binh Duong},
    title = {Hysia: Serving DNN-Based Video-to-Retail Applications in Cloud},
    year = {2020},
    booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
    pages = {4457–4460},
}

About Us

Maintainers

• Huaizheng Zhang
• Yuanming Li
• Qiming Ai

Previous Contributors

• Shengsheng Zhou
• Wenbo Jiang (Now, Shopee)
• Ziyuan Liu (Now, Tencent)
• Yongjie Wang (Now, NTU PhD)