Skip to content

PyTorch implementation for ACL 2021 paper "Maria: A Visual Experience Powered Conversational Agent".

License

Notifications You must be signed in to change notification settings

jokieleung/Maria

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Maria: A Visual Experience Powered Conversational Agent

This repository is the Pytorch implementation of our paper "Maria: A Visual Experience Powered Conversational Agent" in ACL 2021.

In this paper, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e., text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator.

Coming soon!

Summary

Dependencies

  • python 3.7

  • pytorch 1.4.0

  • Ubuntu 18.04

Data

Reddit Conversation Corpus

Please download the Reddit data from google drive here.

Download the Open Images

We will use the Open Images images as candidate images for retrievel. Refer to here to download the images first. You can build the image index with the appropriate size (500,000 in our experiments) as needed.

If you already have Open Images dataset on disk. Save them as

data
|-- open_images
    |-- images
         |-- 14928b4f367c217e.jpg
         |-- 289d643a8761aa83.jpg
         |-- ......

Usage

Text-to-Image Retrieval Model

Please refer to retrieval_model/README.md

Bottom-up Detector Model

Please refer to detector_model/README.md

Dialog Generation Model

Please refer to dialog_model/README.md

Citation

If you find this paper helps your research, please kindly consider citing our paper in your publications.

@inproceedings{liang2021maria,
   title={Maria: A Visual Experience Powered Conversational Agent},
   author={Liang, Zujie and 
           Hu, Huang and 
           Xu, Can and 
           Tao, Chongyang and 
           Geng, Xiubo and 
           Chen, Yining and 
           Liang, Fan and 
           Jiang, Daxin},
   booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)},
   year={2021}
}

Acknowledgment

Special thanks to the authors of OSCAR, vokenization, and py-bottom-up-attention.

About

PyTorch implementation for ACL 2021 paper "Maria: A Visual Experience Powered Conversational Agent".

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published