Skip to content

Multi-view-AE: An extensive collection of multi-modal autoencoders implemented in a modular, scikit-learn style framework.

License

Notifications You must be signed in to change notification settings

alawryaguila/multi-view-AE

Repository files navigation

Multi-modal representation learning using autoencoders

Build Status Documentation Status Python Version DOI version codecov downloads

multi-view-AE is a collection of multi-modal autoencoder models for learning joint representations from multiple modalities of data. The package is structured such that all models have fit, predict_latents and predict_reconstruction methods. All models are built in Pytorch and Pytorch-Lightning.

Many of the models implemented in the multi-view-AE library have been benchmarked against previous implementations, with equal or improved results. See below for more details.

For more information on implemented models and how to use the package, please see the documentation.

Library schematic

Models Implemented

Below is a table with the models contained within this repository and links to the original papers.

Model class Model name Number of views Original work
mcVAE Multi-Channel Variational Autoencoder (mcVAE) >=1 link
AE Multi-view Autoencoder >=1
mAAE Multi-view Adversarial Autoencoder >=1
DVCCA Deep Variational CCA 2 link
mWAE Multi-view Adversarial Autoencoder with a wasserstein loss >=1
mmVAE Variational mixture-of-experts autoencoder (MMVAE) >=1 link
mVAE Multimodal Variational Autoencoder (MVAE) >=1 link
me_mVAE Multimodal Variational Autoencoder (MVAE) with separate ELBO terms for each view >=1 link
JMVAE Joint Multimodal Variational Autoencoder(JMVAE-kl) 2 link
MVTCAE Multi-View Total Correlation Auto-Encoder (MVTCAE) >=1 link
MoPoEVAE Mixture-of-Products-of-Experts VAE >=1 link
mmJSD Multimodal Jensen-Shannon divergence model (mmJSD) >=1 link
weighted_mVAE Generalised Product-of-Experts Variational Autoencoder (gPoE-MVAE) >=1 link
DMVAE Disentangled multi-modal variational autoencoder >=1 link
weighted_DMVAE Disentangled multi-modal variational autoencoder with gPoE joint posterior >=1
mmVAEPlus Mixture-of-experts multimodal VAE Plus (mmVAE+) >=1 link

Installation

To install our package via pip:

pip install multiviewae

Or, clone this repository and move to folder:

git clone https://github.com/alawryaguila/multi-view-AE
cd multi-view-AE

Create the customised python environment:

conda create --name mvae python=3.9

Activate python environment:

conda activate mvae

Install the multi-view-AE package:

pip install ./

Benchmarking results

To illustrate the efficacy of the multi-view-AE implementions, we validated some of the implemented models by reproducing a key result of a previous paper. One of the experiments presented in the paper was reproduced using the \texttt{multi-view-AE} implementations using the same network architectures, modelling choices, and training parameters. The code to reproduce the benchmarking experiments is available in the benchmarking folder. We evaluated performance using the joint log likelihood (↑) and conditional coherence accuracy (↑). Summary of the results of the benchmarking experiments using the BinaryMNIST and PolyMNIST datasets:

Model Experiment Metric Paper Paper results multi-view-AE results
JMVAE BinaryMNIST Joint log likelihood link -86.86 -86.76±0.06
me_mVAE BinaryMNIST Joint log likelihood link -86.26 -86.31±0.08
MoPoEVAE PolyMNIST Conditional Coherence accuracy link 63/75/79/81 68/79/83/84
mmJSD PolyMNIST Conditional Coherence accuracy link 69/57/64/67 75/74/78/80
mmVAE PolyMNIST Conditional Coherence accuracy link 71/71/71/71 71/71/71/71
MVTCAE PolyMNIST Conditional Coherence accuracy link 59/77/83/86 64/81/87/90
mmVAEPlus PolyMNIST Conditional Coherence accuracy link 85.2 86.6±0.07

Citation

If you have used multi-view-AE in your research, please consider citing our JOSS paper:

Lawry Aguila et al., (2023). Multi-view-AE: A Python package for multi-view autoencoder models. Journal of Open Source Software, 8(85), 5093, https://doi.org/10.21105/joss.05093

Bibtex entry:

@article{LawryAguila2023, 
doi = {10.21105/joss.05093}, 
url = {https://doi.org/10.21105/joss.05093}, 
year = {2023}, 
publisher = {The Open Journal}, 
volume = {8}, 
number = {85}, 
pages = {5093}, 
author = {Ana Lawry Aguila and Alejandra Jayme and Nina Montaña-Brown and Vincent Heuveline and Andre Altmann}, 
title = {Multi-view-AE: A Python package for multi-view autoencoder models}, journal = {Journal of Open Source Software} 
}

Contribution guidelines

Contribution guidelines are available at https://multi-view-ae.readthedocs.io/en/latest/