On the Inductive Bias of Masked Language Modeling

Authors

• Tianyi Zhang
• Tatsunori Hashimoto

Overview

This repo containst the official code release for NAACL 2021 paper On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies.

If you find this repo useful, please cite:

@InProceedings{naacl21zhang,
  title = 	 {On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies},
  author = 	 {Zhang, Tianyi and Hashimoto, Tatsunori},
  booktitle	=   {North American Association for Computational Linguistics (NAACL)},
  year = 	 {2021},
}

We organize the code to replicate different experiments into subfolders. The experiments in section 3.2 and 5.1 are in ./finetuning and the experiment in section 5.2 are in ./unsupervised_parsing.

Dependencies

requirements.txt lists the essential requirements for this project.