This project is inspired from Mittens which extends the GloVe model to synthesize general-purpose representations with specialised datasets. The resulting representations of words are arguably more context aware than the pre-trained embeddings.
However, GloVe objective requires a co-occurence matrix of size V2 to be held in-memory, where V is the size of the domain adapted vocabulary. Needless to say, this method becomes difficult to scale with growing vocabulary size.
Replacing the GloVe model with skip-gram reduces the size of the matrix to V×E where E is the embedding dimension and depends on the pre-trained word-embeddings being utilised.