Skip to content

A custom cross encoder used to predict the diseases from an input of symptoms

License

Notifications You must be signed in to change notification settings

chungimungi/DocQA

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

44 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

image

CrossEncoder

A custom cross encoder QA model that gives a diagnosis based on the input of symptoms a user describes in natural language.

Data

This Data was sourced from Kaggle

It contains two columns disease and symptoms, the current dataset has 24 diseases listed with 50 symptoms for each in natural language.

In scripts/s2d CrossEncoder comment the line model = nn.DataParallel(model) if multiple GPUs are unavailable for training.

Requirements

Install all dependencies using pip install -r requirements.txt

Model

Layer (type:depth-idx) Param #
DataParallel --
└─CustomCrossEncoder: 1-1 --
└─Embedding: 2-1 2,568,192
└─LSTM: 2-2 6,299,648
└─MultiheadAttention: 2-3 3,148,800
│ └─NonDynamicallyQuantizableLinear: 3-1 1,049,600
└─Dropout: 2-4 --
└─Linear: 2-5 131,200
└─Linear: 2-6 3,096

Total params: 13,200,536
Trainable params: 13,200,536
Non-trainable params: 0

Changes

  • Added multihead attention to the model improved accuracy by 7% using the same hyperparamters (80.41% to 87.01%)

    • Model now has 1.3M parameters (from 522K)
  • Changed embed_dim to 1024 and hidden_dim to 512, while also changing batch_size to 256.

    • Model trained for 350 epochs this time and improved accuracy by another 7% (87.01% to 94.58%)
    • Model now has 13.2M parameters (from 1.3M)

NOTE: This project is for demo purposes only. For any symptoms/disease, please refer to a Doctor.

About

A custom cross encoder used to predict the diseases from an input of symptoms

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages