Diarization

[laarisyko]

I think Diarization is one of the more important features to realy make it user friendly. It would be great if it could even handle more than two speakers. I'm thinking about team meetings. Majdoddin has done a great job implementing it for two speakers https://github.com/Majdoddin/nlp

[petersterne]

Wow, that's a very cool approach! Do you know if we can run Pyannote locally (since we're running Whisper locally) or would we have to run it in the cloud?

[Majdoddin]

We can run it on CPU efficiently, if we sparsify Whisper and Pyannote models with DeepSparse from Neuralmagic. They have done it for BERT, so here it should work too.

[Majdoddin]

Meanwhile we can use die Openvino model of @zhuzilin, at least for tiny-medium models. Because it is for arm architecture, I suspect it could be adapted to run on Android smartphones.

[laarisyko]

oh, this sounds great, if we could run BERT too. I have seen some scripts which used BERT to genreate summaries of a whisper transcript, which would be the ultimate added value for meeting transcriptions :)

[oelna]

I don't know whether this belongs here, but I just saw openai-whisper-speaker-identification by @zachlatta and I think identifying speakers in the transcripts would be an important feature.

[petersterne]

Thanks @oelna! I agree that speaker diarization is a crucial feature. We were looking into using PyAnnote, but I'm interested to try out @zachlatta's implementation of speaker diarization too.