PointNu-Net

PointNu-Net: Keypoint-assisted Convolutional Neural Network for Simultaneous Multi-tissue Histology Nuclei Segmentation and Classification. Ppaer ArXiv

Kai Yao, Kaizhu Huang, Jie Sun, Amir Hussain, Curran Jude
Both University of Liverpool and Xi'an Jiaotong-liverpool University

Abstract

Automatic nuclei segmentation and classification play a vital role in digital pathology. However, previous works are mostly built on data with limited diversity and small sizes, making the results questionable or misleading in actual downstream tasks. In this paper, we aim to build a reliable and robust method capable of dealing with data from the ‘the clinical wild’. Specifically, we study and design a new method to simultaneously detect, segment, and classify nuclei from Haematoxylin and Eosin (H&E) stained histopathology data, and evaluate our approach using the recent largest dataset: PanNuke. We address the detection and classification of each nuclei as a novel semantic keypoint estimation problem to determine the center point of each nuclei. Next, the corresponding class-agnostic masks for nuclei center points are obtained using dynamic instance segmentation. Meanwhile, we proposed a novel Joint Pyramid Fusion Module (JPFM) to model the cross-scale dependencies, thus enhancing the local feature for better nuclei detection and classification. By decoupling two simultaneous challenging tasks and taking advantage of JPFM, our method can benefit from class-aware detection and class-agnostic segmentation, thus leading to a significant performance boost. We demonstrate the superior performance of our proposed approach for nuclei segmentation and classification across 19 different tissue types, delivering new benchmark results.

News:

[2023/12/25] We add additional experimental results on CPM17 (PQ 0.706, AJI 0.712).
[2023/5/15] Our paper have been accepted by TETCI.
[2023/5/1] We release the training and inference code, and the training instruction.

1. Installation

Clone this repo.

git clone https://github.com/Kaiseem/PointNu-Net.git
cd PointNu-Net/

This code requires PyTorch 1.10+ and python 3+. Please install dependencies by

pip install -r requirements.txt

2. Data preparation

For small dataset Kumar and CoNSeP, we conduct datasets preparation following Hover-Net.

We provide the processed Kumar and CoNSeP datasets. Also, we provide the processed CPM17 dataset.

The PanNuKe datasets can be found here

Download and unzip all the files where the folder structure should look this:

PointNu-Net
├── ...
├── datasets
│   ├── kumar
│   │   ├── train
│   │   ├── test
│   ├── CoNSeP
│   │   ├── train
│   │   ├── test
│   ├── PanNuKe
│   │   ├── images
│   │   │   ├── fold1
│   │   │   │   ├── images.npy
│   │   │   │   ├── types.npy
│   │   │   ├── fold2
│   │   │   │   ├── images.npy
│   │   │   │   ├── types.npy
│   │   │   ├── fold3
│   │   │   │   ├── images.npy
│   │   │   │   ├── types.npy
│   │   ├── masks
│   │   │   ├── fold1
│   │   │   │   ├── masks.npy
│   │   │   ├── fold2
│   │   │   │   ├── masks.npy
│   │   │   ├── fold3
│   │   │   │   ├── masks.npy
├── ...

3. Training and Inference

To reproduce the performance, you need one 3090 GPU at least.

Download ImageNet Pretrain Checkpoints from official HRNet.

1) Kumar Dataset

run the command to train the model

python train.py --name=kumar_exp --seed=888 --config=configs/kumar_notype_large.yaml

run the command to inference

python inference.py --name=kumar_exp

2) CoNSeP Dataset

run the command to train the model

python train.py --name=consep_exp --seed=888 --config=configs/consep_type_large.yaml

run the command to inference

python inference.py --name=consep_exp

2) PanNuKe Dataset

run the command to train the model

python train_pannuke.py --name=pannuke_exp --seed=888 --train_fold={} --val_fold={} --test_fold={}

[train_fold, val_fold, test_fold] should be selected from {[1, 2, 3], [2, 1, 3], [3, 2, 1]}

run the command to inference the model

python infer_pannuke.py --name=pannuke_exp --train_fold={} --test_fold={}

run the command to evaluate the performance

python eval_pannuke.py --name=pannuke_exp --train_fold={} --test_fold={}

Citation

If our work or code helps you, please consider to cite our paper. Thank you!

@article{yao2023pointnu,
  title={PointNu-Net: Keypoint-Assisted Convolutional Neural Network for Simultaneous Multi-Tissue Histology Nuclei Segmentation and Classification},
  author={Yao, Kai and Huang, Kaizhu and Sun, Jie and Hussain, Amir},
  journal={IEEE Transactions on Emerging Topics in Computational Intelligence},
  year={2023},
}