3D Data Augmentation

[tknopp]

This is either a question, or a feature request:

Does Augmentor.jl allow for 3D data augmentation? I tried to pass a 3D array but got an error message.
If this is not possible right now: Would it be complicated to add support for that?

Thanks,

Tobi

[johnnychen94]

The underlying pipeline design allows 3D array input, but not all operations support it by definition (e.g., flipX only makes sense for 2D case). Some operations are supported by upstream packages (e.g., ImageTransformations, Rotations, etc).

It would be clearer if you can provide a minimal example that you want the pipeline to work for the 3D array and then we can figure out which part is missing.

[tknopp]

So first of all: This means that 3D operations would be in the scope of this package?

The minimal example would be to support flipX, which supports flipping a 3D array along the x-axis (and flipY, flipZ) accordingly.

[yeruoforever]

I am a PhD student whose research field is medical image processing. I think @tkopp may refer to some 3d images (such as MRI and CT), rather than a 3d array of multiple 2d images.

Unlike 2d images, medical images have an extra axis,

Many times we need to do data augmentations considering all three axes at the same time.

Augment.jl has the ability to handle higher order arrays, such as an image img with sizes (W, H, d1, d2,..., dn), but only one plane is used when doing enhancements,in other words, almost all the enhancement is only in a slice img[:,:,idx] of the 3D array.

Like augmentation based on affine transformation, 2d images look like this，

and 3d like this

3d medical images have more voxels than 2d images, which makes them expensive to manually label. In order to take full advantage of manual labeling to train models, many deep learning researchers crave a well-designed software package for data augmentation.

Augmentor.jl is currently the only efficient and easy-to-use software package available in the Julia community. Its pipeline API greatly reduces the workloads of researchers without requiring to manipulate ImageTransforms.jl and Rotations.jl separately. At the moment it seems to be used only for augment 2d images, but it shows the potential of 3d images.
Medical image processors are looking forward to using the Augmentor.jl for research, just as TorchIO did with Pytorch.

[tknopp]

Yes, exactly. 3D volumes is what I am looking for. I am currently using synthetic shape phantoms for training 3D data (see https://github.com/JuliaImageRecon/TrainingPhantoms.jl, which includes an ellipsoid and a vessel generator). It would be very cool if Augmentor.jl could be extended to 3D.