augment modes supported by different operations #40

johnnychen94 · 2020-01-29T11:21:26Z

This issue is for reference usage

Operation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
Augmentor.ImageOperation	✅	❌	❌	❌	❌	❌	❌	❌
AggregateThenMapFun	✅	✅	❌	❌	❌	❌	❌	❌
CombineChannels	❌	✅	❌	❌	❌	❌	❌	❌
ConvertEltype	✅	✅	❌	❌	❌	❌	❌	❌
MapFun	✅	✅	❌	❌	❌	❌	❌	❌
PermuteDims	✅	✅	❌	❌	❌	❌	❌	❌
Reshape	❌	✅	❌	❌	❌	❌	❌	❌
SplitChannels	❌	✅	❌	❌	❌	❌	❌	❌

ImageOperation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
Augmentor.AffineOperation	✅	✅	✅	❌	❌	❌	✅	✅
Augmentor.CacheImageInto	✅	✅	❌	❌	❌	❌	❌	❌
CacheImage	✅	❌	❌	❌	❌	❌	❌	❌
Crop	❌	✅	✅	❌	✅	✅	❌	✅
CropNative	❌	✅	✅	❌	✅	✅	❌	✅
CropRatio	❌	✅	✅	❌	✅	✅	❌	✅
CropSize	❌	✅	✅	❌	✅	✅	❌	✅
Either	✅	❌	❌	❌	❌	❌	❌	❌
ElasticDistortion	❌	✅	❌	❌	❌	❌	❌	❌
RCropRatio	❌	✅	✅	❌	✅	✅	❌	✅
Resize	✅	✅	✅	❌	❌	❌	❌	✅
Zoom	❌	✅	✅	❌	❌	❌	❌	✅

AffineOperation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
FlipX	✅	✅	✅	❌	❌	✅	✅	✅
FlipY	✅	✅	✅	❌	❌	✅	✅	✅
NoOp	❌	✅	✅	❌	✅	✅	✅	✅
Rotate	❌	✅	✅	❌	❌	❌	✅	✅
Rotate180	✅	✅	✅	❌	❌	✅	✅	✅
Rotate270	✅	✅	✅	✅	❌	❌	✅	✅
Rotate90	✅	✅	✅	✅	❌	❌	✅	✅
Scale	❌	✅	✅	❌	❌	❌	✅	✅
ShearX	❌	✅	✅	❌	❌	❌	✅	✅
ShearY	❌	✅	✅	❌	❌	❌	✅	✅

script

using Augmentor

check_list = ["lazy",
              "uses_affinemap",
              "eager",
              "permute",
              "view",
              "stepview",
              "affine",
              "affineview",
              ]

op_list = [AggregateThenMapFun,
           CombineChannels,
           ConvertEltype,
           MapFun,
           PermuteDims,
           Reshape,
           SplitChannels,
           # ImageOperation
           Augmentor.ImageOperation,
           Augmentor.CacheImageInto,
           CacheImage,
           Crop,
           CropNative,
           CropRatio,
           CropSize,
           Either,
           ElasticDistortion,
           RCropRatio,
           Resize,
           Zoom,
           # AffineOperation
           Augmentor.AffineOperation,
           FlipX,
           FlipY,
           NoOp,
           Rotate,
           Rotate180,
           Rotate270,
           Rotate90,
           Scale,
           ShearX,
           ShearY,
]

bool2emoji(x) = if x "✅" else "❌" end


open("out.md", "w") do io
    # header
    print(io, "| Name")
    for fn_name in check_list
        print(io, " | ", fn_name)
    end
    println(io, " |")

    print(io, "| ---")
    for _ in check_list
        print(io, " | --- ")
    end
    println(io, " |")

    for op in op_list
        print(io, "| ", op)
        for fn_name in check_list
            if isdefined(Augmentor, Symbol("supports_"*fn_name))
                fn_name = Symbol("supports_"*fn_name)
            else
                fn_name = Symbol(fn_name)
            end
            @show fn_name

            rst = @eval ($fn_name)($op)
            rst = bool2emoji(rst)
            print(io, " | ", rst)
        end
        println(io, " |")
    end
end

barucden · 2021-08-29T19:21:42Z

How about moving this to the Developer's guide section of the docs?

Also, it would be nice if we could describe what each "kind" (view, stepview, etc.) means. I believe it would help new contributors -- and me too 😄.

barucden · 2021-08-30T05:50:05Z

Let me know if I can help. I could, e.g., prepare the page, describe the augment modes that I understand well, and let you complete the remaining modes.

johnnychen94 · 2021-08-30T07:10:46Z

Oh, it would be great if you can help write this up! I think this can be "advanced tutorial", but it also makes sense to have a developer page to document what APIs are needed when someone wants to implement his own operation.

I'm thinking of something like the following but TBH I haven't yet verified if this is still the case.

# Fused pipeline

The secret of Augmentor's performance is the fused pipeline compilation before execution.

<insert some automatically generated table using the script>

## Fused lazy operations

Because many operations are applied in a pointwise sense, Augmentor will detect if we can save intermediate memory allocations by applying it lazily.

<compare `applylazy` and `applyeager` and explain how this saves memory allocation and improve performance>

## Fused Affine operations

Affine operations are internally implemented using `warp` or its lazy version `warpedview`/`invwarpedview`, which are based on 3x3 affine matrix. Thus if there are two or multiple affine operations in a row, we can calculate the result of affine matrix eagerly and fuse them into one. For instance, to apply `pl = Rotate(90) |> Rotate(-90)` to an image, we need to calculate __two__ matrix-vector multiplication for each pixel position, but if we can eagerly fuse them into `pl = Rotate(0)` then we only need to calculate __one__ matrix-vector multiplication, and thus reduces a lot of computations.

<compare `applyeager`, `applylazy` and `applyaffine` and explain how fused affine operations reduces computation>

As far as I understand Augmentor, we currently only have these two optimization applied:

Augmentor.jl/src/codegen.jl

Lines 53 to 63 in 8d80713

 if num_special >= num_affine 

 quote 

 $var_out = unroll_applylazy($(Expr(:tuple, (:(pipeline[$i]) for i in op_offset:op_offset+num_lazy-1)...)), $var_in) 

 $(augment_impl(var_offset+1, op_offset+num_lazy, after_lazy, avoid_eager)) 

 end 

 else 

 quote 

 $var_out = unroll_applyaffine($(Expr(:tuple, (:(pipeline[$i]) for i in op_offset:op_offset+num_affine-1)...)), $var_in) 

 $(augment_impl(var_offset+1, op_offset+num_affine, after_affine, avoid_eager)) 

 end 

 end

johnnychen94 · 2021-08-30T07:43:18Z

but it also makes sense to have a developer page to document what APIs are needed when someone wants to implement his own operation.

To implement a new operator, the minimal requirement is to support eager operation:

decide if this is a generic Operation, ImageOperation or AffineOperation
(Optional) If the operator requires some runtime generated random parameters to work, then define randparam method
If the operator is defined element-wise sense, then it's better to define the lazy mode:
- support_lazy(Type{<:MyOp}) = true
- implement applylazy method
If the operator can't be defined lazily, then must support applyeager method with support_eager(::Type{<:MyOp}) =true

For affine operations, one must implement:

supports_affineview(::Type{<:MyOp}) = true
toaffinemap(op::MyOp, img) defines how to generate the affine matrix on input image img
implement applyaffine

About applyaffine and applyaffineview, I haven't check this very carefully, but the idea I guess is to differentiate operations that satisfy so-called idempotent indexing a[ax][i] == a[ax[i]] and that doesn't. Check the applyaffine and applyaffineview from Crop and CropNative you can notice the difference. In most cases, I believe we can think of them as the same, and maybe deprecate one in favor of the other.

johnnychen94 mentioned this issue Jan 29, 2020

add some comments in codegen #41

Closed

johnnychen94 mentioned this issue Aug 6, 2021

Support segmentation masks #87

Merged

4 tasks

johnnychen94 self-assigned this Aug 30, 2021

barucden linked a pull request Aug 31, 2021 that will close this issue

Docs: Explain fused pipelines & how to add new operations #103

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

augment modes supported by different operations #40

augment modes supported by different operations #40

johnnychen94 commented Jan 29, 2020 •

edited

Loading

barucden commented Aug 29, 2021

barucden commented Aug 30, 2021

johnnychen94 commented Aug 30, 2021 •

edited

Loading

johnnychen94 commented Aug 30, 2021

augment modes supported by different operations #40

augment modes supported by different operations #40

Comments

johnnychen94 commented Jan 29, 2020 • edited Loading

Operation

ImageOperation

AffineOperation

barucden commented Aug 29, 2021

barucden commented Aug 30, 2021

johnnychen94 commented Aug 30, 2021 • edited Loading

johnnychen94 commented Aug 30, 2021

johnnychen94 commented Jan 29, 2020 •

edited

Loading

johnnychen94 commented Aug 30, 2021 •

edited

Loading