WebP: Expose animation frame data

[sghpjuikit]

There is no way to obtain animated webp frame data, such as duration of the frame. This is necessary to be able to read webp and show animation with correct speed. Currently, WebpImageReader does not expose any information about animation duration. This information is already read and stored in WebpImageReader.header: List<AnimationFrame>, however it is not publicly obtainable.

I have managed to obtain this information with reflection and read all frames with correct duration times and display correct animation in a JavaFX Application (using JavaFX Timeline). The pseudocode regarding the reader is:

            (reader.minIndex..Int.MAX_VALUE).asSequence().map { imageIndex ->
                     val (iW, iH) = reader.getWidth(imageIndex) to reader.getHeight(imageIndex)
                     val frame = getFieldValue<List<Any>>(reader, "frames")[imageIndex]
                     val duration = getFieldValue<Int>(frame, "duration")
                     val irp: ImageReadParameters = ...
                     return ImageFrameDto(duration, reader.read(imageIndex, irp)) }
              }

The methods reader.getWidth and reader.getHeight delegate internally to frames[imageIndex].width and frames[imageIndex].height respectively. Similarly, a method reader.getDuration(i: Int) could be added, which is the requested feature here. This would allow clients to avoid using reflection.

The problem however is also that WebpImageReader is not public class and as such, no non-inherited API could be used by clients. In my opinion, making the class public and adding an API is proper solution, as subclasses providing specific APIs is fine design. However, I feel, the Reader classes in imageio were purposefully hidden, so I would like to know what the proper strategy is here.

Another concern is how the new methods would work if the image is not animated. Either return some synthetic value or throw exception.

I can put in necessary work and create PR for this. I'd like to hear from the authors first though, so to avoid implementing a dead end solution.

MetadataReader consistency

In addition, WebPMetadataReader seems inconsistent with GifMetadataReader, which returns directories containing all gif Control tags, with each frame's animation duration. Webp metadata does not contain any of this information, though it does contain flag whether it is animated. The code to print this data is:

         ImageMetadataReader.readMetadata(f)
            .directories.asSequence()
            .filter { it.name!="File" }
            .flatMap { it.tags }
            .joinToString("\n") { "${it.directoryName} > ${it.tagName}: ${it.description}" }

Checking if image is animated

For completeness' sake, here is a way to check whether gif image is animated (maybe there is better way):

GifMetadataReader.readMetadata(f).getDirectoriesOfType(GifControlDirectory.class).size()>1

WebpMetadataReader.readMetadata(f).getFirstDirectoryOfType(WebpDirectory.class).getBoolean(WebpDirectory.TAG_IS_ANIMATION)

I do not like reading all image metadata for this check - it seems like a heavy operation. Particularly for webp, there is a better way to check this flag, but it requires custom reading of the file's input stream as shown here https://stackoverflow.com/a/73170213.
I feel like there is missing API here - WebpMetadataReader.readFirstDirectoryOfType(file: File, directoryType: Class<?>), which would be as efficient as the SO answer, that only reads what's specified. I'd like to hear author's opinion about this.

Ideally, ImageReader would be able to return animation aspect of the loaded image, however since BufferedImage and the like have no proeprties/userData, there is little that can be done here. This is exactly why reading animation frames as BufferedImages does not provide the duration information - there is nowhere to store it.

[sghpjuikit]

I see that GifImageReader is public. It also implements getImageMetadata(index) so it can provide each animation frame's delay:

imageReader.cast<GIFImageReader>().getImageMetadata(ii).cast<GIFImageMetadata>().delayTime

One change I would like to see is for GIFImageReader.getImageMetadata to specify exact return type (GIFImageMetadata) to avoid the cast. I think WebpImageReader should follow suit.

[sghpjuikit]

It appears even the GIFImageMetadata is not com.drew, but JDK and using it gives compilation error due to the java module system:

error: symbol is declared in module 'java.desktop' which does not export package 'com.sun.imageio.plugins.gif'
import com.sun.imageio.plugins.gif.GIFImageMetadata

That is rather unfortunate.

[haraldk]

Hi,

I'm a bit confused right now, but I think you are mixing up a lot of different libraries here... This is the TwelveMonkeys ImageIO issue tracker. We cannot help with OpenJDK GIF plugin (com.sun...) or MetadataExtractor (com.drew...) issues.

But to your initial request, yes, we don't yet expose the WebP animation control data, but the plan is to do so. Through the standard image metadata interface, using a native WebP metadata format. The plan is to model this format kind of similar to the one found in the OpenJDK GIFImageReader. Client code should simply invoke the ImageReader.getImageMetadata(int) interface method to obtain it. We then document the metadata format. But we will not make any of the existing classes public or add new methods to the reader for this purpose.

Feel free to propose a PR with the needed changes!

--
Harald K

[sghpjuikit]

I had a look at the getImageMetadata and IIOMetadataFormat classes and I remain confused.

Pls confirm:

• So com.drew.... is simply an external dependency for imageio, yes?
• com.drew is not related in any way to the issue of obtaining the animation metadata when reading an animated image with ImageReader, yes?
• ImageReader.getImageMetadata returns a generic supertype of the metadata. How am I supposed to read GIFImageMetadata.delayTime?
• ImageMetadataFormat only exposes values as String, right?
• image width/height is also metadata, so why is it in the ImageReader.getWidth(index) API and treated specially?
• ImageMetadataFormat always incurrs xml tree building (for all data) and conversion overheads and pushes value conversion onto the client, right?
• I want to write code without casting. So my idea was to export the reader, have reader declare its metadata type. So clients can call reader.getMetadata(index).getMyImportantData(). What is wrong with this type-safe approach (particularly for ImageReader classes in this library)?
• What is the point of forcing clients consume ImageMetadata using ImageFormat instead of accessing it directly from the eact metadata subclass?

My ideal:

fun ImageReader.getAnimationDelay(index: Int) = when (val m = getImageMetadata(index)) {
  is GifImageMetadata-> m.delay
  is WebpImageMetadata -> m.duration
  else -> fail { "unsupported } // or return null or whatever
}

What I see from your response:

reader.getImageMetadata(ii).getAsTree().get("xmlpath of the delay attribute").getValue().parseStringToWhatYouNeed()

which is ridiculous, particularly if the readers already have this information available in a type-safe way, without parsing, converting, intermediate xmls and so on. Because of the modules, I can not even do an instanceof on the reader to handle gif an webp differently.

I feel I am completely missing something here. As I said, I could create a PR if there is a clear and useful pathway here, but so far I dont see any. I need a bone thrown my way.

So far im using this solution that relies on Java reflection:

@Throws
private fun ImageReader.getDuration(ii: Int, fMime: MimeType): Int = when (fMime) {
   `image∕gif` -> getFieldValue<Int>(getFieldValue(this, "imageMetadata"), "delayTime")
   `image∕webp` -> getFieldValue<Int>(getFieldValue<List<Any?>>(this, "frames")[ii], "duration")
   else -> 0
}

[haraldk]

As I already said:

• com.drew.* is a completely different project that has nothing to do with TwelveMonkeys ImageIO or the javax.imageio API. We don't have any dependencies in either direction.
• The GIF plugin you use is bundled with the JDK as standard. It is not a part of the TwelveMonkeys ImageIO library.

In addition:

• The javax.imageio API is part of the JDK, and was designed around 20 years ago. Our library doesn't provide much API by itself, but rather implementations of this API.
• It's great, because client code don't need a lot of format knowledge to read/write images, and all formats can be treated the same.
• It's less great because some format specific features become tedious and unnatural.
• At the time it was designed XML was a great new thing, and the designers probably thought it was a good idea to use this for metadata. We don't agree today, but the API still is what it is.
• (and no, image dimensions is not just metadata, it's essential data to be able to effectively decode images)

Let's focus on the things we can fix.

1. We need to expose the WebP animation data as IIOMetadata (XML tree) for the general case.
2. To make WebP-specific client code simpler/better we can additionally provide API that expose the same values directly.