Support for Ultra HDR #3799
Comments
ProposalIt looks like support for the extra gain map image could be added pretty easily to the libvips JPEG reader. We could just add an image-valued metadata item called Generating a corrected float HDR image could then be done in python or whatever using the standard libvips tools. |
|
What would be important I think (and here I don't know your API good enough) is that the gain map should be scaled and flipped similarly to the primary image. The gain map doesn't have to be the same size though, it maybe ¼th or less in size. Similarly other operations that just flip, copy, rotate, crop or scale should probably be implemented to support this as well, while other operations (like I think there was one for clipping or so) might either want to remove the metadata or have a switch to operate on the HDR representation of the image or something more complex. Ideally support for gain maps would work by default (as it's just compatible with regular JPEGs) for reading, transforming and writing so all users of the library can just immediately benefit from this without needing to implement any special code for Ultra HDR. |
|
For libvips it'd be best to transform to a HDR float image on load, maybe with a new operation called |
That would loose the SDR image though, which would be bad as you'd want to preserve that SDR image that came directly from the camera unless explicitly specified otherwise. That could maybe be an optional operation if explicitly required. The same thing also stands on the opposite side, you need an HDR and an SDR version on an image to create an Ultra HDR JPEG. Their library also supports creating one from only an HDR image experimentally, but that automatically tonemapped SDR image that is being generated will probably look subpar compared to what the camera initially generated (as the camera just still generated a normal SDR image and the gain map is just an enhancement on top). Ideally just can support SDR + gain map for most stuff and if something more complex is needed I'd advise the following (if possible):
I think the first 3 would be the most important for getting it working, the latter ones are more like nice to have (I don't have any use for them in particular, but maybe you interested in being able to create and do more advanced work with Ultra HDR images). That would also keep compatibility, if you would just convert it to HDR on import, it'd kill any application converting JPEGs to JPEGs instantly as they'd suddenly get really bad images as a result of the HDR to SDR conversation on import. With the proposal above you'd have more work in the transforms, implementing gain map support there (although you should probably just be able to run in on both and therefore the changes shouldn't be too bad), but in return if an Ultra HDR image is imported and later exported, the file would appear as a normal SDR image and as an HDR file in applications that do support Ultra HDR. Worst case if you import an Ultra HDR image and export it as anything other than a JPEG you'll just get a regular SDR image in return, like you also would now with any JPEG. I think this decreases the chance it breaks any of your libraries consumers, better preserves the quality, it would just "magically" work, while still providing advanced controls if wanted by library consumers (assuming you'd provide said APIs). The gain map calculations themselves aren't that hard I believe, I think you just upscale the gain map (if not the same resolution) multiplay the pixel values of the primary image with the gain map values (the gain map with contains one value per pixel (like an alpha channel) or one per channel (like a full RGB image)) and then you have you HDR image. Reverse you'd just take your SDR and HDR image, basically divide every HDR channel through every SDR channel and then you have a gain map. The biggest PITA might only be the parsing of the initial image, in case libjpeg doesn't already support that multi image standard that Ultra HDR uses. I really wish I could be more helpful with an actual implementation of that, but my C/C++ skills are really rusted and never went beyond little more than hello world, im more of a higher level developer usually... |
|
We could leave the gain map image as metadata on the scRGB. It's easy to recover the SDR from that, if necessary. However, many operations on the scRGB image (eg. crop) would make the gain map out of date, so I wonder how useful it would be in practice. It's more likely to be a cause of errors and tricky bugs. Probably the only sane solution is to always regenerate the gainmap on save to ultrahdr. There could be an extra option to the loader to just get the gain map or just get the SDR for applications which needed fine control over processing. |
|
I think you misunderstand, you cannot "regenerate" the gain map (from what would you regenerate it). You'll need to do transformations like crop twice, once on the SDR image and once on the gain map or HDR image. |
|
You can use the gain map as the exponent. It won't match the original SDR + gain map, of course, but it would be a correct representation of the HDR. You could do scRGB -> ultra hdr -> scRGB and get (almost) the same image. libvips treats images as simple arrays, it's just like numpy. There's no way within the API to do an operation on a pair of images, so you must convert the SDR + gain map to scRGB before processing. An application could use libvips to implement ultra HDR processing that maintained a pair of images, but there's no way to do that automatically. |
Yeah well that's the issue. In that case just treating it as an SDR image would probably be better...
That's unfortunate. In that case it probably makes more sense just basically leave it as is and treat them as SDR JPEG and only provide the APIs to
And also import and export the gain map of course. And I'll take this to bimg then and have a look, whether the dual processing for the Ultra HDR image can be handled there. Would have been cool if this could have just worked, but if it doesn't I think producing good looking SDR images makes more sense than trying and preserving the HDR enhancement. Otherwise people will definitely complain, why JPEGs they feed through libvips suddenly look worse, even though the inputs look fine. |
libvips already has quite a bit of HDR support (it supports radiance, scRGB TIFF, HDR JXL and HDR EXR), so loading an ultra HDR as scRGB would allow easy conversion from ultra HDR to any of those formats. Writing an ultra HDR that's also a nice looking SDR is harder and would need a tone mapping stage. That's probably best left for some google library to implement. How about:
|
Yeah something like that, although I'd create the API surface somewhat differently (pseudo/kotlin code, cause I can't write proper C code...): val image = loadImage("path/to/ultra/hdr.jpg") // SDR image with the gain map in metadata
val cropped = crop(image, /* params */)
check(isUltraHdrImage(image)) { "This example assumes that the image is an Ultra HDR image" }
val hdrImage = ultraHdrToHdr(image)
val hdrCropped = crop(image, /* params, possibly transformed because of the different resolution */)
val output = combineToUltraHdrImage(cropped, hdrCropped) // output's an SDR image with the gain map in metadata again
saveImage("path/to/ultra/hdr.cropped.jpg", output)In that case you don't have to decode the image twice for both images but can just extract it from the original image matadata. And as those are separate functions you might be able to shove it into an external module if you choose to use google's library for example. My original Idea would have been to have additionally (again just loose pseudo code): fun crop(image: Image, params: CropParams, transformGainMap: Boolean = true): Image {
val result = /* do crop */
if (isUltraHdr(image)) {
val gainMap = getUltraHdrGainMap(image)
val gainMapParams = /* transform crop params if needed */
val transformedGainMap = crop(gainMap, gainMapParams)
addGainMapToImage(result, transformedGainMap)
}
return result
}But I guess the crop function doesn't get the image metadata and so the
As for that Google explicitly recommends to not do that if possible, most likely for said quality reasons, it'd probably alter the SDR image too much.
Because that's the thing most people wouldn't know that and Ultra HDR is an Ultra HDR, it looks just like any other issue, so when anything changes, it's quite confusing for them and maybe the photo that look great to them before looks pretty be now. Of course ideally every one just had an HDR display and all Application would display Ultra HDRs as HDR, but sadly that won't be the most for the next few year probably until software catches up. That's probably while Google designed it this way in the first place. As a side note from all of that may be quickly: The google lib, if you choose to use it, also supports conversion from and to linear HDR, which is what you need for scRGB i think, although it needs a colorspace, like BT202 or P3, whcih would be smaller than scRGB, although you probably already have that figured out for other formats I guess. |
|
Maybe to emphasize the point that tonemapping is probably a bad idea (at least without the explicit instruction to do so and/or as the only way possible/the default way of handling), this is the original SDR version of the image (do not view on anything that can display Ultra HDRs, otherwise And the automatically tonemapped: As you can see the second image is basically destroyed now, even though in HDR they (should normally) both look the same. I generated them using libultrahdr's CLI (which I did for all the samples here, I could go on a rant on how I needed to convert JXR to AVIF and then to YUV, but hey it works using some random tools off github...). Both the SDR and HDR versions where generated by Xbox Game Bar btw. (it saves a tonemapped PNG and an HDR JXR file, which it does pretty well) I only converted and merged them, like intended with Ultra HDR (although the Pixel Camera does this internally and not with 3 separate CLI tools off github, but Game Bar only does JXR and nothing else for HDR...). This because the Camera or Windows in this case is better at tomemapping, using likely more data than just the HDR image (like Display brightness or in the case of the Pixel Camera for information from the RAW image and Machine Learning probably). Like Camera vendor/programmers/etc. have decades in experience in how to properly tonemap their images and get good results, but it's not a good Idea to try and replicate that using one simple tonemapping algorithm. And ultra HDR itself isn't perfect as well, it improves highlights a bit and can improve areas where it's overblown, but google's current algorithm doesn't seem to use the per-channel approach even, and uses doesn't recover highlights that were present in the HDR, but not the SDR (see details at the bottom). If you want samples from the Pixel camera I can probably get some of those as well, but the result is probably pretty similar. I just have a lot of HDR Cyberpunk screenshot's lying around, so that's my easiest samples to use. But you can create samples with everything that provides you with and HDR and an SDR image. Experiment on HDR details in overblown areas, only works with Chrome on an HDR displaySDR: As you can see the bright image on the right of the ring, which is just white in the SDR, more clearly visible in the pure HDR, is only slightly visible in the Ultra HDR, not as good as in the real HDR. As implemented right now, it's basically better than just SDR, but the HDR isn't quite working completely yet (but I suspect that should work fine if implemented per channel or maybe there's still an issue with chrome and android, the metadata seems rather complex). |
That won't work well with the libvips design -- Crop would perhaps be: sdr = pyvips.Image.new_from_file(filename, access="sequential")
gain = pyvips.Image.new_from_file(filename, page=1) if image.get_typeof("is-ultrahdr") != 0 else None
sdr = sdr.crop(left, top, width, height)
gain = gain.crop(left_adjusted, top_adjusted, ...) if gain != None else None
sdr.write_to_file(new_filename, gain=gain)
That's just a few lines of code, I think. We'd need to integrate it with colour management as well, so this is probably best done in libvips. |
Sounds reasonable. It would be important to not do the parsing twice, if that works with this solution it's nice and good I think. With the design of Ultra HDR itself you should be able to quickly get the normal JPEG header, get the Adobe/GContainer metadata and get the gain map metadata. There should be everything in there aside from the actual gain map itself. One thing that might still be good is on saving additionaly, if you save an Ultra HDR image and find out that it you don't save it with a gain map (e.g. you save a JPEG without gain map, but find Ultra HDR metadata) then please remove the Ultra HDR metadata, right now transformed pictures are detected in Google Photos as if they were Ultra HDRs, even though they lack the gain map, as the EXIF metadata still says it's an Ultra HDR.
Yeah what I meant is you can just let's you load/save an Ultra HDR directly into/from linear HDR, so you don't need to transform it from/to PQ or HLG. |
Agreed 100%. If the image processing affects the SDR or HDR rendition from the gain map (other than simple things like resizing / cropping) it has effectively ruined the image. The purpose of the gain map is so that the image is rendered predictably as SDR, full HDR, or anything in-between. Making any artistic change to the image goes directly against the creative intent and will render an inferior result. A JPG gain map is basically an SDR image (nothing unique there), HDR / gain map metadata (just need to preserve as is) and the gain map stored as a auxiliary "image" (encoded as MPF with optional GContainer header). The key here is to extract the gain map, process it consistent with the base image (similar cropping / scaling but with flexibility to accommodate a gain map encoded at 1/4 or 1/2 resolution in addition to a simple 1/1), and then encode the new results as a JPG with two images and the required metadata. The Adobe specs (functionally same as Google's Ultra HDR JPG spec other than the optional GContainer header), sample gain maps, and a helpful tool to view gain maps are available at https://helpx.adobe.com/camera-raw/using/gain-map.html. More info on creating gain maps with Adobe software: https://gregbenzphotography.com/hdr-images/jpg-hdr-gain-maps-in-adobe-camera-raw/ |
|
Google has shared a reference implementation and other Ultra HDR JPG resources at https://github.com/google/libultrahdr |
|
Hi @gregbenz, thanks for the feedback. Yes, I think that's what's being proposed here. libvips can't automatically process sdr + gainmap images in the general case, but it can make the necessary API available to packages like bimg and sharp. The idea of the "load as HDR" flag is to make it easy to convert uhdr to other HDR formats, like RAD, JXL, EXR, etc. |
|
@jcupitt Sounds good. For conversion to other HDR formats, do you see that actually getting used? The loss of SDR is a huge quality issue in terms of adapting to a range of displays. The quality would be much better working from the source rather than the exported gain map. I suppose you could make the case for analytics such as to generate histogram data or perhaps train AI (though we're flirting with the edge of quality pushing HDR into JPG and this would likely work better with an original source or AVIF in the future). I don't have any concerns with such a feature, I just struggle to see who would use it. |
This would be opt-in then according to the current proposal, so you can either have the SDR/Ultra HDR or you can explicitly load it as HDR to convert to AVIF for example. By default if you would load it and save as AVIF you'd just get a normal SDR image. Ultra HDR to JPEG you'd get just an SDR due to technical constraints, but it would at least be able to build an Ultra HDR to Ultra HDR pipeline in higher level libraries. |
|
Exactly @cromefire, the current proposal is: With higher level libraries (sharp, bimg, etc. etc.) able to implement something fancier if they wish. |
On that topic, it would probably be good as well to have a simple example, that would make it easier for them to adopt it and way easier for me to explain it to them. |
Discussed in #3798
Originally posted by cromefire December 24, 2023
Ultra HDR is Google's new HDR image format based on JPEG and GainMaps (for compatibility) and newer Pixels actually produce Ultra HDR photos by default now (it's pretty cool!)
But the problem is right now libvips is running into that mentioned compatibility mode, where it's taking the primary image and discarding the GainMap, so any image ever processed by libvips looses its HDR and becomes SDR only (it does copy the metadata already so it's identified as Ultra HDR, but it doesn't work).
Here's some random Ultra HDR image I use for testing (it's a screenshot originally at up to 1000nits, so the HDR isn't limited by the camera):
Here it is copied (
vips copy):Here it is flipped (
vips flip):I hope GitHub doesn't modify them(nope looks good on my phone), because in that case the first one should be HDR on an HDR display using Chrome (Desktop and Android) and then you can just download and inspect them, otherwise I'll just upload them as a zip. And that's the exact problem I'm basically facing. Because as soon as Nextcloud for example (which uses Imaginary, which uses bimg, which uses libvips) touches them, it immediately looses the HDR layer.That would be the first step, which should probably not require any API changes, the second one (which I don't have any use for right now as I don't build directly on this library but could be pretty useful in the future) would probably be to support Ultra HDR conversion to other formats directly using libultrahdr using that you could conceivably convert an Ultra HDR image to an HDR AVIF, JXL or HEIC (or JXR for Windows because they don't seem to support anything else...) image.
The text was updated successfully, but these errors were encountered: