Memory for resize

[pdbourke]

This

vips resize scroll123_8.v scroll123_8_small.v 0.01
was at 196GB RAM at 30% complete, creeping up.

vipsheader /mnt/d/scroll123_8.v
/mnt/d/scroll123_8.v: 3828940x440000 uchar, 3 bands, srgb

This is a large as our combined image will get.

I've written my own low level disk based (almost no RAM) versions of what I need: cropping, joining, resizing, splitting along with the custom tools for special warping we need.

The part I worry about is the iiif (dzsave) generation since that is not something I will be able to (reasonably) replicate.

[jcupitt]

I've used dzsave on 500,000 x 500,000 pixels images, so I'd think it would be OK, but of course an actual test would be best.

[pdbourke]

I've asked the person who has been managing this aspect to tell me what settings he has been using for the deep zoom web solution. I realise that what we are attempting is "extreme", we believe it is the highest resolution digital scan of an artwork ever undertaken, almost 3 times the resolution of the scan of the Night Watch, often claimed to date to be the largest. But it is a real, not a theoretical project. Some major exhibitions coming up based upon this scan. I would like to be able to give vips a gold star rating, because it has served us well so far. ps: What I find most delightful is that I've just concatenated the 3 scrolls into a 5TB image, a few months ago and I would have had a heart attack, now it's feeling run-of-the-mill.

…

On Thu, 18 Apr 2024 at 19:27, John Cupitt ***@***.***> wrote: I've used dzsave on 500,000 x 500,000 pixels images, so I'd think it would be OK, but of course an actual test would be best. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325

[jcupitt]

Yes, it's like buying an even bigger TV. 55" looks ridiculous when you first see it in your house, but ordinary just a few days later. Then you see your old TV again and wonder how you managed.

I found the TIFF crop memory black hole, I'll make a PR.

[jcupitt]

Yes, I did a little work on nightwatch too, that was a fun project. I was at the National Gallery in London for years (the birthplace of libvips) doing multispectral imaging of paintings.

[pdbourke]

Ahhh, wonderful! They took a very different approach. The Murten is 10m x 100m, here's the scanning rig https://vimeo.com/903872010 The stitch of each scroll (there are 3) was using PtGui ... camera is the PhaseOne iXH (150MPixels). We would have loved to do multispectral, but it was deemed time prohibitive. The scan as it stands took 2 months. We do have 16 bits, well, 12-14 bits from the iXH.

…

On Thu, 18 Apr 2024 at 19:55, John Cupitt ***@***.***> wrote: Yes, I did a little work on nightwatch too, that was a fun project. I was at the National Gallery in London for years (the birthplace of libvips) doing multispectral imaging of paintings. — Reply to this email directly, view it on GitHub <#3939 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLREF6QNOKWV2AR2LNKWVDY56KBVAVCNFSM6AAAAABGMNOBV6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNJTGA3TC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325

[jcupitt]

Ah very cool! Yes, we used something similar at the NG (camera + lights on a gantry, with a mini-macbeth plus white and black).

Did you have any problems with optical scatter in the camera lens? That was always a pain for us to fix.

[kmartinez]

You can see us with the Vasari scanner in this ancient promo video: https://youtu.be/agCDd9oOPgE?si=fE8oHcFmB6qPFbwB&t=113

[pdbourke]

Ah very cool! Yes, we used something similar at the NG (camera + lights on a gantry, with a mini-macbeth plus white and black). Did you have any problems with optical scatter in the camera lens? That was always a pain for us to fix.

Perhaps not. I guess I got into the groove of writing the smallish set of tools we need, irrespective of whether vips ones were OK or not. I can always swap mine in or the vips ones in once I find which performs the best. Mine are pretty "dumb", but then pretty much limited by disk IO, not CPU limited. Can I ask where you are based? I'm in Melbourne. The Murten project is based at EPFL in Switzerland.

…

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325

[jcupitt]

Ah Australia! Nice. I'm in London still.

Yes, it's all IO limited so it doesn't really matter. The advantage the libvips ones might have is if you use something like python. You could write:

image = pyvips.Image.new_from_file("big.v")
image \
    .colourspace("srgb") \
    .crop(1000000, 1000, 20000, 20000) \
    .resize(0.25) \
    .dzsave("some-pyramid")

And the four operations will execute together, in parallel, and with no intermediate images. In the code above, it looks like colourspace will run on the whole image, but it's all demand-driven, so it'll actually only run on the pixels needed for the output.

You could potentially save a lot of IO.

[pdbourke]

Yes, I totally appreciate this. In the pipeline I'm putting together this doesn't happen (much), there is custom intervention at each stage.

…

On Thu, 18 Apr 2024 at 21:19, John Cupitt ***@***.***> wrote: Ah Australia! Nice. I'm in London still. Yes, it's all IO limited so it doesn't really matter. The advantage the libvips ones might have is if you use something like python. You could write: image = pyvips.Image.new_from_file("big.v")image \ .colourspace("srgb") \ .crop(1000000, 1000, 20000, 20000) \ .resize(0.25) \ .dzsave("some-pyramid") And the four operations will execute together, in parallel, and with no intermediate images. In the code above, it looks like colourspace will run on the whole image, but it's all demand-driven, so it'll actually only run on the pixels needed for the output. You could potentially save a lot of IO. — Reply to this email directly, view it on GitHub <#3939 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLREF655MB34SUFM3QGOVDY56T5DAVCNFSM6AAAAABGMNOBV6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNJTHA2TM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325

[pdbourke]

I've used dzsave on 500,000 x 500,000 pixels images, so I'd think it would be OK, but of course an actual test would be best.

For your records, it works for a 3,800,000 x 440,000 image. :-) vipsheader -a scroll123_decurve.v scroll123_decurve.v: 3828940x440000 uchar, 3 bands, srgb

…

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325

[jcupitt]

Ah that's great! Well done, next time you're in London I'll buy you a beer!

[kmartinez]

Super effort!

[pdbourke]

Nice. Great watching the pioneering work, on what would have been considered top of the line gear at the time.

…

On Fri, 24 May 2024 at 18:18, Kirk Martinez ***@***.***> wrote: You can see us with the Vasari scanner in this ancient promo video: https://youtu.be/agCDd9oOPgE?si=fE8oHcFmB6qPFbwB&t=113 — Reply to this email directly, view it on GitHub <#3939 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLREF2ULTMM555GZQGA7XDZD3ZVFAVCNFSM6AAAAABGMNOBV6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNBUGE4TM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Paul Bourke paulbourke.net ***@***.*** +61 433338325