Black Line artefact with two-point image mosaic

[PHY3PAGEJ]

Bug report

Describe the bug
When mosaicking two images using the two-tie point method, a black line 1 pixel wide, seemingly from the black background used when an image is not completely straight, sometimes appears on the corner of the joined image where it should not be. This line does not go away when adjusting the blur parameter or by tweaking the tie points and has consistently been seen when mosaicking images.
This error has been seen for top bottom and left right joins.

To Reproduce
Steps to reproduce the behavior:

1. Use any two images to join together that using the pyvips.Image.mosaic1(...) function in python, or the 2-tiepoint mosaicing method in nip2
2. The produced image has a little black line at the corners of the join, which is not found if the pyvip.image.mosaic(...) function is used instead

Expected behavior
The images to mosaic together, the same as found with the one-tie-point method in nip2 or using pyvips pyvip.image.mosaic(...) function

Actual behavior
The images to mosaic together, with a 1 pixel black line artefact seemingly from the black background used to store and mosaic images that are not perfectly rectangular.

Screenshots

• 1.png : image 1, the left image
• 2.png : image 2, the right image
• 1tiepoint_join.PNG: the section of the mosaic that has errors for the 2 tie point join, showing it is not present when using 1 tie point.
• 2tiepoint_join_ERROR_LOCATION.PNG : the location the black line appears using the 2 tie point method instead.

Environment

• OS Name: Microsoft Windows 10 Enterprise
• OS Version: 10.0.19044 N/A Build 19044
• NIP2 package: nip2 8.7.0 , linked to vips 8.7.0
• python packages:
  • python [3.11]
  • pyvips [2.21]
  • cffi [1.0.0]
  • pycparser [2.21]

Additional context
Add any other context about the problem here.

[jcupitt]

Hi @PHY3PAGEJ,

Sorry for the delay in responding. Try adding an alpha. I see:

Here's a workspace that does the join:

http://www.rollthepotato.net/~john/join.ws

It looks like this:

Drop your left and right images in as A1 and A2. The A column does a join without the alpha and you'll see the small black tick you talk about at the top.

Column C adds an alpha to each image, then column B does the join again on the alpha images. You can see B8 has no black mark. You must use "drop alpha", not "flatten alpha", to remove the alpha channel again before saving.

Explanation: we added anti-aliased edges to the rotate operator a couple of years ago (they look very nice now), but it hadn't occurred to me that this would mess up two-point mosaics (the black line you are seeing is the anti-alias).

Adding an alpha makes it put the anti-alias into the alpha channel, so your RGB is now unaffected.

[PHY3PAGEJ]

Thank you for getting back to me!

I am struggling to implement this method into pyvips unfortunatly.

Currently, I am adding an alpha channel using the following line:

img = pyvips.Image.new_from_file(filename) 
img = img.addalpha() 

which converts img from <pyvips.Image 1350x1010 ushort, 3 bands, rgb16> to <pyvips.Image 1350x1010 ushort, 4 bands, rgb16>.

Then once the mosaicking is ran, I extract the RGB bands and leave the alpha band using:

#extract the bands
bands = [mosaic.extract_band(i) for i in range(0,4)]
#join the RGB bands back together
imgC = bands[0].bandjoin(bands[1])
imgC = imgC.bandjoin(bands[2])
#re-assurt the colourspace
imgC = imgC.colourspace("rgb16")

which makes imgC have format <pyvips.Image 36858x2299 ushort, 3 bands, rgb16> and the result of imgC.hasalpha() == 0

But the resulting image still has the line artefacts from the two-point join. Am I missing something?

[jcupitt]

extract the RGB bands and leave the alpha band using:

You can write this as:

imgC = mosaic[0:3]

ie. take the first three bands from mosaic.

Hang on, I'll try to make you a demo prog in pyvips.

[jcupitt]

Sorry, I got distracted by another issue.

This program seems to work:

#!/usr/bin/env python

import sys
import pyvips

left = pyvips.Image.new_from_file(sys.argv[1])
right = pyvips.Image.new_from_file(sys.argv[2])

left = left.addalpha()
right = right.addalpha()

join = left.mosaic1(right, "horizontal",
                    1311, 241,
                    21, 189,
                    1310, 733,
                    18, 682)

join = join[0:3]

join.write_to_file(sys.argv[3]);

With:

$ ./two-point-mosaic.py ~/pics/left.png ~/pics/right.png x.png
$ vipsdisp x.png

I see:

[PHY3PAGEJ]

Hi John,

Thanks for getting back to me!

Below is a simplified version of my code to show the black is still an error :(

I suspect it is because as part of automating my mosaicking all the joins have to be right to left rather than left to right as used in your previous examples.

here is a simplified version of my code:

#---------------import modules------------------
#standard libraries
import os
import pyvips

#--------------------state variables---------------
#folder information
dir = os.getcwd()
#filenames
img1_name, img2_name = "image_1.png", "image_2.png"
#reference tie points, where 1010 is the image height in px
x1_ref, y1_ref, x2_ref, y2_ref = 0, 0, 0, 1010
#secondary tie-points and harea
x1_sec, y1_sec, x2_sec, y2_sec = 1272, 49, 1281, 1010+47

#========Mosaicking=========
#1)no alpha mosaic----------------------
#call in images to be mosaicked
img1 = pyvips.Image.new_from_file(dir + "\\"+img1_name) 
img2 = pyvips.Image.new_from_file(dir +  "\\"+img2_name)

#mosaic images 1 to image 2 in a right-left join
join_noalpha = img1.mosaic1(img2, 'horizontal', x1_ref, y1_ref, x1_sec, y1_sec,x2_ref, y2_ref, x2_sec, y2_sec)  		
join_noalpha.write_to_file(dir + "\\"+"noalpha_mosaic.jpg")

#2)mosaic using alpha method---------------
#apply alpha
img1_alpha = img1.addalpha() 
img2_alpha = img2.addalpha() 

#mosaic images 1 to image 2 in a right-left join
join_alpha = img1_alpha.mosaic1(img2_alpha, 'horizontal', x1_ref, y1_ref, x1_sec, y1_sec,x2_ref, y2_ref, x2_sec, y2_sec)   		

#extract RGB bands, remove alpha band
join_alpha_removed = join_alpha[0:3]
#save image
join_alpha_removed.write_to_file(dir + "alpha_mosaic.jpg")

And here are the resulting images, showing the black line is still an issue:

Are there any workarounds you could suggest?

[jcupitt]

all the joins have to be right to left

Yes, that won't work -- it must be left.mosaic1(right). The function does the raised cosine on the right edge of left and the left edge of right, so if you swap them, you won't get a smooth join.

Can't you just swap the two images? I'm probably missing something.

[PHY3PAGEJ]

We have ~900 images to mosaic into chunks and a working code to mosaic them except for this issue with the two-tie point function.

Using the function right to left means the tie points will always be the same for each join no matter how long the image gets, shown by the parameters chosen:

The reference tie points on the right edge of the right image:
x1_ref, y1_ref, x2_ref, y2_ref = 0, 0, 0, 1010

The secondary tie points on the left edge of the left image of dimensions 1350x1010:
x1_sec, y1_sec, x2_sec, y2_sec = 1272, 49, 1281, 1010+47

The issues with joining left to right:
1)
If we mosaicked left to right, then the reference tie point location would change each time a new image is added to the mosaic, as the left image increased in size, meaning the reference would need to be found each time making automation extremely difficult, for example:

first join:
The reference tie points on the left edge of the left image:
x1_ref, y1_ref, x2_ref, y2_ref = 1272, 49, 1281, 1010+47

The secondary tie points on the right edge of the right image of dimensions 1350x1010:
x1_sec, y1_sec, x2_sec, y2_sec = 0, 0, 0, 1010

second join:
The reference tie points on the left edge of the left image:
x1_ref, y1_ref, x2_ref, y2_ref = 2400, 40, 2450, 1010+30

The secondary tie points on the right edge of the right image of dimensions 1350x1010:
x1_sec, y1_sec, x2_sec, y2_sec = 0, 0, 0, 1010

etc.

If we then have our mosaicked row of images and find there is an issue with join 1 say, then this will have a knock on affect on the value of the secondary tie point for every join after 1, meaning the whole row would need to be re-mosaicked and all the new points found.

We have used the right to left method before using the one tie-point mosaic() function with no issues and to change the code to mosaic the other way would be a huge task when the only issue is with this function :( Is there no way to tweak the function to work?

[PHY3PAGEJ]

Hi John,

Thankfully I managed to re-work the code for left-right joins and it seemed to work!

Sadly I have come to the same issue with the Top-Bottom joins that adding the alpha channel does not fix. I have tried a few different ways of solving the issue with no luck:

1. Verifying the black line is still an issue: using Top-Bottom two tie-point mosaic function mosaic1() for two rows with 3 bands produces the blackline artefact as shown previously in this thread and below:

2. Trying the suggested work around used in a left-right join: Adding an alpha channel before joining then removing the alpha channel at the end produces an image with a large black artefact from the top row.

3. Trying keeping the alpha channel from the left-right joins used to make the rows: Saving the .v with an alpha channel, joining the images, then removing the alpha channel at the end produces an image with a different large black artefact from the top row compared to attempt 2:

The only other work around I can think is try bottom-top joining instead, but nip2 does not have the functionality to try this. Do you have any suggestions?

The images used in .jpg form to save space:

[jcupitt]

Hi again, I'll have a look.

(wow your camera is rotated a lot ... are you sure you need a two point mosaic? I'd think a simple 1-point join might work)

[jcupitt]

You can't just add the alpha to the images -- you'll be setting the black areas solid. Try something like:

A1 ++ (A1?1 != 0)

ie. make the alpha 255 in areas where green is non-zero.

Could you upload the uncompressed top and bottom 16-bit images somewhere? Maybe eg. dropbox? I think JPEG compression might be causing the small artefact you're seeing.

[PHY3PAGEJ]

Thanks for your continued help!

There were a few instrumentation issues (Seems like the camera may have been tilting as it mosaicked) so the edges of some image cubes to be mosaicked are not parallel, thus needing the two tie-point method.

When I mosaicked the images they were in 8-bit .v format so I will try again with 16 bit now, and use your suggested method before uploading the files :)

[PHY3PAGEJ]

So with the RGB16.v format version of these images I used your method for setting alpha to before where green is non-zero:

And I am still getting the black line artefact:

Just to get things clear:

• We want the black background and anti-alias edge to be opaque and not included in the mosaic.
• The functions can only be used in the format left-right and top-bottom directions or the work arounds do not work.
• The alpha channel removes the anti-alias but for top-bottom joins is not making the black background opaque, and A1 ++ (A1?1 != 0) sets pixels with the Green channel (The black background) as 0 to opaque but does not seem to remove the anti-alias.

Is there a way to remove the anti-alias and have the black background opaque for the top-bottom join?

I have sent you the RGB16.v files via email.

[PHY3PAGEJ]

Hi John, Has there been any development on work arounds for the top bottom join issue outlined in the bug string #3462 ? Kind regards, John Page

…

________________________________ From: John Cupitt ***@***.***> Sent: 17 May 2023 13:33 To: libvips/libvips ***@***.***> Cc: Page, John ***@***.***>; Mention ***@***.***> Subject: Re: [libvips/libvips] Black Line artefact with two-point image mosaic (Issue #3462) You can't just add the alpha to the images -- you'll be setting the black areas solid. Try something like: A1 ++ (A1?1 != 0) ie. make the alpha 255 in areas where green is non-zero. Could you upload the uncompressed top and bottom 16-bit images somewhere? Maybe eg. dropbox? I think JPEG compression might be causing the small artefact you're seeing. — Reply to this email directly, view it on GitHub<#3462 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3Z4UMND7BSOPRVRLQUMTSDXGTATNANCNFSM6AAAAAAXK2RMUQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> DISCLAIMER: This email is intended solely for the addressee. It may contain private and confidential information. If you are not the intended addressee, please take no action based on it nor show a copy to anyone. In this case, please reply to this email to highlight the error. Opinions and information in this email that do not relate to the official business of Nottingham Trent University shall be understood as neither given nor endorsed by the University. Nottingham Trent University has taken steps to ensure that this email and any attachments are virus-free, but we do advise that the recipient should check that the email and its attachments are actually virus free. This is in keeping with good computing practice.

[PHY3PAGEJ]

Hi John,

You said you would be able to help from today (30/06/2023) onwards, any chance you can have a look at this?

Thanks,

John