Strange merge of fulljones h5parms from killMS2H5parm with H5parm_collector

[tikk3r]

Initiall reported here: lmorabit/lofar-vlbi#70

After converting killMS solutions to fulljones h5parms with killMS2H5parm.py, merging these with H5parm_collector.py behaves a bit strange. For example, in the plots below the merged h5parm shows nothing for 164 MHz at an arbitrary timeslot, yet there are valid solutions there in the specific 164 MHz h5parm.

It appears that it gets confused about the axes or something like that, giving adjacent timeslots different "chunks of frequency". Converting them as only XX and YY with --nofulljones does not show this problem.

The plots show the merged h5parm in the first row and two example h5parms from individual 2 MHz blocks in the second row.

Time slot 2	Time slot 3

150 MHz block timeslot 2	164 MHz block timeslot 2

[revoltek]

yes, most likely it's a problem of axes ordering. I have tested that code only with LBA and only in a specific format. Are all h5parms at least with the same axes order/polarisations? It's dangerous it doesn't show any error though...

[tikk3r]

They have the same structure. I just realize I already mention a possible answer in the other issue. The axes do have the same order, but they do not have the same length necessarily. The kMS solutions can have different time and frequency axes per block. I guess the collector does not like that?

[revoltek]

I use the collector only to merge equal tables, have a look at the H5parm_interpolator.py, which has also a number of limitations. Coding a very flexible tool to merge tables is a bit of a pain, currently I've only these 2 ready

[tikk3r]

I tried H5parm_interpolator out, but it didn't quite do what I expected. What is it intended to do? I tried running it and it produced a single h5parm with apparently all the solutions added/multiplied (it seems) into a single solset and single amplitude and phase soltabs. The frequency axis only has the range of the first one and the amplitudes seem to all have been multiplied or something as now they are suddenly of order 100 where they are of order 1 in the input H5parms.

H5parm_interpolator sounds like what I'm looking for though, so I'll poke around in it a bit more.

[revoltek]

I use it to take all fast phase and slow amp solutions from many directions and combine them into a single h5parm to be used with DDF - I tested it only in a few cases and there's a number of possible restrictions that I probably never though about... if you come up with a more general version that would be great

[tikk3r]

Ah I see, that makes sense.

I think because each h5parm covers a different frequency range it "breaks" the interpolation in this case as well as that makes it not just interpolating to an identical, but faster grid but needing to interpolate different ranges of values to a common interval. I'm definitely in a non-standard messy case I think.