interpolate and smooth are too slow for large dimension soltabs

[wndywllms]

I'm running prefactor on LBA data with an 8hr calibrator run, the losoto bandpass step alone is taking about 10hrs to run (a significant fraction of the entire pipeline run)

The interp and smooth operations each take 4-5 hours because internally it iterates over the time, ant and pol axes and the time axis is long (much longer than 10 min HBA calibrator obs)

I have an update that does this differently... each step runs in <5 minutes

[revoltek]

Hum, why that step is so long? In practice it is just a median, for me (on cep3) it takes:
2019-09-11 21:51:17 - INFO: ^[[32mTime for smooth1 step: 31 s (cpu: 31 s).^[[0m

I don't think I run any interpolation step neither, for what is it needed?

[wndywllms]

It's the step that does
for vals, weights, coord, selection in soltab.getValuesIter(returnAxes=axesToSmooth, weight=True, reference=refAnt):
regardless of what is in the loop it takes ages to go through my h5parm with 7190 times, 976 freqs, 36 ant, 2 pol (each step takes 30ms in a do-nothing loop like this)

I don't think I run any interpolation step neither, for what is it needed?

I'm not sure exactly ... prefactor now has a median smooth in time, a savitzky-golay smooth in frequency and this interpolate step (which regrids the bandpass to 1chan/sb) before another median smooth in time) - I think this was updated in Oct last year by @adrabent

[revoltek]

I can see why it's slow, I need to understand your update a bit better (maybe we should have a chat - but I'm on holiday now). The smooth in time-only is superfast (few seconds), I don't understand what the "savitzky-golay smooth in frequency" is for

[twshimwell]

Hi,

Just wondering if this issue got fixed? Im trying to analyse some pretty long calibrator observations and running into speed issues at this same step.

Thanks,

Tim

[revoltek]

Can you past here the parset of the smooth operation? For things like:
[smooth1]
operation = SMOOTH
soltab = sol000/amplitudeSmooth
pol = XX, YY
axesToSmooth = [time]
mode = median

it should go rather fast

[twshimwell]

This is the parset from prefactor for this step:

ncpu = 32
[duplicateAbkp]
soltabOut = amplitudeOrig000
operation = DUPLICATE
soltab = sol000/amplitude000
[plotA1]
plotFlag = True
axesInPlot = [time,freq]
prefix = /data/shimwell/prefactor_outputs/work_dir//Pre-Facet-Calibrator_L796456/results/inspection/ampBFlag_
soltab = sol000/amplitude000
axisInTable = ant
operation = PLOT
[flag]
operation = FLAG
axesToFlag = [time,freq]
preFlagZeros = False
maxRms = 5
replace = False
mode = smooth
maxCycles = 1
order = [100,40]
soltab = sol000/amplitude000
[flagbp]
ampRange = [0, 0]
operation = FLAGSTATION
mode = bandpass
skipInternational = True
soltab = sol000/amplitude000
[flagextend]
percent = 50
axesToExt = [time,freq]
soltab = sol000/amplitude000
maxCycles = 2
operation = FLAGEXTEND
size = [200,80]
[plotA2]
plotFlag = True
axesInPlot = [time,freq]
prefix = /data/shimwell/prefactor_outputs/work_dir//Pre-Facet-Calibrator_L796456/results/inspection/ampAFlag_
soltab = sol000/amplitude000
axisInTable = ant
operation = PLOT
[merge]
operation = REWEIGHT
soltabImport = amplitude000
mode = copy
soltab = sol000/phase000
[smooth]
axesToSmooth = [time]
log = True
replace = False
pol = XX,YY
mode = median
operation = SMOOTH
soltab = sol000/amplitude000
[bandpass]
axesToSmooth = [freq]
log = True
degree = 2
pol = XX,YY
mode = savitzky-golay
operation = SMOOTH
soltab = sol000/amplitude000
size = [17]
[interp]
newDelta = 195.3125kHz
log = True
maxFlaggedWidth = 30
outSoltab = bandpass
soltab = sol000/amplitude000
delta = 48.82kHz
operation = INTERPOLATE
axisToRegrid = freq
[smoothb]
axesToSmooth = [time]
log = True
replace = True
pol = XX,YY
mode = median
operation = SMOOTH
soltab = sol000/bandpass
[plotB1]
plotFlag = True
axesInPlot = [time,freq]
prefix = /data/shimwell/prefactor_outputs/work_dir//Pre-Facet-Calibrator_L796456/results/inspection/bandpass_
soltab = sol000/bandpass
axisInTable = ant
operation = PLOT
[plotB2]
time.minmaxstep = [0,1e20,500000]
plotFlag = True
axisInCol = pol
axesInPlot = freq
prefix = /data/shimwell/prefactor_outputs/work_dir//Pre-Facet-Calibrator_L796456/results/inspection/bandpass_
soltab = sol000/bandpass
axisInTable = ant
operation = PLOT
[plotB3]
time.minmaxstep = [0,1e20,500000]
plotFlag = True
axisInCol = ant
axesInPlot = freq
prefix = /data/shimwell/prefactor_outputs/work_dir//Pre-Facet-Calibrator_L796456/results/inspection/bandpass_
soltab = sol000/bandpass
operation = PLOT

[twshimwell]

(I think this was about 14hrs on 32cpu to run this parset on 4hrs of HBA calibrator data)

[revoltek]

and the bottleneck is the "smooth" step? (you can check how long each step took in the logfile)

[twshimwell]

I havent broken it down by steps but we were kind of guessing it was the same issue that wendy had previously (unless that had already been fixed).