Order of 'includeConfig' and 'profiles' in nextflow.config seems to affect nested property overrides

[danoreper]

Bug report

Expected behavior and actual behavior

Hi Nextflow team,

I expected that the order in which 'includeConfig' and 'profiles' are declared in nextflow.config would have no effect on property values.
However, I think I'm seeing that 'profiles' must be declared after any 'includeConfig' statements, in order for profile overrides to work correctly. In particular, this seems to affect what I'd describe as nested properties.

Steps to reproduce the problem

Given the following two files:

nextflow.config:

profiles {
    hpc {
        process {

         process_nested.value = "hpc.config"
         process_nonest       = "hpc.config"
       
         withLabel: "labelExample" {
             memory = 400
        }
       }
    }
}

includeConfig "memory.config"

and memory.config:

process {
    process_nested.value = "memory.config"
    process_nonest       = "memory.config"

    withLabel: "labelExample" {
        memory = 4
    }
}

and using the following command line invocation:

nextflow config -profile hpc

The nested properties 'process_nested.value' and 'labelExample' fail to be overridden by the hpc profile, while 'process_nonest' is correctly overridden.

(Of note, the problematic overrides work as expected once the 'includeConfig' declaration is moved above the 'profiles' declaration)

Program output

the unexpected output is as follows:

process {
   process_nested {
      value = 'memory.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 4
   }
}

(No .nextflow.log is generated by nextflow config)

Environment

• Nextflow version: Both 23.10.1 and 23.04.4
• Java version: Java/17.0.4
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release

Additional context

The example provided is for the nextflow config command, but I'm also seeing similar behavior for nextflow run

[bentsherman]

This will be fixed by #4744

[danoreper]

Thanks @bentsherman !

[pditommaso]

I'm not 100% sure this is going to be by #4744. We advice to avoid the pattern you are using for some limitation of the underlying parsing library.

See the red box at this link in the docs.

A better way to handle your need is to use the standard config file profile e.g.

profiles {
    standard {
      includeConfig "memory.config"
    }
    hpc {
        process {

         process_nested.value = "hpc.config"
         process_nonest       = "hpc.config"
       
         withLabel: "labelExample" {
             memory = 400
        }
       }
    }
}

I get the following config

» nextflow config . 
process {
   process_nested {
      value = 'memory.config'
   }
   process_nonest = 'memory.config'
   withLabel:labelExample {
      memory = 4
   }
}


» nextflow config . -profile hpc
process {
   process_nested {
      value = 'hpc.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 400
   }
}

» nextflow config . -profile standard,hpc
process {
   process_nested {
      value = 'hpc.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 400
   }
}

[bentsherman]

It is a suboptimal pattern, but it will be fixed by the new config parser

[pditommaso]

The may new config parser solve this problem, however is still under development and surely it's not going delivered soon.

The current best practice consists of using separate profiles as reported in the documentation