Debugging Why a Target is Rerun Again

[gamlerhart]

I have a quite simple Target, which only depends on a few files.

• If I run in isolation, it is cached, as expected.
• If I run other tasks depending on it, it is cached as expected.
• However, if something a far away task which depends on is chanced, the Target is run again. Despite its inputs not changing.

The target looks like this:

  def nodePackages = T.sources {
    val paths = // List of paths to "package.json", "package-lock.json" etc...
    paths
  }

  def nodeInstall: Target[String] = T {
    val srcs =  nodePackages()
    println(s"Packaged changed: Installing node packages ${millSourcePath}")
    println(s"Sources are: ")
    println(srcs.mkString)
    val hash = srcs.mkString.hashCode
    println(s"HASH: ${hash}")
    os.proc("pnpm", "install").call(cwd = millSourcePath, stdout = Inherit)
    s"Resolved: ${hash}"
  }

I'm not sure if I am wrong / or how to debug why that task is running again:

• My understanding, if the tasks 'inputs' are the same, the it does not run again.

So, the first thing I did is to check its input, to see if I might miss something:

mill visualizePlan nodeInstall

That results in:

Ok, as expected, not other targets/inputs going into the task.

The next thing i tried to do is to see if the input actually changed by accident. Maybe a other process I launch changes a file. For that I did the test 'println' lines.
However, from these lines nothing seems to change between runs, yet the Target runs again:

A run, with a expected execution of the Target

...
./mill codeowners.debug.webpack
[159/206] nodeInstall 
Packaged changed: Installing node packages /home/roman/dev/codeowners-mill-experiment
Sources are: 
ref:v0:cca6bd8e:/home/roman/dev/codeowners-mill-experiment/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/package-lock.jsonref:v0:a0ce60ce:/home/roman/dev/codeowners-mill-experiment/pnpm-lock.yamlref:v0:3723ced5:/home/roman/dev/codeowners-mill-experiment/pnpm-workspace.yamlref:v0:e9ff5ac4:/home/roman/dev/codeowners-mill-experiment/codeowners/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/pnpm-workspace.yamlref:v0:61898468:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/pnpm-workspace.yamlref:v0:8c8a7f9a:/home/roman/dev/codeowners-mill-experiment/cloud/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/pnpm-workspace.yaml
HASH: 1080333695

Then I touch some files why lower in the dependency graph, and the nodeInstall target runs again. Despite no changes:

./mill codeowners.debug.webpack
[159/206] nodeInstall 
Packaged changed: Installing node packages /home/roman/dev/codeowners-mill-experiment
Sources are: 
ref:v0:cca6bd8e:/home/roman/dev/codeowners-mill-experiment/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/package-lock.jsonref:v0:a0ce60ce:/home/roman/dev/codeowners-mill-experiment/pnpm-lock.yamlref:v0:3723ced5:/home/roman/dev/codeowners-mill-experiment/pnpm-workspace.yamlref:v0:e9ff5ac4:/home/roman/dev/codeowners-mill-experiment/codeowners/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/codeowners/pnpm-workspace.yamlref:v0:61898468:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/checklist-buddy/pnpm-workspace.yamlref:v0:8c8a7f9a:/home/roman/dev/codeowners-mill-experiment/cloud/package.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/package-lock.jsonref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/pnpm-lock.yamlref:v0:c984eca8:/home/roman/dev/codeowners-mill-experiment/cloud/pnpm-workspace.yaml
HASH: 1080333695

So, my question is: How do I find out why my target is rerunning again?

[lefou]

You can inspect the file out/mill-profile.json. It contains some values for all targets of the last run, e.g. cached, inputsHash and previousInputsHash.

You can also check the target.json file under out (e.g. out/nodeInstall.json), which contains the inputsHash and the valueHash.

[lefou]

If you by any chance edited the build.sc file or its includes, it may also result in some target invalidations. In newer Mill version only those targets affected by the edit.

[lihaoyi]

Yeah the typical workflow for me would be to look the changed target up in mill-profile.json, check if its inputsHash and previousInputsHash` defer, and if so look up its dependencies. Rinse and repeat until you find the "first" target whose input changed. Could probably be automated but for has not yet been for now.

Given your very simply example you showed, it basically comes down to nodePackages changing, or build.sc changing. Your println should already rule out nodePackages changing, so I would have to assume you made changes to build.sc that invalidate the whole thing. As @lefou mentioned, any Mill version since 0.11.2 - 2023-08-28 does fine-grained invalidation after build.sc changes, but earlier versions would invalidate the entire build. That may be the cause of your observed behavior

[roman-mibex-2]

Thanks for all the hints. Ok, I when it goes wrong, I got this in mill-profile.json:

  {
    "label": "nodeInstall",
    "millis": 0,
    "cached": true,
    "dependencies": [
      "nodePackages"
    ],
    "inputsHash": 1649716480
    "previousInputsHash":  /* some other number /*
  },

This was repeatable, is that it always calculated the inputHash to 1649716480. But in the reruns in had some other previousInputsHash.
That look to me like some other 'run' is calculating a different 'inputHash, and the it flip-flops.

• So, I killed all mill processes, closed my IDE and tried to reproduce my issue with ./mill --no-server to be sure nothing hangs around. And yes, the issue disappears.
• Then I then ran it without my IDE open: No issue.
• As soon as my IDE runs, which imported the project via BSP, the issue start appearing again.

What is most likely to happen:

• IDE runs some BSP actions via mill. This triggers to run the nodeInstall target as well.
  The BSP environment doesn't 11 match, so I get a different hash.
• When rerunning from the outside the IDE, the environment is slightly different, producing other hashes.
• Now the IDE and the command line 'ping-pong' with different hashes.

Follow up questions:

• What is added into the Hashes implicitly / is there way to check that. Because the file hashes are the same, but the Targets output hashes seem to differ between the CLI and BSP runs.
• Is there a good way to see what BSP is doing, like logs what targets it ran?
• Maybe ideas if that could be auto detected.

Anyway, thanks for the help. I probably can get it fixed from here on. Either by ensuring the environment is similar enough between the BSP and the CLI. Or by get that 'target' out of the BSP paths.

[roman-mibex-2]

Answering part of my own questions:

• The BSP logs are in .bsp/mill-bsp.stderr

What I tried but fails. It tried to create my own hacky BSP target, which sets the JDK enironment on launch:

Basically it replaced the first argument of the BSP json which points to the mill script with my wrapper, something like

#!/usr/bin/env sh
export JAVA_HOME=/usr/lib/jvm/java-22-openjdk
export PATH=/usr/lib/jvm/java-22-openjdk/bin:$PATH
exec /home/roman/dev/codeowners-mill-experiment/.bsp/mill-stable-jdk.sh "$@"

However, as soon as the BSP server starts up, it looks like it gets reset to the original 'mill' script, defeating my hack ^_^.
But I probably can extend the hack: I put the actual JDK into the ./mill script.

Will keep digging. I want the BSP server in sync the CLI instances, otherwise they stamp over each other.

Anyway, wondering if that is something to add to the mill.BSP/install target. An option to use the same JDK as the launch command, so that there is less stomping over each other.

[gamlerhart]

Ok, got to a good enough solution.

• Added a info box to my 'nodeInstall' target. Informing that if is shown all the time, the JDKs are mixed up.
• Created my own BSP/install command. It uses the mill.BSP/install, but generates a copy of the './mill' script with a fixed JDK and calls that instead.

However, I might also go with a changed ./mill script which uses SDK man to pick its JDK =).

Thanks again for the thep.

[roman-mibex-2]

My BSP json replacement is still struggling. Some process tends to just insist on using the systems 'java' instead of the one used in the launch scripts. I'll keep looking into it.

[gamlerhart]

Second mistery solved, why things still do not work, despite my special BSP connetion file.

IntelliJ as some basic 'mill' support. Basically, if it sees a mill project, it calls ./mill mill.bsp.BSP/install. So that connection file is rewritten and my work around does not work.

I've to thing about something else.

[roman-mibex-2]

So, yes. My final solution is to pin the JDK version in the ./mill script. That seems the only feasible solution to avoid any ping-pong.