EBUSY on npm install because of rewrite of mounted package.json

[denengineer]

Hi! Thanks for your work of gathering all this stuff in one place!

But how do you managed to make npm install work with direct package.json mounts? I get this errors every time:

 resource busy or locked, rename '/opt/package.json.3249071875' -> '/opt/package.json'

because npm tries to rewrite file from scratch rather than make atomic writes.

[BretFisher]

You'll need to describe your setup. Host, OS and docker versions, etc.

[denengineer]

Windows 10, Docker is 18.03.1-ce and npm v6.1.0 on top of node alpine image.

To reproduce this I think you will need to just mount package.json

volumes:
      - ./src/package.json:/opt/package.json

and make npm install.

My main thought that this is related to docker mechanism of mounting single files and npm mechanism of replacing the whole package.json instead of modifying it. So I assume this is will appear on any version of docker since file mounting enabled and any version of npm.

Maybe it's related to host and on Linux docker volumes work another way, because npm works in the same way everywhere if I remember right.

[BretFisher]

Did you look around for any other windows users having this issue?

Are you using Docker for Windows (Hyper-V) or Docker Toolbox (Virtualbox)?

[denengineer]

Yep, they have same problems and just mounting folders, not files. I'm on Hyper-V. So you are using this conf on Linux and do not have any problems or you not have problems with Windows too?

[BretFisher]

I'm on macOS and no problems last I checked. I knew it would be an issue on Linux, but figured the SMB sharing on Windows would work around the issue like it does on Mac. Sucks that npm replaces the file.

[denengineer]

Hmm, can you please describe what do you mean by SMB sharing in this case? Even some link with the direction where to dig would be awesome :)

[BretFisher]

SMB ("windows file sharing") is the protocol that Windows and the Linux VM running Docker use to share your host code when bind-mounting.

[denengineer]

Hm, seems like in Win it doesn't solve the problem. Maybe it will be great to add comments in compose file for this Windows case?

[adams-family]

I'm can reproduce the same problem on Debian Linux 9 however there is a trick, please see below:

Steps to reproduce:

1. Git clone this repository

2. docker-compose up

3. Break it with Ctrl-C one it's up

4. Execute the following:

docker-compose run node npm install moment --save

Result:

npm WARN checkPermissions Missing write access to /opt/node_app/app/node_modules

5. Execute the following:

docker-compose run node sh -c "cd /opt/node_app; npm install moment --save"

Result:

npm WARN saveError EBUSY: resource busy or locked, rename '/opt/node_app/package.json.3920881860' -> '/opt/node_app/package.json'
npm WARN saveError EBUSY: resource busy or locked, rename '/opt/node_app/package-lock.json.3269793734' -> '/opt/node_app/package-lock.json'

[d3ce1t]

I have the same problem. I'm mounting package.json from host to a docker container. I think the problem is npm install creates a modified copy of package.json named package.json.SOMETHING. And when it comes the time to save it as package.json, it is just renaming the file instead of copying.

[adams-family]

@d3ce1t Yes I'm exactly on the same opinion. Until there is an option in npm to overwrite package json (instead of creating a copy and renaming it) I believe that this approach won't be working anymore :(

[frenzymind]

Meet the same issue.
I also find this answer: https://stackoverflow.com/questions/48960320/locked-package-json-files-in-docker-container-using-docker-compose/58880425#58880425
I try play with links as described, but I don't get success on it.
May be some one else try to do it ? May be I do something wrong.

[andrey-hohlov]

Unanswered issue in npm repository npm/npm#16615

UPD: to make it works need to mount folder with package.json file, not directly package.json

[andreafspeziale]

I created the following example to highlight what I can't make it work:

• https://github.com/andreafspeziale/docker-node

I don't have any local node_modules. I'm running everything through Docker.

As soon as I bootstrap the project with $ docker-compose up everything looks great.
If I try to change the Hello World! sentence in index.js nodemon reloads the server as expected.

But as soon as I want to add a dependency like lodash while developing and while my container is up and running the package.json is not updated.

As explained in this repo (node-docker-good-defaults) I'm using $ docker-compose exec -w /opt/node_app node npm install --save lodash to install the new dependency and reflect the changes on my host project. Unfortunately it isn't working.

If I'm not able to develop locally without having NodeJS installed in the host machine I don't really see the point of using in the docker-compose (at least for development) a service which serves my app.

It makes sense for staging and production since it will use the packaged app image and run everything without mounting anything and hot-reloading anything.

Any hints? Am I missing something also in my assumptions? (@BretFisher moved here the discussion)

[andreafspeziale]

I have updated the upper project code in order to develop with NO NodeJS installed on host machine. It should work. But in the Bret setup the EBUSY problem still exists.

[BretFisher]

This repo is just one example of using docker-compose. In my Docker for Node.js course, I cover two Solutions for designing the compose file for local development and students have thought of several other Solutions along the way.

I also talk about my two "Solutions" at DockerCon. Those slides start here in my talk.

This repo is more aligned with Solution 2 below, but you might consider Solution 1 if the package.json bind-mount issue is a blocker for you.

Solution 1:

• Don't move node_modules up a directory. Leave everything in the standard single dir and do a single bind-mount in the compose file.
• Before doing a docker-compose up the first time, we'll need to do a docker-compose run <app> npm install to see the node_modules that are bind-mounted to the host. This is the main disadvantage of this Solution.
• This also means on macOS/Windows that you can't "dual develop" by sometimes using native host Node.js and sometimes using Docker-based Node.js because the binaries in node_modules (if you have them) would be incompatible. This is a disadvantage if you don't always use docker to run this code.

Solution 2:

• Similar to this repo's setup, you would move node_modules and package*.json up a level in the file path, so they can be separated from host node_modules. This adds complexity to the setup but also provides the flexibility of developing with a different node_modules on the host than what is in the container.

[petrprikryl]

Isn't there way using yarn's --modules-folder? Assuming switch from NPM to yarn.
https://stackoverflow.com/a/53998408/1763888
https://classic.yarnpkg.com/en/docs/cli/install#toc-yarn-install-modules-folder

Then you shouldn't need bind mounts on package*.json because all yarn commands should be feasible from project directory directly, shouldn't?

Disclaimer: I haven't tested it yet.