jenkins pipeline - Docker in Docker - volumes not working: Full of files in 1st level container, empty in 2nd tier

Question

I am running Docker in Docker (specifically to run Jenkins which then runs Docker builder containers to build a project images and then runs these and then the test containers).

This is how the jenkins image is built and started:

docker build --tag bb/ci-jenkins .
mkdir $PWD/volumes/
docker run -d --network=host  
  -v /var/run/docker.sock:/var/run/docker.sock 
  -v /usr/bin/docker:/usr/bin/docker 
  -v $PWD/volumes/jenkins_home:/var/jenkins_home 
  --name ci-jenkins bb/ci-jenkins

Jenkins works fine. But then there is a Jenkinsfile based job, which runs this:

docker run -i --rm -v /var/jenkins_home/workspace/forkMV_jenkins-VOLTRON-3057-KQXKVJNXOU4DGSUG3P27IR3QEDHJ6K7HPDEZYN7W6HCOTCH3QO3Q:/tmp/build collab/collab-services-api-mvn-builder:2a074614 mvn -B -T 2C install

And this ends up with an error:

The goal you specified requires a project to execute but there is no POM in this directory (/tmp/build).

When I do docker exec -it sh to the container, the /tmp/build is empty. But when I am in the Jenkins container, the path /var/jenkins_home/...QO3Q/ exists and it contains the workspace with all the files checked out and prepared.

So I wonder - how can Docker happily mount the volume and then it's empty?*

What's even more confusing, this setup works for my colleague on Mac. I am on Linux, Ubuntu 17.10, Docker latest.

Answer 1

After some research, calming down and thinking, I realized that Docker-in-Docker is not really so much "-in-", as it is rather "Docker-next-to-Docker".

The trick to make a container able to run another container is sharing /var/run/docker.sock through a volume: -v /var/run/docker.sock:/var/run/docker.sock

And then the docker client in the container actually calls Docker on the host.

The volume source path (left of :) does not refer to the middle container, but to the host filesystem!

After realizing that, the fix is to make the paths to the Jenkins workspace directory the same in the host filesystem and the Jenkins (middle) container:

docker run -d --network=host  
   ...
   -v /var/jenkins_home:/var/jenkins_home

And voilá! It works. (I created a symlink instead of moving it, seems to work too.)

It is a bit complicated if you're looking at colleague's Mac, because Docker is implemented a bit differently there - it is running in an Alpine Linux based VM but pretending not to. (Not 100 % sure about that.) On Windows, I read that the paths have another layer of abstraction - mapping from C:/somewhere/... to a Linux-like path.

I hope I'll save someone hours of figuring out :)