Azure batch does not refresh cached containers in the pool

Question

I have a container image prefetched on the Azure batch account pool, but I notice that the pool doesn't auto refresh the container when a new image is uploaded to the container registry, even though I have asked Batch to cache the latest container container (:latest).

Is there any good way to do this (other than deleting and recreating the pool)? The pool machines are currently managed by batch; I don't have access to it. Will refreshing the pool details prefetch the new container?

Answer 1

Container images are loaded on to the machine at the time of the request, e.g., if specified at the pool level, then they are loaded when the node starts up. If it is specified as part of the task, then it is "pulled" as part of the implicit docker run when the task executes. Subsequent task executions would simulate docker run and do not download images if the container image is found locally as per what you would expect if running locally.

There are a few workarounds:

1. Delete or resize down the pool to zero nodes and scale back up to pick up new images.
2. Have a rolling pool strategy where you spin up a new pool with the same image reference and autoscale/drain down the old pool and redirect work to the new pool. As an aside, this is generally a good strategy to have regardless of containers or not to pick up latest Azure Batch Node Agent versions.
3. Create a job preparation task such that you explicitly issue a docker pull of the image when a task within the job referencing the container image will re-pull the image on first run of the job on a node. Note that you'd have to delete the job and recreate it for every instance you update your container.
4. Create a script that remotes into machines and executes an image refresh command across all nodes.