docker - ddev on ZFS: Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument

Question

Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument Happens with any new project after

This error happens when launching any of my ddev projects or when creating a new one with ddev config.

Update: This happens only when docker data (/var/lib/docker by default) is on a ZFS volume. The problem disappeared as soon as I moved it to EXT4.

It does not seem to be a problem with docker or docker compose, as I have launched the example/hello-world containers of both projects successfully.

I have tried removing ~/.ddev, downgrading ddev, and downgrading docker. I was not able to downgrade docker-compose.

Log:

$ ddev start                                                                                                                                     
Starting test... 
mkcert may not be properly installed, we suggest installing it for trusted https support, `brew install mkcert nss`, `choco install -y mkcert`, etc. and then `mkcert -install` 
Running   Command=ip address show dev docker0
Building db 
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument 
Failed to start test: Failed to run docker-compose [-f /home/tester/htdocs/test/.ddev/.ddev-docker-compose-full.yaml up --build -d], err='exit status 1', stdout='Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM $BASE_IMAGE
 ---> 33f3288968e4
Step 3/6 : ARG username
', stderr='Building db
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument' 

config.yml

name: test
type: php
docroot: ""
php_version: "7.3"
webserver_type: nginx-fpm
router_http_port: "80"
router_https_port: "443"
xdebug_enabled: false
additional_hostnames: []
additional_fqdns: []
mariadb_version: "10.2"
mysql_version: ""
provider: default
use_dns_when_possible: true
composer_version: ""

System:

• Docker 20.10.1
• docker-compose 1.27.4
• Manjaro Linux (Arch derivative)
• ZFS 2.0.0-1 (on root)

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 27
 Server Version: 20.10.1
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/root
  Space Used By Parent: 20958640128
  Space Available: 282678809088
  Parent Quota: no
  Compression: off
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.2-2-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.9GiB
 Name: testpc
 ID: BKDP:GAWJ:BUCV:OVP6:QUKY:UQCZ:AESP:V3AN:EXHE:BWJU:4TT7:YPEO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Answer 1

(Don't have enough reputation to comment, but this isn't a full answer.)

I hit the same issue, also with Docker on Arch w/ZFS. I was able to "fix" it by passing DOCKER_BUILDKIT=0 docker build .... It appears to be an issue somewhere in buildkit.

I haven't dug in any further to see what specifically is causing the issue.