Docker for Mac, Docker for Win

Could you please give any feedback on this issue. Thank you.

Ever since MDEV-11520, InnoDB uses posix_fallocate() for extending data files. On GNU/Linux, the GNU libc implementation of it would implement a fallback in case the file system does not support fallocate() natively.

Does it mean it's not a docker-related issue and InnoDB doesn't work on Windows/Mac? Can you advise a workaround?

The fix of MDEV-11520 was actually present in MariaDB 10.1.22, which was claimed to work.
Does 10.1.28 or 10.2.9 work? I would next suspect the refactoring of fil_space_extend_must_retry(). There were some follow-up fixes in MDEV-14244 (10.1.29, 10.2.11).

Yes, 10.1.22, 10.1.28, 10.2.9 work fine. I've tested with two images: debian-based and alpine-based. I've noticed that (it seems) all versions of debian-based 10.2 work fine but not alpine-based. In the output of the debian-based image I've noticed `binary distribution`, may that's the reason.

This is how I run latest 10.2.15:

docker run --rm -e MYSQL_ROOT_PASSWORD=password -v /tmp/mariadb-debian:/var/lib/mysql mariadb:10.2.15

docker run --rm -e MYSQL_ROOT_PASSWORD=password -v /tmp/mariadb-alpine:/var/lib/mysql wodby/mariadb:10.2

The latter gives:

Initializing database

2018-06-08  9:11:28 140437698538376 [ERROR] InnoDB: preallocating 10485760 bytes for file ./ibdata1 failed with error 95

2018-06-08  9:11:28 140437698538376 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space

2018-06-08  9:11:28 140437698538376 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.

2018-06-08  9:11:29 140437698538376 [ERROR] Plugin 'InnoDB' init function returned error.

2018-06-08  9:11:29 140437698538376 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2018-06-08  9:11:29 140437698538376 [ERROR] Unknown/unsupported storage engine: InnoDB

2018-06-08  9:11:29 140437698538376 [ERROR] Aborting

Just released MariaDB 10.1.34 works fine with bind mounts

Any news about this issue? 10.2.16 and 10.3.8 seem do not have this fix for InnoDB.

Could you share your point on this issue? Do you have any plans to fix this or this is a by design behaviour that won't be changed? 10.2.17 and 10.3.9 still have no fix for that. Or maybe you can recommend a workaround. This issue prevents many people from upgrading to 10.2/10.3.

At the moment there are no plans to reject it as a "design behaviour that won't be changed". So you can presume that it will be fixed.

I have the same problem on Linux when I mount an NFS share (v3) on the host and use that as data dir for the Mariadb container. My docker image is based on Alpine and runs Mariadb 10.3.9.

cvicentiu is experienced with Docker and has access to an environment where this could be repeated. I hope that some strace output can be extracted. It should be a rather simple fix, once we know which operation is failing and how.

A workaround from the original community image github issue is to add this argument: --innodb-flush-method=fsync. So it's clearly tied up in the use of linux os commands that assume linux filesystems but in this case docker is federating access to a windows filesystem, because of it being a host volume. This also reproduces using the docker for windows kubernetes front end running a mariadb container. Named volumes are ok because they are actually hosted and managed in the mobylinux vm. I believe this should also happen on mac as well.

The workaround doesn't work for me, I still get "[ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95" when I run my Docker image with datadir on NFS-share.

The issue still persists with 10.2.19

mariadb 10.2.15 on LXD:
This can be reproduced easily on an LXD images:alpine/3.8 container on ZFS storage backend. All you need is a host with LXD (such as Ubuntu 18.04) and a ZFS filesystem.
See https://bugs.alpinelinux.org/issues/9046
It works on BTRFS backend. One difference between ZFS and BTRFS, is that the BTRFS root filesystem of the container is mounted on the host (it can be seen). The ZFS root filesystem of the container does not seem to be mounted on the host. It has a mountpoint, but when I list it, it seems to be empty.

Some strace output:

[pid    30] open("./ibdata1", O_RDWR|O_CREAT|O_EXCL, 0660) = 4

[pid    30] fcntl(4, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0

[pid    30] fstat(4, {st_mode=S_IFREG|0660, st_size=0, ...}) = 0

[pid    30] fallocate(4, 0, 0, 12582912) = -1 EOPNOTSUPP (Not supported)

[pid    30] writev(2, [{iov_base="2018-12-10 19:10:49 139782140689"..., iov_len=44}, {iov_base="InnoDB: preallocating 12582912 b"..., iov_len=76}], 22018-12-10 19:10:49 139782140689288 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95) = 120

fallocate() is not supported on all filesystems. It does not seem to be supported on ZFS, but it is supported on BTRFS.

Just adding another data point...

I see the same problem/symptoms when running MariaDB on Alpine Linux 3.8 on an AWS EC2 instance. The problem does NOT manifest running on Alpine Linux 3.8 in a VirtualBox VM.

Is there any testing I can do to help isolate the issue?

I got the same issue when I did want to upgrade to 10.2 version on alpine.
I'm using an LXD container with a ZFS backend too like Alex.

Not affected:
alpine/3.7 version 10.1.32

Affected versions:
alpine/3.8 version 10.2.15
alpine/edge version 10.3.10

command:

  cmd:

  - mysql_install_db

  - --user=mysql

  - --skip-auth-anonymous-user

  - --datadir=/var/lib/mysql

Stderr:

    2018-12-12  7:33:05 139833824660360 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95

    2018-12-12  7:33:05 139833824660360 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space

    2018-12-12  7:33:05 139833824660360 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.

    2018-12-12  7:33:06 139833824660360 [ERROR] Plugin 'InnoDB' init function returned error.

    2018-12-12  7:33:06 139833824660360 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

    2018-12-12  7:33:06 139833824660360 [ERROR] Unknown/unsupported storage engine: InnoDB

    2018-12-12  7:33:06 139833824660360 [ERROR] Aborting

stdout:

   Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...

    Installation of system tables failed!  Examine the logs in

    /var/lib/mysql for more information.

    The problem could be conflicting information in an external

    my.cnf files. You can ignore these by doing:

        shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf

    You can also try to start the mysqld daemon with:

        shell> /usr/bin/mysqld --skip-grant-tables --general-log &

    and use the command line tool /usr/bin/mysql

    to connect to the mysql database and look at the grant tables:

        shell> /usr/bin/mysql -u root mysql

        mysql> show tables;

    Try 'mysqld --help' if you have problems with paths.  Using

    --general-log gives you a log in /var/lib/mysql that may be helpful.

    The latest information about mysql_install_db is available at

    https://mariadb.com/kb/en/installing-system-tables-mysql_install_db

    You can find the latest source at https://downloads.mariadb.org and

    the maria-discuss email list at https://launchpad.net/~maria-discuss

    Please check all of the above before submitting a bug report

    at http://mariadb.org/jira

This issue hast started after upgrading from mariadb 10.2 to 10.3.

Mariadb now is using native Linux fallocate for creating the ibdata file (punch a hole of xxMB)
It fails in alpine edge and alpine 3.8 (and Docker) running as containers based on ZFS filesystem.
This is a major Issue and I hope the Mariadb team is paying attention to this issue as it persists since mid 2018.

2018-06-29 19:11:12 140013615233928 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95
2018-06-29 19:11:12 140013615233928 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
2018-06-29 19:11:12 140013615233928 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2018-06-29 19:11:13 140013615233928 [ERROR] Plugin 'InnoDB' init function returned error.
2018-06-29 19:11:13 140013615233928 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-06-29 19:11:13 140013615233928 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-06-29 19:11:13 140013615233928 [ERROR] Aborting

I see this issue when using maria-db 10.2.x branch with the, alpine 3.8 docker image and attempting to use a nfs4 mounted aws elastic file system to host the mysql data files. Using either the local bind mount or a volume plugin. The work around, as for others, is to use the maria-db 10.1.x branch with alpine 3.7 docker image.

I guess, this bug is not going to be fixed anytime soon, as it is open for years now.
Mariadb since 10.2 / 10.3 uses the Linux fallocate to create /var/lib/mysql/ibdata1.
Fallocate doesnt work on some storage backends used for Containers (LXD, Docker...). In my Case it was ZFS.

InnoDB should be using the posix_fallocate() wrapper function, which should fall back to write() if fallocate() does not work.

Now that there is a confirmation that this problem can be repeated outside Docker for Mac or Windows, it should be easier for an InnoDB developer to fix this.

Any updates on this issue? Can you maybe advise some workaround? There's a couple of solutions suggested in https://github.com/docker-library/mariadb/issues/95 (changing flush method) but none of them work for the alpine-based image (wodby/mariadb).

Similar isuues:
https://jira.mariadb.org/browse/MDEV-16015
https://github.com/jbergstroem/mariadb-alpine/issues/1
https://gitlab.alpinelinux.org/alpine/aports/-/issues/9046
https://github.com/docker-library/mariadb/issues/95
https://github.com/wodby/docker4wordpress/issues/51

This is not an issue on FreeBSD w/ZFS as MariaDB does not assume it can use Linux's native fallocate().

Up until MariaDB 10.2.5, we could turn off innodb_use_fallocate to mitigate this "Linuxism", but that knob has been deprecated, making this a blocking issue.

It seems that the easy fix here would be to provide a knob to force posix_fallocate on Linux (if fallback to posix_fallocate can't be implemented automatically).

BTW the issue is generally applicable to copy-on-write filesystems, not just ZFS: https://github.com/openzfs/zfs/issues/326#issuecomment-540162402

@Blake Willis
the discussion at openzfs which you have kindly shared, is mainly treating zfs on host.
Whilst you are inside a container, you dont have privileges to set zfs properties on the mounted zfs slice.
Why waiting for zfs developers to do the work?
Just switch to 'truncate -S 80M' whenever 'fallocate' fails.

I realize "Probably out of disk space" is gaining in search engines popularity and becomes a powerful clickbait.

Up until MariaDB 10.2.5, we could turn off innodb_use_fallocate to mitigate this "Linuxism", but that knob has been deprecated, making this a blocking issue.

How about putting back that depreciated 'knob' in place as fallback whenever fallocate fails/missing?

BTW I was able to work around (as opposed to mitigate) this issue by switching the application's database engine to Aria (with TRANSACTIONAL writes) & disabling InnoDB (& turning off NO_ENGINE_SUBSTITUTION). This is not a great solution & certainly won't work for every application (especially those needing full ACID that InnoDB provides), but for simple applications with non-critical data it works.

Was this bug fixed in MariaDB 10.1.48, 10.2.35, 10.3.26, 10.4.16, 10.5.7 by MariaDB Pull Request #1658 a.k.a. adding fall-back logic for the code EOPNOTSUPP?

Hi Marko ,

I confirm the issue is fixed

I was able to reproduce the issue with pure alpine images alpine:latest
/mnt is zfs 4K

docker run --name=test-alpine..db.container.0002 --label=com.opensvc.id=a7beae62-2db6-4e6b-b468-d52f079e220e.container#0002 --net=container:test-alpine..db.container.0001 --volume=/etc/localtime:/etc/localtime:ro --volume=/srv/test-alpine-db/pod01:/data --cgroup-parent /opensvc.slice/test-alpine.slice/db.slice/container.slice/container.0002.slice busybox
s18-fr-1.db.container#2001 docker run -i -t -e MYSQL_ROOT_PASSWORD=mariadb -e MYSQL_INITDB_SKIP_TZINFO=yes --detach --name=test-alpine..db.container.2001 --label=com.opensvc.id=a7beae62-2db6-4e6b-b468-d52f079e220e.container#2001 --net=container:test-alpine..db.container.0001 --volume=/etc/localtime:/etc/localtime:ro --volume=/srv/test-alpine-db/pod01:/mnt --cgroup-parent /opensvc.slice/test-alpine.slice/db.slice/container.slice/container.2001.slice alpine:3 ash
s18-fr-1.db.container#2001 output:
s18-fr-1.db.container#2001 93c32bb31b58ebb78ae34d3011fc114d333e069267836cb3a47ac2a29011adad
s18-fr-1.db.container#2001 wait for up status
s18-fr-1.db.container#2001 wait for container operational
root@s18-fr-1:~# om test-alpine/svc/db docker exec -it

/bin/ash
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.13.0_rc4-29-gbc62523c83 https://dl-cdn.alpinelinux.org/alpine/edge/main
v3.13.0_rc4-25-g08b4300257 https://dl-cdn.alpinelinux.org/alpine/edge/community
OK: 13877 distinct packages available
/ # apk add mariadb
(1/15) Installing mariadb-common (10.5.8-r0)
(2/15) Installing libaio (0.3.112-r1)
(3/15) Installing ca-certificates (20191127-r5)
(4/15) Installing brotli-libs (1.0.9-r3)
(5/15) Installing nghttp2-libs (1.42.0-r1)
(6/15) Installing libcurl (7.74.0-r0)
(7/15) Installing libgcc (10.2.1_pre1-r3)
(8/15) Installing xz-libs (5.2.5-r0)
(9/15) Installing ncurses-terminfo-base (6.2_p20210109-r0)
(10/15) Installing ncurses-libs (6.2_p20210109-r0)
(11/15) Installing linux-pam (1.5.1-r0)
(12/15) Installing pcre2 (10.36-r0)
(13/15) Installing libstdc++ (10.2.1_pre1-r3)
(14/15) Installing libxml2 (2.9.10-r5)
(15/15) Installing mariadb (10.5.8-r0)
Executing mariadb-10.5.8-r0.pre-install
Executing busybox-1.32.0-r8.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 150 MiB in 29 packages
/ # mysql_install_db --datadir=/mnt
Installing MariaDB/MySQL system tables in '/mnt' ...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

Two all-privilege accounts were created.
One is root@localhost, it has no password, but you need to
be system 'root' user to connect. Use, for example, sudo mysql
The second is root@localhost, it has no password either, but
you need to be the system 'root' user to connect.
After connecting you can set the password, if you would need to be
able to connect as any of these users with a password and without sudo

See the MariaDB Knowledgebase at https://mariadb.com/kb or the
MySQL manual for more instructions.

You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mysqld_safe --datadir='/mnt'

You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/mysql-test' ; perl mysql-test-run.pl

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.
You can find additional information about the MySQL part at:
https://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

/ # ls /mnt/
aria_log.00000001 aria_log_control ib_buffer_pool ib_logfile0 ibdata1 mysql performance_schema test
/ # ls -laF /mnt/
total 569
drwxr-xr-x 5 root root 11 Jan 13 11:00 ./
drwxr-xr-x 1 root root 4096 Jan 13 10:59 ../
rw-rr- 1 root root 1 Jan 13 10:59 .opensvc
rw-rw--- 1 root root 24576 Jan 13 11:00 aria_log.00000001
rw-rw--- 1 root root 52 Jan 13 11:00 aria_log_control
rw-r---- 1 root root 972 Jan 13 11:00 ib_buffer_pool
rw-rw--- 1 root root 100663296 Jan 13 11:00 ib_logfile0
rw-rw--- 1 root root 12582912 Jan 13 11:00 ibdata1
drwx------ 2 root root 90 Jan 13 11:00 mysql/
drwx------ 2 root root 3 Jan 13 11:00 performance_schema/
drwx------ 2 root root 3 Jan 13 11:00 test/

stephane@skysql.com, thank you for confirming that this was fixed.

I still can reproduce this issue with alpine-based compiled mariadb image (10.5.8) (wodby/mariadb) on docker for mac (3.0.3, docker 20.10.0) with a bind mount:

$ docker run --rm -e DEBUG=1 -e MYSQL_RANDOM_ROOT_PASSWORD=1 -v /tmp/mariadb123:/var/lib/mysql wodby/mariadb

Initializing database

+ + mysql_install_db grep --help-q

-- --auth-root-authentication-method

+ :

+ installArgs+=(--auth-root-authentication-method=normal)

+ mysql_install_db --datadir=/var/lib/mysql/ --rpm --auth-root-authentication-method=normal

2021-01-13 13:26:16 0 [Warning] The parameter innodb_buffer_pool_instances is deprecated and has no effect.

2021-01-13 13:26:16 0 [Warning] The parameter innodb_log_files_in_group is deprecated and has no effect.

2021-01-13 13:26:16 0 [ERROR] InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 0 pages than specified in the .cnf file: initial 640 pages, max 655360 (relevant if non-zero) pages!

2021-01-13 13:26:16 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

2021-01-13 13:26:17 0 [ERROR] Plugin 'InnoDB' init function returned error.

2021-01-13 13:26:17 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2021-01-13 13:26:17 0 [ERROR] Unknown/unsupported storage engine: InnoDB

2021-01-13 13:26:17 0 [ERROR] Aborting

chingis, can you also reproduce this on Docker for Windows?

I am reassigning this to cvicentiu, who should be able to diagnose this.

For docker images not using standard glibc , Windows and OSX have special emmulation that may be posix_fallocate is not supported at all do not even return EOPNOTSUPP, from the code it's worth testing if innodb_use_trim=ON could be a workaround because it use ftruncate before fallocate

According to MDEV-11254, innodb_use_trim had no effect on 10.2. It was deprecated in 10.2 and removed in 10.3.0.

can it be reproduce with innodb_compression_default=1

tried to reproduce it on Mac, but without any success.

images used:
wodby/mariadb:10.5
mariadb/server:10.5
mariadb/server:10.4
mariadb/server:10.3
mariadb/server:10.2
mariadb:10.5
mariadb:10.3
mariadb:10.2

command used:
docker run --rm -eMYSQL_RANDOM_ROOT_PASSWORD=1 --mount type=bind,source=/tmp/mysql,target=/var/lib/mysql <IMAGE>:<TAG>

MacOS 11.1 Big Sur
Docker Desktop for Mac v3.0.4(51218), engine 20.10.2

stephane@skysql.com, thank you for trying to help, but I do not think that we allow the system tablespace to be a sparse file (page_compressed tables can only be created in .ibd files).

abychko, thank you for confirming that this works for your Docker for Mac. We did not verify the fix on Docker for Windows. I believe that the missing handling for the EOPNOTSUPP error code (which is 95 in GNU/Linux) in posix_fallocate() caused the original reported failure.

chingis, for any remaining problem, please file a separate ticket with the exact symptoms.

It seems the issue is really fixed. The reason it didn't start for me is because I used a bind mount from `/tmp` on macOS, which is a symlink to `/private/tmp`. It works, if you pre-create a directory under `/tmp/` but if it doesn't exists it fails. And it works fine regardless of whether it's pre-created or not for all other non-symlink directories.

Thank you, chingis!