The frequent log scanned up to messages were output due to the bug MDEV-27919. Will the 10.8.3 backup complete eventually? What if you specify a shorter --log-copy-interval?

Note that there is no easy way to downgrade from 10.8 due to MDEV-14425 and MDEV-27199.

10.8.3 completes in about 4 minutes (this is just the lock phase), it was 4 seconds before.

Log size is not a problem. The test with log-copy-interval is to reduce the logs, right? I mean, it will not have any impact on the lock speed, will it?

We can see that the problem is caused by the binary logs, that are now much bigger than before (I don't have the numbers here).

Lock duration with mariadb 10.8.3 look like these:

• Backup with binary logs (3 days retenetion): about 2.5 minutes
• Backup purging all the binary logs at the beginning: 14 seconds
• Backup deactivating binary logs: 8 seconds

With 10.7.3:

• Backup with binary logs (3 days retenetion): 10 seconds

I believe this might be the same bug as described in MDEV 28758. Some of the most up-to-date versions of mariabackup include binary logs into the backup directory, resulting in bloated backups and (depending on binlog size) massively increased lock durations.

Same happening to me. My database is being locked for 30 minutes every day, causing a huge downtime.

I raised MDEV-28955 with many details.

The difference is that I don't have binary logs enabled, so I don't think that's the cause for me.

cocorossello and nunop, did you try specifying a shorter --log-copy-interval like I suggested? If an attempt to copy recently written logs is made less frequently, the backup may take a much longer time. The size of the backed up ib_logfile0 depends on how many persistent changes were made between the latest log checkpoint (at the start of the backup) and the end of the backup.

Thanks marko

I can try, but I wonder if that will just reduce the downtime from 30 minutes to a lower time, but still not "fix" the problem.

I didn't have this problem with 10.5. Do you know if this is a bug, or something that we can't go around anymore?

–

I see there's an option "--no-backup-locks" that I wonder if I could use, even if temporarily: https://mariadb.com/kb/en/mariabackup-options/#-no-backup-locks

But it says: "This option may eventually be removed. See MDEV-19753 for more information."
I would like to read more about why, but MDEV-19753 redirects to MENT-1084, which I can't access.

Thank you!

You can view the archive at https://web.archive.org/web/20200809201840/https://jira.mariadb.org/browse/MDEV-19753 - this appears to refer only to a Percona Server feature, so it doesn't seem likely to help

nunop, it is unclear where the time is being consumed, and I am not convinced that the problem is the time needed for acquiring the locks. Since you are not using binlog, you can’t be affected by MDEV-28758. Many other users are. I would strongly recommend that you try specifying --log-copy-interval=1 or similar. The default was restored to 1 second; it accidentally was 1 millisecond in 10.6 and later, before MDEV-27919 was fixed.

Thanks marko

I would strongly recommend that you try specifying --log-copy-interval=1 or similar

I'll have to postpone this test to tomorrow morning, as now would be too late to make this test (and unnecessarily cause a downtime if it happens again)

the default was restored to 1 second; it accidentally was 1 millisecond in 10.6 and later,

In my raised issue - MDEV-28955 - I mention I was on 10.5, where this didn't happen. So the accidental change you mention on 10.6 (MDEV-27919) didn't apply to me.

it is unclear where the time is being consumed

If it helps, I can provide my full logs to you, but I'd prefer if I could send it through some Private Message or email address, rather than fully public here.

I can understand how "--log-copy-interval=1" would at least reduce the amount of time taken by "log scanned up to", though.
Are there any cons in putting such a short time? (I wonder why it isn't the default)

But again I say, on 10.5 I can see that "log scanned up to" was once per second, yet the database wasn't locking as is now on 10.8.

There were, however, a lot less "log scanned up to" from what I can see, though (not sure why there are so many now on 10.8),
but I'm certain that the database wasn't locking during these 2 minutes the backup was taking (while on 10.5)

Thank you very much!

I will try --log-copy-interval=1 tonight as suggested. I just thoght that this interval affected only to that frequent "scanned up to..." log

Hi marko

I tried now running:

/usr/bin/mariabackup --backup --parallel=4 --log-copy-interval=1 --target-dir=/backup/mysql/

LOADS of log scanned up to entries appearing really fast, and the backup took 1:30 minutes to run.
Absolutely no locked queries while this happened...

I understand how --log-copy-interval=1 made it faster, but I don't understand how that would resolve the database locks???

And is there any cons in keeping --log-copy-interval=1 ? (why is it not the default?)

Thank you very much.

I also tried that without much luck. First I tried with log-copy-interval=1, but after 70 minutes I aborted the backup because it didn't finish.

Then I run with log-copy-interval=100 and the results were not good, almost 1 minute of locks.

log-copy-interval=100

• mariabackup --backup --log-copy-interval=100 --stream=xbstream --target-dir /backup/dump --user ... --password ... --use-memory=12288M

[00] 2022-06-27 22:16:29 Connecting to MariaDB server host: production.travelc.internal, user: onlyscripts, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
[00] 2022-06-27 22:16:29 Using server version 10.8.3-MariaDB-1:10.8.3+maria~jammy-log
mariabackup based on MariaDB server 10.8.3-MariaDB debian-linux-gnu (x86_64)

[00] 2022-06-28 01:30:36 Acquiring BACKUP LOCKS...
[00] 2022-06-28 01:30:37 >> log scanned up to (3665225576819)

[00] 2022-06-28 01:31:34 >> log scanned up to (3665226113706)
[00] 2022-06-28 01:31:34 Executing BACKUP STAGE END
[00] 2022-06-28 01:31:34 All tables unlocked

Full backup duration: 3 hours 16 minutes

no log copy interval:

• mariabackup --backup --stream=xbstream --target-dir /backup/dump --user ... --password ... --use-memory=12288M

[00] 2022-06-26 22:11:40 Connecting to MariaDB server host: production.travelc.internal, user: onlyscripts, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
[00] 2022-06-26 22:11:40 Using server version 10.8.3-MariaDB-1:10.8.3+maria~jammy-log
mariabackup based on MariaDB server 10.8.3-MariaDB debian-linux-gnu (x86_64)

[00] 2022-06-26 22:43:46 Acquiring BACKUP LOCKS...
[00] 2022-06-26 22:43:47 Starting to backup non-InnoDB tables and files
[00] 2022-06-26 22:43:47 >> log scanned up to (3656370393330)

[00] 2022-06-26 22:43:58 Executing BACKUP STAGE END
[00] 2022-06-26 22:43:58 All tables unlocked
[00] 2022-06-26 22:43:58 Streaming /var/lib/mysql/ib_buffer_pool to <STDOUT>

Full backup duration: 32 minutes

The purpose of the backup locks is to prevent any changes to non-InnoDB tables. mariadb-backup is only treating the InnoDB log specially; the Aria storage engine would rely on backup locks. As far as I understand, the problem here is that the steps executed while holding the backup lock are taking longer than expected or acceptable. InnoDB performs some writes also internally (for example when purging the history of committed transactions), so even when the strictest backup lock is being held, some InnoDB writes are possible. That is why we must keep copying the ib_logfile0 in the final phase.

nunop, I see that your question was answered by cocorossello: Too frequent polling of the log file may be a bad idea. One size does not fit all.

cocorossello, if you are enabling the binary log, to work around MDEV-28758 you could try specifying --sst-max-binlogs=1 so that less data will be copied.

Alternatively, you can use the mariadb-backup executable from the 10.8.2 distribution, for executing the --backup step.

Reverting mariadb-backup to 10.8.2 would also revert --log-copy-interval to its buggy meaning (counting microseconds, not milliseconds).

For the --prepare step, I think that it is better to use the 10.8.3 version, because several recovery bugs were fixed since 10.8.2.

I have one more possible explanation or solution. In 10.8.3, MDEV-28111 made the InnoDB redo log use O_DIRECT by default. That would essentially disable the file system cache between mariadbd and mariadb-backup and could ruin the performance. MDEV-28766 (in the upcoming 10.8.4 release) would introduce the parameter innodb_log_file_buffering. It could be a good idea to execute SET GLOBAL innodb_log_file_buffering=OFF; before starting a backup and SET GLOBAL innodb_log_file_buffering=ON; after it. If this is the problem that you both are facing, then maybe this logic should be made part of mariadb-backup itself.

In MariaDB 10.8.3, to disable the use of O_DIRECT for the redo log, you must set innodb_flush_method to something else than O_DIRECT, O_DIRECT_NO_FSYNC, or O_DSYNC. Before MDEV-24854 in 10.6, the default was innodb_flush_method=fsync. This setting will also affect data page writes.

I suggest that you try innodb_flush_method=fsync on both mariadbd and mariadb-backup to check if enabling the file system cache on the ib_logfile0 fixes the performance regression.

Hi marko

Thank you for your reply.

For many many years, I've been using O_DIRECT, and I never had this database lock problem up to v10.5. Only now that I moved to 10.8.3, this problem started happening.

To me, everything indicates that something with 10.8 isn't behaving well, which on 10.5 was working perfectly.

> It could be a good idea to execute SET GLOBAL innodb_log_file_buffering=OFF; before starting a backup and SET GLOBAL innodb_log_file_buffering=ON; after it.

I suppose I can try this, when 10.8.4 is released. (although I hope --log-copy-interval=1 alleviates the issue in the meantime.

Thank you.

nunop, before MDEV-14425 in 10.8 replaced the 512-byte log block structure, it was not possible or feasible to enable O_DIRECT on the redo log, only on the data files. To enable O_DIRECT, we must be able to detect the physical block size, which usually is 512 or 4096 bytes.

marko - thank you for your reply. I'm not sure what do you mean with that (sorry). Are you saying that O_DIRECT was not doing anything before 10.8 (meaning that even though I had this option enabled, it was not taking effect), or that I need to do something now to be able to accommodate this problem on 10.8.3?

Sorry for the confusion.

nunop, I mean that before MDEV-28111 in MariaDB Server 10.8.3, O_DIRECT was never enabled on log files. (To be exact, something equivalent was enabled on Microsoft Windows if the physical block size of the log file was detected to be 512 bytes.) Settings like innodb_flush_method=O_DIRECT only affected the reads and writes of data files.

marko Thanks. That's clear now!

In MDEV-28766, I read this:

O_DIRECT should not be used when innodb_flush_log_at_trx_commit = 2 is set.

I actually always had both of these enabled (O_DIRECT and "innodb_flush_log_at_trx_commit = 2"), for many years.
Could this be one of the reasons for what's happening now?

(what is the reason that O_DIRECT should not be used when innodb_flush_log_at_trx_commit = 2 is set, btw?)

I actually also just raised MDEV-28974 with another problem with mariabackup (cannot be prepared) - not sure if that's another consequence of the same issues.

Thank you very much.

nunop, that opening statement in MDEV-28766 did not seem to hold on my Intel Optane 960 NVMe PCIe drive. Based on the performance tests that I ran on my system (you can find details in the ticket), I did not identify any simple rule that would work with all 3 types of storage that I had at my disposal. Therefore, I thought that it is simplest to control the {{O_DIRECT}}ness of the redo log file with a separate Boolean parameter. While I was at it, I made it changeable without a server restart.

Can you please try to test if disabling O_DIRECT on the redo log (in 10.8.3, you’d have to use innodb_flush_method=fsync) would fix the problem for you? That is, would that work as an alternative to shortening the --log-copy-interval?

I just tried using mariabackup 10.8.2 and the results were not good

full docker command :

echo "apt update; apt install pigz; mariabackup --backup --host="somehost.internal" --parallel=4 --stream=xbstream --target-dir /backup/dump --user ... --password ... --use-memory=12288M | pigz -p 4 > /backup/dump/mariadbbackup-2022-06-29.gz" | docker run --name mariabackup -v /services/mysql/data/mysql:/var/lib/mysql -v /opt/backups/mariadb/dump/:/backup/dump -v /services/mysql/conf:/etc/mysql/conf.d:ro --rm -i --entrypoint sh mariadb:10.8.2

[00] 2022-06-29 07:20:55 Connecting to MariaDB server host: production.travelc.internal, user: onlyscripts, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
[00] 2022-06-29 07:20:55 Using server version 10.8.3-MariaDB-1:10.8.3+maria~jammy-log
mariabackup based on MariaDB server 10.8.2-MariaDB debian-linux-gnu (x86_64)
[00] 2022-06-29 07:20:55 uses posix_fadvise().
[00] 2022-06-29 07:20:55 cd to /var/lib/mysql/

[00] 2022-06-29 07:32:54 Acquiring BACKUP LOCKS...
[00] 2022-06-29 07:32:54 Starting to backup non-InnoDB tables and files
[01] 2022-06-29 07:32:54 Streaming ./travelc/powerbi_bookedhotelservice.frm to <STDOUT>
[00] 2022-06-29 07:32:54 >> log scanned up to (3677498865960)
[01] 2022-06-29 07:32:54 ...done

[00] 2022-06-29 07:33:18 Executing BACKUP STAGE END
[00] 2022-06-29 07:33:18 All tables unlocked
[00] 2022-06-29 07:33:18 Streaming ib_buffer_pool to <STDOUT>

so backup full time is much better at 13 minutes (half) but lock time is 24 seconds, double than 10.8.3.

Maybe I should specify log-copy-interval=1000000 to get the same log copy interval behaviour as with 10.8.3?

I cannot test changing innodb_log_file_buffering as I would have to restart production server and I can't do it unplanned. Another fact is that lock backup times are more or less 2 seconds in the dev environment (just like with 10.7.3), I only get degraded lock time performance in the live server. (I disabled binary logs in the production server when this started to happen, with binary logs lock time is more than 1 minute, I haven't tested with binary logs since then)

Hey

I was waiting for this morning to be able to change to innodb_flush_method = fsync as that requires a restart.

Then I ran /usr/bin/mariabackup --backup --parallel=4 --target-dir=/backup/mysql/ and it went fine - took just 1 minute to run everything and database wasn't locked during that period, but I suspect that because this was right after a restart, the checkpoint has already been made. I don't see the usual table renames, and not so many log scanned up to.

I'll try again tomorrow and let you know, marko.

Thank you

Thank you both. Currently it seems like the fix would be SET GLOBAL innodb_log_file_buffering=OFF during the last phase of backup. It could be done automatically by mariadb-backup itself. Let’s see how it turns out for nunop tomorrow.

cocorossello, yes, you should try to specify --log-copy-interval=1000000 to the 10.8.2 backup, to get the 1-second copying interval that was intended.

Thanks Marko, with 10.8.2 and --log-copy-interval=1000000 I get a 3 second lock, more or less same lock as with 10.7.3.

I will try innodb_log_file_buffering patch when available.

Hello marko

No luck... it seemed to be going very well for a while, but after 1 or 2 minutes, everything started to get locked, with:

Waiting for backup lock

As a reminder, I have:

innodb_doublewrite              = 1

innodb_flush_method             = fsync

I ran:
/usr/bin/mariabackup --backup --parallel=4 --target-dir=/backup/mysql/

It was doing those "loops" of "DDL tracking" and "log scanned up to" for a while, until it started locking all the INSERT/UPDATE queries, so I killed the process right away.

Then I added --log-copy-interval=1,
loads of "log scanned up to" appearing for a while, and in about 1 minute, the backup finished and no locks detected.

–

Then tried to run it again, with --log-copy-interval=1
And this time I did notice for 1 or 2 seconds, some queries locked with Waiting for backup lock, but wasn't so bad. Completed quickly.

EDIT - sorry - the last time I ran was still with --log-copy-interval=1.
I clarified above.

nunop, thank you. Your issue does not seem to be the O_DIRECT then. It might be that thanks to MDEV-14425 and MDEV-27774, the 10.8.3 server can generate log faster than earlier versions, and the 1-second interval is not optimal or sufficient for your workload.

Thank you marko. So, what do you think could be the solution for me? Continue to use --log-copy-interval=1, or eventually try SET GLOBAL innodb_log_file_buffering=OFF ?

nunop, I think that --log-copy-interval=1 could be the right thing for you. It looks like we have to keep this ticket open until 10.8.4 is available and further experiments with SET GLOBAL innodb_log_file_buffering can be performed.

mleich just told me that he is running into something similar when testing on a PMEM device (but not on "fake PMEM", that is, using mmap on a log file in /dev/shm). At the default 1-second interval, the backup could not keep up with the server. It is worth noting that back when we tested MDEV-14425. the log copying interval was 1 millisecond instead of the intended 1 second, and everything worked fine. He got into this because of MDEV-28994, which is some kind of a race condition between backup and the server when using a memory-mapped log. We are still narrowing it down.

I think that the proper fix to all this trouble would be to let the server process be responsible for duplicating its log (something that is a part of MDEV-14992).

Thank you very much for explaining, marko

Yeah, I'm also interested on that MDEV-14992 (I'm watching it). Great job.

Hopefully cocorossello can also find a way to resolve the situation.

For now, I think I'm good, and I guess MDEV-28955 can be closed (maybe it's a duplicate of this one, at this point).
I can give innodb_log_file_buffering a try when 10.8.4 is released.
Although, if there are no side-effects of using --log-copy-interval=1, then I'm happy to keep it, and the backup runs faster as well.

Thank you!

We are fine at the moment. The suggested workaround of using mariabackup 10.8.2 and log-copy-interval=1000000 is enough for us at this moment

I expect 10.8.4 and other quarterly MariaDB server releases to be out within a week or so. While I remember, I’m requesting feedback if this works in 10.8.4 when using suitable settings.

Thank you very much marko.

I have a number of changes to try, pending the 10.8.4 release. And then I'll give feedback.
Thanks for giving me some estimate. I thought about asking a few times about some date, but just kept waiting a bit longer anyway

The only other problem I keep having for now, is the swap usage - MDEV-29097
At least I found that the raise seems to happen every time mariabackup runs! But I have no idea why, or how further I can investigate.

Thanks again!
Have a great day.

marko
Tomorrow I'm going to restart MariaDB with the following options:

innodb_page_size = 16384

innodb_doublewrite = 1

innodb_flush_method = O_DIRECT

innodb_flush_log_at_trx_commit = 2

innodb_log_file_buffering = ON   (default value for innodb_flush_log_at_trx_commit = 2)

Then the idea is to experiment MariaBackup with these, and see if all is good.

I can try to experiment also removing the --log-copy-interval=1, with innodb_log_file_buffering = ON
And if it locks the database, I'll SET GLOBAL innodb_log_file_buffering=OFF and see what happens.

Backup ran OK with the configurations above, with --log-copy-interval=1

Tomorrow I'm going to run it manually, without --log-copy-interval=1 as planned.

I see also that MDEV-28781 is fixed, so I'll look into adding --rsync back as well.

Hi marko

Ran:

/usr/bin/mariabackup --backup --parallel=4 --target-dir=/backup/mysql/

And eventually started getting many queries locked with Waiting for backup lock, so I killed it.

Then I ran:

SET GLOBAL innodb_log_file_buffering=OFF

and confirmed the variable value changed.

Tried again:

/usr/bin/mariabackup --backup --parallel=4 --target-dir=/backup/mysql/

And eventually started getting many queries locked with Waiting for backup lock, again, so I killed it.

So, unfortunately, innodb_log_file_buffering=OFF doesn't seem to do it.

Or do you mean I need to leave innodb_log_file_buffering=OFF for 1 day after a successful backup, to be able to do the test properly?

Hi marko

One full day of innodb_log_file_buffering=OFF since last successful backup.

I ran the backup again with the default --log-copy-interval, and got Waiting for backup lock.

MariaBackup runs fine for a while, always, until it reaches a log line that says:

Waiting for log copy thread to read lsn 10609231931267

Until then, it wasn't locking. I notice the same behavior always. Once that log line appears, the locks start happening.

If I use --log-copy-interval=1, I only notice the Waiting for backup lock at the very end of the backup, but it's so fast that it doesn't affect anyone on the website.

–

Anyway – I'll go ahead and keep --log-copy-interval=1, and the default value for innodb_log_file_buffering.

Thank you.

nunop, thank you. Your findings are in line with what I wrote on 2022-06-28. Because of the "One size does not fit all" observation, I do not think that we should change any default parameters.

Because of recovery regressions (MDEV-29374, MDEV-29383), I cannot recommend 10.8.4, as you know. An unscheduled batch of releases including 10.8.5 should become available for download this week.

Thank you marko, and no problem. I understand. And great to know that 10.8.5 is coming this week!

I guess the only problem outstanding that I have is the swap issue described in MDEV-29097.
I can see if that improves in 10.8.5, but based on the release notes, it probably will not - https://mariadb.com/kb/en/mariadb-1085-release-notes/

Thanks very much!

MariaDB 10.8.5 doesn't solve this; I've saw this ticket after, so I've created MDEV-29631 (feel free to close it if it is duplicate)

acs-ferreira - have you tried to add --log-copy-interval=1 to the command? It resolved the issue for me, and sped up the backup time as well.

Ive tried it. It blocks here :

[00] 2022-09-25 22:22:27 Waiting for log copy thread to read lsn 59357301845

[00] 2022-09-25 22:22:27 >> log scanned up to (59357301845)

[00] 2022-09-25 22:22:27 >> log scanned up to (59357301845)

[00] 2022-09-25 22:22:27 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...

[00] 2022-09-25 22:22:27 mariabackup: The latest check point (for incremental): '57455996308'

mariabackup: Stopping log copying thread.[00] 2022-09-25 22:22:27 >> log scanned up to (59357301845)

For me too, but it's for just 1 second or so. No queries timeout and users don't notice.
Is it locking for longer, to you? (or, 1 second not being acceptable to your usage)

I've stopped the process after ~20sec of locking. We're talking about a nvme disk + 256Gb RAM on a 8Gb dataset (this is a test server).

This has to be a bug/major regression somewhere as the same dataset on a 10.4 server has no such behavior.

Ah ok, that's very strange then!

Yeah, I was on 10.5 before, also in an NVMe with 128GB RAM and never had a problem.

I also started getting these locks (and the website down for several minutes due to everything locking), but adding --log-copy-interval=1 resolved it for me.

One thing I know is that MariaDB says that:

If you would like to use a version of Mariabackup that uses the BACKUP STAGE commands in an efficient way, then one option is to use MariaDB Enterprise Backup that is bundled with MariaDB Enterprise Server.

https://mariadb.com/kb/en/backup-stage/

Have you tried using mariabackup from 10.8.2 and --log-copy-interval=1000000 ?

For me this worked.

acs-ferreira, in the recently released 10.8.5, would SET GLOBAL innodb_log_file_buffering=OFF work for you? (Note: I recommend avoiding 10.8.4; see MDEV-29364.)

cocorossello, thank you for the idea to try the 10.8.2 mariadb-backup for the --backup part. Does it work for you even when the server runs with SET GLOBAL innodb_log_file_buffering=ON?

On Linux or Windows, a later version of mariadb-backup would likely attempt to read the server’s log file in O_DIRECT mode. Because the backup operation is basically polling the log file for updates, it could make sense to open that file in buffered mode, so that if nothing was changed, the unchanged data would be returned from the file system cache. Currently, there is no way to specify innodb_log_file_buffering=OFF for the backup process. An even better solution might be to invoke the Linux system call ppoll(2) or poll(2) in the polling loop, to only read from the file when it has been changed.

mysql -e "SET GLOBAL innodb_log_file_buffering=OFF"

mariabackup --backup --log-copy-interval=1 ....

mysql -e "SET GLOBAL innodb_log_file_buffering=ON"

So setting both

--log-copy-interval=1

and

innodb_log_file_buffering=OFF

works fine on 10.8.5. Missing either option results in long blocking.

acs-ferreira, thank you, this like I expected. I did not debug mariadb-backup yet, so I do not know for sure if it would read the server’s ib_logfile0 via O_DIRECT in this case, but I assume that it would. I think that the 1-millisecond polling interval that worked for you can cause trouble for others who have different type of storage.

Could you please also test the 10.8.5 server with the default innodb_log_buffering=OFF and a copy of mariadb-backup from MariaDB Server 10.8.2, for executing the --backup part, like cocorossello suggested? In that version, the log copying interval was incorrectly specified in microseconds instead of milliseconds (MDEV-27919), so you have to specify it as 1000000 to get the intended default value of 1000 milliseconds (1 second).

I’d appreciate it if all of you (acs-ferreira, nunop, cocorossello) could conduct this experiment.

Marko, where can I find mariadb-backup from MariaDB Server 10.8.2 binary for aarch64?

acs-ferreira, I did not get any response from the maintainers of the package repositories yet. I see that the web UI only offers the latest version in each series. Which package or GNU/Linux distribution are you using? It should be possible to get the package straight from a repository mirror. I successfully did that to get some older .deb or .rpm for analyzing some core dumps or checking the addresses in some stack traces.

You should find packages of all types in https://archive.mariadb.org/mariadb-10.8.2/. For example for Debian, the .deb files for all CPU architectures and supported Debian releases are in https://archive.mariadb.org/mariadb-10.8.2/repo/debian/pool/main/m/mariadb-10.8/.

Ubuntu 22.04 Jammy. Can't find the binary at https://archive.mariadb.org/mariadb-10.8.2/repo/ubuntu/pool/main/m/mariadb-10.8/

One of these packages, maybe?

mariadb-backup_10.8.2+maria~bionic_amd64.deb 6.3 MiB 2022-Feb-12 03:42
mariadb-backup_10.8.2+maria~bionic_arm64.deb 5.9 MiB 2022-Feb-12 03:48
mariadb-backup_10.8.2+maria~bionic_ppc64el.deb 6.4 MiB 2022-Feb-12 03:43
mariadb-backup_10.8.2+maria~focal_amd64.deb 6.3 MiB 2022-Feb-12 03:51
mariadb-backup_10.8.2+maria~focal_arm64.deb 5.9 MiB 2022-Feb-12 03:54
mariadb-backup_10.8.2+maria~focal_ppc64el.deb 6.4 MiB 2022-Feb-12 03:52
mariadb-backup_10.8.2+maria~focal_s390x.deb 5.8 MiB 2022-Feb-12 03:57
mariadb-backup_10.8.2+maria~impish_amd64.deb

None of them are for Jammy. I can find such packages on the 10.8.3+ branch, maybe before Jammy packages where not available?

Correct, Ubuntu 22.04 GA was after MariaDB Community Server 10.8.2 was released

Thanks Ralf. So i'm sorry but I can't test your case Marko. My only staging machine runs on Jammy+ARM64.

I could provide a custom build once I get to implement this. I would first like to hear from nunop and cocorossello and anyone else if disabling O_DIRECT in mariadb-backup --backup (but enabling it in the server) would be the right fix. That is the purpose of the 10.8.2 experiment.

I'm testing this now. If I understand you want me to test with 10.8.5 in the server and 10.8.2 in the backup. Something like:

  echo "apt update; apt install pigz; mariabackup --backup --log-copy-interval=1000000 --host="mariadb-host" --parallel=4 --stream=xbstream  --target-dir /backup/dump --user xxxx --password xxxx --use-memory=12288M | pigz -p 4 > /backup/dump/mariadbbackup-2022-10-04.gz" | docker run --name mariabackup -v /services/mysql/data/mysql:/var/lib/mysql -v /opt/backups/mariadb/dump/:/backup/dump -v /services/mysql/conf:/etc/mysql/conf.d:ro --rm -i --entrypoint sh mariadb:10.8.2