[MDEV-26110] mtr failure innodb.innodb_scrub (assertion) aix RelWithDebug build Created: 2021-07-08  Updated: 2022-05-04  Resolved: 2021-07-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5, 10.6
Fix Version/s: 10.5.12, 10.6.4

Type: Bug Priority: Major
Reporter: Daniel Black Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None
Environment:

AIX

Issue Links:
Problem/Incident
causes MDEV-28473 field_ref_zero is not initialized in ... Closed

 Description   

EGuesnet can I get a hand with this please. Its constantly failing and with standard kind of debugging I've got a corrupted stack trace.

As its an assertion its not just a test failure.
https://buildbot.mariadb.org/#/builders/121/builds/1585/steps/6/logs/stdio

Also reproducable on the CMAKE_BUILD_TYPE=Debug

$ mysql-test/mtr --manual-gdb innodb.innodb_scrub 
Logging: /home/mdborg/mariadb-server/mysql-test/mysql-test-run.pl  --manual-gdb innodb.innodb_scrub
 
vardir: /home/mdborg/build-mariadb-server/mysql-test/var
 
Checking leftover processes...
 
Removing old var directory...
 
Creating var directory '/home/mdborg/build-mariadb-server/mysql-test/var'...
 
Checking supported features...
 
2021-07-08  2:06:40 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
 
2021-07-08  2:06:40 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
 
2021-07-08  2:06:40 0 [Warning] WSREP: Guessing address for incoming client connections failed. Try setting wsrep_node_incoming_address explicitly.
 
2021-07-08  2:06:40 0 [Note] WSREP: Node addr: 
MariaDB Version 10.5.12-MariaDB
 
 - SSL connections supported
 
 - binaries built with wsrep patch
 
Collecting tests...
 
Installing system database...
 
 
 
==============================================================================
 
 
 
TEST                                      RESULT   TIME (ms) or COMMENT
 
--------------------------------------------------------------------------
 
 
 
worker[1] Using MTR_BUILD_THREAD 800, with reserved ports 26000..26019
 
 
 
To start gdb for mysqld.1, type in another window:
 
gdb -x /home/mdborg/build-mariadb-server/mysql-test/var/tmp/gdbinit.mysqld.1   /home/mdborg/build-mariadb-server/sql/mariadbd
 



gdb -x /home/mdborg/build-mariadb-server/mysql-test/var/tmp/gdbinit.mysqld.1   /home/mdborg/build-mariadb-server/sql/mariadbd
 
GNU gdb (GDB) 8.1.1
 
Copyright (C) 2018 Free Software Foundation, Inc.
 
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 
This is free software: you are free to change and redistribute it.
 
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 
and "show warranty" for details.
 
This GDB was configured as "powerpc64-ibm-aix6.1.0.0".
 
Type "show configuration" for configuration details.
 
For bug reporting instructions, please see:
 
<http://www.gnu.org/software/gdb/bugs/>.
 
Find the GDB manual and other documentation resources online at:
 
<http://www.gnu.org/software/gdb/documentation/>.
 
For help, type "help".
 
Type "apropos word" to search for commands related to "word"...
 
Reading symbols from /home/mdborg/build-mariadb-server/sql/mariadbd...done.
 
(gdb) run
 
Starting program: /home/mdborg/build-mariadb-server/sql/mariadbd --defaults-group-suffix=.1 --defaults-file=/home/mdborg/build-mariadb-server/mysql-test/var/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --innodb-immediate-scrub-data-uncompressed=ON --core-file --loose-debug-sync-timeout=300 --loose-gdb < /dev/null
 
BFD: BFD (GNU Binutils) 2.29.51 assertion fail format.c:453
 
warning: "/usr/lib/libdl.a": member "shr_64.o" missing.
 
warning: Could not load shared library symbols for /usr/lib/libdl.a(shr_64.o).
 
Do you need "set solib-search-path" or "set sysroot"?
 
[New Thread 1]
 
2021-07-08  2:07:15 0 [Note] /home/mdborg/build-mariadb-server/sql/mariadbd (mysqld 10.5.12-MariaDB-log) starting as process 13172792 ...
 
[New Thread 258]
 
[New Thread 515]
 
[New Thread 772]
 
[New Thread 1030]
 
[New Thread 1287]
 
[New Thread 1544]
 
[New Thread 1801]
 
[New Thread 2058]
 
[New Thread 2315]
 
[New Thread 2572]
 
[New Thread 2829]
 
[New Thread 3086]
 
[New Thread 3343]
 
[New Thread 3600]
 
[New Thread 3857]
 
[New Thread 4114]
 
[New Thread 4371]
 
[New Thread 4628]
 
[New Thread 4885]
 
[New Thread 5142]
 
[New Thread 5399]
 
[New Thread 5656]
 
[New Thread 5913]
 
[New Thread 6170]
 
[New Thread 6427]
 
[New Thread 6684]
 
[New Thread 6941]
 
[New Thread 7198]
 
[New Thread 7455]
 
[New Thread 7712]
 
[New Thread 7969]
 
 
 
Thread 2 received signal SIGABRT, Aborted.
 
[Switching to Thread 1]
 
0x0000000000000000 in ?? ()
 
(gdb) bt
 
#0  0x0000000000000000 in ?? ()
 
(gdb) bt full
 
#0  0x0000000000000000 in ?? ()
 
No symbol table info available.

notes on other aix errors:

  • main.long_unique / main.query_cache memory rlimit issue that I'll correct shortly
  • plugins.server_audit / innodb.temporary_table / type_inet.type_inet6 / type_test.type_test_double - probably something simple however a non-functional debugging environment makes it hard to track down.

 Comments   
Comment by Etienne Guesnet [ 2021-07-08 ]

This fail occurs since version 10.5.9 according my backlogs.

I can reproduce it. I have the following backtrace from core:

#0  0x0900000000536f94 in pthread_kill () from /usr/lib/libpthreads.a(shr_xpg5_64.o)
 
#1  0x09000000054cdaf4 in my_write_core () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#2  0x09000000054cd3a0 in handle_fatal_signal () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#3  <signal handler called>
 
#4  0x0900000000536f94 in pthread_kill () from /usr/lib/libpthreads.a(shr_xpg5_64.o)
 
#5  0x09000000005367e8 in _p_raise () from /usr/lib/libpthreads.a(shr_xpg5_64.o)
 
#6  0x0900000000039c6c in raise () from /usr/lib/libc.a(shr_64.o)
 
#7  0x0900000000056668 in abort () from /usr/lib/libc.a(shr_64.o)
 
#8  0x090000000493786c in ut_dbg_assertion_failed(char const*, char const*, unsigned int) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#9  0x0900000004b31d68 in os_aio(IORequest const&, void*, unsigned long, unsigned long) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#10 0x0900000004a52b8c in fil_space_t::io(IORequest const&, unsigned long, unsigned long, void*, buf_page_t*) ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#11 0x0900000004a2aa90 in buf_flush_freed_pages(fil_space_t*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#12 0x0900000004a2b69c in buf_flush_space(unsigned int) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#13 0x0900000004a2be70 in buf_flush_LRU_list_batch(unsigned long, flush_counters_t*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#14 0x0900000004a2c3f8 in buf_do_LRU_batch(unsigned long) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#15 0x0900000004a2d480 in buf_flush_lists(unsigned long, unsigned long) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#16 0x09000000049e7b60 in buf_LRU_get_free_block(bool) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#17 0x0900000004974864 in buf_page_init_for_read(unsigned long, page_id_t, unsigned long, bool) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#18 0x0900000004975324 in buf_read_page_low(dberr_t*, fil_space_t*, bool, unsigned long, page_id_t, unsigned long, bool) ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#19 0x0900000004975ed4 in buf_read_page(page_id_t, unsigned long) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#20 0x09000000049fab04 in buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool) ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#21 0x09000000049fc33c in buf_page_get_gen(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool) ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#22 0x0900000004d070f0 in trx_rsegf_get(fil_space_t*, unsigned int, mtr_t*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#23 0x0900000004d092a4 in trx_purge_truncate_rseg_history(trx_rseg_t&, purge_sys_t::iterator const&) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#24 0x0900000004d09be4 in trx_purge_truncate_history() () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#25 0x0900000004d0d018 in trx_purge(unsigned long, bool) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#26 0x0900000004997e30 in srv_do_purge(unsigned long*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#27 0x0900000004998684 in purge_coordinator_callback_low() () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#28 0x090000000499885c in purge_coordinator_callback(void*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#29 0x0900000004b43854 in tpool::task_group::execute(tpool::task*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#30 0x0900000004b3420c in tpool::task::execute() () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#31 0x0900000004b35da0 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#32 0x0900000004b40bf4 in void std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(std::__invoke_memfun_deref, void (tpool::thread_pool_generic::*&&)(tpool::worker_data*), tpool::thread_pool_generic*&&, tpool::worker_data*&&) () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#33 0x0900000004b409f0 in std::__invoke_result<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>::type std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(void (tpool::thread_pool_generic::*&&)(tpool::worker_data*), tpool::thread_pool_generic*&&, tpool::worker_data*&&)
 
    () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#34 0x0900000004b3fae4 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#35 0x0900000004b3f9b0 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator()() ()
 
   from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#36 0x0900000004b3f8d8 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run() () from /opt/freeware/src/packages/BUILD/mariadb-10.5.9/debug/sql/libserver.a(libserver.so)
 
#37 0x0900000006befddc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/freeware/lib/pthread/ppc64/libstdc++.a(libstdc++.so.6)
 
#38 0x090000000051ee14 in _pthread_body () from /usr/lib/libpthreads.a(shr_xpg5_64.o)
 
#39 0x0000000000000000 in ?? ()

2 threads. No backtrace for the second, only

aix-thread: ptrace (52, 70713541) returned -1 (errno = 3 No such process)

If I correctly understand the error, it fails at storage/innobase/os/os0file.cc:4141

ut_a(reinterpret_cast<size_t>(cb->m_buffer) % OS_FILE_LOG_BLOCK_SIZE== 0);

with OS_FILE_LOG_BLOCK_SIZE = 512U.

The test mysql-test/suite/innodb/t/innodb_scrub.test repeats 500 times an insert, then commits and does some stuff.

I do not understand the reason of the error as I see in log debug/mysql-test/var/log/innodb.innodb_scrub-innodb/mysqld.1/mysqld.log:

4 Query    INSERT INTO t1(f2,f3) VALUES ('repairman', repeat('unicycle', 1000))

500 times. Last command is

4 Query    SET GLOBAL innodb_max_purge_lag_wait= 0

that seems OK.

It is the first time I investigate this error, and I do not understand where it is from.

Comment by Daniel Black [ 2021-07-09 ]

I've been trying to bisect where it came into today without much luck.

A p *cb at the point of assertion may provide some hints. Its the buf arg at the breakpoint fil_space_t::io. Does block_size (global get get 512 multiple size?

Comment by Daniel Black [ 2021-07-09 ]

Tips most welcome as to what I need to get a decent backtrace on AIX.

Cmake-3.16 is problematic and the LIBPATH solution is hackish (which is what I've done in buildbot). I'm investigating a cmake upgrade if this is related. Failing that will look at a downgrade.

Comment by Etienne Guesnet [ 2021-07-12 ] 

I've been trying to bisect where it came into today without much luck.

The same. I have proceed from 10.6.1 to avoid rebuild all, and I have the following :

$ /opt/freeware/bin/perl_64 ./mysql-test-run.pl --force --suite-timeout=900 --testcase-timeout=30 --force-restart --shutdown-timeout=60 --max-test-fail=9999 --ssl innodb_scrub --manual-gdb
 
$ # In another terminal
 
$ gdb -x /opt/freeware/src/packages/BUILD/mariadb-10.6.1/debug/mysql-test/var/tmp/gdbinit.mysqld.1   /opt/freeware/src/packages/BUILD/mariadb-10.6.1/debug/sql/mariadbd


(gdb) b  storage/innobase/os/os0file.cc:3857
 
No source file named storage/innobase/os/os0file.cc.
 
Make breakpoint pending on future shared library load? (y or [n]) y
 
Breakpoint 1 (storage/innobase/os/os0file.cc:3857) pending.
 
(gdb) r
 
Starting program: /opt/freeware/src/packages/BUILD/mariadb-10.6.1/debug/sql/mariadbd --defaults-group-suffix=.1 --defaults-file=/opt/freeware/src/packages/BUILD/mariadb-10.6.1/debug/mysql-test/var/my.cnf --user=root --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --innodb-immediate-scrub-data-uncompressed=ON --core-file --loose-debug-sync-timeout=300 --loose-gdb < /dev/null
 
[New Thread 1]
 
 
 
[...]
 
 
 
Thread 16 hit Breakpoint 1, os_aio (type=..., buf=0x11574f250, offset=4654953128, n=720575940395319296) at /opt/freeware/src/packages/BUILD/mariadb-10.6.1/storage/innobase/os/os0file.cc:3857
 
3857            ut_a(reinterpret_cast<size_t>(cb->m_buffer) % OS_FILE_LOG_BLOCK_SIZE
 
(gdb) c 295
 
Will ignore next 294 crossings of breakpoint 1.  Continuing.
 
[New Thread 1040]
 
[New Thread 1297]
 
[New Thread 1554]
 
[New Thread 1299]
 
[New Thread 1556]
 
[New Thread 1557]
 
[New Thread 1814]
 
[New Thread 2071]
 
[Switching to Thread 1814]
 
 
 
Thread 23 hit Breakpoint 1, os_aio (type=..., buf=0x1153169a0, offset=4650527224, n=720575940393353216) at /opt/freeware/src/packages/BUILD/mariadb-10.6.1/storage/innobase/os/os0file.cc:3857
 
3857            ut_a(reinterpret_cast<size_t>(cb->m_buffer) % OS_FILE_LOG_BLOCK_SIZE
 
(gdb) p cb
 
$1 = (aiocb *) 0x1145d2830
 
(gdb) p *cb
 
$2 = {m_fh = 12, m_opcode = AIO_PWRITE, m_offset = 491520, *m_buffer = 0xa00000000d6c000*, m_len = 16384, m_callback = @0x8001000a04d01b0: 0x900000008338ba0 <io_callback(tpool::aiocb*)>,
 
  m_group = 0x1145c2958, m_ret_len = 16384, m_err = 0, m_internal = 0x8f8f8f8f8f8f8f8f, m_internal_task = warning: can't find linker symbol for virtual table for `task' value
 
{_vptr.task = 0x8001000a041fbb0 <_ZTVN5tpool4taskE+16>,
 
    m_func = @0x8001000a04cf238: 0x900000008356068 <tpool::simulated_aio::simulated_aio_callback(void*)>, m_arg = 0x1145d2830, m_group = 0x1145c2958},
 
  m_userdata = "\n\000\000\000\000\246|@\000\000\000\001\024t`\250\000\000\000\061\000\000\000"}
 
(gdb) p (size_t) cb->m_buffer
 
$3 = 720575940393353216
 
(gdb) p (size_t) cb->m_buffer % 512
 
$4 = 0
 
(gdb) c
 
Continuing.
 
[New Thread 1048]
 
 
 
Thread 23 hit Breakpoint 1, os_aio (type=..., buf=0x1153168d0, offset=4650527016, n=648518346494843680) at /opt/freeware/src/packages/BUILD/mariadb-10.6.1/storage/innobase/os/os0file.cc:3857
 
3857            ut_a(reinterpret_cast<size_t>(cb->m_buffer) % OS_FILE_LOG_BLOCK_SIZE
 
(gdb) p (size_t) cb->m_buffer % 512
 
$5 = 288
 
(gdb) p *cb
 
$6 = {m_fh = 10, m_opcode = AIO_PWRITE, m_offset = 5062656, m_buffer = 0x900000009261b20 <field_ref_zero>, m_len = 16384,
 
  m_callback = @0x8001000a04d01b0: 0x900000008338ba0 <io_callback(tpool::aiocb*)>, m_group = 0x1145c2958, m_ret_len = 16384, m_err = 0, m_internal = 0x8f8f8f8f8f8f8f8f, m_internal_task = warning: can't find linker symbol for virtual table for `task' value
 
{
 
    _vptr.task = 0x8001000a041fbb0 <_ZTVN5tpool4taskE+16>, m_func = @0x8001000a04cf238: 0x900000008356068 <tpool::simulated_aio::simulated_aio_callback(void*)>, m_arg = 0x1145d2830,
 
    m_group = 0x1145c2958}, m_userdata = '\000' <repeats 11 times>, "\001\024h\341\350\000\000\000\021\000\000\000\002"}
 
(gdb)

The value 295 in continue was found by manual search.

Buffer value seems to be kind of field_ref_zero (that refers to storage/innobase/buf/buf0buf.cc, function buf_is_zeroes).
We never had trouble with memcmp function.

I do not have idea about where this bug is from.

Cmake-3.16 is problematic and the LIBPATH solution is hackish (which is what I've done in buildbot). I'm investigating a cmake upgrade if this is related. Failing that will look at a downgrade.

LIBPATH is a know bug. Trouble between pthread and not pthread stdlib is a common trouble on AIX. I will speak with our GCC expert (he is currently on vacation). As far as I know, it is not corrected in higher version.
CMake higher than 3.16 has not been distributed by Bullfreeware (my team) or AIX Toolbox (different of AIXTools team!, i), and we distribute a patched CMake. I do not know if MariaDB can be build with vanilla CMake. Recent CMake (3.20) are broken on AIX: the whole plugin system does not work when I have checked. Unfortunately, I have not spent time to work on.
Link to forum is about rootvg, so AIXTools. M. Felt provides some AIX packages, but they can be incompatible with ours. I do not recommend to mix opensource package source.

Comment by Daniel Black [ 2021-07-15 ]

Thanks EGuesnet. Sounds like the cmake version isn't the case of my lack of symbols in backtracking so if I'm missing essential build options on https://buildbot.mariadb.org/#/builders/121 I'm keen to know.

Thanks for the debug info here.

Yes I'm trying to be careful with these packages (saved by environment paths so far) but will clean them up. Thanks for clearing up my knowlege of AIX terms.

Comment by Etienne Guesnet [ 2021-07-16 ]

Your options seem good.
I have other priorities right now, but I'll let you know as soon as I have news.

Comment by Marko Mäkelä [ 2021-07-21 ]

In the file log0log.h, I remember that we had to replace alignas with MY_ALIGNED because the 128-byte cache line size on s390x would be too large for alignas according to GCC 5. In the GDB output above, I see that the address of field_ref_zero is only aligned to 32 bytes, even though the code requested 1024-byte alignment. (The assertion expects 512-byte alignment.) Would the following work?

diff --git a/storage/innobase/page/page0zip.cc b/storage/innobase/page/page0zip.cc
 
index 1ed9efafc69..1bc8fa64a2d 100644
 
--- a/storage/innobase/page/page0zip.cc
 
+++ b/storage/innobase/page/page0zip.cc
 
@@ -38,7 +38,7 @@ using st_::span;
 
 /** A BLOB field reference full of zero, for use in assertions and tests.
 
 Initially, BLOB field references are set to zero, in
 
 dtuple_convert_big_rec(). */
 
-alignas(UNIV_PAGE_SIZE_MIN)
 
+MY_ALIGNED(UNIV_PAGE_SIZE_MIN)
 
 const byte field_ref_zero[UNIV_PAGE_SIZE_MAX] = { 0, };
 
 
 #ifndef UNIV_INNOCHECKSUM

Comment by Etienne Guesnet [ 2021-07-21 ]

The test fails again with this change.

Comment by Daniel Black [ 2021-07-21 ]

Thanks Etienne, i found that too, just didn't update this quick enough.

Marko notes:

Come to think of it, that ut_a on alignment can be disabled on any platform that does not support O_DIRECT. If s390x would silently ignore the MY_ALIGNED then scrubbing should be broken there as well. But Linux on s390x would actually support O_DIRECT.

But, if we now know that MY_ALIGNED does not work, then maybe we should use aligned_malloc for these as well:

extra/mariabackup/xtrabackup.cc:        byte MY_ALIGNED(OS_FILE_LOG_BLOCK_SIZE) log_hdr_buf[LOG_FILE_HDR_SIZE];
 
storage/innobase/include/log0log.h:     MY_ALIGNED(OS_FILE_LOG_BLOCK_SIZE)
 
storage/innobase/log/log0log.cc:  MY_ALIGNED(OS_FILE_LOG_BLOCK_SIZE) byte buf[OS_FILE_LOG_BLOCK_SIZE] = {0};

I'll look at this tomorrow

Comment by Marko Mäkelä [ 2021-07-22 ]

I will take over this. I suspect that also s390x on Linux may be affected, and the problems should not be limited to scrubbing.

Possibly the difference between alignas and MY_ALIGNED is that the latter may be silently ignored by the compiler when the requested alignment cannot be guaranteed. (On Debian GNU/Linux on s390x, GCC 5 complained about alignas, but MY_ALIGNED was fine. We did not check the actual alignment.)

10.5 might appear to work, because it did not yet default to innodb_flush_method=O_DIRECT (MDEV-25121). I do not know if we support O_DIRECT on AIX, or whether AIX would allow unaligned buffers to be used with it.

Comment by Marko Mäkelä [ 2021-07-22 ]

Apart from the special 64KiB all-zero buffer field_ref_zero, which should now be 4KiB-aligned, there were a few 512-byte aligned log buffers related to ib_logfile0, which were previously allocated statically or from the stack. Now we use guaranteed aligned heap memory allocation for those as well.

Comment by Etienne Guesnet [ 2021-07-26 ]

Hi @marko,
If you have a pull request available, I will try to backport it on 10.6.3 for MariaDB users on AIX (I distribute packages).
Moreover, you say 10.5 might appear to work, but I am able to reproduce the bug in 10.5.9. I am checking more recent version.

Comment by Daniel Black [ 2021-07-26 ]

10.6.4 is coming out very soon.

commit is - https://github.com/MariaDB/server/commit/82d5994520c239da1b6edf1b24e08138ae0c753d

10.5 - test with mtr -mysqld=-innodb_flush_method=O_DIRECT --suite=innodb

Comment by Etienne Guesnet [ 2021-07-26 ]

I confirm all is OK with the commit.
I will wait Friday and official release before distributing it.
Thanks.

Generated at Thu Feb 08 09:42:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.