[MDEV-24922] 10.5.8 fails to run with GLIBC 2.32 and 2.33 Created: 2021-02-19  Updated: 2021-04-19  Resolved: 2021-02-23

Status: Closed
Project: MariaDB Server
Component/s: N/A
Affects Version/s: 10.5.8
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Noel Assignee: Daniel Black
Resolution: Not a Bug Votes: 0
Labels: crash
Environment:

linux kernel 5.10.16-smp 32 bit,GLIBC2.33
slackware -current

Attachments: HTML File FFFFFFFF     HTML File GGGGGGGG    
Issue Links:
Duplicate
duplicates MDEV-24745 Fallback CRC-32C computation wrongly ... Closed

 Description   

mariadb fails on startup with latest glibc 2.32/2.33

The exact same version (10.5.8) ran fine on older GLIBC, prior to all packages rebuilt for latest glibc 2.33 and 2.32, everything worked fine on glibc 2.30 prior to that.

Do you use lchmod? This is identical problem with rsync where Wayne D, says lchmod is used in latest glibc and may be cause.

2021-02-16 17:37:16 0 [Note] InnoDB: Using Linux native AIO
2021-02-16 17:37:16 0 [Note] InnoDB: Uses event mutexes
2021-02-16 17:37:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-02-16 17:37:16 0 [Note] InnoDB: Number of pools: 1
2021-02-16 17:37:16 0 [Note] InnoDB: Using generic crc32 instructions
2021-02-16 17:37:16 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-02-16 17:37:16 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-02-16 17:37:16 0 [Note] InnoDB: Completed initialization of buffer pool
2021-02-16 17:37:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
210216 17:37:16 [ERROR] mysqld got signal 4 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.8-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466473 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
??:0(my_print_stacktrace)[0x12b78fe]
??:0(handle_fatal_signal)[0xc1c992]
addr2line: 'linux-gate.so.1': No such file
linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7f16554]
??:0(my_dlerror)[0x12d2180]
??:0(std::unique_lock<std::mutex>::unlock())[0x1166e3c]
??:0(std::unique_lock<std::mutex>::unlock())[0x115c41f]
??:0(std::unique_lock<std::mutex>::unlock())[0x11de6d3]
??:0(std::unique_lock<std::mutex>::unlock())[0x11e2588]
??:0(std::unique_lock<std::mutex>::unlock())[0x11e2b87]
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x83ef98]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0xfcbbdc]
??:0(ha_initialize_handlerton(st_plugin_int*))[0xc1fb2d]
??:0(sys_var_pluginvar::sys_var_pluginvar(sys_var_chain*, char const*, st_plugin_int*, st_mysql_sys_var*))[0x9c30cc]
??:0(plugin_init(int*, char**, int))[0x9c47da]
??:0(unireg_abort)[0x8c056c]
??:0(mysqld_main(int, char**))[0x8c6b35]
??:0(main)[0x883467]
??:0(__libc_start_main)[0xb75ad889]
??:0(_start)[0x8b9811]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15945 15945 processes
Max open files 32184 32184 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15945 15945 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

Note, there is no core file generated.

 Comments   
Comment by Daniel Black [ 2021-02-19 ]

There's no lchmod used in mariadb server.

Are you able to ulimit -c unlimited and get a backtrace from where this occurred?

Or an strace to see if the `std::unique_lock<std::mutex>::unlock` calls resulted in a futex syscall before termination.

Comment by Noel [ 2021-02-19 ]

Not that I can see, but I might be reading it all wrong,
I have attached 2 files

GGGGGGGG is strace std output
FFFFFFFFFF is strace with -f output

Comment by Daniel Black [ 2021-02-19 ]

An extract from FFFFFF (-f is needed to capture the mysqld process rather than the mysqld_safe script).

19347 pread64(9,  <unfinished ...>
 
19351 write(2, "2021-02-19 13:31:19 0 [Note] InnoDB: If the mysqld execution user is authorized,"..., 161) = 161
 
19351 futex_time64(0x197def4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1613705480, tv_nsec=423349000}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
 
19347 <... pread64 resumed>"\2540\352\371\0\0\2N\377\377\377\377\377\377\377\377\0\0\0Nb\274\264\275\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\21\t2\377\377\377\377\0\0\377\377\377\377\0\0\0\2\10W\0\0\0\0\0\0\2\201!2\0\0\0\1\0\0\2N\0,"..., 1048576, 1048576) = 1048576
 
19347 pread64(9, "\376\223;\251\0\0\2\213\377\377\377\377\377\377\377\377\0\0\0Nb\274\241)\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10O\10p\377\377\377\377\0\0\377\377\377\377\0\0\0\2\7\225\0\0\0\0\0\0\2\201\t\362\0\0\0\1\0\0\2\213\0,"..., 1048576, 2097152) = 1048576
 
19347 --- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x1215250} ---

So file descriptor 9 is the ibdata1 file. As you are running the same code as before you upgraded glibc the SIGILL isn't part of mariadb otherwise it would of failed earlier. Note the futex call here never returns. Its quite possible that the futex call in the kernel or glibc has the SIGILL.

Given the duplicate bug references here is also a x86 reference on slackware I'd be looking closely at their build process and bug reports for something similar.

Comment by Noel [ 2021-02-19 ]

OK, i'll pass it back to them, they didnt seem interested before sigh will see what happens now

Thanks Daniel

Comment by Daniel Black [ 2021-02-23 ]

https://sourceware.org/glibc/wiki/Testing/ABI_checker maybe. I'm not sure this calls the APIs but maybe the interface checks is sufficient.
Otherwise

Comment by Daniel Black [ 2021-02-23 ]

Closing NOTABUG but really just NOT-OUR-BUG.

Comment by Daniel Black [ 2021-04-16 ]

nobby6 thanks for your interest in further testing. Development/testing versions produced by our CI are available here http://hasky.askmonty.org/archive/10.5/.
Apologizes for preemptively closing this issue.

Comment by Noel [ 2021-04-18 ]

No problem,s thanks, happy to confirm mariadb started correctly, and has been running now since 30 hours on 32bit dev box.

Comment by Daniel Black [ 2021-04-19 ]

Thanks for confirming nobby6.

Generated at Thu Feb 08 09:33:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.