[MDEV-20430] SST transfer issue with rsync Created: 2019-08-27  Updated: 2019-09-23

Status: Open
Project: MariaDB Server
Component/s: Galera, Galera SST, Storage Engine - InnoDB
Affects Version/s: 10.2.26
Fix Version/s: 10.2

Type: Bug Priority: Major
Reporter: Franky Yustanto Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Centos 7.6



 Description   

Hello,

I started configure galera cusltering on production server on different data center.
i started with 3 node, but when starting node #2, i got following issue.
It seems that issue on transfering the database on SST process, FYI database size currently arroung 3GB.
Here is the log that i receive from joiner node :

Aug 26 17:59:22 exploregreenline sh: This could be because you hit a bug. It is also possible that this binary
Aug 26 17:59:22 exploregreenline sh: or one of the libraries it was linked against is corrupt, improperly built,
Aug 26 17:59:22 exploregreenline sh: or misconfigured. This error can also be caused by malfunctioning hardware.
Aug 26 17:59:22 exploregreenline sh: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Aug 26 17:59:22 exploregreenline sh: We will try our best to scrape up some info that will hopefully help
Aug 26 17:59:22 exploregreenline sh: diagnose the problem, but since we have already crashed,
Aug 26 17:59:22 exploregreenline sh: something is definitely wrong and this may fail.
Aug 26 17:59:22 exploregreenline sh: Server version: 10.2.26-MariaDB
Aug 26 17:59:22 exploregreenline sh: key_buffer_size=134217728
Aug 26 17:59:22 exploregreenline sh: read_buffer_size=131072
Aug 26 17:59:22 exploregreenline sh: max_used_connections=0
Aug 26 17:59:22 exploregreenline sh: max_threads=153
Aug 26 17:59:22 exploregreenline sh: thread_count=0
Aug 26 17:59:22 exploregreenline sh: It is possible that mysqld could use up to
Aug 26 17:59:22 exploregreenline sh: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K bytes of memory
Aug 26 17:59:22 exploregreenline sh: Hope that's ok; if not, decrease some variables in the equation.
Aug 26 17:59:22 exploregreenline sh: Thread pointer: 0x0
Aug 26 17:59:22 exploregreenline sh: Attempting backtrace. You can use the following information to find out
Aug 26 17:59:22 exploregreenline sh: where mysqld died. If you see no messages after this, something went
Aug 26 17:59:22 exploregreenline sh: terribly wrong...
Aug 26 17:59:22 exploregreenline sh: stack_bottom = 0x0 thread_stack 0x49000
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dfb513b22e]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x55dfb4bc01dd]
Aug 26 17:59:22 exploregreenline sh: sigaction.c:0(__restore_rt)[0x7f779cfe45d0]
Aug 26 17:59:22 exploregreenline sh: :0(__GI_raise)[0x7f779b2b72c7]
Aug 26 17:59:22 exploregreenline sh: :0(__GI_abort)[0x7f779b2b89b8]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x97f883)[0x55dfb4ec2883]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x42fae6)[0x55dfb4972ae6]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0xa357b6)[0x55dfb4f787b6]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x9e64bd)[0x55dfb4f294bd]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x87f8b1)[0x55dfb4dc28b1]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x9369c0)[0x55dfb4e799c0]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x81bef1)[0x55dfb4d5eef1]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x55dfb4bc29d4]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x4f6810)[0x55dfb4a39810]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x9a2)[0x55dfb4a3b0e2]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x450ff1)[0x55dfb4993ff1]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1e59)[0x55dfb4999449]
Aug 26 17:59:22 exploregreenline sh: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f779b2a3495]
Aug 26 17:59:22 exploregreenline sh: /usr/sbin/mysqld(+0x44a3cd)[0x55dfb498d3cd]
Aug 26 17:59:22 exploregreenline sh: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 26 17:59:22 exploregreenline sh: information that should help you find out what is causing the crash.
Aug 26 17:59:22 exploregreenline sh: Writing a core file...
Aug 26 17:59:22 exploregreenline sh: Working directory at /var/lib/mysql
Aug 26 17:59:22 exploregreenline sh: Resource Limits:
Aug 26 17:59:22 exploregreenline sh: Limit Soft Limit Hard Limit Units
Aug 26 17:59:22 exploregreenline sh: Max cpu time unlimited unlimited seconds
Aug 26 17:59:22 exploregreenline sh: Max file size unlimited unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max data size unlimited unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max stack size 8388608 unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max core file size 0 unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max resident set unlimited unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max processes 15075 15075 processes
Aug 26 17:59:22 exploregreenline sh: Max open files 65536 65536 files
Aug 26 17:59:22 exploregreenline sh: Max locked memory 65536 65536 bytes
Aug 26 17:59:22 exploregreenline sh: Max address space unlimited unlimited bytes
Aug 26 17:59:22 exploregreenline sh: Max file locks unlimited unlimited locks
Aug 26 17:59:22 exploregreenline sh: Max pending signals 15075 15075 signals
Aug 26 17:59:22 exploregreenline sh: Max msgqueue size 819200 819200 bytes
Aug 26 17:59:22 exploregreenline sh: Max nice priority 0 0
Aug 26 17:59:22 exploregreenline sh: Max realtime priority 0 0
Aug 26 17:59:22 exploregreenline sh: Max realtime timeout unlimited unlimited us
Aug 26 17:59:22 exploregreenline sh: Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
Aug 26 17:59:22 exploregreenline sh: /usr/bin/galera_recovery: line 71: 16509 Aborted /usr/sbin/mysqld --user=mysql --wsrep_recover --disable-log-error'
Aug 26 17:59:22 exploregreenline systemd: mariadb.service: control process exited, code=exited status=1
Aug 26 17:59:22 exploregreenline systemd: Failed to start MariaDB 10.2.26 database server.
Aug 26 17:59:22 exploregreenline systemd: Unit mariadb.service entered failed state.
Aug 26 17:59:22 exploregreenline systemd: mariadb.service failed.

Thank you in advance.

Regards,
Franky Yu



 Comments   
Comment by Elena Stepanova [ 2019-09-23 ]

Assuming I got the right binary,

(__GI_raise)[0x7f779b2b72c7]
(__GI_abort)[0x7f779b2b89b8]
/usr/sbin/mysqld(0x97f883 ib::fatal::~fatal() + 67)[0x55dfb4ec2883]
/usr/sbin/mysqld(0x42fae6 fil_report_invalid_page_access(unsigned long, unsigned long, char const*, unsigned long, unsigned long, bool) + 319)[0x55dfb4972ae6]
/usr/sbin/mysqld(0xa357b6 fil_io(IORequest const&, bool, page_id_t, page_size_t const&, unsigned long, unsigned long, void*, void*, bool) + 2486)[0x55dfb4f787b6]
/usr/sbin/mysqld(0x9e64bd buf_read_recv_pages(bool, unsigned long, unsigned long const*, unsigned long) + 909)[0x55dfb4f294bd]
/usr/sbin/mysqld(0x87f8b1 recv_apply_hashed_log_recs(bool) + 4513)[0x55dfb4dc28b1]
/usr/sbin/mysqld(0x9369c0 innobase_start_or_create_for_mysql() + 9008)[0x55dfb4e799c0]
/usr/sbin/mysqld(0x81bef1 innobase_init(void*) + 4753)[0x55dfb4d5eef1]
/usr/sbin/mysqld(ha_initialize_handlerton(st_plugin_int*)+0x64)[0x55dfb4bc29d4]
/usr/sbin/mysqld(0x4f6810 plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool) + 352)[0x55dfb4a39810]
/usr/sbin/mysqld(plugin_init(int*, char**, int)+0x9a2)[0x55dfb4a3b0e2]
/usr/sbin/mysqld(0x450ff1 init_server_components() + 1153)[0x55dfb4993ff1]
/usr/sbin/mysqld(mysqld_main(int, char**)+0x1e59)[0x55dfb4999449]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f779b2a3495]
/usr/sbin/mysqld(0x44a3cd _start + 41)[0x55dfb498d3cd]

Generated at Thu Feb 08 08:59:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.