Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.0.23, 10.1.11
Description
When replicating from our 10.0 server, MariaDB 10.1.11 crashes. After the crash, it will not startup again unless all relay logs are deleted.
Servers running Ubuntu, both with 200G ram.
2016-02-15 15:12:47 140328625293056 [Warning] Slave SQL: Could not execute Write_rows_v1 event on table Voorraadbeheer.InklaarLog; Unknown error, Error_code: 1105; handler error No Error!; the event's master log mysql-bin.056373, end_log_pos 217054452, Gtid 0-765-1299574704, Internal MariaDB error code: 1105
|
2016-02-15 15:12:47 140328625293056 [ERROR] Slave SQL: Could not read field 'Valuta' of table 'Voorraadbeheer.InklaarPrijs', Gtid 0-765-1299574704, Internal MariaDB error code: 1610
|
160215 15:12:47 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.11-MariaDB-1~trusty-log
|
key_buffer_size=16777216
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=502
|
thread_count=1
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1118986 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x7f95a6c2b008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fa0cddcb590 thread_stack 0x80000
|
2016-02-15 15:12:47 140328649512704 [Note] Slave I/O thread: connected to master 'slave@jen.managementboek.nl:3306',replication started in log 'mysql-bin.056376' at position 692280882
|
addr2line: 'mysqld': No such file
|
mysqld(my_print_stacktrace+0x2e)[0x559e045013fe]
|
mysqld(handle_fatal_signal+0x38d)[0x559e0402ef9d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fa0d02e5340]
|
mysqld(_Z10unpack_rowP14rpl_group_infoP5TABLEjPKhPK9st_bitmapPS4_PmS4_+0x89)[0x559e0410f0b9]
|
mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x220)[0x559e0410be80]
|
mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x31c)[0x559e040fe2cc]
|
mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x1e1)[0x559e03e3a2e1]
|
mysqld(handle_slave_sql+0x261b)[0x559e03e3dd1b]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fa0d02dd182]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa0cfa0047d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): is an invalid pointer
|
Connection ID (thread ID): 5
|
Status: NOT_KILLED
|
gdb shows the following:
(gdb) bt
|
#0 0x0000555555be30b9 in unpack_row(rpl_group_info*, TABLE*, unsigned int, unsigned char const*, st_bitmap const*, unsigned char const**, unsigned long*, unsigned char const*) ()
|
#1 0x0000555555bdfe80 in Update_rows_log_event::do_exec_row(rpl_group_info*) ()
|
#2 0x0000555555bd22cc in Rows_log_event::do_apply_event(rpl_group_info*) ()
|
#3 0x000055555590e2e1 in apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*, rpl_parallel_thread*) ()
|
#4 0x0000555555911d1b in handle_slave_sql ()
|
#5 0x00007ffff6afe182 in start_thread (arg=0x7ffff5c85700) at pthread_create.c:312
|
#6 0x00007ffff622147d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
Slave my.cnf
[client]
|
port = 3306
|
socket = /var/run/mysqld/mysqld.sock
|
back_log=1500
|
|
|
# # UTF-8
|
character_set_server = utf8
|
collation_server = utf8_general_ci
|
|
|
#
|
# * Logging and Replication
|
#
|
#log_slow_queries = /var/log/mysql/mysql-slow.log
|
long_query_time = 1
|
log-queries-not-using-indexes
|
log-output = TABLE
|
log_warnings = 2
|
|
|
sync_binlog = 0
|
binlog_checksum = crc32
|
binlog_commit_wait_usec = 50000
|
binlog_commit_wait_count= 20
|
server-id = 345
|
log_bin = /data/bigmomma/mysql/log/mysql-bin.log
|
binlog_format = MIXED
|
expire_logs_days = 7
|
max_binlog_size = 1G
|
log-slave-updates
|
slave_compressed_protocol = 1
|
relay-log = /data/bigmomma/mysql/log/mysql-relay.log
|
|
|
|
|
# INNODB #
|
innodb_log_files_in_group = 2
|
innodb_flush_log_at_trx_commit = 0
|
innodb_buffer_pool_size = 40G
|
innodb_file_per_table
|
innodb_log_file_size=1G
|
innodb_flush_method=O_DIRECT
|
innodb_file_format=barracuda
|
innodb_io_capacity = 20000
|
innodb_buffer_pool_instances=10
|
innodb_log_buffer_size=512M
|
#innodb_adaptive_checkpoint = keep_average
|
#innodb_flush_neighbor_pages = 0
|
group_concat_max_len = 8096
|
|
|
[mysqldump]
|
quick
|
quote-names
|
max_allowed_packet = 16M
|
ignore-table = telefoongids.white_subscriber
|
ignore-table = telefoongids.pink_subscriber
|
Master my.cnf
[client]
|
port = 3306
|
socket = /var/run/mysqld/mysqld.sock
|
|
|
# Here is entries for some specific programs
|
# The following values assume you have at least 32M ram
|
|
|
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
|
[mysqld_safe]
|
socket = /var/run/mysqld/mysqld.sock
|
nice = 0
|
|
|
[mysqld]
|
replicate-wild-ignore-table = %.Voorafgaand%_Klanten
|
replicate-wild-ignore-table = tmp.%
|
transaction-isolation = READ-COMMITTED
|
|
|
user = mysql
|
pid-file = /var/run/mysqld/mysqld.pid
|
socket = /var/run/mysqld/mysqld.sock
|
port = 3306
|
basedir = /usr
|
datadir = /var/lib/mysql
|
tmpdir = /tmp
|
lc-messages-dir = /usr/share/mysql
|
skip-external-locking
|
|
|
log-error = /data/bigmomma/mysql/log/error.log
|
innodb_stats_on_metadata = 0
|
lock_wait_timeout = 20
|
innodb_lock_wait_timeout = 20
|
|
|
#
|
# Instead of skip-networking the default is now to listen only on
|
# localhost which is more compatible and is not less secure.
|
bind-address = 0.0.0.0
|
|
|
# GTIDs
|
master_info_repository = table
|
relay_log_info_repository = table
|
master-verify-checksum = on
|
slave-sql-verify-checksum = on
|
|
|
|
|
#
|
# * Fine Tuning
|
#
|
key_buffer = 16M
|
#max_allowed_packet = 16M
|
thread_stack = 512K
|
thread_cache_size = 8
|
myisam-recover = BACKUP
|
|
|
# SAFETY #
|
max_allowed_packet = 64M
|
max_connect_errors = 1000000
|
sysdate_is_now = 1
|
|
|
max_connections=500
|
event_scheduler=ON
|
table_cache = 3000
|
|
|
tmp_table_size = 64M
|
max_heap_table_size = 64M
|
query_cache_type = 1
|
query_cache_size = 8M
|
max_connections = 500
|
thread_cache_size = 50
|
open_files_limit = 65535
|
table_definition_cache = 4096
|
table_open_cache = 4096
|
|
|
max_sp_recursion_depth = 2
|
|
|
back_log=1500
|
|
|
|
|
# UTF-8
|
character_set_server = utf8
|
collation_server = utf8_general_ci
|
|
|
|
|
#
|
# * Logging and Replication
|
#
|
#log_slow_queries = /var/log/mysql/mysql-slow.log
|
long_query_time = 1
|
log-queries-not-using-indexes
|
log-output = TABLE
|
log_warnings = 2
|
|
|
sync_binlog = 0
|
binlog_checksum = crc32
|
binlog_commit_wait_usec = 50000
|
binlog_commit_wait_count= 20
|
server-id = 765
|
log_bin = /data/bigmomma/mysql/log/mysql-bin.log
|
binlog_format = MIXED
|
expire_logs_days = 7
|
max_binlog_size = 1G
|
log-slave-updates
|
slave_compressed_protocol = 1
|
relay-log = /data/bigmomma/mysql/log/mysql-relay.log
|
|
|
|
|
# INNODB #
|
innodb_log_files_in_group = 2
|
innodb_flush_log_at_trx_commit = 0
|
innodb_buffer_pool_size = 40G
|
innodb_file_per_table
|
innodb_log_file_size=1G
|
innodb_flush_method=O_DIRECT
|
innodb_file_format=barracuda
|
innodb_io_capacity = 20000
|
innodb_buffer_pool_instances=10
|
innodb_log_buffer_size=32M
|
|
|
group_concat_max_len = 8096
|
|
|
[mysqldump]
|
quick
|
quote-names
|
max_allowed_packet = 16M
|
gtid = ON
|
ignore-table = telefoongids.white_subscriber
|
ignore-table = telefoongids.pink_subscriber
|
|
|
[mysql]
|
#no-auto-rehash # faster start of mysql but no tab completition
|
|
|
[isamchk]
|
key_buffer = 16M
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-
Activity
If you start your replication from a clean (empty) datadir, --disable-mysql56-temporal-format mentioned above might be a good workaround for you for now, even before you get fixes for crashes.
If you start replication from an already existing 10.1 directory where tables with microseconds have already been created, the workaround will be a little more complicated, tables will need to be ALTER-ed. If there are many tables, it should be possible to generate ALTER statements automatically based on the contents of INFORMATION_SCHEMA.COLUMNS.
The patch is Ok to push.
I'd only add tests for the other data types (TIME and DATETIME).
Currently it tests only TIMESTAMP.
Is it safe to assume a fix will be in the next version? Currently replication is working 10.0->10.0 and I can wait a few weeks before trying 10.1 again.
Yes, it will be.
By the way, the release schedule is on our jira main page (http://mariadb.org/jira) and next 10.1 is planned for this week.
if you don't see the roadmap, please log in — it's a jira bug JRA-59335 that hopefully will be fixed soon
Problem 3: Replication failure on 10.1 release server
This is a combined effect of
MDEV-5377andMDEV-5528.Because of
MDEV-5377temporal times could not be always replicated using RBR. And they were incompatible with MySQL temporal types (because MariaDB had them before MySQL). Later we have added MySQL-compatible temporal types and inMDEV-5528provided an option to use them by default. MySQL-compatible types can be safely replicated using RBR events.So your 10.0 server creates old MariaDB timestamp(3) field. While your 10.1 server creates new MariaDB timestamp(3). If you use RBR, old timestamp(3) can be only replicated to identically defined table. But your 10.0 and 10.1 tables aren't identical, that's why the replication fails.
I'll see if we can recognize this situation on the slave and automatically create the compatible table structure.
But until then you can start your 10.1 server with --disable-mysql56-temporal-format option. The replication will work.