[MDEV-16923] 10.3.8 crashes Created: 2018-08-08  Updated: 2023-10-19

Status: Open
Project: MariaDB Server
Component/s: Admin statements
Affects Version/s: 10.3
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Sander Pilon Assignee: Elena Stepanova
Resolution: Unresolved Votes: 0
Labels: need_verification

Issue Links:
Duplicate
duplicates MDEV-16924 10.3.8 FLUSH TABLES WITH READ LOCK ha... Closed
Relates
relates to MDEV-14915 Attempted to upgrade from 5.5.52 to 1... Open
relates to MDEV-15488 Crashes after upgrade from 5.5.52 to ... Closed
relates to MDEV-16554 InnoDB: Assertion failure in file mar... Closed

 Description   

We skipped 10.2 because of MDEV-13983. So we decided to test 10.3.8 and it crashes in the first few seconds after brining it online.

Server version: 10.3.8-MariaDB-1:10.3.8+maria~trusty-log
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=20
max_threads=502
thread_count=38
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1119930 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fb2700009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb3b1769e40 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fbfb2b37f2e]
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x7fbfb25d60a7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fbfb0784330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fbfafdd7c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fbfafddb028]
/usr/sbin/mysqld(+0x4c8d18)[0x7fbfb2321d18]
/usr/sbin/mysqld(+0x9e69c3)[0x7fbfb283f9c3]
/usr/sbin/mysqld(+0xa15f01)[0x7fbfb286ef01]
/usr/sbin/mysqld(+0x945f4a)[0x7fbfb279ef4a]
/usr/sbin/mysqld(_Z8closefrmP5TABLE+0x131)[0x7fbfb24aab41]
/usr/sbin/mysqld(_Z8tc_purgeb+0x69)[0x7fbfb2558c39]
/usr/sbin/mysqld(_Z19close_cached_tablesP3THDP10TABLE_LISTbm+0x87)[0x7fbfb23a8b97]
/usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x2ff)[0x7fbfb24f7abf]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1379)[0x7fbfb24016d9]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x202)[0x7fbfb2409562]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1c38)[0x7fbfb240c068]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x7fbfb240cefe]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x7fbfb24dd62a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x7fbfb24dd74d]
/usr/sbin/mysqld(+0x929ead)[0x7fbfb2782ead]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fbfb077c184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fbfafe9f03d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fb2700116e0): FLUSH TABLES WITH READ LOCK
 
Connection ID (thread ID): 7916
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,mat
erialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cach
e_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

our slave server also crashes with 10.3.8 installed

180808 21:51:29 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.8-MariaDB-1:10.3.8+maria~trusty-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=3
max_threads=502
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1234618 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556816bdef2e]
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x55681667d0a7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f434b3a7330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f434a9fac37]
linux/raise.c:56(__GI_raise)[0x7f434a9fe028]
/usr/sbin/mysqld(+0x4c8d18)[0x5568163c8d18]
/usr/sbin/mysqld(+0x9e69c3)[0x5568168e69c3]
/usr/sbin/mysqld(+0xa15f01)[0x556816915f01]
/usr/sbin/mysqld(+0x945f4a)[0x556816845f4a]
/usr/sbin/mysqld(_Z8closefrmP5TABLE+0x131)[0x556816551b41]
/usr/sbin/mysqld(_Z8tc_purgeb+0x69)[0x5568165ffc39]
/usr/sbin/mysqld(_Z19close_cached_tablesP3THDP10TABLE_LISTbm+0x87)[0x55681644fb97]
/usr/sbin/mysqld(+0x4ecd8f)[0x5568163ecd8f]
/usr/sbin/mysqld(_Z10unireg_endv+0x3b)[0x5568163ed07b]
/usr/sbin/mysqld(+0x4f09a9)[0x5568163f09a9]
/usr/sbin/mysqld(kill_server_thread+0xe)[0x5568163f09ee]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f434b39f184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f434aac203d]



 Comments   
Comment by Elena Stepanova [ 2018-08-08 ]

Could you please paste or attach your cnf file(s) and say whatever you can about the workflow that's happening on the server?
If you could enable the coredump and produce all threads' stack trace, it would be very helpful.

Comment by Sander Pilon [ 2018-08-08 ]

my.cnf

(Witht the note that the parralell threads was changed to 1 because of another crash, which I may or may not be the same. MDEV-16924

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
 
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
 
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
 
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0
 
[mysqld]
replicate-wild-ignore-table = %.Voorafgaand%_Klanten
replicate-wild-ignore-table = tmp.%
replicate-wild-ignore-table = Voorraadbeheer.VBMIH_tekst_cache          
replicate-wild-ignore-table = Managementboek_Klanten.OrderschermRegels  
replicate-wild-ignore-table = php_sessions.%                            
 
transaction-isolation =  READ-COMMITTED
 
slave-parallel-threads=1              
slave-parallel-mode=none       
slave_skip_errors=1062,1032,1964,1452 
sql-mode=NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION  
 
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
 
log-error = /data/bigmomma/mysql/log/error.log
innodb_stats_on_metadata = 0
lock_wait_timeout = 20
innodb_lock_wait_timeout = 20
 
#performance_schema=on
 
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 0.0.0.0
 
# GTIDs
master_info_repository = table
relay_log_info_repository = table
master-verify-checksum = on
slave-sql-verify-checksum = on
 
 
#
# * Fine Tuning
#
key_buffer              = 16M
#max_allowed_packet     = 16M
thread_stack            = 512K
thread_cache_size       = 8
myisam-recover         = BACKUP
 
# SAFETY #
max_allowed_packet             = 64M
max_connect_errors             = 1000000
sysdate_is_now                 = 1
 
max_connections=100
event_scheduler=ON
#table_cache     = 3000
 
tmp_table_size                 = 64M
max_heap_table_size            = 64M
query_cache_type               = 1
query_cache_size               = 8M
max_connections                = 500
thread_cache_size              = 50
open_files_limit               = 65535
table_definition_cache         = 4096
table_open_cache               = 4096
 
max_sp_recursion_depth         = 5
 
back_log=1500
 
 
# UTF-8
character_set_server = utf8
collation_server = utf8_general_ci
 
 
#
# * Logging and Replication
#
#log_slow_queries       = /var/log/mysql/mysql-slow.log
long_query_time         = 1
log-queries-not-using-indexes
log-output              = TABLE
log_warnings            = 2
 
sync_binlog             = 0
binlog_checksum         = crc32
binlog_commit_wait_usec = 50000
binlog_commit_wait_count= 20
binlog_cache_size       = 32M
server-id               = 765
log_bin                 = /data/bigmomma/mysql/log/mysql-bin.log
binlog_format           = MIXED
expire_logs_days        = 6
max_binlog_size         = 5G
# log-slave-updates [[WAAROM ZOUDEN WE DIT NODIG HEBBEN??]]
slave_compressed_protocol = 0
relay-log               = /data/bigmomma/mysql/log/mysql-relay.log
 
 
# INNODB #
innodb_log_files_in_group      = 2
innodb_flush_log_at_trx_commit = 0
innodb_buffer_pool_size        = 45G
innodb_file_per_table
innodb_log_file_size=1G
innodb_flush_method=O_DIRECT
#innodb_io_capacity=20000
innodb_buffer_pool_instances=10
innodb_log_buffer_size=32M
 
group_concat_max_len = 8096
 
[mysqldump]
quick
quote-names
max_allowed_packet      = 16M
gtid                    = ON
ignore-table            = telefoongids.white_subscriber
ignore-table            = telefoongids.pink_subscriber
 
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
 
[isamchk]
key_buffer              = 16M
 
[mysqld]
stack-trace
#core-file
[mysqld-safe]
#core-file-size=unlimited

Comment by Sander Pilon [ 2018-08-08 ]

As for the workflow - it is quite a busy server. Lots of things happening, it is the backend for a large e-commerce shop in the netherlands.

Comment by Sander Pilon [ 2018-08-09 ]

It has crashed numerous times last night. We're going back to 10.1.

Comment by Sander Pilon [ 2018-08-09 ]

While we wait for the restore, it keeps crashing every few minutes. Here's another dump:

I increased the stack_size to 1M, but that has no effect.

2018-08-09 10:02:08 0x7fdbef2f9700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/que/que0que.cc line 563
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180809 10:02:08 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.8-MariaDB-1:10.3.8+maria~trusty-log
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=22
max_threads=102
thread_count=33
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 240592 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fdb180010e8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fdbef2f8e70 thread_stack 0x100000
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7329f)[0x7fe7c84c029f]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fe7c855b87c]
/lib/x86_64-linux-gnu/libc.so.6(+0x10d750)[0x7fe7c855a750]
/lib/x86_64-linux-gnu/libc.so.6(+0x10e7c7)[0x7fe7c855b7c7]
/usr/sbin/mysqld(my_addr_resolve+0xda)[0x7fe7cb1faa0a]
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7fe7cb1e40c2]
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x7fe7cac820a7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fe7c8e30330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe7c8483c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe7c8487028]
/usr/sbin/mysqld(+0x4c8d18)[0x7fe7ca9cdd18]
/usr/sbin/mysqld(+0x9e69c3)[0x7fe7caeeb9c3]
/usr/sbin/mysqld(+0xa15f01)[0x7fe7caf1af01]
/usr/sbin/mysqld(+0x945f4a)[0x7fe7cae4af4a]
/usr/sbin/mysqld(_Z8closefrmP5TABLE+0x131)[0x7fe7cab56b41]
/usr/sbin/mysqld(+0x6fe9e9)[0x7fe7cac039e9]
/usr/sbin/mysqld(_Z12tc_add_tableP3THDP5TABLE+0x1e8)[0x7fe7cac03c18]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xb05)[0x7fe7caa54485]
/usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xe0a)[0x7fe7caa5721a]
/usr/sbin/mysqld(_Z29mysqld_show_create_get_fieldsP3THDP10TABLE_LISTP4ListI4ItemEP6String+0xdf)[0x7fe7cab1052f]
/usr/sbin/mysqld(_Z18mysqld_show_createP3THDP10TABLE_LIST+0x132)[0x7fe7cab10e52]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d4f)[0x7fe7caaae0af]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x202)[0x7fe7caab5562]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1c38)[0x7fe7caab8068]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x7fe7caab8efe]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x7fe7cab8962a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x7fe7cab8974d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fe7c8e28184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe7c854b03d]
======= Memory map: ========
7fda94000000-7fda95d21000 rw-p 00000000 00:00 0 
7fda95d21000-7fda98000000 ---p 00000000 00:00 0 
7fda9c000000-7fda9fff9000 rw-p 00000000 00:00 0 
7fda9fff9000-7fdaa0000000 ---p 00000000 00:00 0 
7fdaa0000000-7fdaa026c000 rw-p 00000000 00:00 0 
7fdaa026c000-7fdaa4000000 ---p 00000000 00:00 0 
7fdaa4000000-7fdaa58ad000 rw-p 00000000 00:00 0 
7fdaa58ad000-7fdaa8000000 ---p 00000000 00:00 0 
7fdaa8000000-7fdaab82c000 rw-p 00000000 00:00 0 
7fdaab82c000-7fdaac000000 ---p 00000000 00:00 0 
7fdaac000000-7fdaadcf3000 rw-p 00000000 00:00 0 
7fdaadcf3000-7fdab0000000 ---p 00000000 00:00 0 
7fdab0000000-7fdab194d000 rw-p 00000000 00:00 0 
7fdab194d000-7fdab4000000 ---p 00000000 00:00 0 
7fdab4000000-7fdab7ffe000 rw-p 00000000 00:00 0 
7fdab7ffe000-7fdab8000000 ---p 00000000 00:00 0 
7fdab8000000-7fdab9945000 rw-p 00000000 00:00 0 
7fdab9945000-7fdabc000000 ---p 00000000 00:00 0 
7fdabc000000-7fdabfffe000 rw-p 00000000 00:00 0 
7fdabfffe000-7fdac0000000 ---p 00000000 00:00 0 
7fdac0000000-7fdac3b9b000 rw-p 00000000 00:00 0 
7fdac3b9b000-7fdac4000000 ---p 00000000 00:00 0 
7fdac4000000-7fdac6d1a000 rw-p 00000000 00:00 0 
7fdac6d1a000-7fdac8000000 ---p 00000000 00:00 0 
7fdac8000000-7fdacadc0000 rw-p 00000000 00:00 0 
7fdacadc0000-7fdacc000000 ---p 00000000 00:00 0 
7fdad0000000-7fdad3610000 rw-p 00000000 00:00 0 
7fdad3610000-7fdad4000000 ---p 00000000 00:00 0 
7fdad8000000-7fdad912b000 rw-p 00000000 00:00 0 
7fdad912b000-7fdadc000000 ---p 00000000 00:00 0 
7fdae0000000-7fdae1c51000 rw-p 00000000 00:00 0 
7fdae1c51000-7fdae4000000 ---p 00000000 00:00 0 
7fdae5fff000-7fdae8000000 rw-p 00000000 00:00 0 
7fdae8000000-7fdaeb714000 rw-p 00000000 00:00 0 
7fdaeb714000-7fdaec000000 ---p 00000000 00:00 0 
7fdaec000000-7fdaee573000 rw-p 00000000 00:00 0 
7fdaee573000-7fdaf0000000 ---p 00000000 00:00 0 
7fdaf0000000-7fdaf7ffc000 rw-p 00000000 00:00 0 
7fdaf7ffc000-7fdaf8000000 ---p 00000000 00:00 0 
7fdaf8000000-7fdaffff4000 rw-p 00000000 00:00 0 
7fdaffff4000-7fdb00000000 ---p 00000000 00:00 0 
7fdb00000000-7fdb03ffb000 rw-p 00000000 00:00 0 
7fdb03ffb000-7fdb04000000 ---p 00000000 00:00 0 
7fdb04000000-7fdb0739c000 rw-p 00000000 00:00 0 
7fdb0739c000-7fdb08000000 ---p 00000000 00:00 0 
7fdb08000000-7fdb0a27f000 rw-p 00000000 00:00 0 
7fdb0a27f000-7fdb0c000000 ---p 00000000 00:00 0 
7fdb0c000000-7fdb0ffff000 rw-p 00000000 00:00 0 
7fdb0ffff000-7fdb10000000 ---p 00000000 00:00 0 
7fdb10000000-7fdb13fff000 rw-p 00000000 00:00 0 
7fdb13fff000-7fdb14000000 ---p 00000000 00:00 0 
7fdb14000000-7fdb18000000 rw-p 00000000 00:00 0 
7fdb18000000-7fdb1bffe000 rw-p 00000000 00:00 0 
7fdb1bffe000-7fdb1c000000 ---p 00000000 00:00 0 
7fdb20000000-7fdb22fb0000 rw-p 00000000 00:00 0 
7fdb22fb0000-7fdb24000000 ---p 00000000 00:00 0 
7fdb24000000-7fdb27be6000 rw-p 00000000 00:00 0 
7fdb27be6000-7fdb28000000 ---p 00000000 00:00 0 
7fdb29fff000-7fdb2c000000 rw-p 00000000 00:00 0 
7fdb2c000000-7fdb2fffb000 rw-p 00000000 00:00 0 
7fdb2fffb000-7fdb30000000 ---p 00000000 00:00 0 
7fdb30000000-7fdb329a2000 rw-p 00000000 00:00 0 
7fdb329a2000-7fdb34000000 ---p 00000000 00:00 0 
7fdb34000000-7fdb3bfff000 rw-p 00000000 00:00 0 
7fdb3bfff000-7fdb3c000000 ---p 00000000 00:00 0 
7fdb3c000000-7fdb3fb54000 rw-p 00000000 00:00 0 
7fdb3fb54000-7fdb40000000 ---p 00000000 00:00 0 
7fdb40000000-7fdb431c4000 rw-p 00000000 00:00 0 
7fdb431c4000-7fdb44000000 ---p 00000000 00:00 0 
7fdb44000000-7fdb44021000 rw-p 00000000 00:00 0 
7fdb44021000-7fdb48000000 ---p 00000000 00:00 0 
7fdb48000000-7fdb48021000 rw-p 00000000 00:00 0 
7fdb48021000-7fdb4c000000 ---p 00000000 00:00 0 
7fdb4c000000-7fdb4c021000 rw-p 00000000 00:00 0 
7fdb4c021000-7fdb50000000 ---p 00000000 00:00 0 
7fdb50000000-7fdb50021000 rw-p 00000000 00:00 0 
7fdb50021000-7fdb54000000 ---p 00000000 00:00 0 
7fdb54000000-7fdb56b46000 rw-p 00000000 00:00 0 
7fdb56b46000-7fdb58000000 ---p 00000000 00:00 0 
7fdb58000000-7fdb5c000000 rw-p 00000000 00:00 0 
7fdb5c000000-7fdb5c03e000 rw-p 00000000 00:00 0 
7fdb5c03e000-7fdb60000000 ---p 00000000 00:00 0 
7fdb60000000-7fdb60021000 rw-p 00000000 00:00 0 
7fdb60021000-7fdb64000000 ---p 00000000 00:00 0 
7fdb64000000-7fdb64021000 rw-p 00000000 00:00 0 
7fdb64021000-7fdb68000000 ---p 00000000 00:00 0 
7fdb68000000-7fdb68021000 rw-p 00000000 00:00 0 
7fdb68021000-7fdb6c000000 ---p 00000000 00:00 0 
7fdb6c000000-7fdb6c021000 rw-p 00000000 00:00 0 
7fdb6c021000-7fdb70000000 ---p 00000000 00:00 0 
7fdb70000000-7fdb70021000 rw-p 00000000 00:00 0 
7fdb70021000-7fdb74000000 ---p 00000000 00:00 0 
7fdb74000000-7fdb74021000 rw-p 00000000 00:00 0 
7fdb74021000-7fdb78000000 ---p 00000000 00:00 0 
7fdb78000000-7fdb78021000 rw-p 00000000 00:00 0 
7fdb78021000-7fdb7c000000 ---p 00000000 00:00 0 
7fdb7c000000-7fdb7c021000 rw-p 00000000 00:00 0 
7fdb7c021000-7fdb80000000 ---p 00000000 00:00 0 
7fdb80000000-7fdb80021000 rw-p 00000000 00:00 0 
7fdb80021000-7fdb84000000 ---p 00000000 00:00 0 
7fdb84000000-7fdb84021000 rw-p 00000000 00:00 0 
7fdb84021000-7fdb88000000 ---p 00000000 00:00 0 
7fdb8899c000-7fdb94000000 rw-p 00000000 00:00 0 
7fdb94000000-7fdb94021000 rw-p 00000000 00:00 0 
7fdb94021000-7fdb98000000 ---p 00000000 00:00 0 
7fdb98000000-7fdb98021000 rw-p 00000000 00:00 0 
7fdb98021000-7fdb9c000000 ---p 00000000 00:00 0 
7fdb9c000000-7fdb9c021000 rw-p 00000000 00:00 0 
7fdb9c021000-7fdba0000000 ---p 00000000 00:00 0 
7fdba07fc000-7fdba27fd000 rw-p 00000000 00:00 0 
7fdba27fd000-7fdba27fe000 ---p 00000000 00:00 0 
7fdba27fe000-7fdba2ffe000 rw-p 00000000 00:00 0                          [stack:16758]
7fdba2ffe000-7fdba2fff000 ---p 00000000 00:00 0 
7fdba2fff000-7fdba37ff000 rw-p 00000000 00:00 0                          [stack:16753]
7fdba37ff000-7fdba3800000 ---p 00000000 00:00 0 
7fdba3800000-7fdba4000000 rw-p 00000000 00:00 0                          [stack:16718]
7fdba4000000-7fdba4021000 rw-p 00000000 00:00 0 
7fdba4021000-7fdba8000000 ---p 00000000 00:00 0 
7fdba8000000-7fdba8021000 rw-p 00000000 00:00 0 
7fdba8021000-7fdbac000000 ---p 00000000 00:00 0 
7fdbac000000-7fdbac021000 rw-p 00000000 00:00 0 
7fdbac021000-7fdbb0000000 ---p 00000000 00:00 0 
7fdbb0543000-7fdbb0748000 rw-p 00000000 00:00 0 
7fdbb0748000-7fdbb0749000 ---p 00000000 00:00 0 
7fdbb0749000-7fdbb0f49000 rw-p 00000000 00:00 0                          [stack:16717]
7fdbb0f49000-7fdbb0f4a000 ---p 00000000 00:00 0 
7fdbb0f4a000-7fdbb174a000 rw-p 00000000 00:00 0                          [stack:16716]
7fdbb174a000-7fdbb174b000 ---p 00000000 00:00 0 
7fdbb174b000-7fdbb1f4b000 rw-p 00000000 00:00 0                          [stack:16715]
7fdbb1f4b000-7fdbb1f4c000 ---p 00000000 00:00 0 
7fdbb1f4c000-7fdbb274c000 rw-p 00000000 00:00 0                          [stack:16714]
7fdbb274c000-7fdbb274d000 ---p 00000000 00:00 0 
7fdbb274d000-7fdbb2f4d000 rw-p 00000000 00:00 0                          [stack:16713]
7fdbb2f4d000-7fdbb2f4e000 ---p 00000000 00:00 0 
7fdbb2f4e000-7fdbb374e000 rw-p 00000000 00:00 0                          [stack:16712]
7fdbb374e000-7fdbb374f000 ---p 00000000 00:00 0 
7fdbb374f000-7fdbb3f4f000 rw-p 00000000 00:00 0                          [stack:16711]
7fdbb3f4f000-7fdbb3f50000 ---p 00000000 00:00 0 
7fdbb3f50000-7fdbb4750000 rw-p 00000000 00:00 0                          [stack:16710]
7fdbb4750000-7fdbb4751000 ---p 00000000 00:00 0 
7fdbb4751000-7fdbb4f51000 rw-p 00000000 00:00 0                          [stack:16709]
7fdbb4f51000-7fdbb4f52000 ---p 00000000 00:00 0 
7fdbb4f52000-7fdbb5752000 rw-p 00000000 00:00 0                          [stack:16708]
7fdbb5752000-7fdbb5753000 ---p 00000000 00:00 0 
7fdbb5753000-7fdbb5f53000 rw-p 00000000 00:00 0                          [stack:16707]
7fdbb5f53000-7fdbb5f54000 ---p 00000000 00:00 0 
7fdbb5f54000-7fdbb6754000 rw-p 00000000 00:00 0                          [stack:16706]
7fdbb6754000-7fdbb6755000 ---p 00000000 00:00 0 
7fdbb6755000-7fdbcc552000 rw-p 00000000 00:00 0                          [stack:16705]
7fdbcdfff000-7fdbd0000000 rw-p 00000000 00:00 0 
7fdbd0000000-7fdbd0021000 rw-p 00000000 00:00 0 
7fdbd0021000-7fdbd4000000 ---p 00000000 00:00 0 
7fdbd4000000-7fdbd4021000 rw-p 00000000 00:00 0 
7fdbd4021000-7fdbd8000000 ---p 00000000 00:00 0 
7fdbd8000000-7fdbda56a000 rw-p 00000000 00:00 0 
7fdbda56a000-7fdbdc000000 ---p 00000000 00:00 0 
7fdbdc000000-7fdbdc021000 rw-p 00000000 00:00 0 
7fdbdc021000-7fdbe0000000 ---p 00000000 00:00 0 
7fdbe0000000-7fdbe1062000 rw-p 00000000 00:00 0 
7fdbe1062000-7fdbe4000000 ---p 00000000 00:00 0 
7fdbe4000000-7fdbe5067000 rw-p 00000000 00:00 0 
7fdbe5067000-7fdbe8000000 ---p 00000000 00:00 0 
7fdbe8000000-7fdbe8be3000 rw-p 00000000 00:00 0 
7fdbe8be3000-7fdbec000000 ---p 00000000 00:00 0 
7fdbecdf4000-7fdbecdf5000 ---p 00000000 00:00 0 
7fdbecdf5000-7fdbecef5000 rw-p 00000000 00:00 0                          [stack:17019]
7fdbecef5000-7fdbecef6000 ---p 00000000 00:00 0 
7fdbecef6000-7fdbecff6000 rw-p 00000000 00:00 0                          [stack:17017]
7fdbecff6000-7fdbecff7000 ---p 00000000 00:00 0 
7fdbecff7000-7fdbed0f7000 rw-p 00000000 00:00 0                          [stack:17016]
7fdbed0f7000-7fdbed0f8000 ---p 00000000 00:00 0 
7fdbed0f8000-7fdbed1f8000 rw-p 00000000 00:00 0                          [stack:17005]
7fdbed1f8000-7fdbed1f9000 ---p 00000000 00:00 0 
7fdbed1f9000-7fdbed2f9000 rw-p 00000000 00:00 0                          [stack:17015]
7fdbee2ea000-7fdbee2eb000 ---p 00000000 00:00 0 
7fdbee2eb000-7fdbee3eb000 rw-p 00000000 00:00 0 
7fdbee3eb000-7fdbee3ec000 ---p 00000000 00:00 0 
7fdbee3ec000-7fdbee4ec000 rw-p 00000000 00:00 0                          [stack:20709]
7fdbee4ec000-7fdbee4ed000 ---p 00000000 00:00 0 
7fdbee4ed000-7fdbee5ed000 rw-p 00000000 00:00 0                          [stack:19862]
7fdbee5ed000-7fdbee5ee000 ---p 00000000 00:00 0 
7fdbee5ee000-7fdbee6ee000 rw-p 00000000 00:00 0                          [stack:19225]
7fdbee6ee000-7fdbee6ef000 ---p 00000000 00:00 0 
7fdbee6ef000-7fdbee7ef000 rw-p 00000000 00:00 0                          [stack:19217]
7fdbee7ef000-7fdbee7f0000 ---p 00000000 00:00 0 
7fdbee7f0000-7fdbee8f0000 rw-p 00000000 00:00 0                          [stack:19216]
7fdbee8f0000-7fdbee8f1000 ---p 00000000 00:00 0 
7fdbee8f1000-7fdbee9f1000 rw-p 00000000 00:00 0                          [stack:19122]
7fdbee9f1000-7fdbee9f2000 ---p 00000000 00:00 0 
7fdbee9f2000-7fdbeeaf2000 rw-p 00000000 00:00 0                          [stack:19102]
7fdbeeaf2000-7fdbeeaf3000 ---p 00000000 00:00 0 
7fdbeeaf3000-7fdbeebf3000 rw-p 00000000 00:00 0                          [stack:19121]
7fdbeebf3000-7fdbeebf4000 ---p 00000000 00:00 0 
7fdbeebf4000-7fdbeecf4000 rw-p 00000000 00:00 0                          [stack:18199]
7fdbeecf4000-7fdbeecf5000 ---p 00000000 00:00 0 
7fdbeecf5000-7fdbeedf5000 rw-p 00000000 00:00 0                          [stack:18198]
7fdbeedf5000-7fdbeedf6000 ---p 00000000 00:00 0 
7fdbeedf6000-7fdbeeef6000 rw-p 00000000 00:00 0                          [stack:17753]
7fdbeeef6000-7fdbeeef7000 ---p 00000000 00:00 0 
7fdbeeef7000-7fdbeeff7000 rw-p 00000000 00:00 0                          [stack:17218]
7fdbeeff7000-7fdbeeff8000 ---p 00000000 00:00 0 
7fdbeeff8000-7fdbef0f8000 rw-p 00000000 00:00 0                          [stack:17217]
7fdbef0f8000-7fdbef0f9000 ---p 00000000 00:00 0 
7fdbef0f9000-7fdbef1f9000 rw-p 00000000 00:00 0                          [stack:17216]
7fdbef1f9000-7fdbef1fa000 ---p 00000000 00:00 0 
7fdbef1fa000-7fdbef2fa000 rw-p 00000000 00:00 0                          [stack:17215]
7fdbef2fa000-7fdbef2fb000 ---p 00000000 00:00 0 
7fdbef2fb000-7fdbef3fb000 rw-p 00000000 00:00 0                          [stack:16776]
7fdbef3fb000-7fdbef3fc000 ---p 00000000 00:00 0 
7fdbef3fc000-7fdbef4fc000 rw-p 00000000 00:00 0                          [stack:16774]
7fdbef4fc000-7fdbef4fd000 ---p 00000000 00:00 0 
7fdbef4fd000-7fdbef5fd000 rw-p 00000000 00:00 0                          [stack:16773]
7fdbef5fd000-7fdbef5fe000 ---p 00000000 00:00 0 
7fdbef5fe000-7fdbef6fe000 rw-p 00000000 00:00 0                          [stack:16772]
7fdbef6fe000-7fdbef6ff000 ---p 00000000 00:00 0 
7fdbef6ff000-7fdbef7ff000 rw-p 00000000 00:00 0                          [stack:16771]
7fdbef7ff000-7fdbef800000 ---p 00000000 00:00 0 
7fdbef800000-7fdbf0000000 rw-p 00000000 00:00 0                          [stack:16762]
7fdbf0000000-7fdbf1963000 rw-p 00000000 00:00 0 
7fdbf1963000-7fdbf4000000 ---p 00000000 00:00 0 
7fdbf40cf000-7fdbf40d0000 ---p 00000000 00:00 0 
7fdbf40d0000-7fdbf41d0000 rw-p 00000000 00:00 0                          [stack:16770]
7fdbf41d0000-7fdbf41d1000 ---p 00000000 00:00 0 
7fdbf41d1000-7fdbf42d1000 rw-p 00000000 00:00 0                          [stack:16769]
7fdbf42d1000-7fdbf42d2000 r-xp 00000000 08:02 1712898                    /usr/lib/mysql/plugin/libfnv_udf.so
7fdbf42d2000-7fdbf44d1000 ---p 00001000 08:02 1712898                    /usr/lib/mysql/plugin/libfnv_udf.so
7fdbf44d1000-7fdbf44d2000 rw-p 00000000 08:02 1712898                    /usr/lib/mysql/plugin/libfnv_udf.so
7fdbf44d2000-7fdbf44d3000 r-xp 00000000 08:02 1712897                    /usr/lib/mysql/plugin/libfnv1a_udf.so
7fdbf44d3000-7fdbf46d2000 ---p 00001000 08:02 1712897                    /usr/lib/mysql/plugin/libfnv1a_udf.so
7fdbf46d2000-7fdbf46d3000 rw-p 00000000 08:02 1712897                    /usr/lib/mysql/plugin/libfnv1a_udf.so
7fdbf46d3000-7fdbf46d4000 r-xp 00000000 08:02 1708927                    /usr/lib/mysql/plugin/libmysqldameraulevenshtein.so
7fdbf46d4000-7fdbf48d3000 ---p 00001000 08:02 1708927                    /usr/lib/mysql/plugin/libmysqldameraulevenshtein.so
7fdbf48d3000-7fdbf48d4000 r--p 00000000 08:02 1708927                    /usr/lib/mysql/plugin/libmysqldameraulevenshtein.so
7fdbf48d4000-7fdbf48d5000 rw-p 00001000 08:02 1708927                    /usr/lib/mysql/plugin/libmysqldameraulevenshtein.so
7fdbf48d5000-7fdbf48d6000 ---p 00000000 00:00 0 
7fdbf48d6000-7fdbf50d6000 rw-p 00000000 00:00 0                          [stack:16766]
7fdbf50d6000-7fdbf50d7000 ---p 00000000 00:00 0 
7fdbf50d7000-7fdbf58d7000 rw-p 00000000 00:00 0                          [stack:16765]
7fdbf58d7000-7fdbf58d8000 ---p 00000000 00:00 0 
7fdbf58d8000-7fdbf60d8000 rw-p 00000000 00:00 0                          [stack:16764]
7fdbf60d8000-7fdbf60d9000 ---p 00000000 00:00 0 
7fdbf60d9000-7fdbf68d9000 rw-p 00000000 00:00 0                          [stack:16763]
7fdbf68d9000-7fdbf68da000 ---p 00000000 00:00 0 
7fdbf68da000-7fdbf70da000 rw-p 00000000 00:00 0                          [stack:16761]
7fdbf70da000-7fdbf70db000 ---p 00000000 00:00 0 
7fdbf70db000-7fdbf78db000 rw-p 00000000 00:00 0                          [stack:16760]
7fdbf78db000-7fdbf78dc000 ---p 00000000 00:00 0 
7fdbf78dc000-7fdbf80dc000 rw-p 00000000 00:00 0                          [stack:16759]
7fdbf80dc000-7fdbf80dd000 ---p 00000000 00:00 0 
7fdbf80dd000-7fdbf88dd000 rw-p 00000000 00:00 0                          [stack:16757]
7fdbf88dd000-7fdbf88de000 ---p 00000000 00:00 0 
7fdbf88de000-7fdbf90de000 rw-p 00000000 00:00 0                          [stack:16756]
7fdbf90de000-7fdbf90df000 ---p 00000000 00:00 0 
7fdbf90df000-7fdbf98df000 rw-p 00000000 00:00 0                          [stack:16755]
7fdbf98df000-7fdbf98e0000 ---p 00000000 00:00 0 
7fdbf98e0000-7fdbfa0e0000 rw-p 00000000 00:00 0                          [stack:16754]
7fdbfa0e0000-7fdbfe0e0000 rw-p 00000000 00:00 0 
7fdbfe0e0000-7fdc2c390000 rw-p 00000000 00:00 0 
7fdc2c390000-7fdd52f90000 rw-p 00000000 00:00 0 
7fdd52f90000-7fdd538b0000 rw-p 00000000 00:00 0 
7fdd538b0000-7fde7a4b0000 rw-p 00000000 00:00 0 
7fde7a4b0000-7fde7add0000 rw-p 00000000 00:00 0 
7fde7add0000-7fdfa19d0000 rw-p 00000000 00:00 0 
7fdfa19d0000-7fdfa22f0000 rw-p 00000000 00:00 0 
7fdfa22f0000-7fe0c8ef0000 rw-p 00000000 00:00 0 
7fe0c8ef0000-7fe0c9810000 rw-p 00000000 00:00 0 
7fe0c9810000-7fe1f0410000 rw-p 00000000 00:00 0 
7fe1f0410000-7fe1f0d30000 rw-p 00000000 00:00 0 
7fe1f0d30000-7fe317930000 rw-p 00000000 00:00 0 
7fe317930000-7fe318250000 rw-p 00000000 00:00 0 
7fe318250000-7fe43ee50000 rw-p 00000000 00:00 0 
7fe43ee50000-7fe43f770000 rw-p 00000000 00:00 0 
7fe43f770000-7fe566370000 rw-p 00000000 00:00 0 
7fe566370000-7fe566800000 rw-p 00000000 00:00 0 
7fe566800000-7fe7b4000000 rw-p 00000000 00:00 0 
7fe7b4000000-7fe7b4021000 rw-p 00000000 00:00 0 
7fe7b4021000-7fe7b8000000 ---p 00000000 00:00 0 
7fe7b80bb000-7fe7c0000000 rw-p 00000000 00:00 0 
7fe7c0000000-7fe7c0021000 rw-p 00000000 00:00 0 
7fe7c0021000-7fe7c4000000 ---p 00000000 00:00 0 
7fe7c4097000-7fe7c4098000 ---p 00000000 00:00 0 
7fe7c4098000-7fe7c4eb9000 rw-p 00000000 00:00 0                          [stack:16768]
7fe7c4eb9000-7fe7c4eba000 ---p 00000000 00:00 0 
7fe7c4eba000-7fe7c5def000 rw-p 00000000 00:00 0                          [stack:16683]
7fe7c5def000-7fe7c5df1000 r-xp 00000000 08:02 1723355                    /usr/lib/mysql/plugin/metadata_lock_info.so
7fe7c5df1000-7fe7c5ff0000 ---p 00002000 08:02 1723355                    /usr/lib/mysql/plugin/metadata_lock_info.so
7fe7c5ff0000-7fe7c5ff1000 r--p 00001000 08:02 1723355                    /usr/lib/mysql/plugin/metadata_lock_info.so
7fe7c5ff1000-7fe7c5ff2000 rw-p 00002000 08:02 1723355                    /usr/lib/mysql/plugin/metadata_lock_info.so
7fe7c5ff2000-7fe7c6ff3000 rw-p 00000000 00:00 0 
7fe7c6ff3000-7fe7c6ff4000 ---p 00000000 00:00 0 
7fe7c6ff4000-7fe7c77f4000 rw-p 00000000 00:00 0                          [stack:16682]
7fe7c77f4000-7fe7c77fe000 r-xp 00000000 08:02 2883636                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe7c77fe000-7fe7c79fd000 ---p 0000a000 08:02 2883636                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe7c79fd000-7fe7c79fe000 r--p 00009000 08:02 2883636                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe7c79fe000-7fe7c79ff000 rw-p 0000a000 08:02 2883636                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe7c79ff000-7fe7c7a0a000 r-xp 00000000 08:02 2887753                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe7c7a0a000-7fe7c7c09000 ---p 0000b000 08:02 2887753                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe7c7c09000-7fe7c7c0a000 r--p 0000a000 08:02 2887753                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe7c7c0a000-7fe7c7c0b000 rw-p 0000b000 08:02 2887753                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe7c7c0b000-7fe7c7c22000 r-xp 00000000 08:02 2887766                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe7c7c22000-7fe7c7e21000 ---p 00017000 08:02 2887766                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe7c7e21000-7fe7c7e22000 r--p 00016000 08:02 2887766                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe7c7e22000-7fe7c7e23000 rw-p 00017000 08:02 2887766                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe7c7e23000-7fe7c7e25000 rw-p 00000000 00:00 0 
7fe7c7e25000-7fe7c7e2e000 r-xp 00000000 08:02 2887755                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe7c7e2e000-7fe7c802d000 ---p 00009000 08:02 2887755                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe7c802d000-7fe7c802e000 r--p 00008000 08:02 2887755                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe7c802e000-7fe7c802f000 rw-p 00009000 08:02 2887755                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe7c802f000-7fe7c8045000 r-xp 00000000 08:02 2890704                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe7c8045000-7fe7c8244000 ---p 00016000 08:02 2890704                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe7c8244000-7fe7c8245000 rw-p 00015000 08:02 2890704                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe7c8245000-7fe7c824c000 r-xp 00000000 08:02 2887764                    /lib/x86_64-linux-gnu/librt-2.19.so
7fe7c824c000-7fe7c844b000 ---p 00007000 08:02 2887764                    /lib/x86_64-linux-gnu/librt-2.19.so
7fe7c844b000-7fe7c844c000 r--p 00006000 08:02 2887764                    /lib/x86_64-linux-gnu/librt-2.19.so
7fe7c844c000-7fe7c844d000 rw-p 00007000 08:02 2887764                    /lib/x86_64-linux-gnu/librt-2.19.so
7fe7c844d000-7fe7c860b000 r-xp 00000000 08:02 2887771                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe7c860b000-7fe7c880b000 ---p 001be000 08:02 2887771                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe7c880b000-7fe7c880f000 r--p 001be000 08:02 2887771                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe7c880f000-7fe7c8811000 rw-p 001c2000 08:02 2887771                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe7c8811000-7fe7c8816000 rw-p 00000000 00:00 0 
7fe7c8816000-7fe7c891b000 r-xp 00000000 08:02 2887757                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe7c891b000-7fe7c8b1a000 ---p 00105000 08:02 2887757                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe7c8b1a000-7fe7c8b1b000 r--p 00104000 08:02 2887757                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe7c8b1b000-7fe7c8b1c000 rw-p 00105000 08:02 2887757                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe7c8b1c000-7fe7c8c02000 r-xp 00000000 08:02 1049450                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7fe7c8c02000-7fe7c8e01000 ---p 000e6000 08:02 1049450                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7fe7c8e01000-7fe7c8e09000 r--p 000e5000 08:02 1049450                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7fe7c8e09000-7fe7c8e0b000 rw-p 000ed000 08:02 1049450                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7fe7c8e0b000-7fe7c8e20000 rw-p 00000000 00:00 0 
7fe7c8e20000-7fe7c8e39000 r-xp 00000000 08:02 2887763                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe7c8e39000-7fe7c9038000 ---p 00019000 08:02 2887763                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe7c9038000-7fe7c9039000 r--p 00018000 08:02 2887763                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe7c9039000-7fe7c903a000 rw-p 00019000 08:02 2887763                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe7c903a000-7fe7c903e000 rw-p 00000000 00:00 0 
7fe7c903e000-7fe7c9041000 r-xp 00000000 08:02 2887767                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe7c9041000-7fe7c9240000 ---p 00003000 08:02 2887767                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe7c9240000-7fe7c9241000 r--p 00002000 08:02 2887767                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe7c9241000-7fe7c9242000 rw-p 00003000 08:02 2887767                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe7c9242000-7fe7c93f5000 r-xp 00000000 08:02 2884017                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe7c93f5000-7fe7c95f4000 ---p 001b3000 08:02 2884017                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe7c95f4000-7fe7c960f000 r--p 001b2000 08:02 2884017                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe7c960f000-7fe7c961a000 rw-p 001cd000 08:02 2884017                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe7c961a000-7fe7c961e000 rw-p 00000000 00:00 0 
7fe7c961e000-7fe7c9673000 r-xp 00000000 08:02 2888870                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe7c9673000-7fe7c9873000 ---p 00055000 08:02 2888870                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe7c9873000-7fe7c9876000 r--p 00055000 08:02 2888870                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe7c9876000-7fe7c987d000 rw-p 00058000 08:02 2888870                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe7c987d000-7fe7c987f000 r-xp 00000000 08:02 2887811                    /lib/x86_64-linux-gnu/libsystemd-daemon.so.0.0.10
7fe7c987f000-7fe7c9a7f000 ---p 00002000 08:02 2887811                    /lib/x86_64-linux-gnu/libsystemd-daemon.so.0.0.10
7fe7c9a7f000-7fe7c9a80000 r--p 00002000 08:02 2887811                    /lib/x86_64-linux-gnu/libsystemd-daemon.so.0.0.10
7fe7c9a80000-7fe7c9a81000 rw-p 00003000 08:02 2887811                    /lib/x86_64-linux-gnu/libsystemd-daemon.so.0.0.10
7fe7c9a81000-7fe7c9a8a000 r-xp 00000000 08:02 2887754                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe7c9a8a000-7fe7c9c8a000 ---p 00009000 08:02 2887754                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe7c9c8a000-7fe7c9c8b000 r--p 00009000 08:02 2887754                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe7c9c8b000-7fe7c9c8c000 rw-p 0000a000 08:02 2887754                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe7c9c8c000-7fe7c9cba000 rw-p 00000000 00:00 0 
7fe7c9cba000-7fe7c9cd2000 r-xp 00000000 08:02 2888367                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe7c9cd2000-7fe7c9ed1000 ---p 00018000 08:02 2888367                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe7c9ed1000-7fe7c9ed2000 r--p 00017000 08:02 2888367                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe7c9ed2000-7fe7c9ed3000 rw-p 00018000 08:02 2888367                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe7c9ed3000-7fe7c9edd000 r-xp 00000000 08:02 1051257                    /usr/lib/x86_64-linux-gnu/libnuma.so.1
7fe7c9edd000-7fe7ca0dc000 ---p 0000a000 08:02 1051257                    /usr/lib/x86_64-linux-gnu/libnuma.so.1
7fe7ca0dc000-7fe7ca0dd000 r--p 00009000 08:02 1051257                    /usr/lib/x86_64-linux-gnu/libnuma.so.1
7fe7ca0dd000-7fe7ca0de000 rw-p 0000a000 08:02 1051257                    /usr/lib/x86_64-linux-gnu/libnuma.so.1
7fe7ca0de000-7fe7ca0df000 r-xp 00000000 08:02 2883726                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe7ca0df000-7fe7ca2de000 ---p 00001000 08:02 2883726                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe7ca2de000-7fe7ca2df000 r--p 00000000 08:02 2883726                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe7ca2df000-7fe7ca2e0000 rw-p 00001000 08:02 2883726                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe7ca2e0000-7fe7ca303000 r-xp 00000000 08:02 2887760                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe7ca305000-7fe7ca306000 ---p 00000000 00:00 0 
7fe7ca306000-7fe7ca502000 rw-p 00000000 00:00 0                          [stack:16767]
7fe7ca502000-7fe7ca503000 r--p 00022000 08:02 2887760                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe7ca503000-7fe7ca504000 rw-p 00023000 08:02 2887760                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe7ca504000-7fe7ca505000 rw-p 00000000 00:00 0 
7fe7ca505000-7fe7cb7aa000 r-xp 00000000 08:02 1049092                    /usr/sbin/mysqld
7fe7cb7e2000-7fe7cb7e3000 rw-p 00000000 00:00 0 
7fe7cb7e3000-7fe7cb7e8000 rw-s 00000000 00:0b 4105727355                 /[aio] (deleted)
7fe7cb7e8000-7fe7cb7ed000 rw-s 00000000 00:0b 4105727354                 /[aio] (deleted)
7fe7cb7ed000-7fe7cb7f2000 rw-s 00000000 00:0b 4105727353                 /[aio] (deleted)
7fe7cb7f2000-7fe7cb7f7000 rw-s 00000000 00:0b 4105727352                 /[aio] (deleted)
7fe7cb7f7000-7fe7cb7fc000 rw-s 00000000 00:0b 4105727351                 /[aio] (deleted)
7fe7cb7fc000-7fe7cb801000 rw-s 00000000 00:0b 4105727350                 /[aio] (deleted)
7fe7cb801000-7fe7cb806000 rw-s 00000000 00:0b 4105727349                 /[aio] (deleted)
7fe7cb806000-7fe7cb80b000 rw-s 00000000 00:0b 4105727348                 /[aio] (deleted)
7fe7cb80b000-7fe7cb810000 rw-s 00000000 00:0b 4105727347                 /[aio] (deleted)
7fe7cb810000-7fe7cb99e000 rw-p 00000000 00:00 0 
7fe7cb99e000-7fe7cb9a0000 rw-s 00000000 00:0b 4105727356                 /[aio] (deleted)
7fe7cb9a0000-7fe7cb9a5000 rw-s 00000000 00:0b 4105727346                 /[aio] (deleted)
7fe7cb9a5000-7fe7cb9a7000 rw-s 00000000 00:0b 4105727345                 /[aio] (deleted)
7fe7cb9a7000-7fe7cb9aa000 rw-p 00000000 00:00 0 
7fe7cb9aa000-7fe7cbaaa000 r--p 012a5000 08:02 1049092                    /usr/sbin/mysqld
7fe7cbaaa000-7fe7cbb63000 rw-p 013a5000 08:02 1049092                    /usr/sbin/mysqld
7fe7cbb63000-7fe7cc3fc000 rw-p 00000000 00:00 0 
7fe7cde63000-7fe812e11000 rw-p 00000000 00:00 0                          [heap]
7fff48327000-7fff48348000 rw-p 00000000 00:00 0                          [stack]
7fff483fb000-7fff483fd000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
2018-08-09 10:02:09 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2018-08-09 10:02:09 0 [ERROR] mysqld: Can't open shared library '/usr/lib/mysql/plugin/ha_connect.so' (errno: 22, cannot open shared object file: No such file or directory)
2018-08-09 10:02:09 0 [ERROR] Couldn't load plugins from 'ha_connect.so'.
2018-08-09 10:02:09 0 [Note] InnoDB: Using Linux native AIO
2018-08-09 10:02:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-08-09 10:02:09 0 [Note] InnoDB: Uses event mutexes
2018-08-09 10:02:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-08-09 10:02:09 0 [Note] InnoDB: Number of pools: 1
2018-08-09 10:02:09 0 [Note] InnoDB: Using SSE2 crc32 instructions
2018-08-09 10:02:09 0 [Note] InnoDB: Initializing buffer pool, total size = 45G, instances = 8, chunk size = 128M
2018-08-09 10:02:10 0 [Note] InnoDB: Completed initialization of buffer pool
2018-08-09 10:02:10 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-08-09 10:02:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=38590165237902
2018-08-09 10:02:13 0 [Note] InnoDB: Starting final batch to recover 8899 pages from redo log.
2018-08-09 10:02:15 0 [Note] InnoDB: Last binlog file '/data/bigmomma/mysql/log/mysql-bin.071997', position 84646484
2018-08-09 10:02:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-08-09 10:02:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-08-09 10:02:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-08-09 10:02:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-08-09 10:02:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-08-09 10:02:15 0 [Note] InnoDB: Waiting for purge to start
2018-08-09 10:02:15 0 [Note] InnoDB: 10.3.8 started; log sequence number 38590165278293; transaction id 9916476118
2018-08-09 10:02:15 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-08-09 10:02:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2018-08-09 10:02:15 0 [Note] Recovering after a crash using /data/bigmomma/mysql/log/mysql-bin
2018-08-09 10:02:16 0 [Note] Starting crash recovery...
2018-08-09 10:02:16 0 [Note] Crash recovery finished.

Comment by Elena Stepanova [ 2018-08-09 ]

If it crashes every few minutes, would you be able to enable the general log for one round, so that we could see what exactly is happening there?

Comment by Sander Pilon [ 2018-08-09 ]

I seem to be able to trigger a crash when executing stored procedures.

Total length well exceeds 1000+ lines so I wonder if it would even help if I posted the routines here.

Comment by Sander Pilon [ 2018-08-09 ]

I added the general logfile.

Comment by Elena Stepanova [ 2018-08-09 ]

Thanks, I've downloaded it. If you want (that is, if you don't want to keep it there in public), you can remove it now.

And yes, if you can, please paste or attach the procedure in question anyway. Would you be able to provide the schema dump (only structures, without the data, but all of them, tables, routines, triggers, etc.)?

The fact that stored procedures seem to be involved is extremely important. So far it wasn't the main theory, because the stack traces don't show any involvement of stored procedures, but if it's the case, we have something else to look at. We've had similar bugs before (e.g. MDEV-15746, it was relevant for stored procedures as well, prepared statements and stored procedures often share the same flaws), the ones that we could reliably reproduce are fixed now, but apparently there are more.

Comment by Elena Stepanova [ 2018-08-09 ]

Strangely, the general log didn't contain any flush tables with read lock or any other statements. While locking could be hidden inside the procedure, unlock couldn't.
Did that round also end up with a crash on flush tables ..., or was there something else?

Comment by Sander Pilon [ 2018-08-10 ]

As we've rolled back all servers to 10.1, I cannot verify this anymore.

Unfortunately this puts us in a position where both 10.2 and 10.3 crash our production environment within minutes of coming online. We will stay with 10.1 for now and maybe evaluate what it will take to move forward in 2019.
(An episode like yesterday costs us a lot of time, unfortunately.)

Comment by Elena Stepanova [ 2018-08-10 ]

I'm very sorry about the trouble you experienced, and I understand the frustration.
Most users seem to migrate well, however there is something, either in the workflow, or in the data layout, or in their combination, that causes this problem for some setups or configurations. We are trying to hunt it down.

Comment by Sander Pilon [ 2018-08-13 ]

I understand. Low traffic tests on our testing setup work fine. This is about the heart of the problem, as we would love to debug this with you but at the moment (as last year) this only occurs on production level traffic. We simply cannot afford to have our business offline for this purpose - unfortunately.

Comment by Marko Mäkelä [ 2018-12-31 ]

On a too quick look, it seemed that this failure could share a root cause with the following:
MDEV-12023 Assertion failure sym_node->table != NULL on startup

But, it is claimed that the failure only occurs when the system is heavily loaded. The MDEV-12023 scenario only applies on the very first startup when upgrading the server.

There was no core dump on all threads, so I cannot be sure whether this could duplicate MDEV-15488, which is reported for 10.2, but I think that it is plausible.

Comment by Marko Mäkelä [ 2023-10-04 ]

I think that my previous comment was in response to CrewOne’s message that included an InnoDB crash (which could have been fixed in MariaDB Server 10.3.11 by MDEV-12023), and a stack trace of a possibly unrelated thread:

2018-08-09 10:02:08 0x7fdbef2f9700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/que/que0que.cc line 563

The stack trace in the Description (crash during FLUSH TABLES WITH READ LOCK) could be unrelated to MDEV-12023. There have been some bugs fixed in the .frm file parsing. Some memory corruption due to such bugs might be the cause of the crash.

Comment by Mason Sharp [ 2023-10-19 ]

marko so yes ok to close?

Generated at Thu Feb 08 08:32:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.