[MDEV-18223] Crash during insert (?) Created: 2019-01-13  Updated: 2020-05-04  Resolved: 2020-03-17

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.21
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Javier Zon Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 1
Labels: need_feedback
Environment:

CentOS 7

Attachments: File gdb.txt.gz    
Issue Links:
Relates
relates to MDEV-4609 mariadb server repeatedly crashes Closed
relates to MDEV-10640 Failure to start after power failure ... Closed
relates to MDEV-19666 Maria DB server crases during an insert Closed
relates to MDEV-22397 repeatable crashes: InnoDB: Assertion... Closed
relates to MDEV-22397 repeatable crashes: InnoDB: Assertion... Closed

 Description    

2019-01-13 13:06:16 0x7fb5106ba700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc line 348
 
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
190113 13:06:16 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.21-MariaDB-log
 
key_buffer_size=65536
 
read_buffer_size=2097152
 
max_used_connections=131
 
max_threads=1002
 
thread_count=53
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18490261 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7fb4086c7338
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fb5106b9d30 thread_stack 0x3c000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5585674bcdbe]
 
/usr/sbin/mysqld(handle_fatal_signal+0x355)[0x558566f43ec5]
 
/lib64/libpthread.so.0(+0xf6d0)[0x7fba650446d0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7fba63318277]
 
/lib64/libc.so.6(abort+0x148)[0x7fba63319968]
 
/usr/sbin/mysqld(+0x423b9b)[0x558566cf8b9b]
 
/usr/sbin/mysqld(+0x9c4f70)[0x558567299f70]
 
/usr/sbin/mysqld(+0x9cc1b3)[0x5585672a11b3]
 
/usr/sbin/mysqld(+0x900114)[0x5585671d5114]
 
/usr/sbin/mysqld(+0x9042e2)[0x5585671d92e2]
 
/usr/sbin/mysqld(+0x906821)[0x5585671db821]
 
/usr/sbin/mysqld(+0x91651a)[0x5585671eb51a]
 
/usr/sbin/mysqld(+0x85779f)[0x55856712c79f]
 
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x413)[0x558566f4e923]
 
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6f)[0x558566d9371f]
 
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x140e)[0x558566d99b6e]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d0e)[0x558566daef9e]
 
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4ee)[0x558566dc9e2e]
 
/usr/sbin/mysqld(+0x4f50b0)[0x558566dca0b0]
 
/usr/sbin/mysqld(+0x4f5e67)[0x558566dcae67]
 
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558566dcaef7]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xdbc)[0x558566db874c]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x558566dba785]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x558566e81d2a]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x558566e81e4d]
 
/lib64/libpthread.so.0(+0x7e25)[0x7fba6503ce25]
 
/lib64/libc.so.6(clone+0x6d)[0x7fba633e0bad]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fb4086bc0d0): insert into `oauth_access_tokens` (`id`, `user_id`, `client_id`, `scopes`, `revoked`, `created_at`, `updated_at`, `expires_at`) values ('1cb23df69eecf5ba05520955cca2b253502a0f193bb8507378f8c647654e9ea48656658dd26a3b67', 1, '5', '[\"*\"]', 0, '2019-01-13 12:06:16', '2019-01-13 12:06:16', '2020-01-13 12:06:16')
 
Connection ID (thread ID): 28992
 
Status: NOT_KILLED



 Comments   
Comment by Javier Zon [ 2019-01-13 ]

server.cnf

[mysqld]
 
 
 
# generic configuration options
 
port= 3306
 
socket= /data/mysql/mysql.sock
 
log-error                      = /data/logs/error.log
 
datadir=/data/mysql
 
open-files-limit = 1024000
 
innodb_undo_logs=1
 
innodb_safe_truncate=OFF
 
back_log = 5
 
max_connections = 1000
 
max_connect_errors = 100000
 
table_open_cache = 4098
 
sql_mode                       = NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION  
max_allowed_packet = 64M
 
binlog_cache_size = 1M
 
max_heap_table_size = 256M
 
read_buffer_size = 2M
 
read_rnd_buffer_size = 16M
 
sort_buffer_size = 16M
 
join_buffer_size = 16M
 
thread_cache_size = 12
 
thread_concurrency = 8
 
query_cache_size = 0
 
query_cache_limit = 2M
 
default-storage-engine = INNODB
 
thread_stack = 240K
 
transaction_isolation = READ-COMMITTED
 
tmp_table_size = 256M
 
log-bin=/data/binlogs/mysql-bin
 
binlog_format=row
 
expire_logs_days = 1
 
slow_query_log
 
long_query_time = 10
 
tmpdir = /data/tmp
 
server-id = 10050
 
#read_only
 
key_buffer_size = 64K
 
bulk_insert_buffer_size = 64M
 
myisam_sort_buffer_size = 64M
 
myisam_max_sort_file_size = 64M
 
myisam_repair_threads = 1
 
innodb_buffer_pool_size = 20G
 
innodb_data_file_path = ibdata1:10M:autoextend
 
innodb_data_home_dir = /data/mysql
 
innodb_thread_concurrency = 8
 
innodb_flush_log_at_trx_commit = 2
 
innodb_log_buffer_size = 16M
 
innodb_log_file_size = 2G
 
innodb_log_files_in_group = 2
 
innodb_log_group_home_dir = /data/iblogfiles
 
innodb_max_dirty_pages_pct = 90
 
innodb_flush_method=O_DIRECT
 
innodb_lock_wait_timeout = 120
 
 
 
[mysql]
 
no-auto-rehash

Comment by Elena Stepanova [ 2019-01-13 ]

Is it reproducible by re-running the same or similar INSERT?

Comment by Javier Zon [ 2019-01-13 ]

Yes, doing the same insert crashed

Comment by Javier Zon [ 2019-01-13 ] 

2019-01-13 14:30:39 0x7f5e5c07f700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc line 348
 
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
190113 14:30:39 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.21-MariaDB-log
 
key_buffer_size=65536
 
read_buffer_size=2097152
 
max_used_connections=178
 
max_threads=1002
 
thread_count=27
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18490261 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f5ddc3abb48
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f5e5c07ed30 thread_stack 0x3c000
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f63a9f836e7]
 
/lib64/libc.so.6(+0x116862)[0x7f63a9f81862]
 
/lib64/libc.so.6(+0x118647)[0x7f63a9f83647]
 
/usr/sbin/mysqld(my_addr_resolve+0xda)[0x56117a42087a]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x56117a409f52]
 
/usr/sbin/mysqld(handle_fatal_signal+0x355)[0x561179e90ec5]
 
/lib64/libpthread.so.0(+0xf6d0)[0x7f63abbcd6d0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7f63a9ea1277]
 
/lib64/libc.so.6(abort+0x148)[0x7f63a9ea2968]
 
/usr/sbin/mysqld(+0x423b9b)[0x561179c45b9b]
 
/usr/sbin/mysqld(+0x9c4f70)[0x56117a1e6f70]
 
/usr/sbin/mysqld(+0x9cc1b3)[0x56117a1ee1b3]
 
/usr/sbin/mysqld(+0x900114)[0x56117a122114]
 
/usr/sbin/mysqld(+0x9042e2)[0x56117a1262e2]
 
/usr/sbin/mysqld(+0x906821)[0x56117a128821]
 
/usr/sbin/mysqld(+0x91651a)[0x56117a13851a]
 
/usr/sbin/mysqld(+0x85779f)[0x56117a07979f]
 
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x413)[0x561179e9b923]
 
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6f)[0x561179ce071f]
 
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x140e)[0x561179ce6b6e]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d0e)[0x561179cfbf9e]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x561179d0390e]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x21ea)[0x561179d06b7a]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x561179d07785]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x561179dced2a]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x561179dcee4d]
 
/lib64/libpthread.so.0(+0x7e25)[0x7f63abbc5e25]
 
/lib64/libc.so.6(clone+0x6d)[0x7f63a9f69bad]
 
======= Memory map: ========
 
561179822000-56117a93e000 r-xp 00000000 fd:00 50872                      /usr/sbin/mysqld
 
56117ab3e000-56117ac17000 r--p 0111c000 fd:00 50872                      /usr/sbin/mysqld
 
56117ac17000-56117acce000 rw-p 011f5000 fd:00 50872                      /usr/sbin/mysqld
 
56117acce000-56117b562000 rw-p 00000000 00:00 0 
56117d145000-56119a334000 rw-p 00000000 00:00 0                          [heap]
 
7f5d44000000-7f5d455e9000 rw-p 00000000 00:00 0 
7f5d455e9000-7f5d48000000 ---p 00000000 00:00 0 
7f5d48000000-7f5d48f62000 rw-p 00000000 00:00 0 
7f5d48f62000-7f5d4c000000 ---p 00000000 00:00 0 
7f5d4c000000-7f5d4d5bf000 rw-p 00000000 00:00 0 
7f5d4d5bf000-7f5d50000000 ---p 00000000 00:00 0 
7f5d50000000-7f5d511cc000 rw-p 00000000 00:00 0 
7f5d511cc000-7f5d54000000 ---p 00000000 00:00 0 
7f5d54000000-7f5d54b9a000 rw-p 00000000 00:00 0 
7f5d54b9a000-7f5d58000000 ---p 00000000 00:00 0 
7f5d58000000-7f5d58eb2000 rw-p 00000000 00:00 0 
7f5d58eb2000-7f5d5c000000 ---p 00000000 00:00 0 
7f5d5c000000-7f5d5ce8d000 rw-p 00000000 00:00 0 
7f5d5ce8d000-7f5d60000000 ---p 00000000 00:00 0 
7f5d60000000-7f5d61671000 rw-p 00000000 00:00 0 
7f5d61671000-7f5d64000000 ---p 00000000 00:00 0 
7f5d64000000-7f5d65220000 rw-p 00000000 00:00 0 
7f5d65220000-7f5d68000000 ---p 00000000 00:00 0 
7f5d68000000-7f5d69cc9000 rw-p 00000000 00:00 0 
7f5d69cc9000-7f5d6c000000 ---p 00000000 00:00 0 
7f5d6c000000-7f5d6cd05000 rw-p 00000000 00:00 0 
7f5d6cd05000-7f5d70000000 ---p 00000000 00:00 0 
7f5d70000000-7f5d711d7000 rw-p 00000000 00:00 0 
7f5d711d7000-7f5d74000000 ---p 00000000 00:00 0 
7f5d74000000-7f5d74cde000 rw-p 00000000 00:00 0 
7f5d74cde000-7f5d78000000 ---p 00000000 00:00 0 
7f5d78000000-7f5d79081000 rw-p 00000000 00:00 0 
7f5d79081000-7f5d7c000000 ---p 00000000 00:00 0 
7f5d7c000000-7f5d7da57000 rw-p 00000000 00:00 0 
7f5d7da57000-7f5d80000000 ---p 00000000 00:00 0 
7f5d80000000-7f5d81ffb000 rw-p 00000000 00:00 0 
7f5d81ffb000-7f5d84000000 ---p 00000000 00:00 0 
7f5d84000000-7f5d855b9000 rw-p 00000000 00:00 0 
7f5d855b9000-7f5d88000000 ---p 00000000 00:00 0 
7f5d88000000-7f5d88f83000 rw-p 00000000 00:00 0 
7f5d88f83000-7f5d8c000000 ---p 00000000 00:00 0 
7f5d8c000000-7f5d8db87000 rw-p 00000000 00:00 0 
7f5d8db87000-7f5d90000000 ---p 00000000 00:00 0 
7f5d90000000-7f5d9096a000 rw-p 00000000 00:00 0 
7f5d9096a000-7f5d94000000 ---p 00000000 00:00 0 
7f5d94000000-7f5d94b17000 rw-p 00000000 00:00 0 
7f5d94b17000-7f5d98000000 ---p 00000000 00:00 0 
7f5d98000000-7f5d9a0c7000 rw-p 00000000 00:00 0 
7f5d9a0c7000-7f5d9c000000 ---p 00000000 00:00 0 
7f5d9c000000-7f5d9d479000 rw-p 00000000 00:00 0 
7f5d9d479000-7f5da0000000 ---p 00000000 00:00 0 
7f5da0000000-7f5da1145000 rw-p 00000000 00:00 0 
7f5da1145000-7f5da4000000 ---p 00000000 00:00 0 
7f5da4000000-7f5da4f35000 rw-p 00000000 00:00 0 
7f5da4f35000-7f5da8000000 ---p 00000000 00:00 0 
7f5da8000000-7f5da93a9000 rw-p 00000000 00:00 0 
7f5da93a9000-7f5dac000000 ---p 00000000 00:00 0 
7f5dac000000-7f5dad4bc000 rw-p 00000000 00:00 0 
7f5dad4bc000-7f5db0000000 ---p 00000000 00:00 0 
7f5db0000000-7f5db12ab000 rw-p 00000000 00:00 0 
7f5db12ab000-7f5db4000000 ---p 00000000 00:00 0 
7f5db4000000-7f5db4a44000 rw-p 00000000 00:00 0 
7f5db4a44000-7f5db8000000 ---p 00000000 00:00 0 
7f5db8000000-7f5db99f9000 rw-p 00000000 00:00 0 
7f5db99f9000-7f5dbc000000 ---p 00000000 00:00 0 
7f5dbc000000-7f5dbcc93000 rw-p 00000000 00:00 0 
7f5dbcc93000-7f5dc0000000 ---p 00000000 00:00 0 
7f5dc0000000-7f5dc16f7000 rw-p 00000000 00:00 0 
7f5dc16f7000-7f5dc4000000 ---p 00000000 00:00 0 
7f5dc4000000-7f5dc5474000 rw-p 00000000 00:00 0 
7f5dc5474000-7f5dc8000000 ---p 00000000 00:00 0 
7f5dc8000000-7f5dc8965000 rw-p 00000000 00:00 0 
7f5dc8965000-7f5dcc000000 ---p 00000000 00:00 0 
7f5dcc000000-7f5dccc7e000 rw-p 00000000 00:00 0 
7f5dccc7e000-7f5dd0000000 ---p 00000000 00:00 0 
7f5dd0000000-7f5dd1455000 rw-p 00000000 00:00 0 
7f5dd1455000-7f5dd4000000 ---p 00000000 00:00 0 
7f5dd4000000-7f5dd5522000 rw-p 00000000 00:00 0 
7f5dd5522000-7f5dd8000000 ---p 00000000 00:00 0 
7f5dd8000000-7f5dd88f7000 rw-p 00000000 00:00 0 
7f5dd88f7000-7f5ddc000000 ---p 00000000 00:00 0 
7f5ddc000000-7f5ddd846000 rw-p 00000000 00:00 0 
7f5ddd846000-7f5de0000000 ---p 00000000 00:00 0 
7f5de0000000-7f5de11ce000 rw-p 00000000 00:00 0 
7f5de11ce000-7f5de4000000 ---p 00000000 00:00 0 
7f5de4000000-7f5de4f09000 rw-p 00000000 00:00 0 
7f5de4f09000-7f5de8000000 ---p 00000000 00:00 0 
7f5de8000000-7f5de917d000 rw-p 00000000 00:00 0 
7f5de917d000-7f5dec000000 ---p 00000000 00:00 0 
7f5dec000000-7f5decdf1000 rw-p 00000000 00:00 0 
7f5decdf1000-7f5df0000000 ---p 00000000 00:00 0 
7f5df0000000-7f5df1dfa000 rw-p 00000000 00:00 0 
7f5df1dfa000-7f5df4000000 ---p 00000000 00:00 0 
7f5df4000000-7f5df47be000 rw-p 00000000 00:00 0 
7f5df47be000-7f5df8000000 ---p 00000000 00:00 0 
7f5df8000000-7f5df93d0000 rw-p 00000000 00:00 0 
7f5df93d0000-7f5dfc000000 ---p 00000000 00:00 0 
7f5dfc000000-7f5dfd82d000 rw-p 00000000 00:00 0 
7f5dfd82d000-7f5e00000000 ---p 00000000 00:00 0 
7f5e00000000-7f5e0091f000 rw-p 00000000 00:00 0 
7f5e0091f000-7f5e04000000 ---p 00000000 00:00 0 
7f5e04000000-7f5e05689000 rw-p 00000000 00:00 0 
7f5e05689000-7f5e08000000 ---p 00000000 00:00 0 
7f5e08000000-7f5e08f07000 rw-p 00000000 00:00 0 
7f5e08f07000-7f5e0c000000 ---p 00000000 00:00 0 
7f5e0c000000-7f5e0cc7b000 rw-p 00000000 00:00 0 
7f5e0cc7b000-7f5e10000000 ---p 00000000 00:00 0 
7f5e10000000-7f5e114c2000 rw-p 00000000 00:00 0 
7f5e114c2000-7f5e14000000 ---p 00000000 00:00 0 
7f5e14000000-7f5e156aa000 rw-p 00000000 00:00 0 
7f5e156aa000-7f5e18000000 ---p 00000000 00:00 0 
7f5e18000000-7f5e18907000 rw-p 00000000 00:00 0 
7f5e18907000-7f5e1c000000 ---p 00000000 00:00 0 
7f5e1c000000-7f5e1c63e000 rw-p 00000000 00:00 0 
7f5e1c63e000-7f5e20000000 ---p 00000000 00:00 0 
7f5e21778000-7f5e24000000 rw-p 00000000 00:00 0 
7f5e24000000-7f5e24e44000 rw-p 00000000 00:00 0 
7f5e24e44000-7f5e28000000 ---p 00000000 00:00 0 
7f5e29778000-7f5e2c000000 rw-p 00000000 00:00 0 
7f5e2c000000-7f5e2ce34000 rw-p 00000000 00:00 0 
7f5e2ce34000-7f5e30000000 ---p 00000000 00:00 0 
7f5e30000000-7f5e30d17000 rw-p 00000000 00:00 0 
7f5e30d17000-7f5e34000000 ---p 00000000 00:00 0 
7f5e34000000-7f5e34fd6000 rw-p 00000000 00:00 0 
7f5e34fd6000-7f5e38000000 ---p 00000000 00:00 0 
7f5e38cc1000-7f5e38ec6000 rw-p 00000000 00:00 0 
7f5e38ec6000-7f5e38ec7000 ---p 00000000 00:00 0 
7f5e38ec7000-7f5e396c7000 rw-p 00000000 00:00 0 
7f5e396c7000-7f5e396c8000 ---p 00000000 00:00 0 
7f5e396c8000-7f5e39ec8000 rw-p 00000000 00:00 0 
7f5e39ec8000-7f5e39ec9000 ---p 00000000 00:00 0 
7f5e39ec9000-7f5e3a6c9000 rw-p 00000000 00:00 0 
7f5e3a6c9000-7f5e3a6ca000 ---p 00000000 00:00 0 
7f5e3a6ca000-7f5e3aeca000 rw-p 00000000 00:00 0 
7f5e3aeca000-7f5e3aecb000 ---p 00000000 00:00 0 
7f5e3aecb000-7f5e3b6cb000 rw-p 00000000 00:00 0 
7f5e3b6cb000-7f5e3b6cc000 ---p 00000000 00:00 0 
7f5e3b6cc000-7f5e3becc000 rw-p 00000000 00:00 0 
7f5e3becc000-7f5e3becd000 ---p 00000000 00:00 0 
7f5e3becd000-7f5e3c6cd000 rw-p 00000000 00:00 0 
7f5e3c6cd000-7f5e3c6ce000 ---p 00000000 00:00 0 
7f5e3c6ce000-7f5e3cece000 rw-p 00000000 00:00 0 
7f5e3cece000-7f5e3cecf000 ---p 00000000 00:00 0 
7f5e3cecf000-7f5e3d6cf000 rw-p 00000000 00:00 0 
7f5e3d6cf000-7f5e3d6d0000 ---p 00000000 00:00 0 
7f5e3d6d0000-7f5e3ded0000 rw-p 00000000 00:00 0 
7f5e3ded0000-7f5e3ded1000 ---p 00000000 00:00 0 
7f5e3ded1000-7f5e3e6d1000 rw-p 00000000 00:00 0 
7f5e3e6d1000-7f5e3e6d2000 ---p 00000000 00:00 0 
7f5e3e6d2000-7f5e3eed2000 rw-p 00000000 00:00 0 
7f5e3eed2000-7f5e3eed3000 ---p 00000000 00:00 0 
7f5e3eed3000-7f5e3f6d3000 rw-p 00000000 00:00 0 
7f5e3f6d3000-7f5e3f6d4000 ---p 00000000 00:00 0 
7f5e3f6d4000-7f5e496d5000 rw-p 00000000 00:00 0 
7f5e4bab0000-7f5e4bab1000 ---p 00000000 00:00 0 
7f5e4bab1000-7f5e4baee000 rw-p 00000000 00:00 0 
7f5e4baee000-7f5e4baef000 ---p 00000000 00:00 0 
7f5e4baef000-7f5e4bb2c000 rw-p 00000000 00:00 0 
7f5e4bb2c000-7f5e4bb2d000 ---p 00000000 00:00 0 
7f5e4bb2d000-7f5e4bb6a000 rw-p 00000000 00:00 0 
7f5e4bb6a000-7f5e4bb6b000 ---p 00000000 00:00 0 
7f5e4bb6b000-7f5e4bba8000 rw-p 00000000 00:00 0 
7f5e4bba8000-7f5e4bba9000 ---p 00000000 00:00 0 
7f5e4bba9000-7f5e4bbe6000 rw-p 00000000 00:00 0 
7f5e4bbe6000-7f5e4bbe7000 ---p 00000000 00:00 0 
7f5e4bbe7000-7f5e4bc24000 rw-p 00000000 00:00 0 
7f5e4bc24000-7f5e4bc25000 ---p 00000000 00:00 0 
7f5e4bc25000-7f5e4bc62000 rw-p 00000000 00:00 0 
7f5e4bc62000-7f5e4bc63000 ---p 00000000 00:00 0 
7f5e4bc63000-7f5e4bca0000 rw-p 00000000 00:00 0 
7f5e4bca0000-7f5e4bca1000 ---p 00000000 00:00 0 
7f5e4bca1000-7f5e4bcde000 rw-p 00000000 00:00 0 
7f5e4bcde000-7f5e4bcdf000 ---p 00000000 00:00 0 
7f5e4bcdf000-7f5e4bd1c000 rw-p 00000000 00:00 0 
7f5e4bd1c000-7f5e4bd1d000 ---p 00000000 00:00 0 
7f5e4bd1d000-7f5e4bd5a000 rw-p 00000000 00:00 0 
7f5e4bd5a000-7f5e4bd5b000 ---p 00000000 00:00 0 
7f5e4bd5b000-7f5e4bd98000 rw-p 00000000 00:00 0 
7f5e4bd98000-7f5e4bd99000 ---p 00000000 00:00 0 
7f5e4bd99000-7f5e4bdd6000 rw-p 00000000 00:00 0 
7f5e4bdd6000-7f5e4bdd7000 ---p 00000000 00:00 0 
7f5e4bdd7000-7f5e4be14000 rw-p 00000000 00:00 0 
7f5e4be14000-7f5e4be15000 ---p 00000000 00:00 0 
7f5e4be15000-7f5e4be52000 rw-p 00000000 00:00 0 
7f5e4be52000-7f5e4be53000 ---p 00000000 00:00 0 
7f5e4be53000-7f5e4be90000 rw-p 00000000 00:00 0 
7f5e4be90000-7f5e4be91000 ---p 00000000 00:00 0 
7f5e4be91000-7f5e4bece000 rw-p 00000000 00:00 0 
7f5e4bece000-7f5e4becf000 ---p 00000000 00:00 0 
7f5e4becf000-7f5e4bf0c000 rw-p 00000000 00:00 0 
7f5e4bf0c000-7f5e4bf0d000 ---p 00000000 00:00 0 
7f5e4bf0d000-7f5e4bf4a000 rw-p 00000000 00:00 0 
7f5e4bf4a000-7f5e4bf4b000 ---p 00000000 00:00 0 
7f5e4bf4b000-7f5e4bf88000 rw-p 00000000 00:00 0 
7f5e4bf88000-7f5e4bf89000 ---p 00000000 00:00 0 
7f5e4bf89000-7f5e4bfc6000 rw-p 00000000 00:00 0 
7f5e4bfc6000-7f5e4bfc7000 ---p 00000000 00:00 0 
7f5e4bfc7000-7f5e4c004000 rw-p 00000000 00:00 0 
7f5e4c004000-7f5e4c005000 ---p 00000000 00:00 0 
7f5e4c005000-7f5e4c042000 rw-p 00000000 00:00 0 
7f5e4c042000-7f5e4c043000 ---p 00000000 00:00 0 
7f5e4c043000-7f5e4c080000 rw-p 00000000 00:00 0 
7f5e4c080000-7f5e4c081000 ---p 00000000 00:00 0 
7f5e4c081000-7f5e4c0be000 rw-p 00000000 00:00 0 
7f5e4c0be000-7f5e4c0bf000 ---p 00000000 00:00 0 
7f5e4c0bf000-7f5e4c0fc000 rw-p 00000000 00:00 0 
7f5e4c0fc000-7f5e4c0fd000 ---p 00000000 00:00 0 
7f5e4c0fd000-7f5e4c13a000 rw-p 00000000 00:00 0 
7f5e4c13a000-7f5e4c13b000 ---p 00000000 00:00 0 
7f5e4c13b000-7f5e4c178000 rw-p 00000000 00:00 0 
7f5e4c178000-7f5e4c179000 ---p 00000000 00:00 0 
7f5e4c179000-7f5e4c1b6000 rw-p 00000000 00:00 0 
7f5e4c1b6000-7f5e4c1b7000 ---p 00000000 00:00 0 
7f5e4c1b7000-7f5e4c1f4000 rw-p 00000000 00:00 0 
7f5e4c1f4000-7f5e4c1f5000 ---p 00000000 00:00 0 
7f5e4c1f5000-7f5e4c232000 rw-p 00000000 00:00 0 
7f5e4c232000-7f5e4c233000 ---p 00000000 00:00 0 
7f5e4c233000-7f5e4c270000 rw-p 00000000 00:00 0 
7f5e4c270000-7f5e4c271000 ---p 00000000 00:00 0 
7f5e4c271000-7f5e4c2ae000 rw-p 00000000 00:00 0 
7f5e4c2ae000-7f5e4c2af000 ---p 00000000 00:00 0 
7f5e4c2af000-7f5e4c2ec000 rw-p 00000000 00:00 0 
7f5e4c2ec000-7f5e4c2ed000 ---p 00000000 00:00 0 
7f5e4c2ed000-7f5e4c32a000 rw-p 00000000 00:00 0 
7f5e4c32a000-7f5e4c32b000 ---p 00000000 00:00 0 
7f5e4c32b000-7f5e4c368000 rw-p 00000000 00:00 0 
7f5e4c368000-7f5e4c369000 ---p 00000000 00:00 0 
7f5e4c369000-7f5e4c3a6000 rw-p 00000000 00:00 0 
7f5e4c3a6000-7f5e4c3a7000 ---p 00000000 00:00 0 
7f5e4c3a7000-7f5e4c3e4000 rw-p 00000000 00:00 0 
7f5e4c3e4000-7f5e4c3e5000 ---p 00000000 00:00 0 
7f5e4c3e5000-7f5e4c422000 rw-p 00000000 00:00 0 
7f5e4c422000-7f5e4c423000 ---p 00000000 00:00 0 
7f5e4c423000-7f5e4c460000 rw-p 00000000 00:00 0 
7f5e4c460000-7f5e4c461000 ---p 00000000 00:00 0 
7f5e4c461000-7f5e4c49e000 rw-p 00000000 00:00 0 
7f5e4c49e000-7f5e4c49f000 ---p 00000000 00:00 0 
7f5e4c49f000-7f5e4c4dc000 rw-p 00000000 00:00 0 
7f5e4c4dc000-7f5e4c4dd000 ---p 00000000 00:00 0 
7f5e4c4dd000-7f5e4c51a000 rw-p 00000000 00:00 0 
7f5e4c51a000-7f5e4c51b000 ---p 00000000 00:00 0 
7f5e4c51b000-7f5e4c558000 rw-p 00000000 00:00 0 
7f5e4c558000-7f5e4c559000 ---p 00000000 00:00 0 
7f5e4c559000-7f5e4c596000 rw-p 00000000 00:00 0 
7f5e4c596000-7f5e4c597000 ---p 00000000 00:00 0 
7f5e4c597000-7f5e4c5d4000 rw-p 00000000 00:00 0 
7f5e4c5d4000-7f5e4c5d5000 ---p 00000000 00:00 0 
7f5e4c5d5000-7f5e4c612000 rw-p 00000000 00:00 0 
7f5e4c612000-7f5e4c613000 ---p 00000000 00:00 0 
7f5e4c613000-7f5e4c650000 rw-p 00000000 00:00 0 
7f5e4c650000-7f5e4c651000 ---p 00000000 00:00 0 
7f5e4c651000-7f5e4c68e000 rw-p 00000000 00:00 0 
7f5e4c68e000-7f5e4c68f000 ---p 00000000 00:00 0 
7f5e4c68f000-7f5e4c6cc000 rw-p 00000000 00:00 0 
7f5e4c6cc000-7f5e4c6cd000 ---p 00000000 00:00 0 
7f5e4c6cd000-7f5e4c70a000 rw-p 00000000 00:00 0 
7f5e4c70a000-7f5e4c70b000 ---p 00000000 00:00 0 
7f5e4c70b000-7f5e4c748000 rw-p 00000000 00:00 0 
7f5e4c748000-7f5e4c749000 ---p 00000000 00:00 0 
7f5e4c749000-7f5e4c786000 rw-p 00000000 00:00 0 
7f5e4c786000-7f5e4c787000 ---p 00000000 00:00 0 
7f5e4c787000-7f5e4c7c4000 rw-p 00000000 00:00 0 
7f5e4c7c4000-7f5e4c7c5000 ---p 00000000 00:00 0 
7f5e4c7c5000-7f5e4c802000 rw-p 00000000 00:00 0 
7f5e4c802000-7f5e4c803000 ---p 00000000 00:00 0 
7f5e4c803000-7f5e4c840000 rw-p 00000000 00:00 0 
7f5e4c840000-7f5e4c841000 ---p 00000000 00:00 0 
7f5e4c841000-7f5e4c87e000 rw-p 00000000 00:00 0 
7f5e4c87e000-7f5e4c87f000 ---p 00000000 00:00 0 
7f5e4c87f000-7f5e4c8bc000 rw-p 00000000 00:00 0 
7f5e4c8bc000-7f5e4c8bd000 ---p 00000000 00:00 0 
7f5e4c8bd000-7f5e4c8fa000 rw-p 00000000 00:00 0 
7f5e4c8fa000-7f5e4c8fb000 ---p 00000000 00:00 0 
7f5e4c8fb000-7f5e4c938000 rw-p 00000000 00:00 0 
7f5e4c938000-7f5e4c939000 ---p 00000000 00:00 0 
7f5e4c939000-7f5e4c976000 rw-p 00000000 00:00 0 
7f5e4c976000-7f5e4c977000 ---p 00000000 00:00 0 
7f5e4c977000-7f5e4c9b4000 rw-p 00000000 00:00 0 
7f5e4c9b4000-7f5e4c9b5000 ---p 00000000 00:00 0 
7f5e4c9b5000-7f5e4c9f2000 rw-p 00000000 00:00 0 
7f5e4c9f2000-7f5e4c9f3000 ---p 00000000 00:00 0 
7f5e4c9f3000-7f5e4ca30000 rw-p 00000000 00:00 0 
7f5e4ca30000-7f5e4ca31000 ---p 00000000 00:00 0 
7f5e4ca31000-7f5e4ca6e000 rw-p 00000000 00:00 0 
7f5e4ca6e000-7f5e4ca6f000 ---p 00000000 00:00 0 
7f5e4ca6f000-7f5e4caac000 rw-p 00000000 00:00 0 
7f5e4caac000-7f5e4caad000 ---p 00000000 00:00 0 
7f5e4caad000-7f5e4caea000 rw-p 00000000 00:00 0 
7f5e4caea000-7f5e4caeb000 ---p 00000000 00:00 0 
7f5e4caeb000-7f5e4cb28000 rw-p 00000000 00:00 0 
7f5e4cb28000-7f5e4cb29000 ---p 00000000 00:00 0 
7f5e4cb29000-7f5e4cb66000 rw-p 00000000 00:00 0 
7f5e4cb66000-7f5e4cb67000 ---p 00000000 00:00 0 
7f5e4cb67000-7f5e4cba4000 rw-p 00000000 00:00 0 
7f5e4cba4000-7f5e4cba5000 ---p 00000000 00:00 0 
7f5e4cba5000-7f5e4cbe2000 rw-p 00000000 00:00 0 
7f5e4cbe2000-7f5e4cbe3000 ---p 00000000 00:00 0 
7f5e4cbe3000-7f5e4cc20000 rw-p 00000000 00:00 0 
7f5e4cc20000-7f5e4cc21000 ---p 00000000 00:00 0 
7f5e4cc21000-7f5e4cc5e000 rw-p 00000000 00:00 0 
7f5e4cc5e000-7f5e4cc5f000 ---p 00000000 00:00 0 
7f5e4cc5f000-7f5e4cc9c000 rw-p 00000000 00:00 0 
7f5e4cc9c000-7f5e4cc9d000 ---p 00000000 00:00 0 
7f5e4cc9d000-7f5e4ccda000 rw-p 00000000 00:00 0 
7f5e4ccda000-7f5e4ccdb000 ---p 00000000 00:00 0 
7f5e4ccdb000-7f5e4cd18000 rw-p 00000000 00:00 0 
7f5e4cd18000-7f5e4cd19000 ---p 00000000 00:00 0 
7f5e4cd19000-7f5e4cd56000 rw-p 00000000 00:00 0 
7f5e4cd56000-7f5e4cd57000 ---p 00000000 00:00 0 
7f5e4cd57000-7f5e4cd94000 rw-p 00000000 00:00 0 
7f5e4cd94000-7f5e4cd95000 ---p 00000000 00:00 0 
7f5e4cd95000-7f5e4cdd2000 rw-p 00000000 00:00 0 
7f5e4cdd2000-7f5e4cdd3000 ---p 00000000 00:00 0 
7f5e4cdd3000-7f5e4ce10000 rw-p 00000000 00:00 0 
7f5e4ce10000-7f5e4ce11000 ---p 00000000 00:00 0 
7f5e4ce11000-7f5e4ce4e000 rw-p 00000000 00:00 0 
7f5e4ce4e000-7f5e4ce4f000 ---p 00000000 00:00 0 
7f5e4ce4f000-7f5e4ce8c000 rw-p 00000000 00:00 0 
7f5e4ce8c000-7f5e4ce8d000 ---p 00000000 00:00 0 
7f5e4ce8d000-7f5e4ceca000 rw-p 00000000 00:00 0 
7f5e4ceca000-7f5e4cecb000 ---p 00000000 00:00 0 
7f5e4cecb000-7f5e4cf08000 rw-p 00000000 00:00 0 
7f5e4cf08000-7f5e4cf09000 ---p 00000000 00:00 0 
7f5e4cf09000-7f5e4cf46000 rw-p 00000000 00:00 0 
7f5e4cf46000-7f5e4cf47000 ---p 00000000 00:00 0 
7f5e4cf47000-7f5e4cf84000 rw-p 00000000 00:00 0 
7f5e4cf84000-7f5e4cf85000 ---p 00000000 00:00 0 
7f5e4cf85000-7f5e4cfc2000 rw-p 00000000 00:00 0 
7f5e4cfc2000-7f5e4cfc3000 ---p 00000000 00:00 0 
7f5e4cfc3000-7f5e4d000000 rw-p 00000000 00:00 0 
7f5e4d000000-7f5e4d001000 ---p 00000000 00:00 0 
7f5e4d001000-7f5e4d03e000 rw-p 00000000 00:00 0 
7f5e4d03e000-7f5e4d03f000 ---p 00000000 00:00 0 
7f5e4d03f000-7f5e4d07c000 rw-p 00000000 00:00 0 
7f5e4d07c000-7f5e4d07d000 ---p 00000000 00:00 0 
7f5e4d07d000-7f5e4d0ba000 rw-p 00000000 00:00 0 
7f5e4d0ba000-7f5e4d0bb000 ---p 00000000 00:00 0 
7f5e4d0bb000-7f5e4d0f8000 rw-p 00000000 00:00 0 
7f5e4d0f8000-7f5e4d0f9000 ---p 00000000 00:00 0 
7f5e4d0f9000-7f5e4d136000 rw-p 00000000 00:00 0 
7f5e4d136000-7f5e4d137000 ---p 00000000 00:00 0 
7f5e4d137000-7f5e4d174000 rw-p 00000000 00:00 0 
7f5e4d174000-7f5e4d175000 ---p 00000000 00:00 0 
7f5e4d175000-7f5e4d1b2000 rw-p 00000000 00:00 0 
7f5e4d1b2000-7f5e4d1b3000 ---p 00000000 00:00 0 
7f5e4d1b3000-7f5e4d1f0000 rw-p 00000000 00:00 0 
7f5e4d1f0000-7f5e4d1f1000 ---p 00000000 00:00 0 
7f5e4d1f1000-7f5e4d22e000 rw-p 00000000 00:00 0 
7f5e4d22e000-7f5e4d22f000 ---p 00000000 00:00 0 
7f5e4d22f000-7f5e4d26c000 rw-p 00000000 00:00 0 
7f5e4d26c000-7f5e4d26d000 ---p 00000000 00:00 0 
7f5e4d26d000-7f5e4d2aa000 rw-p 00000000 00:00 0 
7f5e4d2aa000-7f5e4d2ab000 ---p 00000000 00:00 0 
7f5e4d2ab000-7f5e4d2e8000 rw-p 00000000 00:00 0 
7f5e4d2e8000-7f5e4d2e9000 ---p 00000000 00:00 0 
7f5e4d2e9000-7f5e4d326000 rw-p 00000000 00:00 0 
7f5e4d326000-7f5e4d327000 ---p 00000000 00:00 0 
7f5e4d327000-7f5e4d364000 rw-p 00000000 00:00 0 
7f5e4d364000-7f5e4d365000 ---p 00000000 00:00 0 
7f5e4d365000-7f5e4d3a2000 rw-p 00000000 00:00 0 
7f5e4d3a2000-7f5e4d3a3000 ---p 00000000 00:00 0 
7f5e4d3a3000-7f5e4d3e0000 rw-p 00000000 00:00 0 
7f5e4d3e0000-7f5e4d3e1000 ---p 00000000 00:00 0 
7f5e4d3e1000-7f5e4d41e000 rw-p 00000000 00:00 0 
7f5e4d41e000-7f5e4d41f000 ---p 00000000 00:00 0 
7f5e4d41f000-7f5e4d45c000 rw-p 00000000 00:00 0 
7f5e4d45c000-7f5e4d45d000 ---p 00000000 00:00 0 
7f5e4d45d000-7f5e4d49a000 rw-p 00000000 00:00 0 
7f5e4d49a000-7f5e4d49b000 ---p 00000000 00:00 0 
7f5e4d49b000-7f5e4d4d8000 rw-p 00000000 00:00 0 
7f5e4d4d8000-7f5e4d4d9000 ---p 00000000 00:00 0 
7f5e4d4d9000-7f5e4d516000 rw-p 00000000 00:00 0 
7f5e4d516000-7f5e4d517000 ---p 00000000 00:00 0 
7f5e4d517000-7f5e4d554000 rw-p 00000000 00:00 0 
7f5e4d554000-7f5e4d555000 ---p 00000000 00:00 0 
7f5e4d555000-7f5e4d592000 rw-p 00000000 00:00 0 
7f5e4d592000-7f5e4d593000 ---p 00000000 00:00 0 
7f5e4d593000-7f5e4d5d0000 rw-p 00000000 00:00 0 
7f5e4d5d0000-7f5e4d5d1000 ---p 00000000 00:00 0 
7f5e4d5d1000-7f5e4d60e000 rw-p 00000000 00:00 0 
7f5e4d60e000-7f5e4d60f000 ---p 00000000 00:00 0 
7f5e4d60f000-7f5e4d64c000 rw-p 00000000 00:00 0 
7f5e4d64c000-7f5e4d64d000 ---p 00000000 00:00 0 
7f5e4d64d000-7f5e4d68a000 rw-p 00000000 00:00 0 
7f5e4d68a000-7f5e4d68b000 ---p 00000000 00:00 0 
7f5e4d68b000-7f5e4d6c8000 rw-p 00000000 00:00 0 
7f5e4d6c8000-7f5e4d6c9000 ---p 00000000 00:00 0 
7f5e4d6c9000-7f5e4d706000 rw-p 00000000 00:00 0 
7f5e4d706000-7f5e4d707000 ---p 00000000 00:00 0 
7f5e4d707000-7f5e4d744000 rw-p 00000000 00:00 0 
7f5e4d744000-7f5e4d745000 ---p 00000000 00:00 0 
7f5e4d745000-7f5e4d782000 rw-p 00000000 00:00 0 
7f5e4d782000-7f5e4d783000 ---p 00000000 00:00 0 
7f5e4d783000-7f5e4d7c0000 rw-p 00000000 00:00 0 
7f5e4d7c0000-7f5e4d7c1000 ---p 00000000 00:00 0 
7f5e4d7c1000-7f5e4d7fe000 rw-p 00000000 00:00 0 
7f5e4d7fe000-7f5e4d7ff000 ---p 00000000 00:00 0 
7f5e4d7ff000-7f5e4d83c000 rw-p 00000000 00:00 0 
7f5e4d83c000-7f5e4d83d000 ---p 00000000 00:00 0 
7f5e4d83d000-7f5e4d87a000 rw-p 00000000 00:00 0 
7f5e4d87a000-7f5e4d87b000 ---p 00000000 00:00 0 
7f5e4d87b000-7f5e4d8b8000 rw-p 00000000 00:00 0 
7f5e4d8b8000-7f5e4d8b9000 ---p 00000000 00:00 0 
7f5e4d8b9000-7f5e4d8f6000 rw-p 00000000 00:00 0 
7f5e4d8f6000-7f5e4d8f7000 ---p 00000000 00:00 0 
7f5e4d8f7000-7f5e4d934000 rw-p 00000000 00:00 0 
7f5e4d934000-7f5e4d935000 ---p 00000000 00:00 0 
7f5e4d935000-7f5e4d972000 rw-p 00000000 00:00 0 
7f5e4d972000-7f5e4d973000 ---p 00000000 00:00 0 
7f5e4d973000-7f5e4d9b0000 rw-p 00000000 00:00 0 
7f5e4d9b0000-7f5e4d9b1000 ---p 00000000 00:00 0 
7f5e4d9b1000-7f5e4d9ee000 rw-p 00000000 00:00 0 
7f5e4d9ee000-7f5e4d9ef000 ---p 00000000 00:00 0 
7f5e4d9ef000-7f5e4da2c000 rw-p 00000000 00:00 0 
7f5e4da2c000-7f5e4da2d000 ---p 00000000 00:00 0 
7f5e4da2d000-7f5e4da6a000 rw-p 00000000 00:00 0 
7f5e4da6a000-7f5e4da6b000 ---p 00000000 00:00 0 
7f5e4da6b000-7f5e4daa8000 rw-p 00000000 00:00 0 
7f5e4daa8000-7f5e4daa9000 ---p 00000000 00:00 0 
7f5e4daa9000-7f5e4dae6000 rw-p 00000000 00:00 0 
7f5e4dae6000-7f5e4dae7000 ---p 00000000 00:00 0 
7f5e4dae7000-7f5e4db24000 rw-p 00000000 00:00 0 
7f5e4db24000-7f5e4db25000 ---p 00000000 00:00 0 
7f5e4db25000-7f5e4db62000 rw-p 00000000 00:00 0 
7f5e4db62000-7f5e4db63000 ---p 00000000 00:00 0 
7f5e4db63000-7f5e4dba0000 rw-p 00000000 00:00 0 
7f5e4dba0000-7f5e4dba1000 ---p 00000000 00:00 0 
7f5e4dba1000-7f5e4dbde000 rw-p 00000000 00:00 0 
7f5e4dbde000-7f5e4dbdf000 ---p 00000000 00:00 0 
7f5e4dbdf000-7f5e4dc1c000 rw-p 00000000 00:00 0 
7f5e4dc1c000-7f5e4dc1d000 ---p 00000000 00:00 0 
7f5e4dc1d000-7f5e4dc5a000 rw-p 00000000 00:00 0 
7f5e4dc5a000-7f5e4dc5b000 ---p 00000000 00:00 0 
7f5e4dc5b000-7f5e4dc98000 rw-p 00000000 00:00 0 
7f5e4dc98000-7f5e4dc99000 ---p 00000000 00:00 0 
7f5e4dc99000-7f5e4dcd6000 rw-p 00000000 00:00 0 
7f5e4dcd6000-7f5e4dcd7000 ---p 00000000 00:00 0 
7f5e4dcd7000-7f5e4dd14000 rw-p 00000000 00:00 0 
7f5e4dd14000-7f5e4dd15000 ---p 00000000 00:00 0 
7f5e4dd15000-7f5e4dd52000 rw-p 00000000 00:00 0 
7f5e4dd52000-7f5e4dd53000 ---p 00000000 00:00 0 
7f5e4dd53000-7f5e4dd90000 rw-p 00000000 00:00 0 
7f5e4dd90000-7f5e4dd91000 ---p 00000000 00:00 0 
7f5e4dd91000-7f5e4ddce000 rw-p 00000000 00:00 0 
7f5e4ddce000-7f5e4ddcf000 ---p 00000000 00:00 0 
7f5e4ddcf000-7f5e4de0c000 rw-p 00000000 00:00 0 
7f5e4de0c000-7f5e4de0d000 ---p 00000000 00:00 0 
7f5e4de0d000-7f5e4de4a000 rw-p 00000000 00:00 0 
7f5e4de4a000-7f5e4de4b000 ---p 00000000 00:00 0 
7f5e4de4b000-7f5e4de88000 rw-p 00000000 00:00 0 
7f5e4de88000-7f5e4de89000 ---p 00000000 00:00 0 
7f5e4de89000-7f5e4dec6000 rw-p 00000000 00:00 0 
7f5e4dec6000-7f5e4dec7000 ---p 00000000 00:00 0 
7f5e4dec7000-7f5e4df04000 rw-p 00000000 00:00 0 
7f5e4df04000-7f5e4df05000 ---p 00000000 00:00 0 
7f5e4df05000-7f5e4df42000 rw-p 00000000 00:00 0 
7f5e4df42000-7f5e4df43000 ---p 00000000 00:00 0 
7f5e4df43000-7f5e4df80000 rw-p 00000000 00:00 0 
7f5e4df80000-7f5e4df81000 ---p 00000000 00:00 0 
7f5e4df81000-7f5e4dfbe000 rw-p 00000000 00:00 0 
7f5e4dfbe000-7f5e4dfbf000 ---p 00000000 00:00 0 
7f5e4dfbf000-7f5e4dffc000 rw-p 00000000 00:00 0 
7f5e4dffc000-7f5e4dffd000 ---p 00000000 00:00 0 
7f5e4dffd000-7f5e4e7fd000 rw-p 00000000 00:00 0 
7f5e4e7fd000-7f5e4e7fe000 ---p 00000000 00:00 0 
7f5e4e7fe000-7f5e4effe000 rw-p 00000000 00:00 0 
7f5e4effe000-7f5e4efff000 ---p 00000000 00:00 0 
7f5e4efff000-7f5e4f7ff000 rw-p 00000000 00:00 0 
7f5e4f7ff000-7f5e4f800000 ---p 00000000 00:00 0 
7f5e4f800000-7f5e50000000 rw-p 00000000 00:00 0 
7f5e50000000-7f5e5154e000 rw-p 00000000 00:00 0 
7f5e5154e000-7f5e54000000 ---p 00000000 00:00 0 
7f5e5403b000-7f5e5403c000 ---p 00000000 00:00 0 
7f5e5403c000-7f5e54079000 rw-p 00000000 00:00 0 
7f5e54079000-7f5e5407a000 ---p 00000000 00:00 0 
7f5e5407a000-7f5e540b7000 rw-p 00000000 00:00 0 
7f5e540b7000-7f5e540b8000 ---p 00000000 00:00 0 
7f5e540b8000-7f5e540f5000 rw-p 00000000 00:00 0 
7f5e540f5000-7f5e540f6000 ---p 00000000 00:00 0 
7f5e540f6000-7f5e54133000 rw-p 00000000 00:00 0 
7f5e54133000-7f5e54134000 ---p 00000000 00:00 0 
7f5e54134000-7f5e54171000 rw-p 00000000 00:00 0 
7f5e54171000-7f5e54172000 ---p 00000000 00:00 0 
7f5e54172000-7f5e541af000 rw-p 00000000 00:00 0 
7f5e541af000-7f5e541b0000 ---p 00000000 00:00 0 
7f5e541b0000-7f5e541ed000 rw-p 00000000 00:00 0 
7f5e541ed000-7f5e541ee000 ---p 00000000 00:00 0 
7f5e541ee000-7f5e5422b000 rw-p 00000000 00:00 0 
7f5e5422b000-7f5e5422c000 ---p 00000000 00:00 0 
7f5e5422c000-7f5e54269000 rw-p 00000000 00:00 0 
7f5e54269000-7f5e5426a000 ---p 00000000 00:00 0 
7f5e5426a000-7f5e542a7000 rw-p 00000000 00:00 0 
7f5e542a7000-7f5e542a8000 ---p 00000000 00:00 0 
7f5e542a8000-7f5e542e5000 rw-p 00000000 00:00 0 
7f5e542e5000-7f5e542e6000 ---p 00000000 00:00 0 
7f5e542e6000-7f5e54323000 rw-p 00000000 00:00 0 
7f5e54323000-7f5e54324000 ---p 00000000 00:00 0 
7f5e54324000-7f5e54361000 rw-p 00000000 00:00 0 
7f5e54361000-7f5e54362000 ---p 00000000 00:00 0 
7f5e54362000-7f5e5439f000 rw-p 00000000 00:00 0 
7f5e5439f000-7f5e543a0000 ---p 00000000 00:00 0 
7f5e543a0000-7f5e543dd000 rw-p 00000000 00:00 0 
7f5e543dd000-7f5e543ef000 r-xp 00000000 fd:00 577                        /usr/lib64/libnss_myhostname.so.2
 
7f5e543ef000-7f5e545ee000 ---p 00012000 fd:00 577                        /usr/lib64/libnss_myhostname.so.2
 
7f5e545ee000-7f5e545f1000 r--p 00011000 fd:00 577                        /usr/lib64/libnss_myhostname.so.2
 
7f5e545f1000-7f5e545f2000 rw-p 00014000 fd:00 577                        /usr/lib64/libnss_myhostname.so.2
 
7f5e545f2000-7f5e545f7000 r-xp 00000000 fd:00 8802                       /usr/lib64/libnss_dns-2.17.so
 
7f5e545f7000-7f5e547f7000 ---p 00005000 fd:00 8802                       /usr/lib64/libnss_dns-2.17.so
 
7f5e547f7000-7f5e547f8000 r--p 00005000 fd:00 8802                       /usr/lib64/libnss_dns-2.17.so
 
7f5e547f8000-7f5e547f9000 rw-p 00006000 fd:00 8802                       /usr/lib64/libnss_dns-2.17.so
 
7f5e547f9000-7f5e547fa000 ---p 00000000 00:00 0 
7f5e547fa000-7f5e54ffa000 rw-p 00000000 00:00 0 
7f5e54ffa000-7f5e54ffb000 ---p 00000000 00:00 0 
7f5e54ffb000-7f5e557fb000 rw-p 00000000 00:00 0 
7f5e557fb000-7f5e557fc000 ---p 00000000 00:00 0 
7f5e557fc000-7f5e55ffc000 rw-p 00000000 00:00 0 
7f5e55ffc000-7f5e55ffd000 ---p 00000000 00:00 0 
7f5e55ffd000-7f5e567fd000 rw-p 00000000 00:00 0 
7f5e567fd000-7f5e567fe000 ---p 00000000 00:00 0 
7f5e567fe000-7f5e56ffe000 rw-p 00000000 00:00 0 
7f5e56ffe000-7f5e56fff000 ---p 00000000 00:00 0 
7f5e56fff000-7f5e577ff000 rw-p 00000000 00:00 0 
7f5e577ff000-7f5e57800000 ---p 00000000 00:00 0 
7f5e57800000-7f5e58000000 rw-p 00000000 00:00 0 
7f5e58000000-7f5e58f17000 rw-p 00000000 00:00 0 
7f5e58f17000-7f5e5c000000 ---p 00000000 00:00 0 
7f5e5c004000-7f5e5c005000 ---p 00000000 00:00 0 
7f5e5c005000-7f5e5c042000 rw-p 00000000 00:00 0 
7f5e5c042000-7f5e5c043000 ---p 00000000 00:00 0 
7f5e5c043000-7f5e5c080000 rw-p 00000000 00:00 0 
7f5e5c080000-7f5e5c081000 ---p 00000000 00:00 0 
7f5e5c081000-7f5e5c0be000 rw-p 00000000 00:00 0 
7f5e5c0be000-7f5e5c0bf000 ---p 00000000 00:00 0 
7f5e5c0bf000-7f5e5c0fc000 rw-p 00000000 00:00 0 
7f5e5c0fc000-7f5e5c108000 r-xp 00000000 fd:00 8804                       /usr/lib64/libnss_files-2.17.so
 
7f5e5c108000-7f5e5c307000 ---p 0000c000 fd:00 8804                       /usr/lib64/libnss_files-2.17.so
 
7f5e5c307000-7f5e5c308000 r--p 0000b000 fd:00 8804                       /usr/lib64/libnss_files-2.17.so
 
7f5e5c308000-7f5e5c309000 rw-p 0000c000 fd:00 8804                       /usr/lib64/libnss_files-2.17.so
 
7f5e5c309000-7f5e5c30f000 rw-p 00000000 00:00 0 
7f5e5c30f000-7f5e5c310000 ---p 00000000 00:00 0 
7f5e5c310000-7f5e5cb10000 rw-p 00000000 00:00 0 
7f5e5cb10000-7f5e5cb11000 ---p 00000000 00:00 0 
7f5e5cb11000-7f5e5d311000 rw-p 00000000 00:00 0 
7f5e5d311000-7f5e5d312000 ---p 00000000 00:00 0 
7f5e5d312000-7f6394000000 rw-p 00000000 00:00 0 
7f6394000000-7f63955c3000 rw-p 00000000 00:00 0 
7f63955c3000-7f6398000000 ---p 00000000 00:00 0 
7f6398039000-7f63a0000000 rw-p 00000000 00:00 0 
7f63a0000000-7f63a193a000 rw-p 00000000 00:00 0 
7f63a193a000-7f63a4000000 ---p 00000000 00:00 0 
7f63a4015000-7f63a4016000 ---p 00000000 00:00 0 
7f63a4016000-7f63a4053000 rw-p 00000000 00:00 0 
7f63a4053000-7f63a4054000 ---p 00000000 00:00 0 
7f63a4054000-7f63a4091000 rw-p 00000000 00:00 0 
7f63a4091000-7f63a4092000 ---p 00000000 00:00 0 
7f63a4092000-7f63a40cf000 rw-p 00000000 00:00 0 
7f63a40cf000-7f63a40d0000 ---p 00000000 00:00 0 
7f63a40d0000-7f63a5bdf000 rw-p 00000000 00:00 0 
7f63a5bdf000-7f63a5be0000 ---p 00000000 00:00 0 
7f63a5be0000-7f63a6b15000 rw-p 00000000 00:00 0 
7f63a6b15000-7f63a6b16000 ---p 00000000 00:00 0 
7f63a6b16000-7f63a7316000 rw-p 00000000 00:00 0 
7f63a7316000-7f63a7319000 r-xp 00000000 fd:00 8940                       /usr/lib64/libkeyutils.so.1.5
 
7f63a7319000-7f63a7518000 ---p 00003000 fd:00 8940                       /usr/lib64/libkeyutils.so.1.5
 
7f63a7518000-7f63a7519000 r--p 00002000 fd:00 8940                       /usr/lib64/libkeyutils.so.1.5
 
7f63a7519000-7f63a751a000 rw-p 00003000 fd:00 8940                       /usr/lib64/libkeyutils.so.1.5
 
7f63a751a000-7f63a7527000 r-xp 00000000 fd:00 614                        /usr/lib64/libkrb5support.so.0.1
 
7f63a7527000-7f63a7726000 ---p 0000d000 fd:00 614                        /usr/lib64/libkrb5support.so.0.1
 
7f63a7726000-7f63a7727000 r--p 0000c000 fd:00 614                        /usr/lib64/libkrb5support.so.0.1
 
7f63a7727000-7f63a7728000 rw-p 0000d000 fd:00 614                        /usr/lib64/libkrb5support.so.0.1
 
7f63a7728000-7f63a7737000 r-xp 00000000 fd:00 9075                       /usr/lib64/libbz2.so.1.0.6
 
7f63a7737000-7f63a7936000 ---p 0000f000 fd:00 9075                       /usr/lib64/libbz2.so.1.0.6
 
7f63a7936000-7f63a7937000 r--p 0000e000 fd:00 9075                       /usr/lib64/libbz2.so.1.0.6
 
7f63a7937000-7f63a7938000 rw-p 0000f000 fd:00 9075                       /usr/lib64/libbz2.so.1.0.6
 
7f63a7938000-7f63a794f000 r-xp 00000000 fd:00 9100                       /usr/lib64/libelf-0.170.so
 
7f63a794f000-7f63a7b4e000 ---p 00017000 fd:00 9100                       /usr/lib64/libelf-0.170.so
 
7f63a7b4e000-7f63a7b4f000 r--p 00016000 fd:00 9100                       /usr/lib64/libelf-0.170.so
 
7f63a7b4f000-7f63a7b50000 rw-p 00017000 fd:00 9100                       /usr/lib64/libelf-0.170.so
 
7f63a7b50000-7f63a7bb0000 r-xp 00000000 fd:00 8889                       /usr/lib64/libpcre.so.1.2.0
 
7f63a7bb0000-7f63a7db0000 ---p 00060000 fd:00 8889                       /usr/lib64/libpcre.so.1.2.0
 
7f63a7db0000-7f63a7db1000 r--p 00060000 fd:00 8889                       /usr/lib64/libpcre.so.1.2.0
 
7f63a7db1000-7f63a7db2000 rw-p 00061000 fd:00 8889                       /usr/lib64/libpcre.so.1.2.0
 
7f63a7db2000-7f63a7db6000 r-xp 00000000 fd:00 8911                       /usr/lib64/libattr.so.1.1.0
 
7f63a7db6000-7f63a7fb5000 ---p 00004000 fd:00 8911                       /usr/lib64/libattr.so.1.1.0
 
7f63a7fb5000-7f63a7fb6000 r--p 00003000 fd:00 8911                       /usr/lib64/libattr.so.1.1.0
 
7f63a7fb6000-7f63a7fb7000 rw-p 00004000 fd:00 8911                       /usr/lib64/libattr.so.1.1.0
 
7f63a7fb7000-7f63a7fe8000 r-xp 00000000 fd:00 606                        /usr/lib64/libk5crypto.so.3.1
 
7f63a7fe8000-7f63a81e7000 ---p 00031000 fd:00 606                        /usr/lib64/libk5crypto.so.3.1
 
7f63a81e7000-7f63a81e9000 r--p 00030000 fd:00 606                        /usr/lib64/libk5crypto.so.3.1
 
7f63a81e9000-7f63a81ea000 rw-p 00032000 fd:00 606                        /usr/lib64/libk5crypto.so.3.1
 
7f63a81ea000-7f63a81ed000 r-xp 00000000 fd:00 600                        /usr/lib64/libcom_err.so.2.1
 
7f63a81ed000-7f63a83ec000 ---p 00003000 fd:00 600                        /usr/lib64/libcom_err.so.2.1
 
7f63a83ec000-7f63a83ed000 r--p 00002000 fd:00 600                        /usr/lib64/libcom_err.so.2.1
 
7f63a83ed000-7f63a83ee000 rw-p 00003000 fd:00 600                        /usr/lib64/libcom_err.so.2.1
 
7f63a83ee000-7f63a84c6000 r-xp 00000000 fd:00 612                        /usr/lib64/libkrb5.so.3.3
 
7f63a84c6000-7f63a86c5000 ---p 000d8000 fd:00 612                        /usr/lib64/libkrb5.so.3.3
 
7f63a86c5000-7f63a86d3000 r--p 000d7000 fd:00 612                        /usr/lib64/libkrb5.so.3.3
 
7f63a86d3000-7f63a86d6000 rw-p 000e5000 fd:00 612                        /usr/lib64/libkrb5.so.3.3
 
7f63a86d6000-7f63a8720000 r-xp 00000000 fd:00 602                        /usr/lib64/libgssapi_krb5.so.2.2
 
7f63a8720000-7f63a8920000 ---p 0004a000 fd:00 602                        /usr/lib64/libgssapi_krb5.so.2.2
 
7f63a8920000-7f63a8921000 r--p 0004a000 fd:00 602                        /usr/lib64/libgssapi_krb5.so.2.2
 
7f63a8921000-7f63a8923000 rw-p 0004b000 fd:00 602                        /usr/lib64/libgssapi_krb5.so.2.2
 
7f63a8923000-7f63a8938000 r-xp 00000000 fd:00 574                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f63a8938000-7f63a8b37000 ---p 00015000 fd:00 574                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f63a8b37000-7f63a8b38000 r--p 00014000 fd:00 574                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f63a8b38000-7f63a8b39000 rw-p 00015000 fd:00 574                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f63a8b39000-7f63a8b7d000 r-xp 00000000 fd:00 214688                     /usr/lib64/libdw-0.170.so
 
7f63a8b7d000-7f63a8d7d000 ---p 00044000 fd:00 214688                     /usr/lib64/libdw-0.170.so
 
7f63a8d7d000-7f63a8d7f000 r--p 00044000 fd:00 214688                     /usr/lib64/libdw-0.170.so
 
7f63a8d7f000-7f63a8d80000 rw-p 00046000 fd:00 214688                     /usr/lib64/libdw-0.170.so
 
7f63a8d80000-7f63a8d96000 r-xp 00000000 fd:00 8814                       /usr/lib64/libresolv-2.17.so
 
7f63a8d96000-7f63a8f95000 ---p 00016000 fd:00 8814                       /usr/lib64/libresolv-2.17.so
 
7f63a8f95000-7f63a8f96000 r--p 00015000 fd:00 8814                       /usr/lib64/libresolv-2.17.so
 
7f63a8f96000-7f63a8f97000 rw-p 00016000 fd:00 8814                       /usr/lib64/libresolv-2.17.so
 
7f63a8f97000-7f63a8f99000 rw-p 00000000 00:00 0 
7f63a8f99000-7f63a8f9d000 r-xp 00000000 fd:00 9092                       /usr/lib64/libgpg-error.so.0.10.0
 
7f63a8f9d000-7f63a919c000 ---p 00004000 fd:00 9092                       /usr/lib64/libgpg-error.so.0.10.0
 
7f63a919c000-7f63a919d000 r--p 00003000 fd:00 9092                       /usr/lib64/libgpg-error.so.0.10.0
 
7f63a919d000-7f63a919e000 rw-p 00004000 fd:00 9092                       /usr/lib64/libgpg-error.so.0.10.0
 
7f63a919e000-7f63a921b000 r-xp 00000000 fd:00 9094                       /usr/lib64/libgcrypt.so.11.8.2
 
7f63a921b000-7f63a941a000 ---p 0007d000 fd:00 9094                       /usr/lib64/libgcrypt.so.11.8.2
 
7f63a941a000-7f63a941b000 r--p 0007c000 fd:00 9094                       /usr/lib64/libgcrypt.so.11.8.2
 
7f63a941b000-7f63a941e000 rw-p 0007d000 fd:00 9094                       /usr/lib64/libgcrypt.so.11.8.2
 
7f63a941e000-7f63a941f000 rw-p 00000000 00:00 0 
7f63a941f000-7f63a9433000 r-xp 00000000 fd:00 58323                      /usr/lib64/liblz4.so.1.7.5
 
7f63a9433000-7f63a9632000 ---p 00014000 fd:00 58323                      /usr/lib64/liblz4.so.1.7.5
 
7f63a9632000-7f63a9633000 r--p 00013000 fd:00 58323                      /usr/lib64/liblz4.so.1.7.5
 
7f63a9633000-7f63a9634000 rw-p 00014000 fd:00 58323                      /usr/lib64/liblz4.so.1.7.5
 
7f63a9634000-7f63a9658000 r-xp 00000000 fd:00 8900                       /usr/lib64/libselinux.so.1
 
7f63a9658000-7f63a9857000 ---p 00024000 fd:00 8900                       /usr/lib64/libselinux.so.1
 
7f63a9857000-7f63a9858000 r--p 00023000 fd:00 8900                       /usr/lib64/libselinux.so.1
 
7f63a9858000-7f63a9859000 rw-p 00024000 fd:00 8900                       /usr/lib64/libselinux.so.1
 
7f63a9859000-7f63a985b000 rw-p 00000000 00:00 0 
7f63a985b000-7f63a9862000 r-xp 00000000 fd:00 8816                       /usr/lib64/librt-2.17.so
 
7f63a9862000-7f63a9a61000 ---p 00007000 fd:00 8816                       /usr/lib64/librt-2.17.so
 
7f63a9a61000-7f63a9a62000 r--p 00006000 fd:00 8816                       /usr/lib64/librt-2.17.so
 
7f63a9a62000-7f63a9a63000 rw-p 00007000 fd:00 8816                       /usr/lib64/librt-2.17.so
 
7f63a9a63000-7f63a9a67000 r-xp 00000000 fd:00 8915                       /usr/lib64/libcap.so.2.22
 
7f63a9a67000-7f63a9c66000 ---p 00004000 fd:00 8915                       /usr/lib64/libcap.so.2.22
 
7f63a9c66000-7f63a9c67000 r--p 00003000 fd:00 8915                       /usr/lib64/libcap.so.2.22
 
7f63a9c67000-7f63a9c68000 rw-p 00004000 fd:00 8915                       /usr/lib64/libcap.so.2.22
 
7f63a9c68000-7f63a9c6a000 r-xp 00000000 fd:00 8142                       /usr/lib64/libfreebl3.so
 
7f63a9c6a000-7f63a9e69000 ---p 00002000 fd:00 8142                       /usr/lib64/libfreebl3.so
 
7f63a9e69000-7f63a9e6a000 r--p 00001000 fd:00 8142                       /usr/lib64/libfreebl3.so
 
7f63a9e6a000-7f63a9e6b000 rw-p 00002000 fd:00 8142                       /usr/lib64/libfreebl3.so
 
7f63a9e6b000-7f63aa02e000 r-xp 00000000 fd:00 8786                       /usr/lib64/libc-2.17.so
 
7f63aa02e000-7f63aa22d000 ---p 001c3000 fd:00 8786                       /usr/lib64/libc-2.17.so
 
7f63aa22d000-7f63aa231000 r--p 001c2000 fd:00 8786                       /usr/lib64/libc-2.17.so
 
7f63aa231000-7f63aa233000 rw-p 001c6000 fd:00 8786                       /usr/lib64/libc-2.17.so
 
7f63aa233000-7f63aa238000 rw-p 00000000 00:00 0 
7f63aa238000-7f63aa339000 r-xp 00000000 fd:00 8794                       /usr/lib64/libm-2.17.so
 
7f63aa339000-7f63aa538000 ---p 00101000 fd:00 8794                       /usr/lib64/libm-2.17.so
 
7f63aa538000-7f63aa539000 r--p 00100000 fd:00 8794                       /usr/lib64/libm-2.17.so
 
7f63aa539000-7f63aa53a000 rw-p 00101000 fd:00 8794                       /usr/lib64/libm-2.17.so
 
7f63aa53a000-7f63aa623000 r-xp 00000000 fd:00 8840                       /usr/lib64/libstdc++.so.6.0.19
 
7f63aa623000-7f63aa822000 ---p 000e9000 fd:00 8840                       /usr/lib64/libstdc++.so.6.0.19
 
7f63aa822000-7f63aa82a000 r--p 000e8000 fd:00 8840                       /usr/lib64/libstdc++.so.6.0.19
 
7f63aa82a000-7f63aa82c000 rw-p 000f0000 fd:00 8840                       /usr/lib64/libstdc++.so.6.0.19
 
7f63aa82c000-7f63aa841000 rw-p 00000000 00:00 0 
7f63aa841000-7f63aa843000 r-xp 00000000 fd:00 8792                       /usr/lib64/libdl-2.17.so
 
7f63aa843000-7f63aaa43000 ---p 00002000 fd:00 8792                       /usr/lib64/libdl-2.17.so
 
7f63aaa43000-7f63aaa44000 r--p 00002000 fd:00 8792                       /usr/lib64/libdl-2.17.so
 
7f63aaa44000-7f63aaa45000 rw-p 00003000 fd:00 8792                       /usr/lib64/libdl-2.17.so
 
7f63aaa45000-7f63aac79000 r-xp 00000000 fd:00 9016                       /usr/lib64/libcrypto.so.1.0.2k
 
7f63aac79000-7f63aae79000 ---p 00234000 fd:00 9016                       /usr/lib64/libcrypto.so.1.0.2k
 
7f63aae79000-7f63aae95000 r--p 00234000 fd:00 9016                       /usr/lib64/libcrypto.so.1.0.2k
 
7f63aae95000-7f63aaea2000 rw-p 00250000 fd:00 9016                       /usr/lib64/libcrypto.so.1.0.2k
 
7f63aaea2000-7f63aaea6000 rw-p 00000000 00:00 0 
7f63aaea6000-7f63aaf0d000 r-xp 00000000 fd:00 9018                       /usr/lib64/libssl.so.1.0.2k
 
7f63aaf0d000-7f63ab10d000 ---p 00067000 fd:00 9018                       /usr/lib64/libssl.so.1.0.2k
 
7f63ab10d000-7f63ab111000 r--p 00067000 fd:00 9018                       /usr/lib64/libssl.so.1.0.2k
 
7f63ab111000-7f63ab118000 rw-p 0006b000 fd:00 9018                       /usr/lib64/libssl.so.1.0.2k
 
7f63ab118000-7f63ab147000 r-xp 00000000 fd:00 589                        /usr/lib64/libsystemd.so.0.6.0
 
7f63ab147000-7f63ab347000 ---p 0002f000 fd:00 589                        /usr/lib64/libsystemd.so.0.6.0
 
7f63ab347000-7f63ab348000 r--p 0002f000 fd:00 589                        /usr/lib64/libsystemd.so.0.6.0
 
7f63ab348000-7f63ab349000 rw-p 00030000 fd:00 589                        /usr/lib64/libsystemd.so.0.6.0
 
7f63ab349000-7f63ab351000 r-xp 00000000 fd:00 8790                       /usr/lib64/libcrypt-2.17.so
 
7f63ab351000-7f63ab550000 ---p 00008000 fd:00 8790                       /usr/lib64/libcrypt-2.17.so
 
7f63ab550000-7f63ab551000 r--p 00007000 fd:00 8790                       /usr/lib64/libcrypt-2.17.so
 
7f63ab551000-7f63ab552000 rw-p 00008000 fd:00 8790                       /usr/lib64/libcrypt-2.17.so
 
7f63ab552000-7f63ab580000 rw-p 00000000 00:00 0 
7f63ab580000-7f63ab595000 r-xp 00000000 fd:00 8833                       /usr/lib64/libz.so.1.2.7
 
7f63ab595000-7f63ab794000 ---p 00015000 fd:00 8833                       /usr/lib64/libz.so.1.2.7
 
7f63ab794000-7f63ab795000 r--p 00014000 fd:00 8833                       /usr/lib64/libz.so.1.2.7
 
7f63ab795000-7f63ab796000 rw-p 00015000 fd:00 8833                       /usr/lib64/libz.so.1.2.7
 
7f63ab796000-7f63ab797000 r-xp 00000000 fd:00 143979                     /usr/lib64/libaio.so.1.0.1
 
7f63ab797000-7f63ab996000 ---p 00001000 fd:00 143979                     /usr/lib64/libaio.so.1.0.1
 
7f63ab996000-7f63ab997000 r--p 00000000 fd:00 143979                     /usr/lib64/libaio.so.1.0.1
 
7f63ab997000-7f63ab998000 rw-p 00001000 fd:00 143979                     /usr/lib64/libaio.so.1.0.1
 
7f63ab998000-7f63ab9bd000 r-xp 00000000 fd:00 9069                       /usr/lib64/liblzma.so.5.2.2
 
7f63ab9bd000-7f63abbbc000 ---p 00025000 fd:00 9069                       /usr/lib64/liblzma.so.5.2.2
 
7f63abbbc000-7f63abbbd000 r--p 00024000 fd:00 9069                       /usr/lib64/liblzma.so.5.2.2
 
7f63abbbd000-7f63abbbe000 rw-p 00025000 fd:00 9069                       /usr/lib64/liblzma.so.5.2.2
 
7f63abbbe000-7f63abbd5000 r-xp 00000000 fd:00 8812                       /usr/lib64/libpthread-2.17.so
 
7f63abbd5000-7f63abdd4000 ---p 00017000 fd:00 8812                       /usr/lib64/libpthread-2.17.so
 
7f63abdd4000-7f63abdd5000 r--p 00016000 fd:00 8812                       /usr/lib64/libpthread-2.17.so
 
7f63abdd5000-7f63abdd6000 rw-p 00017000 fd:00 8812                       /usr/lib64/libpthread-2.17.so
 
7f63abdd6000-7f63abdda000 rw-p 00000000 00:00 0 
7f63abdda000-7f63abdfc000 r-xp 00000000 fd:00 8779                       /usr/lib64/ld-2.17.so
 
7f63abe0f000-7f63abe10000 ---p 00000000 00:00 0 
7f63abe10000-7f63abe4d000 rw-p 00000000 00:00 0 
7f63abe4d000-7f63abe4e000 ---p 00000000 00:00 0 
7f63abe4e000-7f63abe8b000 rw-p 00000000 00:00 0 
7f63abe8b000-7f63abe8c000 ---p 00000000 00:00 0 
7f63abe8c000-7f63abec9000 rw-p 00000000 00:00 0 
7f63abec9000-7f63abeca000 ---p 00000000 00:00 0 
7f63abeca000-7f63abf08000 rw-p 00000000 00:00 0 
7f63abf08000-7f63abf0b000 rw-s 00000000 00:0b 552140                     /[aio] (deleted)
 
7f63abf0b000-7f63abf0e000 rw-s 00000000 00:0b 552139                     /[aio] (deleted)
 
7f63abf0e000-7f63abf11000 rw-s 00000000 00:0b 552138                     /[aio] (deleted)
 
7f63abf11000-7f63abf14000 rw-s 00000000 00:0b 552137                     /[aio] (deleted)
 
7f63abf14000-7f63abf17000 rw-s 00000000 00:0b 552136                     /[aio] (deleted)
 
7f63abf17000-7f63abf1a000 rw-s 00000000 00:0b 552135                     /[aio] (deleted)
 
7f63abf1a000-7f63abf1d000 rw-s 00000000 00:0b 552134                     /[aio] (deleted)
 
7f63abf1d000-7f63abf20000 rw-s 00000000 00:0b 552133                     /[aio] (deleted)
 
7f63abf20000-7f63abf23000 rw-s 00000000 00:0b 552132                     /[aio] (deleted)
 
7f63abf23000-7f63abff5000 rw-p 00000000 00:00 0 
7f63abff5000-7f63abff6000 rw-s 00000000 00:0b 552141                     /[aio] (deleted)
 
7f63abff6000-7f63abff9000 rw-s 00000000 00:0b 552131                     /[aio] (deleted)
 
7f63abff9000-7f63abffa000 rw-s 00000000 00:0b 552130                     /[aio] (deleted)
 
7f63abffa000-7f63abffb000 rw-p 00000000 00:00 0 
7f63abffb000-7f63abffc000 r--p 00021000 fd:00 8779                       /usr/lib64/ld-2.17.so
 
7f63abffc000-7f63abffd000 rw-p 00022000 fd:00 8779                       /usr/lib64/ld-2.17.so
 
7f63abffd000-7f63abffe000 rw-p 00000000 00:00 0 
7ffd60ed0000-7ffd60ef1000 rw-p 00000000 00:00 0                          [stack]
 
7ffd60f95000-7ffd60f97000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Uses event mutexes
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Using Linux native AIO
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Number of pools: 1
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Using SSE2 crc32 instructions
 
2019-01-13 14:30:44 139803099113664 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M
 
2019-01-13 14:30:45 139803099113664 [Note] InnoDB: Completed initialization of buffer pool
 
2019-01-13 14:30:46 139779722630912 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2019-01-13 14:30:46 139803099113664 [Note] InnoDB: Highest supported file format is Barracuda.
 
2019-01-13 14:30:46 139803099113664 [Note] InnoDB: Starting crash recovery from checkpoint LSN=22131348930
 
2019-01-13 14:30:47 139803099113664 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
 
2019-01-13 14:30:47 139803099113664 [Note] InnoDB: Trx id counter is 25405696
 
2019-01-13 14:30:47 139803099113664 [Note] InnoDB: Starting final batch to recover 173 pages from redo log.
 
2019-01-13 14:30:48 139803099113664 [Note] InnoDB: Last binlog file '/data/binlogs/mysql-bin.000116', position 83644

Comment by Javier Zon [ 2019-01-13 ]

For now, the table got renamed and we create a new one. Possible solution, but still think the server should not crash like that, right?

Comment by Elena Stepanova [ 2019-01-13 ]

It definitely shouldn't crash, we just need to find out why it does. Unless there was a genuine hardware problem (disk failure which caused data corruption), it's a bug.
Please preserve the renamed table for further investigation.

marko, I'm passing it over to you, as I don't want to suggest anything that will change the table so that the problem is no longer reproducible.

Comment by Javier Zon [ 2019-01-14 ]

I'll keep the table and a physical backup for further investigation. Thanks!

Comment by Marko Mäkelä [ 2019-01-15 ]

javier.zon, would it be possible for you to resolve the stack trace? Or even better, to attach a debugger to the server process before it crashes?
On this table, does it crash always when trying to perform that INSERT?

If you are able to attach a debugger, I would suggest the following:

gdb -p $(pgrep mysqld)
 
continue
 
# after the crash
 
set log on
 
set height 0
 
thread apply all backtrace
 
# multiple times, until you reach the correct stack frame where {{ut_dbg_assertion_failed}} was called
 
up
 
print *get_block
 
print *cursor->index
 
dump binary memory /tmp/g.bin get_block->frame get_block->frame+srv_page_size
 
dump binary memory /tmp/page.bin page page+srv_page_size
 
quit # or detach

The output from this (the files gdb.txt, g.bin and page.bin would help quite a bit.

Comment by Javier Zon [ 2019-01-17 ]

Hello @marko! The server is under production load and I'll not be able to perform any crash without maintenance, which is complicated after we get all this events.
Do you think uploading the ibd file with the structure or .frm to some FTP place someone else can perform that process?

Comment by Javier Zon [ 2019-01-21 ]

The server crashed one more time with the "same" INSERT on a new ibd file created last time

2019-01-21  1:06:02 139685099910912 [Warning] Aborted connection 912524 to db: 'contabilidad' user: 'advansapp' host: '10.5.53.51' (Got an error reading communication packets)
 
2019-01-21 01:06:27 0x7f0b0255b700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc line 321
 
InnoDB: Failing assertion: btr_page_get_next( latch_leaves.blocks[0]->frame, mtr) == page_get_page_no(page)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
190121  1:06:27 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.21-MariaDB-log
 
key_buffer_size=65536
 
read_buffer_size=2097152
 
max_used_connections=1001
 
max_threads=1002
 
thread_count=567
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18490261 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f0bc8aa6f48
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f0b0255ad30 thread_stack 0x3c000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555df78c2dbe]
 
/usr/sbin/mysqld(handle_fatal_signal+0x355)[0x555df7349ec5]
 
/lib64/libpthread.so.0(+0xf6d0)[0x7f1163d956d0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7f1162069277]
 
/lib64/libc.so.6(abort+0x148)[0x7f116206a968]
 
/usr/sbin/mysqld(+0x423b9b)[0x555df70feb9b]
 
/usr/sbin/mysqld(+0x9c4fe8)[0x555df769ffe8]
 
/usr/sbin/mysqld(+0x9cc1b3)[0x555df76a71b3]
 
/usr/sbin/mysqld(+0x900114)[0x555df75db114]
 
/usr/sbin/mysqld(+0x9042e2)[0x555df75df2e2]
 
/usr/sbin/mysqld(+0x906821)[0x555df75e1821]
 
/usr/sbin/mysqld(+0x91651a)[0x555df75f151a]
 
/usr/sbin/mysqld(+0x85779f)[0x555df753279f]
 
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x413)[0x555df7354923]
 
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6f)[0x555df719971f]
 
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x140e)[0x555df719fb6e]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d0e)[0x555df71b4f9e]
 
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4ee)[0x555df71cfe2e]
 
/usr/sbin/mysqld(+0x4f50b0)[0x555df71d00b0]
 
/usr/sbin/mysqld(+0x4f5e67)[0x555df71d0e67]
 
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555df71d0ef7]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xdbc)[0x555df71be74c]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x555df71c0785]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x555df7287d2a]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x555df7287e4d]
 
/lib64/libpthread.so.0(+0x7e25)[0x7f1163d8de25]
 
/lib64/libc.so.6(clone+0x6d)[0x7f1162131bad]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f0bc89e7170): insert into `oauth_access_tokens` (`id`, `user_id`, `client_id`, `scopes`, `revoked`, `created_at`, `updated_at`, `expires_at`) values ('07bc71cf438961f255bb29c8c76b93bfe29e8eb8c032f2a4107a12078552cba587d6b86e3aec99c7', 1, '4', '[\"*\"]', 0, '2019-01-21 06:06:27', '2019-01-21 06:06:27', '2020-01-21 06:06:27')
 
Connection ID (thread ID): 913740
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

Comment by Marko Mäkelä [ 2019-01-21 ]

javier.zon, yes, a data file might help narrow it down. Ultimately, we need to be able to reproduce the corruption by using SQL. You can upload an archive of your files to ftp://ftp.mariadb.com/uploads and let me know the file name.

You should be able to use Mariabackup to export the files from the production environment with minimal impact on the server.

Comment by Javier Zon [ 2019-01-21 ]

My installation does not have symbols, or they are not recognized.
Anyone around to help me on that?

Comment by Javier Zon [ 2019-01-21 ]

Ok, I attached my gdb, that is the only thing I got, the dumps on /tmp didn't work because of the missing symbols.
Let me know if we can do something else with this.

#5  0x000055a33ba223fd in _start ()
 
(gdb) print *get_block
 
No symbol "get_block" in current context.
 
(gdb) print *cursor->index
 
No symbol "cursor" in current context.
 
(gdb) dump binary memory /tmp/g.bin get_block->frame get_block->frame+srv_page_size
 
No symbol "get_block" in current context.
 
(gdb) dump binary memory /tmp/page.bin page page+srv_page_size
 
No symbol "page" in current context.
 
(gdb) quit
 
A debugging session is active.
 
 
 
	Inferior 1 [process 229261] will be detached.
 
 
 
Quit anyway? (y or n) y
 
Detaching from program: /usr/sbin/mysqld, process 229261

Comment by Javier Zon [ 2019-01-21 ] 

Thread 90 (Thread 0x7f30667fc700 (LWP 229364)):
 
#0  0x00007f35c307d277 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 
#1  0x00007f35c307e968 in __GI_abort () at abort.c:90
 
#2  0x000055a33ba05b9b in ut_dbg_assertion_failed (expr=expr@entry=0x55a33c30fa30 "btr_page_get_next( latch_leaves.blocks[0]->frame, mtr) == page_get_page_no(page)", 
    file=file@entry=0x55a33c30f7d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc", line=line@entry=321)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/ut/ut0dbg.cc:60
 
#3  0x000055a33bfa6fe8 in btr_cur_latch_leaves (block=<optimized out>, page_id=..., page_size=..., latch_mode=<optimized out>, cursor=0x7f30667f7870, mtr=0x7f30667f8700)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc:319
 
#4  0x000055a33bfae1b3 in btr_cur_search_to_nth_level (index=index@entry=0x7f2fcc23fb40, level=level@entry=0, tuple=tuple@entry=0x7f2fcc241818, mode=mode@entry=PAGE_CUR_LE, latch_mode=33, 
    latch_mode@entry=545, cursor=cursor@entry=0x7f30667f7870, has_search_latch=has_search_latch@entry=0, 
    file=file@entry=0x55a33c2fdae8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc", line=line@entry=2903, 
    mtr=mtr@entry=0x7f30667f8700, autoinc=autoinc@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/btr/btr0cur.cc:1494
 
#5  0x000055a33bee2114 in row_ins_sec_index_entry_low (flags=flags@entry=0, mode=mode@entry=33, index=index@entry=0x7f2fcc23fb40, offsets_heap=offsets_heap@entry=0x7f2f9c2faf40, 
    heap=heap@entry=0x7f2f9c22d590, entry=entry@entry=0x7f2fcc241818, trx_id=trx_id@entry=0, thr=thr@entry=0x7f2fcc2410c0, dup_chk_only=dup_chk_only@entry=false)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:2903
 
#6  0x000055a33bee62e2 in row_ins_sec_index_entry (index=index@entry=0x7f2fcc23fb40, entry=entry@entry=0x7f2fcc241818, thr=thr@entry=0x7f2fcc2410c0, dup_chk_only=dup_chk_only@entry=false)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:3263
 
#7  0x000055a33bee8821 in row_ins_index_entry (thr=0x7f2fcc2410c0, entry=0x7f2fcc241818, index=0x7f2fcc23fb40)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:3294
 
#8  row_ins_index_entry_step (thr=0x7f2fcc2410c0, node=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:3442
 
#9  row_ins (thr=<optimized out>, node=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:3585
 
#10 row_ins_step (thr=thr@entry=0x7f2fcc2410c0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0ins.cc:3811
 
#11 0x000055a33bef851a in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7f2fcc15c3a0 "\302P", prebuilt=0x7f2fcc2403a0)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/row/row0mysql.cc:1413
 
#12 0x000055a33be3979f in ha_innobase::write_row (this=0x7f2fcc15bc80, record=0x7f2fcc15c3a0 "\302P")
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/storage/innobase/handler/ha_innodb.cc:8234
 
#13 0x000055a33bc5b923 in handler::ha_write_row (this=0x7f2fcc15bc80, buf=0x7f2fcc15c3a0 "\302P")
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/handler.cc:5961
 
#14 0x000055a33baa071f in write_record (thd=thd@entry=0x7f2f9c01c448, table=table@entry=0x7f2fcc15b0e8, info=info@entry=0x7f30667f97f0)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_insert.cc:1930
 
#15 0x000055a33baa6b6e in mysql_insert (thd=thd@entry=0x7f2f9c01c448, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_insert.cc:1057
 
#16 0x000055a33babbf9e in mysql_execute_command (thd=thd@entry=0x7f2f9c01c448) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_parse.cc:4438
 
#17 0x000055a33bac390e in mysql_parse (thd=0x7f2f9c01c448, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x7f30667fb100, is_com_multi=<optimized out>, is_next_command=<optimized out>)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_parse.cc:8015
 
#18 0x000055a33bac6b7a in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f2f9c01c448, 
    packet=packet@entry=0x7f2f9c13dc89 "insert into cc.`oauth_access_tokens` (`id`, `user_id`, `client_id`, `scopes`, `revoked`, `created_at`, `updated_at`, `expires_at`) values ('07bc71cf438961f255bb29c8c76b93bfe29e8eb8c032f2a4107a12078552"..., packet_length=packet_length@entry=313, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false)
 
    at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_parse.cc:1826
 
#19 0x000055a33bac7785 in do_command (thd=0x7f2f9c01c448) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_parse.cc:1379
 
#20 0x000055a33bb8ed2a in do_handle_one_connection (connect=connect@entry=0x55a35983b798) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_connect.cc:1335
 
#21 0x000055a33bb8ee4d in handle_one_connection (arg=0x55a35983b798) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.21/sql/sql_connect.cc:1241
 
#22 0x00007f35c4da1e25 in start_thread (arg=0x7f30667fc700) at pthread_create.c:308
 
#23 0x00007f35c3145bad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Comment by Javier Zon [ 2019-01-21 ]

This is not isolated issue, this table is generated by PHP Laravel framework. We found multiples databases using the same table and some of them are having this issue.
DROP and RECREATE fixed for now..

MariaDB [cc]> show create table oauth_access_tokens\G
 
*************************** 1. row ***************************
 
       Table: oauth_access_tokens
 
Create Table: CREATE TABLE `oauth_access_tokens` (
 
  `id` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
 
  `user_id` int(11) DEFAULT NULL,
 
  `client_id` int(11) NOT NULL,
 
  `name` varchar(191) COLLATE utf8_unicode_ci DEFAULT NULL,
 
  `scopes` text COLLATE utf8_unicode_ci DEFAULT NULL,
 
  `revoked` tinyint(1) NOT NULL,
 
  `created_at` timestamp NULL DEFAULT NULL,
 
  `updated_at` timestamp NULL DEFAULT NULL,
 
  `expires_at` datetime DEFAULT NULL,
 
  PRIMARY KEY (`id`),
 
  KEY `oauth_access_tokens_user_id_index` (`user_id`)
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
 
1 row in set (0.00 sec)

Comment by Marko Mäkelä [ 2019-02-18 ]

It looks like the secondary index on the column user_id of the table oauth_access_tokens is corrupted. At some level (I cannot determine from the stack trace if it is the leaf level), the FIL_PAGE_PREV link of the next page is not pointing to the current page.

If this corruption is above the leaf level, then you can probably repair the table with ALTER TABLE…FORCE or OPTIMIZE TABLE or similar. This is because when copying the table, we would only dive to the first leaf-level page and then scan through the leaf-level pages. While doing so, we would not visit any other non-leaf pages than those along the path from the root to the first leaf page.

The challenge is how to trigger this corruption with repeatable SQL. I am not aware of other cases of this kind. Have you ruled out hardware problems, such as intermittent faults in the main memory?

CHECK TABLE should detect this kind of corruption, but beware: a corrupted page can cause InnoDB to crash.

Comment by Elena Stepanova [ 2019-06-01 ]

marko, MDEV-19666 might be another case of the same problem.

Generated at Thu Feb 08 08:42:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.