[MDEV-13060] Server Audit Plugin Crashes with AWS KMS plugin Created: 2017-06-12  Updated: 2017-09-18  Resolved: 2017-07-05

Status: Closed
Project: MariaDB Server
Component/s: Plugin - Audit, Plugin - AWS key management
Affects Version/s: 10.2.6
Fix Version/s: 10.2.7

Type: Bug Priority: Major
Reporter: Dave Juntgen Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: encryption, plugins
Environment:

CentOS Linux release 7.3.1611 (Core)


 Description   

MariaDB 10.2.6 Crashes loading Audit plugin after AWS KMS plugin

Vagrant Configuration:

  1. Requires barebones CentOS7
  2. Add this line to Vagrant file 

    config.vm.provision "shell", path: "mariadb-10-aws-audit.sh"

  3. GIST Vagrant up shell, includes MariaDB configuration: mariadb-10-aws-audit.sh
  4. Don't forget to update the KMS API Key and Security Key

Crash Summary

Starting MariaDB database server...
 
2017-06-12 13:10:51 139713030162560 [Note] /usr/sbin/mysqld (mysqld 10.2.6-MariaDB-log) starting as process 1830 ...
 
2017-06-12 13:10:52 139713030162560 [Note] mysqld: AWS KMS plugin: loaded key 1, version 1, key length 256 bit
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Uses event mutexes
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Using Linux native AIO
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Number of pools: 1
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Using SSE2 crc32 instructions
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Completed initialization of buffer pool
 
2017-06-12 13:10:52 139712137037568 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Highest supported file format is Barracuda.
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: S: (0x050C05060D0B000F010C090B0A0C0A0D07070B0D09040E03050D06040E0A000A0F010A0E040D0D020B0D020D0204040B030F09050F050205090903010308030802040D07090E0B08050A05080F0F020900060A0A020B0A03000C010900030105080006070502040E0B08010C00090303060D070A040106030E030708080B0503060D000308000C0A090F0301020B0C0A0F0B000A03040B030D040C0C0F0D0B0A0206040D0B040A00030F02020F060A0A0F09080606070D080B0D04010B060602010C0E0D0F000E000000060F070C0F0A0D0607050C0A0E0B0A0E040E060804020F0C000B040D0F060003010703090004010D000D0403050103070005090E070004080D0102090F0D0A060C0D0C0E04070A03020509020E020E04070A09020100040E08030E05000D0B0E01020F0006050A0B01000505060C0B070D0A030B0D0B090B0D0404070D0B06040901040B0809010F03040A020B090704000F030F0C030C0E00070705090B0D0F0E02010E0405090B0F0307070F040A0007020A0505040A010507090101000F010100010503070C0E0F04090D0707040C00040B090D040109030D0008040900060C0A0F0A090402020B0102070F0F080D0D0E0E080105070200060F03000A08050F0C040E0C0F070D010905010F0D080C0A0E0F0B03010C03090F050F00060F040B080B0E03010601040A080109080
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: c: (0x0008000C000000070F0F0F020004000C00040004000E000F0400050002000F0F0F0F0200050006000400050008000F0F0F0F020005000C00040005000E000F0400060002000F0F0F0F0200060006000400060008000F0F0F0F020006000C00040006000E000F0400070002000F0F0F0F0200070006000400070008000F0F0F0F020007000C00040007000E000F0400080002000F0F0F0F0200080006000400080008000F0F0F0F020008000C00040008000E000F0A0B030302040E06040A030E0A0E08020F020401030F020B080B0D04020607080B060D0B080C090F0F07070D070B05070E07050D0C060A0B080D04020E0E0706000602080C01000E030A01050F0C07060C060E010C0A0D0F020A0A070B0E0D030D040A0002080C0508000E0D07030F0B00090B0B090D08060D070E080507030F04080C000400020A06080F0D0D0B0604030003090604060C08020A020B0E0F0807020303080B0B0F010E060701050D0408080F0907060700000200000801010F060304030C08050E09060E06000E09040D07000000040D010204090308070C020F090C08050C020A0003010E040D0507080C0C060403030A000B04070701070301060F0D0A09000C090A0F0F07010C0E0D050E090C0D0A0609010C050A0A0D0E0104000E0D0D020F000B060C03000C0B0F0C08010709020E090300010A0A0C0608010B0C0
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: S: (0x050C05060D0B000F010C090B0A0C0A0D07070B0D09040E03050D06040E0A000A0F010A0E040D0D020B0D020D0204040B030F09050F050205090903010308030802040D07090E0B08050A05080F0F020900060A0A020B0A03000C010900030105080006070502040E0B08010C00090303060D070A040106030E030708080B0503060D000308000C0A090F0301020B0C0A0F0B000A03040B030D040C0C0F0D0B0A0206040D0B040A00030F02020F060A0A0F09080606070D080B0D04010B060602010C0E0D0F000E000000060F070C0F0A0D0607050C0A0E0B0A0E040E060804020F0C000B040D0F060003010703090004010D000D0403050103070005090E070004080D0102090F0D0A060C0D0C0E04070A03020509020E020E04070A09020100040E08030E05000D0B0E01020F0006050A0B01000505060C0B070D0A030B0D0B090B0D0404070D0B06040901040B0809010F03040A020B090704000F030F0C030C0E00070705090B0D0F0E02010E0405090B0F0307070F040A0007020A0505040A010507090101000F010100010503070C0E0F04090D0707040C00040B090D040109030D0008040900060C0A0F0A090402020B0102070F0F080D0D0E0E080105070200060F03000A08050F0C040E0C0F070D010905010F0D080C0A0E0F0B03010C03090F050F00060F040B080B0E03010601040A080109080
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: c: (0x0008000C000000070F0F0F020004000C00040004000E000F0400050002000F0F0F0F0200050006000400050008000F0F0F0F020005000C00040005000E000F0400060002000F0F0F0F0200060006000400060008000F0F0F0F020006000C00040006000E000F0400070002000F0F0F0F0200070006000400070008000F0F0F0F020007000C00040007000E000F0400080002000F0F0F0F0200080006000400080008000F0F0F0F020008000C00040008000E000F0A0B030302040E06040A030E0A0E08020F020401030F020B080B0D04020607080B060D0B080C090F0F07070D070B05070E07050D0C060A0B080D04020E0E0706000602080C01000E030A01050F0C07060C060E010C0A0D0F020A0A070B0E0D030D040A0002080C0508000E0D07030F0B00090B0B090D08060D070E080507030F04080C000400020A06080F0D0D0B0604030003090604060C08020A020B0E0F0807020303080B0B0F010E060701050D0408080F0907060700000200000801010F060304030C08050E09060E06000E09040D07000000040D010204090308070C020F090C08050C020A0003010E040D0507080C0C060403030A000B04070701070301060F0D0A09000C090A0F0F07010C0E0D050E090C0D0A0609010C050A0A0D0E0104000E0D0D020F000B060C03000C0B0F0C08010709020E090300010A0A0C0608010B0C0
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: 128 out of 128 rollback segments are active.
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Waiting for purge to start
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: 5.7.14 started; log sequence number 1641656
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Creating 1 encryption thread id 139711846799104 total threads 4.
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Creating 2 encryption thread id 139711838406400 total threads 4.
 
2017-06-12 13:10:52 139711855191808 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/data/ib_buffer_pool
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Creating 3 encryption thread id 139711830013696 total threads 4.
 
2017-06-12 13:10:52 139713030162560 [Note] InnoDB: Creating 4 encryption thread id 139711821620992 total threads 4.
 
2017-06-12 13:10:52 139711855191808 [Note] InnoDB: Buffer pool(s) load completed at 170612 13:10:52
 
2017-06-12 13:10:52 139713030162560 [Note] Plugin 'FEEDBACK' is disabled.
 
170612 13:10:52 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
 
170612 13:10:52 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
Server version: 10.2.6-MariaDB-log
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=2002
 
thread_count=5
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4529570 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f117a4dec9e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x7f1179f2d93d]
 
/lib64/libpthread.so.0(+0xf370)[0x7f117949c370]
 
/usr/lib64/mysql/plugin/server_audit.so(+0x465d)[0x7f11746a265d]
 
/usr/lib64/mysql/plugin/server_audit.so(auditing+0xc06)[0x7f11746a7326]
 
/usr/sbin/mysqld(+0x5a0d01)[0x7f1179e6bd01]
 
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x178)[0x7f1179db2d68]
 
/usr/sbin/mysqld(_Z18mysql_audit_notifyP3THDjPKv+0xac)[0x7f1179e6c31c]
 
/usr/sbin/mysqld(+0x442805)[0x7f1179d0d805]
 
/usr/sbin/mysqld(my_message_sql+0xf3)[0x7f1179d0d913]
 
/usr/sbin/mysqld(my_printf_error+0xcb)[0x7f117a4d558b]
 
/usr/lib64/mysql/plugin/aws_key_management.so(+0x273218)[0x7f1174283218]
 
/usr/lib64/mysql/plugin/aws_key_management.so(+0x273c86)[0x7f1174283c86]
 
/usr/sbin/mysqld(_Z24init_io_cache_encryptionv+0x58)[0x7f1179ea9ab8]
 
/usr/sbin/mysqld(+0x4483cb)[0x7f1179d133cb]
 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1b51)[0x7f1179d176b1]
 
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1177a11b35]
 
/usr/sbin/mysqld(+0x440e3d)[0x7f1179d0be3d]
 
mariadb.service: main process exited, code=killed, status=11/SEGV
 
Failed to start MariaDB database server.
 
Unit mariadb.service entered failed state.
 
mariadb.service failed.
 
mariadb.service holdoff time over, scheduling restart.
 
Starting MariaDB database server...
 
2017-06-12 13:10:58 140196398078080 [Note] /usr/sbin/mysqld (mysqld 10.2.6-MariaDB-log) starting as process 2064 ...
 
2017-06-12 13:10:59 140196398078080 [Note] mysqld: AWS KMS plugin: loaded key 1, version 1, key length 256 bit
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Uses event mutexes
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Using Linux native AIO
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Number of pools: 1
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Using SSE2 crc32 instructions
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Completed initialization of buffer pool
 
2017-06-12 13:10:59 140195505399552 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Highest supported file format is Barracuda.
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: S: (0x050F0A080E0802060A040B0F060D0C010B0A070C0E0C0D0604030B030B03080D0C0600000C0F0B0C080702070F0D040F070B0507070A0D0C0B050D060B060C080D0701000109000501000D0D0407050F0F030109090F0E030408070F01070003090B03070A01040D0708080A090F0A010E0B060D0D05030F07070303080F030F0C0607030001080A0B0F020F090C04040E0100010904030E0A0209000A070B0E050D0B070D080A0405080D0C040706020A0A060A020A0502010D0C09010203030809090D0E0706030D09040D040D0C0A0B030503070003000A01030D05070A0808080108030C020109060500030308040305070E0C0B030209010F07040A0A07070E000103050E0B0903020B020E0F030A0900030E01030F0500020B000B0B0C050F0406040A0F0E0C0C0D0F0800080B0603090F000B06040C0704050D090F050A00070E0508020A080E070B06020B0B08040F0E06010B0E0905090B0409040101060D050D0C04050D010B03010A0F03050B0603090E0A0D0F06010200070108010D0D0C0C0E030B0302020B080806020E09090D08070B030B0D09090C0D0C0E0305060604070B0D0D01070B0202000A020D0C070800060F0405030F0706000105090A0907020608090909050F05070804080D080E070F010A020A0E0A05070205020300040200060F080B0101040803050A0A0F000F000B0
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: c: (0x0C01070F090E060F0109030D0C0104070C0C0B00070E0E040E0509020C0D0D01030A090C08010403070E060608090F070F01060C06000E03040A020F0C08000D0C0F0F020F0D080D020E0203010F080C070A0E0A0106070C0304030B0F0205090C040C0007050804050B0C0F0906050D0400050700000F080009060D010A03050000010C000308090C060100040B0B0B0C0E040D03010708040B090902010706010302050401070A000C0F030A050808020C0402060B0E0300050603090601090A09080A0707050506070905030C010C0A03050203030D090D08090E05030C000E02000B0C0D050601060C050B070A0D0401060F0E080B090D020A000602020F0E010508080003020B0B0E0B0E0A0205060A00020A060C0D03040C0208090E0B0B0E05010C0B0303040A0E0408040F02020E08000A0802040F0A0A00040C0E08020A05080D020B0D00090B0A000D000B04080D0E0B000C080B0C0600030D0A03040F0D010E01000E080302040708080A010F0803000A0A0D090A070402090500090A050109000A0701010C0D01060C00090F0802050D07060B0D0F0207060302070F0701090A0C010307020F0B050F05030B08010D0C0D0F05070F0B0A060E050600080A0D0806060A0808060A020D02040C0801020B020709010C08050B04040C0701080F04040D0700060609060C0209010203000F020F0
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1641675
 
2017-06-12 13:10:59 140196398078080 [ERROR] InnoDB: Incorrect log record type:85
 
2017-06-12 13:10:59 140196398078080 [Warning] InnoDB: Log scan aborted at LSN 1641472
 
2017-06-12 13:10:59 140196398078080 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
2017-06-12 13:10:59 140196398078080 [Note] InnoDB: Starting shutdown...
 
2017-06-12 13:11:00 140196398078080 [ERROR] Plugin 'InnoDB' init function returned error.
 
2017-06-12 13:11:00 140196398078080 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
2017-06-12 13:11:00 140196398078080 [Note] Plugin 'FEEDBACK' is disabled.
 
170612 13:11:00 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
 
2017-06-12 13:11:00 140196398078080 [Note] mysqld: AWS KMS plugin: loaded key 2, version 1, key length 256 bit
 
2017-06-12 13:11:00 140196398078080 [Note] Using encryption key id 2 for temporary files
 
2017-06-12 13:11:00 140196398078080 [ERROR] Unknown/unsupported storage engine: InnoDB
 
2017-06-12 13:11:00 140196398078080 [ERROR] Aborting
 
170612 13:11:00 server_audit: STOPPED
 
[INFO] 2017-06-12 13:10:58 Aws::Config::AWSConfigFileProfileConfigLoader [140196398078080] Initializing config loader against fileName /var/lib/mysql//.aws/config and using profilePrefix = 1
 
[INFO] 2017-06-12 13:10:58 Aws::Config::AWSConfigFileProfileConfigLoader [140196398078080] Initializing config loader against fileName /var/lib/mysql//.aws/credentials and using profilePrefix = 0
 
[INFO] 2017-06-12 13:10:58 ProfileConfigFileAWSCredentialsProvider [140196398078080] Setting provider to read credentials from /var/lib/mysql//.aws/credentials for credentials file and /var/lib/mysql//.aws/config for the config file , for use with profile default
 
[INFO] 2017-06-12 13:10:58 EC2MetadataClient [140196398078080] Creating HttpClient with max connections 2 and scheme http
 
[INFO] 2017-06-12 13:10:58 CurlHandleContainer [140196398078080] Initializing CurlHandleContainer with size 2
 
[INFO] 2017-06-12 13:10:58 InstanceProfileCredentialsProvider [140196398078080] Creating Instance with default EC2MetadataClient and refresh rate 900000
 
[INFO] 2017-06-12 13:10:58 CurlHandleContainer [140196398078080] Initializing CurlHandleContainer with size 25
 
[INFO] 2017-06-12 13:10:58 Aws::Config::AWSProfileConfigLoader [140196398078080] Successfully reloaded configuration.
 
[INFO] 2017-06-12 13:10:58 CurlHandleContainer [140196398078080] Pool successfully grown by 2
 
[INFO] 2017-06-12 13:10:58 CurlHandleContainer [140196398078080] Connection has been released. Continuing.
 
[INFO] 2017-06-12 13:11:00 CurlHandleContainer [140196398078080] Connection has been released. Continuing.
 
[INFO] 2017-06-12 13:11:00 CurlHandleContainer [140196398078080] Cleaning up CurlHandleContainer.
 
[INFO] 2017-06-12 13:11:00 CurlHandleContainer [140196398078080] Cleaning up CurlHandleContainer.
 
mariadb.service: main process exited, code=exited, status=1/FAILURE
 
Failed to start MariaDB database server.
 
Unit mariadb.service entered failed state.
 
mariadb.service failed.
 
Starting MariaDB database server...
 
2017-06-12 13:14:19 139926267807872 [Note] /usr/sbin/mysqld (mysqld 10.2.6-MariaDB-log) starting as process 2287 ...
 
2017-06-12 13:14:19 139926267807872 [Note] mysqld: AWS KMS plugin: loaded key 1, version 1, key length 256 bit
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Uses event mutexes
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Using Linux native AIO
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Number of pools: 1
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Using SSE2 crc32 instructions
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Completed initialization of buffer pool
 
2017-06-12 13:14:19 139925367076608 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Highest supported file format is Barracuda.
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: S: (0x050F0A080E0802060A040B0F060D0C010B0A070C0E0C0D0604030B030B03080D0C0600000C0F0B0C080702070F0D040F070B0507070A0D0C0B050D060B060C080D0701000109000501000D0D0407050F0F030109090F0E030408070F01070003090B03070A01040D0708080A090F0A010E0B060D0D05030F07070303080F030F0C0607030001080A0B0F020F090C04040E0100010904030E0A0209000A070B0E050D0B070D080A0405080D0C040706020A0A060A020A0502010D0C09010203030809090D0E0706030D09040D040D0C0A0B030503070003000A01030D05070A0808080108030C020109060500030308040305070E0C0B030209010F07040A0A07070E000103050E0B0903020B020E0F030A0900030E01030F0500020B000B0B0C050F0406040A0F0E0C0C0D0F0800080B0603090F000B06040C0704050D090F050A00070E0508020A080E070B06020B0B08040F0E06010B0E0905090B0409040101060D050D0C04050D010B03010A0F03050B0603090E0A0D0F06010200070108010D0D0C0C0E030B0302020B080806020E09090D08070B030B0D09090C0D0C0E0305060604070B0D0D01070B0202000A020D0C070800060F0405030F0706000105090A0907020608090909050F05070804080D080E070F010A020A0E0A05070205020300040200060F080B0101040803050A0A0F000F000B0
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: c: (0x0C01070F090E060F0109030D0C0104070C0C0B00070E0E040E0509020C0D0D01030A090C08010403070E060608090F070F01060C06000E03040A020F0C08000D0C0F0F020F0D080D020E0203010F080C070A0E0A0106070C0304030B0F0205090C040C0007050804050B0C0F0906050D0400050700000F080009060D010A03050000010C000308090C060100040B0B0B0C0E040D03010708040B090902010706010302050401070A000C0F030A050808020C0402060B0E0300050603090601090A09080A0707050506070905030C010C0A03050203030D090D08090E05030C000E02000B0C0D050601060C050B070A0D0401060F0E080B090D020A000602020F0E010508080003020B0B0E0B0E0A0205060A00020A060C0D03040C0208090E0B0B0E05010C0B0303040A0E0408040F02020E08000A0802040F0A0A00040C0E08020A05080D020B0D00090B0A000D000B04080D0E0B000C080B0C0600030D0A03040F0D010E01000E080302040708080A010F0803000A0A0D090A070402090500090A050109000A0701010C0D01060C00090F0802050D07060B0D0F0207060302070F0701090A0C010307020F0B050F05030B08010D0C0D0F05070F0B0A060E050600080A0D0806060A0808060A020D02040C0801020B020709010C08050B04040C0701080F04040D0700060609060C0209010203000F020F0
 
2017-06-12 13:14:19 139926267807872 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1641675
 
2017-06-12 13:14:19 139926267807872 [ERROR] InnoDB: Incorrect log record type:85
 
2017-06-12 13:14:19 139926267807872 [Warning] InnoDB: Log scan aborted at LSN 1641472
 
2017-06-12 13:14:19 139926267807872 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
2017-06-12 13:14:20 139926267807872 [Note] InnoDB: Starting shutdown...
 
2017-06-12 13:14:20 139926267807872 [ERROR] Plugin 'InnoDB' init function returned error.
 
2017-06-12 13:14:20 139926267807872 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
2017-06-12 13:14:20 139926267807872 [Note] Plugin 'FEEDBACK' is disabled.
 
170612 13:14:20 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
 
2017-06-12 13:14:20 139926267807872 [Note] mysqld: AWS KMS plugin: loaded key 2, version 1, key length 256 bit
 
2017-06-12 13:14:20 139926267807872 [Note] Using encryption key id 2 for temporary files
 
2017-06-12 13:14:20 139926267807872 [ERROR] Unknown/unsupported storage engine: InnoDB
 
2017-06-12 13:14:20 139926267807872 [ERROR] Aborting
 
170612 13:14:20 server_audit: STOPPED
 
[INFO] 2017-06-12 13:14:19 Aws::Config::AWSConfigFileProfileConfigLoader [139926267807872] Initializing config loader against fileName /var/lib/mysql//.aws/config and using profilePrefix = 1
 
[INFO] 2017-06-12 13:14:19 Aws::Config::AWSConfigFileProfileConfigLoader [139926267807872] Initializing config loader against fileName /var/lib/mysql//.aws/credentials and using profilePrefix = 0
 
[INFO] 2017-06-12 13:14:19 ProfileConfigFileAWSCredentialsProvider [139926267807872] Setting provider to read credentials from /var/lib/mysql//.aws/credentials for credentials file and /var/lib/mysql//.aws/config for the config file , for use with profile default
 
[INFO] 2017-06-12 13:14:19 EC2MetadataClient [139926267807872] Creating HttpClient with max connections 2 and scheme http
 
[INFO] 2017-06-12 13:14:19 CurlHandleContainer [139926267807872] Initializing CurlHandleContainer with size 2
 
[INFO] 2017-06-12 13:14:19 InstanceProfileCredentialsProvider [139926267807872] Creating Instance with default EC2MetadataClient and refresh rate 900000
 
[INFO] 2017-06-12 13:14:19 CurlHandleContainer [139926267807872] Initializing CurlHandleContainer with size 25
 
[INFO] 2017-06-12 13:14:19 Aws::Config::AWSProfileConfigLoader [139926267807872] Successfully reloaded configuration.
 
[INFO] 2017-06-12 13:14:19 CurlHandleContainer [139926267807872] Pool successfully grown by 2
 
[INFO] 2017-06-12 13:14:19 CurlHandleContainer [139926267807872] Connection has been released. Continuing.
 
[INFO] 2017-06-12 13:14:20 CurlHandleContainer [139926267807872] Connection has been released. Continuing.
 
[INFO] 2017-06-12 13:14:20 CurlHandleContainer [139926267807872] Cleaning up CurlHandleContainer.
 
[INFO] 2017-06-12 13:14:20 CurlHandleContainer [139926267807872] Cleaning up CurlHandleContainer.
 
mariadb.service: main process exited, code=exited, status=1/FAILURE
 
Failed to start MariaDB database server.
 
Unit mariadb.service entered failed state.
 
mariadb.service failed.



 Comments   
Comment by Dave Juntgen [ 2017-06-16 ]

How do I get traction on this issue?

Comment by Dave Juntgen [ 2017-06-21 ]

I have been working to compile a debug version of MariaDB 10.2.6 to generate a core file and symbols to help identify the issue.

Please reference this public Google Drive Folder to gain access to config, core file and mysqld compiled with debugging enabled.

GDB:

[root@localhost data]# gdb /usr/sbin/mysqld core.22014
 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
 
Copyright (C) 2013 Free Software Foundation, Inc.
 
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 
This is free software: you are free to change and redistribute it.
 
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 
and "show warranty" for details.
 
This GDB was configured as "x86_64-redhat-linux-gnu".
 
For bug reporting instructions, please see:
 
<http://www.gnu.org/software/gdb/bugs/>...
 
Reading symbols from /usr/sbin/mysqld...done.
 
[New LWP 22014]
 
[New LWP 22019]
 
[New LWP 22020]
 
[New LWP 22026]
 
[New LWP 22042]
 
[New LWP 22038]
 
[New LWP 22015]
 
[New LWP 22018]
 
[New LWP 22037]
 
[New LWP 22023]
 
[New LWP 22036]
 
[New LWP 22033]
 
[New LWP 22031]
 
[New LWP 22017]
 
[New LWP 22024]
 
[New LWP 22021]
 
[New LWP 22022]
 
[New LWP 22025]
 
[New LWP 22027]
 
[New LWP 22029]
 
[New LWP 22030]
 
[New LWP 22032]
 
[New LWP 22044]
 
[New LWP 22034]
 
[New LWP 22043]
 
[New LWP 22040]
 
[New LWP 22039]
 
[New LWP 22035]
 
[New LWP 22041]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib64/libthread_db.so.1".
 
Core was generated by `mysqld --debug=d,info,error,query:o,/tmp/mysqld.trace'.
 
Program terminated with signal 6, Aborted.
 
#0  0x00007f3fce397741 in pthread_kill () from /lib64/libpthread.so.0
 
Missing separate debuginfos, use: debuginfo-install MariaDB-server-10.2.6-1.el7.centos.x86_64
 
(gdb) where
 
#0  0x00007f3fce397741 in pthread_kill () from /lib64/libpthread.so.0
 
#1  0x00007f3fcf746358 in my_write_core (sig=6) at /vagrant/server/mysys/stacktrace.c:477
 
#2  0x00007f3fcefe8ced in handle_fatal_signal (sig=6) at /vagrant/server/sql/signal_handler.cc:299
 
#3  <signal handler called>
 
#4  0x00007f3fcc90f1d7 in raise () from /lib64/libc.so.6
 
#5  0x00007f3fcc9108c8 in abort () from /lib64/libc.so.6
 
#6  0x00007f3fcc94ef07 in __libc_message () from /lib64/libc.so.6
 
#7  0x00007f3fcc956503 in _int_free () from /lib64/libc.so.6
 
#8  0x00007f3fcf74012b in my_free (ptr=0x7f3fd2f469b8) at /vagrant/server/mysys/my_malloc.c:217
 
#9  0x00007f3fced93b82 in cleanup_variables (vars=0x7f3fd35b20e0) at /vagrant/server/sql/sql_plugin.cc:3172
 
#10 0x00007f3fced93ce7 in plugin_thdvar_cleanup (thd=0x7f3fd35b1a08) at /vagrant/server/sql/sql_plugin.cc:3198
 
#11 0x00007f3fced2c947 in THD::free_connection (this=0x7f3fd35b1a08) at /vagrant/server/sql/sql_class.cc:1511
 
#12 0x00007f3fced2cc91 in THD::~THD (this=0x7f3fd35b1a08, __in_chrg=<optimized out>) at /vagrant/server/sql/sql_class.cc:1582
 
#13 0x00007f3fced2d284 in THD::~THD (this=0x7f3fd35b1a08, __in_chrg=<optimized out>) at /vagrant/server/sql/sql_class.cc:1636
 
#14 0x00007f3fceeb5df5 in servers_init (dont_read_servers_table=false) at /vagrant/server/sql/sql_servers.cc:175
 
#15 0x00007f3fcec9d07e in init_server_components () at /vagrant/server/sql/mysqld.cc:5469
 
#16 0x00007f3fcec9d926 in mysqld_main (argc=73, argv=0x7f3fd2ab4ca8) at /vagrant/server/sql/mysqld.cc:5843
 
#17 0x00007f3fcec92685 in main (argc=2, argv=0x7ffddbad2838) at /vagrant/server/sql/main.cc:25
 
(gdb) backtrace full
 
#0  0x00007f3fce397741 in pthread_kill () from /lib64/libpthread.so.0
 
No symbol table info available.
 
#1  0x00007f3fcf746358 in my_write_core (sig=6) at /vagrant/server/mysys/stacktrace.c:477
 
No locals.
 
#2  0x00007f3fcefe8ced in handle_fatal_signal (sig=6) at /vagrant/server/sql/signal_handler.cc:299
 
        curr_time = 1498074608
 
        tm = {tm_sec = 8, tm_min = 50, tm_hour = 19, tm_mday = 21, tm_mon = 5, tm_year = 117, tm_wday = 3, tm_yday = 171, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x7f3fd2aacb80 "UTC"}
 
        thd = 0x7f3fd35b1a08
 
        print_invalid_query_pointer = false
 
#3  <signal handler called>
 
No symbol table info available.
 
#4  0x00007f3fcc90f1d7 in raise () from /lib64/libc.so.6
 
No symbol table info available.
 
#5  0x00007f3fcc9108c8 in abort () from /lib64/libc.so.6
 
No symbol table info available.
 
#6  0x00007f3fcc94ef07 in __libc_message () from /lib64/libc.so.6
 
No symbol table info available.
 
#7  0x00007f3fcc956503 in _int_free () from /lib64/libc.so.6
 
No symbol table info available.
 
#8  0x00007f3fcf74012b in my_free (ptr=0x7f3fd2f469b8) at /vagrant/server/mysys/my_malloc.c:217
 
        old_size = 8
 
        old_flags = 0 '\000'
 
        _db_stack_frame_ = {func = 0x7f3fcf81ca8e "plugin_thdvar_cleanup", file = 0x7f3fcf81bee0 "/vagrant/server/sql/sql_plugin.cc", level = 2147483653, line = -1,
 
          prev = 0x7ffddbad1c70}
 
#9  0x00007f3fced93b82 in cleanup_variables (vars=0x7f3fd35b20e0) at /vagrant/server/sql/sql_plugin.cc:3172
 
        ptr = 0x7f3fd2f8fc70
 
        v = 0x7f3fd2f4ed00
 
        idx = 13
 
        __PRETTY_FUNCTION__ = "void cleanup_variables(system_variables*)"
 
#10 0x00007f3fced93ce7 in plugin_thdvar_cleanup (thd=0x7f3fd35b1a08) at /vagrant/server/sql/sql_plugin.cc:3198
 
        idx = 2381
 
        list = 0x7ffddbad1cf0
 
        _db_stack_frame_ = {func = 0x7f3fcf80f2a0 "~THD()", file = 0x7f3fcf80eff8 "/vagrant/server/sql/sql_class.cc", level = 2147483652, line = -1, prev = 0x7ffddbad1cf0}
 
#11 0x00007f3fced2c947 in THD::free_connection (this=0x7f3fd35b1a08) at /vagrant/server/sql/sql_class.cc:1511
 
        __PRETTY_FUNCTION__ = "void THD::free_connection()"
 
#12 0x00007f3fced2cc91 in THD::~THD (this=0x7f3fd35b1a08, __in_chrg=<optimized out>) at /vagrant/server/sql/sql_class.cc:1582
 
        orig_thd = 0x7f3fd35b1a08
 
        _db_stack_frame_ = {func = 0x7f3fcf845da2 "servers_init", file = 0x7f3fcf845d28 "/vagrant/server/sql/sql_servers.cc", level = 2147483651, line = -1, prev = 0x7ffddbad1d90}
 
        __PRETTY_FUNCTION__ = "virtual THD::~THD()"
 
#13 0x00007f3fced2d284 in THD::~THD (this=0x7f3fd35b1a08, __in_chrg=<optimized out>) at /vagrant/server/sql/sql_class.cc:1636
 
        __PRETTY_FUNCTION__ = "virtual THD::~THD()"
 
#14 0x00007f3fceeb5df5 in servers_init (dont_read_servers_table=false) at /vagrant/server/sql/sql_servers.cc:175
 
---Type <return> to continue, or q <return> to quit---
 
        thd = 0x7f3fd35b1a08
 
        return_val = false
 
        _db_stack_frame_ = {func = 0x7f3fcf7f333d "init_server_components", file = 0x7f3fcf7f158a "/vagrant/server/sql/mysqld.cc", level = 2147483650, line = -1, prev = 0x7ffddbad1e20}
 
#15 0x00007f3fcec9d07e in init_server_components () at /vagrant/server/sql/mysqld.cc:5469
 
        _db_stack_frame_ = {func = 0x7f3fcfab8125 "?func", file = 0x7f3fcfab812b "?file", level = 2147483649, line = -1, prev = 0x0}
 
        __PRETTY_FUNCTION__ = "int init_server_components()"
 
#16 0x00007f3fcec9d926 in mysqld_main (argc=73, argv=0x7f3fd2ab4ca8) at /vagrant/server/sql/mysqld.cc:5843
 
        ho_error = 0
 
        new_thread_stack_size = 299008
 
#17 0x00007f3fcec92685 in main (argc=2, argv=0x7ffddbad2838) at /vagrant/server/sql/main.cc:25
 
No locals.

Comment by Andrii Nikitin (Inactive) [ 2017-06-22 ]

I've confirmed with kind of exact steps, just got different crashes, e.g. during mysql_install_db as well
It looks it is some timing / concurrency issue during plugins initialization; working on this now

Comment by Andrii Nikitin (Inactive) [ 2017-06-22 ]

djuntgen, could you confirm if any troubleshooting is expected about the fact that InnoDB doesn't start anymore? (because it looks slightly different problem than original crash).
If you have chance to try original test case once again - could you confirm:
1. if the system has file /etc/my.cnf.d/enable_encryption.preset
2. whether the crash happens if you comment out line with encrypt-tmp-files from /etc/my.cnf.d/enable_encryption.preset ?

Comment by Andrii Nikitin (Inactive) [ 2017-06-22 ]

I observed the problem both in 10.2.6 on Centos 7 and in current 10.2 on Ubuntu

According to my analysis memory corruption happens with encrypt-tmp-files=1 around init_io_cache_encryption() when AWS plugin tries to print a Note into Error log that Encryption Key 2 (for encrypt-tmp-files) is loaded .
That note is redirected to Audit plugin, (which possibly is not fully initialized yet?).

No problem was observed when Audit plugin is loaded dynamically after server startup or when encrypt-tmp-files=0

I am leaving ticket as verified with instructions from original description, with a note that installer on Centos7 does generate enable_encryption.preset which has encrypt-tmp-files=1 , which triggers the problem according to my analysis.

When memory corruption happens, stack traces and crash dump often are irrelevant (because they indicate state some time after corruption).
Anyway, below are few examples of stack traces I observed.

(gdb) c
 
Continuing.
 
*** Error in `/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld': free(): invalid next size (fast): 0x0000555557248970 ***
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x790cb)[0x7ffff60be0cb]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x82c9a)[0x7ffff60c7c9a]
 
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ffff60cbd8c]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(+0x4b5040)[0x555555a09040]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_Z21plugin_thdvar_cleanupP3THD+0x7b)[0x555555a0be1b]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_ZN3THD15free_connectionEv+0x68)[0x5555559d1f28]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_ZN3THDD1Ev+0x510)[0x5555559d2470]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_ZN3THDD0Ev+0x9)[0x5555559d24f9]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_Z12servers_initb+0x11e)[0x555555abc87e]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(+0x4252b7)[0x5555559792b7]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_Z11mysqld_mainiPPc+0x1c11)[0x55555597d311]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7ffff60653f1]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld(_start+0x2a)[0x555555971c8a]
 
======= Memory map: ========
 
555555554000-5555565ac000 r-xp 00000000 08:23 1639168594                 /home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld
 
5555567ac000-55555687b000 r--p 01058000 08:23 1639168594                 /home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld
 
55555687b000-555556931000 rw-p 01127000 08:23 1639168594                 /home/a/test1/mariadb-environs/m9-10.2/build/sql/mysqld
 
555556931000-555558fdb000 rw-p 00000000 00:00 0                          [heap]
 
7fff84000000-7fff84021000 rw-p 00000000 00:00 0 
7fff84021000-7fff88000000 ---p 00000000 00:00 0 
7fff8c000000-7fff8c021000 rw-p 00000000 00:00 0 
7fff8c021000-7fff90000000 ---p 00000000 00:00 0 
7fff9054b000-7fff98000000 rw-p 00000000 00:00 0 
7fff98000000-7fff98021000 rw-p 00000000 00:00 0 
7fff98021000-7fff9c000000 ---p 00000000 00:00 0 
7fff9c000000-7fff9c021000 rw-p 00000000 00:00 0 
7fff9c021000-7fffa0000000 ---p 00000000 00:00 0 
7fffa0000000-7fffa0021000 rw-p 00000000 00:00 0 
7fffa0021000-7fffa4000000 ---p 00000000 00:00 0 
7fffa47f9000-7fffa47fa000 ---p 00000000 00:00 0 
7fffa47fa000-7fffa4ffa000 rw-p 00000000 00:00 0 
7fffa4ffa000-7fffa4ffb000 ---p 00000000 00:00 0 
7fffa4ffb000-7fffa57fb000 rw-p 00000000 00:00 0 
7fffa57fb000-7fffa57fc000 ---p 00000000 00:00 0 
7fffa57fc000-7fffa5ffc000 rw-p 00000000 00:00 0 
7fffa5ffc000-7fffa5ffd000 ---p 00000000 00:00 0 
7fffa5ffd000-7fffa67fd000 rw-p 00000000 00:00 0 
7fffa67fd000-7fffa67fe000 ---p 00000000 00:00 0 
7fffa67fe000-7fffa6ffe000 rw-p 00000000 00:00 0 
7fffa6ffe000-7fffa6fff000 ---p 00000000 00:00 0 
7fffa6fff000-7fffa77ff000 rw-p 00000000 00:00 0 
7fffa77ff000-7fffa7800000 ---p 00000000 00:00 0 
7fffa7800000-7fffa8000000 rw-p 00000000 00:00 0 
7fffa8000000-7fffa8021000 rw-p 00000000 00:00 0 
7fffa8021000-7fffac000000 ---p 00000000 00:00 0 
7fffac000000-7fffac021000 rw-p 00000000 00:00 0 
7fffac021000-7fffb0000000 ---p 00000000 00:00 0 
7fffb0000000-7fffb0021000 rw-p 00000000 00:00 0 
7fffb0021000-7fffb4000000 ---p 00000000 00:00 0 
7fffb4000000-7fffb4021000 rw-p 00000000 00:00 0 
7fffb4021000-7fffb8000000 ---p 00000000 00:00 0 
7fffb8000000-7fffb8021000 rw-p 00000000 00:00 0 
7fffb8021000-7fffbc000000 ---p 00000000 00:00 0 
7fffbc000000-7fffbc021000 rw-p 00000000 00:00 0 
7fffbc021000-7fffc0000000 ---p 00000000 00:00 0 
7fffc0586000-7fffc05f8000 r-xp 00000000 08:23 536923275                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fffc05f8000-7fffc07f7000 ---p 00072000 08:23 536923275                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fffc07f7000-7fffc07f8000 r--p 00071000 08:23 536923275                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fffc07f8000-7fffc07f9000 rw-p 00072000 08:23 536923275                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fffc07f9000-7fffc07fa000 ---p 00000000 00:00 0 
7fffc07fa000-7fffc0ffa000 rw-p 00000000 00:00 0 
7fffc0ffa000-7fffc0ffb000 ---p 00000000 00:00 0 
7fffc0ffb000-7fffc17fb000 rw-p 00000000 00:00 0 
7fffc17fb000-7fffc17fc000 ---p 00000000 00:00 0 
7fffc17fc000-7fffc1ffc000 rw-p 00000000 00:00 0 
7fffc1ffc000-7fffc1ffd000 ---p 00000000 00:00 0 
7fffc1ffd000-7fffc27fd000 rw-p 00000000 00:00 0 
7fffc27fd000-7fffc27fe000 ---p 00000000 00:00 0 
7fffc27fe000-7fffc2ffe000 rw-p 00000000 00:00 0 
7fffc2ffe000-7fffc2fff000 ---p 00000000 00:00 0 
7fffc2fff000-7fffc37ff000 rw-p 00000000 00:00 0 
7fffc37ff000-7fffc3800000 ---p 00000000 00:00 0 
7fffc3800000-7fffc4000000 rw-p 00000000 00:00 0 
7fffc4000000-7fffc4021000 rw-p 00000000 00:00 0 
7fffc4021000-7fffc8000000 ---p 00000000 00:00 0 
7fffc8000000-7fffc8021000 rw-p 00000000 00:00 0 
7fffc8021000-7fffcc000000 ---p 00000000 00:00 0 
7fffcc000000-7fffcc021000 rw-p 00000000 00:00 0 
7fffcc021000-7fffd0000000 ---p 00000000 00:00 0 
7fffd0069000-7fffd0070000 r-xp 00000000 08:23 570993512                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7fffd0070000-7fffd026f000 ---p 00007000 08:23 570993512                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7fffd026f000-7fffd0270000 r--p 00006000 08:23 570993512                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7fffd0270000-7fffd0271000 rw-p 00007000 08:23 570993512                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7fffd0271000-7fffd07bd000 rw-p 00000000 00:00 0 
7fffd07bd000-7fffd07be000 ---p 00000000 00:00 0 
7fffd07be000-7fffd0fbe000 rw-p 00000000 00:00 0 
7fffd0fbe000-7fffd0fbf000 ---p 00000000 00:00 0 
7fffd0fbf000-7fffd17bf000 rw-p 00000000 00:00 0 
7fffd17bf000-7fffd17c0000 ---p 00000000 00:00 0 
7fffd17c0000-7fffd1fc0000 rw-p 00000000 00:00 0 
7fffd1fc0000-7fffd1fc1000 ---p 00000000 00:00 0 
7fffd1fc1000-7fffd27c1000 rw-p 00000000 00:00 0 
7fffd27c1000-7fffd27c2000 ---p 00000000 00:00 0 
7fffd27c2000-7fffd2fc2000 rw-p 00000000 00:00 0 
7fffd2fc2000-7fffd2fc3000 ---p 00000000 00:00 0 
7fffd2fc3000-7fffd39c8000 rw-p 00000000 00:00 0 
7fffd39c8000-7fffd39c9000 ---p 00000000 00:00 0 
7fffd39c9000-7fffd41c9000 rw-p 00000000 00:00 0 
7fffd41c9000-7fffd41ca000 ---p 00000000 00:00 0 
7fffd41ca000-7fffd49ca000 rw-p 00000000 00:00 0 
7fffd49ca000-7fffd49cb000 ---p 00000000 00:00 0 
7fffd49cb000-7fffd51cb000 rw-p 00000000 00:00 0 
7fffd51cb000-7fffd51cc000 ---p 00000000 00:00 0 
7fffd51cc000-7fffd59cc000 rw-p 00000000 00:00 0 
7fffd59cc000-7fffd59cd000 ---p 00000000 00:00 0 
7fffd59cd000-7fffd61cd000 rw-p 00000000 00:00 0 
7fffd61cd000-7fffd61ce000 ---p 00000000 00:00 0 
7fffd61ce000-7fffd69ce000 rw-p 00000000 00:00 0 
7fffd69ce000-7fffd69cf000 ---p 00000000 00:00 0 
7fffd69cf000-7fffd71cf000 rw-p 00000000 00:00 0 
7fffd71cf000-7fffd71d0000 ---p 00000000 00:00 0 
7fffd71d0000-7fffd79d0000 rw-p 00000000 00:00 0 
7fffd79d0000-7fffd79d1000 ---p 00000000 00:00 0 
7fffd79d1000-7fffd81d1000 rw-p 00000000 00:00 0 
7fffd81d1000-7fffd81d2000 ---p 00000000 00:00 0 
7fffd81d2000-7fffd89d2000 rw-p 00000000 00:00 0 
7fffd89d2000-7fffd89d3000 ---p 00000000 00:00 0 
7fffd89d3000-7fffd92c9000 rw-p 00000000 00:00 0 
7fffd945f000-7fffd9482000 r-xp 00000000 08:23 536923302                  /lib/x86_64-linux-gnu/libselinux.so.1
 
7fffd9482000-7fffd9681000 ---p 00023000 08:23 536923302                  /lib/x86_64-linux-gnu/libselinux.so.1
 
7fffd9681000-7fffd9682000 r--p 00022000 08:23 536923302                  /lib/x86_64-linux-gnu/libselinux.so.1
 
7fffd9682000-7fffd9683000 rw-p 00023000 08:23 536923302                  /lib/x86_64-linux-gnu/libselinux.so.1
 
7fffd9683000-7fffd9685000 rw-p 00000000 00:00 0 
7fffd9685000-7fffd96c5000 r-xp 00000000 08:23 536920584                  /lib/x86_64-linux-gnu/libnss_resolve.so.2
 
7fffd96c5000-7fffd96c8000 r--p 0003f000 08:23 536920584                  /lib/x86_64-linux-gnu/libnss_resolve.so.2
 
7fffd96c8000-7fffd96c9000 rw-p 00042000 08:23 536920584                  /lib/x86_64-linux-gnu/libnss_resolve.so.2
 
7fffd96e7000-7fffe3ff5000 rw-p 00000000 00:00 0 
7fffe3ff5000-7fffe4000000 r-xp 00000000 08:23 68012418                   /home/a/test1/mariadb-environs/m9-10.2/build/plugin/server_audit/server_audit.so
 
7fffe4000000-7fffe41ff000 ---p 0000b000 08:23 68012418                   /home/a/test1/mariadb-environs/m9-10.2/build/plugin/server_audit/server_audit.so
 
7fffe41ff000-7fffe4200000 r--p 0000a000 08:23 68012418                   /home/a/test1/mariadb-environs/m9-10.2/build/plugin/server_audit/server_audit.so
 
7fffe4200000-7fffe4202000 rw-p 0000b000 08:23 68012418                   /home/a/test1/mariadb-environs/m9-10.2/build/plugin/server_audit/server_audit.so
 
7fffe4202000-7fffe4204000 rw-p 00000000 00:00 0 
7fffe4204000-7fffe4300000 r-xp 00000000 08:23 1074798257                 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 
7fffe4300000-7fffe4500000 ---p 000fc000 08:23 1074798257                 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 
7fffe4500000-7fffe4503000 r--p 000fc000 08:23 1074798257                 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 
7fffe4503000-7fffe4505000 rw-p 000ff000 08:23 1074798257                 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 
7fffe4505000-7fffe4506000 rw-p 00000000 00:00 0 
7fffe4506000-7fffe454c000 r-xp 00000000 08:23 1074797730                 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 
7fffe454c000-7fffe474c000 ---p 00046000 08:23 1074797730                 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 
7fffe474c000-7fffe474e000 r--p 00046000 08:23 1074797730                 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 
7fffe474e000-7fffe4750000 rw-p 00048000 08:23 1074797730                 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 
7fffe4750000-7fffe4751000 rw-p 00000000 00:00 0 
7fffe4751000-7fffe475f000 r-xp 00000000 08:23 1074797710                 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 
7fffe475f000-7fffe495e000 ---p 0000e000 08:23 1074797710                 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 
7fffe495e000-7fffe495f000 r--p 0000d000 08:23 1074797710                 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 
7fffe495f000-7fffe4960000 rw-p 0000e000 08:23 1074797710                 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 
7fffe4960000-7fffe4987000 r-xp 00000000 08:23 1074798484                 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 
7fffe4987000-7fffe4b87000 ---p 00027000 08:23 1074798484                 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 
7fffe4b87000-7fffe4b88000 r--p 00027000 08:23 1074798484                 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 
7fffe4b88000-7fffe4b89000 rw-p 00028000 08:23 1074798484                 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 
7fffe4b89000-7fffe4b9e000 r-xp 00000000 08:23 1074798173                 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 
7fffe4b9e000-7fffe4d9d000 ---p 00015000 08:23 1074798173                 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 
7fffe4d9d000-7fffe4d9e000 r--p 00014000 08:23 1074798173                 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 
7fffe4d9e000-7fffe4d9f000 rw-p 00015000 08:23 1074798173                 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 
7fffe4d9f000-7fffe4dcf000 r-xp 00000000 08:23 1074797708                 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 
7fffe4dcf000-7fffe4fcf000 ---p 00030000 08:23 1074797708                 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 
7fffe4fcf000-7fffe4fd0000 r--p 00030000 08:23 1074797708                 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 
7fffe4fd0000-7fffe4fd1000 rw-p 00031000 08:23 1074797708                 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 
7fffe4fd1000-7fffe4fd2000 rw-p 00000000 00:00 0 
7fffe4fd2000-7fffe5071000 r-xp 00000000 08:23 1074797128                 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 
7fffe5071000-7fffe5270000 ---p 0009f000 08:23 1074797128                 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 
7fffe5270000-7fffe5271000 r--p 0009e000 08:23 1074797128                 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 
7fffe5271000-7fffe5274000 rw-p 0009f000 08:23 1074797128                 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 
7fffe5274000-7fffe52f8000 r-xp 00000000 08:23 1074797820                 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 
7fffe52f8000-7fffe54f7000 ---p 00084000 08:23 1074797820                 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 
7fffe54f7000-7fffe54fa000 r--p 00083000 08:23 1074797820                 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 
7fffe54fa000-7fffe54fd000 rw-p 00086000 08:23 1074797820                 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 
7fffe54fd000-7fffe54fe000 rw-p 00000000 00:00 0 
7fffe54fe000-7fffe5506000 r-xp 00000000 08:23 1074797712                 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 
7fffe5506000-7fffe5705000 ---p 00008000 08:23 1074797712                 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 
7fffe5705000-7fffe5706000 r--p 00007000 08:23 1074797712                 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 
7fffe5706000-7fffe5707000 rw-p 00008000 08:23 1074797712                 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 
7fffe5707000-7fffe570a000 r-xp 00000000 08:23 536923206                  /lib/x86_64-linux-gnu/libkeyutils.so.1.5
 
7fffe570a000-7fffe5909000 ---p 00003000 08:23 536923206                  /lib/x86_64-linux-gnu/libkeyutils.so.1.5
 
7fffe5909000-7fffe590a000 r--p 00002000 08:23 536923206                  /lib/x86_64-linux-gnu/libkeyutils.so.1.5
 
7fffe590a000-7fffe590b000 rw-p 00003000 08:23 536923206                  /lib/x86_64-linux-gnu/libkeyutils.so.1.5
 
7fffe590b000-7fffe5912000 r-xp 00000000 08:23 1074797435                 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
 
7fffe5912000-7fffe5b11000 ---p 00007000 08:23 1074797435                 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
 
7fffe5b11000-7fffe5b12000 r--p 00006000 08:23 1074797435                 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
 
7fffe5b12000-7fffe5b13000 rw-p 00007000 08:23 1074797435                 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
 
7fffe5b13000-7fffe5b50000 r-xp 00000000 08:23 1074797631                 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 
7fffe5b50000-7fffe5d50000 ---p 0003d000 08:23 1074797631                 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 
7fffe5d50000-7fffe5d51000 r--p 0003d000 08:23 1074797631                 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 
7fffe5d51000-7fffe5d53000 rw-p 0003e000 08:23 1074797631                 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 
7fffe5d53000-7fffe5d54000 rw-p 00000000 00:00 0 
7fffe5d54000-7fffe5d6d000 r-xp 00000000 08:23 1074798201                 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 
7fffe5d6d000-7fffe5f6d000 ---p 00019000 08:23 1074798201                 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 
7fffe5f6d000-7fffe5f6e000 r--p 00019000 08:23 1074798201                 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 
7fffe5f6e000-7fffe5f6f000 rw-p 0001a000 08:23 1074798201                 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 
7fffe5f6f000-7fffe5f86000 r-xp 00000000 08:23 570993511                  /lib/x86_64-linux-gnu/libresolv-2.24.so
 
7fffe5f86000-7fffe6186000 ---p 00017000 08:23 570993511                  /lib/x86_64-linux-gnu/libresolv-2.24.so
 
7fffe6186000-7fffe6187000 r--p 00017000 08:23 570993511                  /lib/x86_64-linux-gnu/libresolv-2.24.so
 
7fffe6187000-7fffe6188000 rw-p 00018000 08:23 570993511                  /lib/x86_64-linux-gnu/libresolv-2.24.so
 
7fffe6188000-7fffe618a000 rw-p 00000000 00:00 0 
7fffe618a000-7fffe6194000 r-xp 00000000 08:23 1074794119                 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 
7fffe6194000-7fffe6393000 ---p 0000a000 08:23 1074794119                 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 
7fffe6393000-7fffe6394000 r--p 00009000 08:23 1074794119                 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 
7fffe6394000-7fffe6395000 rw-p 0000a000 08:23 1074794119                 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 
7fffe6395000-7fffe6398000 r-xp 00000000 08:23 536923158                  /lib/x86_64-linux-gnu/libcom_err.so.2.1
 
7fffe6398000-7fffe6597000 ---p 00003000 08:23 536923158                  /lib/x86_64-linux-gnu/libcom_err.so.2.1
 
7fffe6597000-7fffe6598000 r--p 00002000 08:23 536923158                  /lib/x86_64-linux-gnu/libcom_err.so.2.1
 
7fffe6598000-7fffe6599000 rw-p 00003000 08:23 536923158                  /lib/x86_64-linux-gnu/libcom_err.so.2.1
 
7fffe6599000-7fffe65c5000 r-xp 00000000 08:23 1074794121                 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 
7fffe65c5000-7fffe67c4000 ---p 0002c000 08:23 1074794121                 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 
7fffe67c4000-7fffe67c6000 r--p 0002b000 08:23 1074794121                 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 
7fffe67c6000-7fffe67c7000 rw-p 0002d000 08:23 1074794121                 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 
7fffe67c7000-7fffe67c8000 rw-p 00000000 00:00 0 
7fffe67c8000-7fffe688c000 r-xp 00000000 08:23 1074794114                 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 
7fffe688c000-7fffe6a8b000 ---p 000c4000 08:23 1074794114                 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 
7fffe6a8b000-7fffe6a99000 r--p 000c3000 08:23 1074794114                 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 
7fffe6a99000-7fffe6a9b000 rw-p 000d1000 08:23 1074794114                 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 
7fffe6a9b000-7fffe6aac000 r-xp 00000000 08:23 1074798276                 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.2
 
7fffe6aac000-7fffe6cac000 ---p 00011000 08:23 1074798276                 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.2
 
7fffe6cac000-7fffe6cad000 r--p 00011000 08:23 1074798276                 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.2
 
7fffe6cad000-7fffe6cae000 rw-p 00012000 08:23 1074798276                 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.2
 
7fffe6cae000-7fffe6d07000 r-xp 00000000 08:23 1074798035                 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
 
7fffe6d07000-7fffe6f06000 ---p 00059000 08:23 1074798035                 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
 
7fffe6f06000-7fffe6f10000 r--p 00058000 08:23 1074798035                 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
 
7fffe6f10000-7fffe6f12000 rw-p 00062000 08:23 1074798035                 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
 
7fffe6f12000-7fffe6f91000 r-xp 00000000 08:23 1074797576                 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.1
 
7fffe6f91000-7fffe7190000 ---p 0007f000 08:23 1074797576                 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.1
 
7fffe7190000-7fffe7191000 r--p 0007e000 08:23 1074797576                 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.1
 
7fffe7191000-7fffe7192000 rw-p 0007f000 08:23 1074797576                 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.1
 
7fffe7192000-7fffe71c4000 r-xp 00000000 08:23 1074793959                 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
 
7fffe71c4000-7fffe73c3000 ---p 00032000 08:23 1074793959                 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
 
7fffe73c3000-7fffe73c4000 r--p 00031000 08:23 1074793959                 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
 
7fffe73c4000-7fffe73c5000 rw-p 00032000 08:23 1074793959                 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
 
7fffe73c5000-7fffe7412000 r-xp 00000000 08:23 1074797835                 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
 
7fffe7412000-7fffe7611000 ---p 0004d000 08:23 1074797835                 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
 
7fffe7611000-7fffe7613000 r--p 0004c000 08:23 1074797835                 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
 
7fffe7613000-7fffe7614000 rw-p 0004e000 08:23 1074797835                 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
 
7fffe7614000-7fffe7616000 rw-p 00000000 00:00 0 
7fffe7616000-7fffe7623000 r-xp 00000000 08:23 1074797830                 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
 
7fffe7623000-7fffe7823000 ---p 0000d000 08:23 1074797830                 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
 
7fffe7823000-7fffe7824000 r--p 0000d000 08:23 1074797830                 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
 
7fffe7824000-7fffe7825000 rw-p 0000e000 08:23 1074797830                 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
 
7fffe7825000-7fffe786c000 r-xp 00000000 08:23 1074794110                 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 
7fffe786c000-7fffe7a6c000 ---p 00047000 08:23 1074794110                 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 
7fffe7a6c000-7fffe7a6d000 r--p 00047000 08:23 1074794110                 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 
7fffe7a6d000-7fffe7a6f000 rw-p 00048000 08:23 1074794110                 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 
7fffe7a6f000-7fffe7bbc000 r-xp 00000000 08:23 1074792606                 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.9.0
 
7fffe7bbc000-7fffe7dbb000 ---p 0014d000 08:23 1074792606                 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.9.0
 
7fffe7dbb000-7fffe7dc7000 r--p 0014c000 08:23 1074792606                 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.9.0
 
7fffe7dc7000-7fffe7dc8000 rw-p 00158000 08:23 1074792606                 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.9.0
 
7fffe7dc8000-7fffe7dc9000 rw-p 00000000 00:00 0 
7fffe7dc9000-7fffe7dfd000 r-xp 00000000 08:23 1074793966                 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
 
7fffe7dfd000-7fffe7ffc000 ---p 00034000 08:23 1074793966                 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
 
7fffe7ffc000-7fffe7ffe000 r--p 00033000 08:23 1074793966                 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
 
7fffe7ffe000-7fffe7fff000 rw-p 00035000 08:23 1074793966                 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
 
7fffe7fff000-7ffff0000000 rw-p 00000000 00:00 0 
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 
7ffff4017000-7ffff4166000 rw-p 00000000 00:00 0 
7ffff4166000-7ffff4181000 r-xp 00000000 08:23 1074798181                 /usr/lib/x86_64-linux-gnu/librtmp.so.1
 
7ffff4181000-7ffff4380000 ---p 0001b000 08:23 1074798181                 /usr/lib/x86_64-linux-gnu/librtmp.so.1
 
7ffff4380000-7ffff4381000 r--p 0001a000 08:23 1074798181                 /usr/lib/x86_64-linux-gnu/librtmp.so.1
 
7ffff4381000-7ffff4382000 rw-p 0001b000 08:23 1074798181                 /usr/lib/x86_64-linux-gnu/librtmp.so.1
 
7ffff4382000-7ffff43b3000 r-xp 00000000 08:23 536923199                  /lib/x86_64-linux-gnu/libidn.so.11.6.16
 
7ffff43b3000-7ffff45b3000 ---p 00031000 08:23 536923199                  /lib/x86_64-linux-gnu/libidn.so.11.6.16
 
7ffff45b3000-7ffff45b4000 r--p 00031000 08:23 536923199                  /lib/x86_64-linux-gnu/libidn.so.11.6.16
 
7ffff45b4000-7ffff45b5000 rw-p 00032000 08:23 536923199                  /lib/x86_64-linux-gnu/libidn.so.11.6.16
 
7ffff45b5000-7ffff45b9000 r-xp 00000000 08:23 536922938                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
 
7ffff45b9000-7ffff47b8000 ---p 00004000 08:23 536922938                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
 
7ffff47b8000-7ffff47b9000 r--p 00003000 08:23 536922938                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
 
7ffff47b9000-7ffff47ba000 rw-p 00004000 08:23 536922938                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
 
7ffff47ba000-7ffff4825000 r-xp 00000000 08:23 1074794123                 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
 
7ffff4825000-7ffff4a24000 ---p 0006b000 08:23 1074794123                 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
 
7ffff4a24000-7ffff4a27000 r--p 0006a000 08:23 1074794123                 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
 
7ffff4a27000-7ffff4a28000 rw-p 0006d000 08:23 1074794123                 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
 
7ffff4a28000-7ffff4de9000 r-xp 00000000 08:23 1126477837                 /home/a/test1/mariadb-environs/m9-10.2/plugin/aws_key_management.so
 
7ffff4de9000-7ffff4fe8000 ---p 003c1000 08:23 1126477837                 /home/a/test1/mariadb-environs/m9-10.2/plugin/aws_key_management.so
 
7ffff4fe8000-7ffff5002000 r--p 003c0000 08:23 1126477837                 /home/a/test1/mariadb-environs/m9-10.2/plugin/aws_key_management.so
 
7ffff5002000-7ffff5003000 rw-p 003da000 08:23 1126477837                 /home/a/test1/mariadb-environs/m9-10.2/plugin/aws_key_management.so
 
7ffff5003000-7ffff5004000 rw-p 00000000 00:00 0 
7ffff5004000-7ffff5005000 ---p 00000000 00:00 0 
7ffff5005000-7ffff5805000 rw-p 00000000 00:00 0 
7ffff5805000-7ffff5810000 r-xp 00000000 08:23 570993505                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7ffff5810000-7ffff5a0f000 ---p 0000b000 08:23 570993505                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7ffff5a0f000-7ffff5a10000 r--p 0000a000 08:23 570993505                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7ffff5a10000-7ffff5a11000 rw-p 0000b000 08:23 570993505                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7ffff5a11000-7ffff5a17000 rw-p 00000000 00:00 0 
7ffff5a17000-7ffff5a22000 r-xp 00000000 08:23 570993507                  /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7ffff5a22000-7ffff5c21000 ---p 0000b000 08:23 570993507                  /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7ffff5c21000-7ffff5c22000 r--p 0000a000 08:23 570993507                  /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7ffff5c22000-7ffff5c23000 rw-p 0000b000 08:23 570993507                  /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7ffff5c23000-7ffff5c39000 r-xp 00000000 08:23 570984670                  /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7ffff5c39000-7ffff5e38000 ---p 00016000 08:23 570984670                  /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7ffff5e38000-7ffff5e39000 r--p 00015000 08:23 570984670                  /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7ffff5e39000-7ffff5e3a000 rw-p 00016000 08:23 570984670                  /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7ffff5e3a000-7ffff5e3c000 rw-p 00000000 00:00 0 
7ffff5e3c000-7ffff5e44000 r-xp 00000000 08:23 570984671                  /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7ffff5e44000-7ffff6043000 ---p 00008000 08:23 570984671                  /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7ffff6043000-7ffff6044000 r--p 00007000 08:23 570984671                  /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7ffff6044000-7ffff6045000 rw-p 00008000 08:23 570984671                  /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7ffff6045000-7ffff6203000 r-xp 00000000 08:23 567564780                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7ffff6203000-7ffff6402000 ---p 001be000 08:23 567564780                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7ffff6402000-7ffff6406000 r--p 001bd000 08:23 567564780                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7ffff6406000-7ffff6408000 rw-p 001c1000 08:23 567564780                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7ffff6408000-7ffff640c000 rw-p 00000000 00:00 0 
7ffff640c000-7ffff6422000 r-xp 00000000 08:23 536923187                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7ffff6422000-7ffff6621000 ---p 00016000 08:23 536923187                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7ffff6621000-7ffff6622000 r--p 00015000 08:23 536923187                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7ffff6622000-7ffff6623000 rw-p 00016000 08:23 536923187                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7ffff6623000-7ffff672b000 r-xp 00000000 08:23 570984667                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7ffff672b000-7ffff692a000 ---p 00108000 08:23 570984667                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7ffff692a000-7ffff692b000 r--p 00107000 08:23 570984667                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7ffff692b000-7ffff692c000 rw-p 00108000 08:23 570984667                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7ffff692c000-7ffff6aa4000 r-xp 00000000 08:23 1074798266                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7ffff6aa4000-7ffff6ca4000 ---p 00178000 08:23 1074798266                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7ffff6ca4000-7ffff6cae000 r--p 00178000 08:23 1074798266                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7ffff6cae000-7ffff6cb0000 rw-p 00182000 08:23 1074798266                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7ffff6cb0000-7ffff6cb4000 rw-p 00000000 00:00 0 
7ffff6cb4000-7ffff6ccc000 r-xp 00000000 08:23 570993510                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7ffff6ccc000-7ffff6ecc000 ---p 00018000 08:23 570993510                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7ffff6ecc000-7ffff6ecd000 r--p 00018000 08:23 570993510                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7ffff6ecd000-7ffff6ece000 rw-p 00019000 08:23 570993510                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7ffff6ece000-7ffff6ed2000 rw-p 00000000 00:00 0 
7ffff6ed2000-7ffff6ed5000 r-xp 00000000 08:23 570984666                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7ffff6ed5000-7ffff70d4000 ---p 00003000 08:23 570984666                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7ffff70d4000-7ffff70d5000 r--p 00002000 08:23 570984666                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7ffff70d5000-7ffff70d6000 rw-p 00003000 08:23 570984666                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7ffff70d6000-7ffff72ef000 r-xp 00000000 08:23 536923365                  /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 
7ffff72ef000-7ffff74ef000 ---p 00219000 08:23 536923365                  /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 
7ffff74ef000-7ffff750b000 r--p 00219000 08:23 536923365                  /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 
7ffff750b000-7ffff7517000 rw-p 00235000 08:23 536923365                  /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 
7ffff7517000-7ffff751a000 rw-p 00000000 00:00 0 
7ffff751a000-7ffff7578000 r-xp 00000000 08:23 536924914                  /lib/x86_64-linux-gnu/libssl.so.1.0.0
 
7ffff7578000-7ffff7778000 ---p 0005e000 08:23 536924914                  /lib/x86_64-linux-gnu/libssl.so.1.0.0
 
7ffff7778000-7ffff777c000 r--p 0005e000 08:23 536924914                  /lib/x86_64-linux-gnu/libssl.so.1.0.0
 
7ffff777c000-7ffff7783000 rw-p 00062000 08:23 536924914                  /lib/x86_64-linux-gnu/libssl.so.1.0.0
 
7ffff7783000-7ffff778c000 r-xp 00000000 08:23 570984665                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7ffff778c000-7ffff798b000 ---p 00009000 08:23 570984665                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7ffff798b000-7ffff798c000 r--p 00008000 08:23 570984665                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7ffff798c000-7ffff798d000 rw-p 00009000 08:23 570984665                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7ffff798d000-7ffff79bb000 rw-p 00000000 00:00 0 
7ffff79bb000-7ffff79d4000 r-xp 00000000 08:23 589977864                  /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7ffff79d4000-7ffff7bd3000 ---p 00019000 08:23 589977864                  /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7ffff7bd3000-7ffff7bd4000 r--p 00018000 08:23 589977864                  /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7ffff7bd4000-7ffff7bd5000 rw-p 00019000 08:23 589977864                  /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7ffff7bd5000-7ffff7bd6000 r-xp 00000000 08:23 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7ffff7bd6000-7ffff7dd5000 ---p 00001000 08:23 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7ffff7dd5000-7ffff7dd6000 r--p 00000000 08:23 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7ffff7dd6000-7ffff7dd7000 rw-p 00000000 00:00 0 
7ffff7dd7000-7ffff7dfc000 r-xp 00000000 08:23 536923383                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7ffff7dff000-7ffff7e05000 rw-s 00000000 08:23 69091509                   /home/a/test1/mariadb-environs/m9-10.2/dt/tc.log
 
7ffff7e05000-7ffff7e27000 rw-p 00000000 00:00 0 
7ffff7e27000-7ffff7e2c000 rw-s 00000000 00:10 1421240                    /[aio] (deleted)
 
7ffff7e2c000-7ffff7e31000 rw-s 00000000 00:10 1421239                    /[aio] (deleted)
 
7ffff7e31000-7ffff7e36000 rw-s 00000000 00:10 1421238                    /[aio] (deleted)
 
7ffff7e36000-7ffff7e3b000 rw-s 00000000 00:10 1421237                    /[aio] (deleted)
 
7ffff7e3b000-7ffff7e74000 rw-p 00000000 00:00 0 
7ffff7e74000-7ffff7e79000 rw-s 00000000 00:10 1421236                    /[aio] (deleted)
 
7ffff7e79000-7ffff7fd7000 rw-p 00000000 00:00 0 
7ffff7fd8000-7ffff7fd9000 rw-p 00000000 00:00 0 
7ffff7fd9000-7ffff7fdb000 rw-s 00000000 00:10 1421241                    /[aio] (deleted)
 
7ffff7fdb000-7ffff7fe0000 rw-s 00000000 00:10 1421235                    /[aio] (deleted)
 
7ffff7fe0000-7ffff7fe5000 rw-s 00000000 00:10 1421234                    /[aio] (deleted)
 
7ffff7fe5000-7ffff7fea000 rw-s 00000000 00:10 1421233                    /[aio] (deleted)
 
7ffff7fea000-7ffff7fef000 rw-s 00000000 00:10 1421232                    /[aio] (deleted)
 
7ffff7fef000-7ffff7ff4000 rw-s 00000000 00:10 1421231                    /[aio] (deleted)
 
7ffff7ff4000-7ffff7ff5000 rw-s 00000000 00:10 1421230                    /[aio] (deleted)
 
7ffff7ff5000-7ffff7ff8000 rw-p 00000000 00:00 0 
7ffff7ff8000-7ffff7ffa000 r--p 00000000 00:00 0                          [vvar]
 
7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
 
7ffff7ffc000-7ffff7ffd000 r--p 00025000 08:23 536923383                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7ffff7ffd000-7ffff7ffe000 rw-p 00026000 08:23 536923383                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
 
 
Thread 1 "mysqld" received signal SIGABRT, Aborted.
 
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
 
58	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
 
(gdb) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
 
#1  0x00007ffff607c3ea in __GI_abort () at abort.c:89
 
#2  0x00007ffff60be0d0 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff61d3f80 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
 
#3  0x00007ffff60c7c9a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7ffff61d3ff8 "free(): invalid next size (fast)", action=3) at malloc.c:5048
 
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3904
 
#5  0x00007ffff60cbd8c in __GI___libc_free (mem=<optimized out>) at malloc.c:2984
 
#6  0x0000555555a09040 in cleanup_variables (vars=vars@entry=0x5555579d2ca8) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_plugin.cc:3172
 
#7  0x0000555555a0be1b in plugin_thdvar_cleanup (thd=thd@entry=0x5555579d2678) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_plugin.cc:3198
 
#8  0x00005555559d1f28 in THD::free_connection (this=this@entry=0x5555579d2678) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1511
 
#9  0x00005555559d2470 in THD::~THD (this=0x5555579d2678, __in_chrg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1582
 
#10 0x00005555559d24f9 in THD::~THD (this=0x5555579d2678, __in_chrg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1636
 
#11 0x0000555555abc87e in servers_init (dont_read_servers_table=dont_read_servers_table@entry=false) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_servers.cc:175
 
#12 0x00005555559792b7 in init_server_components () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/mysqld.cc:5469
 
#13 0x000055555597d311 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/mysqld.cc:5843
 
#14 0x00007ffff60653f1 in __libc_start_main (main=0x55555595d860 <main(int, char**)>, argc=3, argv=0x7fffffffdec8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8)
 
    at ../csu/libc-start.c:291
 
#15 0x0000555555971c8a in _start ()
 
(gdb) thread apply all bt
 
 
 
Thread 33 (Thread 0x7fffa57fa700 (LWP 305)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555f8ca0e in inline_mysql_cond_timedwait (src_file=0x5555561b25c8 "/home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fffa57f9e10, 
    mutex=0x5555571aabe0 <LOCK_checkpoint>, that=0x5555571aaba0 <COND_checkpoint>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/include/mysql/psi/mysql_thread.h:1186
 
#2  my_service_thread_sleep (control=control@entry=0x5555568b56a0 <checkpoint_control>, sleep_time=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/maria/ma_servicethread.c:115
 
#3  0x0000555555f862d9 in ma_checkpoint_background (arg=0x1e) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/maria/ma_checkpoint.c:708
 
#4  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa57fa700) at pthread_create.c:333
 
#5  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 32 (Thread 0x7fffa5ffb700 (LWP 304)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555cd8ae2 in inline_mysql_cond_wait (src_file=0x55555616f9c8 "/home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/handler/ha_innodb.cc", src_line=344, mutex=0x7fffa5ffae70, that=0x7fffa5ffaea0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/include/mysql/psi/mysql_thread.h:1149
 
#2  thd_destructor_proxy () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/handler/ha_innodb.cc:344
 
#3  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa5ffb700) at pthread_create.c:333
 
#4  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 31 (Thread 0x7fffa67fc700 (LWP 303)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a5240) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a5240, reset_sig_count=1, reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555e832ef in buf_resize_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/buf/buf0buf.cc:3085
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa67fc700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 30 (Thread 0x7fffa6ffd700 (LWP 302)):
 
#0  0x00007ffff6cc4ced in nanosleep () at ../sysdeps/unix/syscall-template.S:84
 
#1  0x0000555555d5a9ec in os_thread_sleep (tm=tm@entry=1000000) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0thread.cc:230
 
#2  0x0000555555e75422 in btr_defragment_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/btr/btr0defragment.cc:771
 
#3  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa6ffd700) at pthread_create.c:333
 
#4  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 29 (Thread 0x7fffa77fe700 (LWP 301)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557687620, abstime=abstime@entry=0x7fffa77fd6c0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557687620, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1, reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555eff25a in fil_crypt_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0crypt.cc:2131
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa77fe700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 28 (Thread 0x7fffa7fff700 (LWP 300)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557687620, abstime=abstime@entry=0x7fffa7ffe6c0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557687620, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1, reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555eff25a in fil_crypt_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0crypt.cc:2131
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffa7fff700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 27 (Thread 0x7fffc0ff9700 (LWP 32767)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557687620, abstime=abstime@entry=0x7fffc0ff86c0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557687620, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1, reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555eff25a in fil_crypt_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0crypt.cc:2131
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc0ff9700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 26 (Thread 0x7fffc17fa700 (LWP 32766)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557687620, abstime=abstime@entry=0x7fffc17f96c0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557687620, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1, reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555eff25a in fil_crypt_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0crypt.cc:2131
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc17fa700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 25 (Thread 0x7fffc1ffb700 (LWP 32765)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a5140) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a5140, reset_sig_count=1, reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
---Type <return> to continue, or q <return> to quit---
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555e8c1e6 in buf_dump_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/buf/buf0dump.cc:815
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc1ffb700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 24 (Thread 0x7fffc27fc700 (LWP 32764)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a4fc0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a4fc0, reset_sig_count=1, reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555debc85 in srv_resume_thread (slot=slot@entry=0x5555568b3968 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:951
 
#5  0x0000555555df145d in srv_worker_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2656
 
#6  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc27fc700) at pthread_create.c:333
 
#7  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 23 (Thread 0x7fffc2ffd700 (LWP 32763)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a4f40) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a4f40, reset_sig_count=1, reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555debc85 in srv_resume_thread (slot=slot@entry=0x5555568b3930 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:951
 
#5  0x0000555555df145d in srv_worker_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2656
 
#6  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc2ffd700) at pthread_create.c:333
 
#7  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 22 (Thread 0x7fffc37fe700 (LWP 32762)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a4ec0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a4ec0, reset_sig_count=1, reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555debc85 in srv_resume_thread (slot=slot@entry=0x5555568b38f8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:951
 
#5  0x0000555555df145d in srv_worker_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2656
 
#6  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc37fe700) at pthread_create.c:333
 
#7  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 21 (Thread 0x7fffc3fff700 (LWP 32761)):
 
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#1  0x0000555555d5a386 in os_event::wait (this=0x5555572a4e40) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:166
 
#2  os_event::wait_low (this=0x5555572a4e40, reset_sig_count=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:336
 
#3  0x0000555555d5a825 in os_event_wait_low (event=<optimized out>, reset_sig_count=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:535
 
#4  0x0000555555debc85 in srv_resume_thread (slot=slot@entry=0x5555568b38c0 <srv_sys+192>, sig_count=<optimized out>, wait=true, timeout_usec=<optimized out>)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:951
 
#5  0x0000555555df1ad9 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2809
 
#6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2896
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffc3fff700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 20 (Thread 0x7fffd0fbd700 (LWP 32760)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557686cc0, abstime=abstime@entry=0x7fffd0fbcc20) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557686cc0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555e43a47 in ib_wqueue_timedwait (wq=wq@entry=0x55555764eb00, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/ut/ut0wqueue.cc:163
 
#5  0x0000555555f3125e in fts_optimize_thread (arg=0x55555764eb00) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fts/fts0opt.cc:3041
 
#6  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd0fbd700) at pthread_create.c:333
 
#7  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 19 (Thread 0x7fffd17be700 (LWP 32759)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557644c20, abstime=abstime@entry=0x7fffd17bdda0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x555557644c20, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555ee13aa in dict_stats_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/dict/dict0stats_bg.cc:451
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd17be700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 18 (Thread 0x7fffd1fbf700 (LWP 32758)):
 
#0  0x00007ffff6cc4ced in nanosleep () at ../sysdeps/unix/syscall-template.S:84
 
#1  0x0000555555d5a9ec in os_thread_sleep (tm=tm@entry=1000000) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0thread.cc:230
 
#2  0x0000555555df0e9c in srv_master_sleep () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2467
 
#3  srv_master_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:2512
 
#4  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd1fbf700) at pthread_create.c:333
 
#5  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 17 (Thread 0x7fffd27c0700 (LWP 32757)):
 
---Type <return> to continue, or q <return> to quit---
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x5555572a50c0, abstime=abstime@entry=0x7fffd27bfe90) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x5555572a50c0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555df0864 in srv_monitor_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:1753
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd27c0700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 16 (Thread 0x7fffd37c2700 (LWP 32756)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x5555572a5040, abstime=abstime@entry=0x7fffd37c1c60) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x5555572a5040, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555defa66 in srv_error_monitor_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0srv.cc:1978
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd37c2700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 15 (Thread 0x7fffd2fc1700 (LWP 32755)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x555557641440, abstime=abstime@entry=0x7fffd2fc0d80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=this@entry=0x555557641440, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=event@entry=0x555557641440, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555d2b9d8 in lock_wait_timeout_thread () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/lock/lock0wait.cc:530
 
#5  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd2fc1700) at pthread_create.c:333
 
#6  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 13 (Thread 0x7fffd41c8700 (LWP 32753)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555555d5a251 in os_event::timed_wait (this=this@entry=0x5555572a51c0, abstime=abstime@entry=0x7fffd41c7b10) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:286
 
#2  0x0000555555d5a4b1 in os_event::wait_time_low (this=0x5555572a51c0, time_in_usec=<optimized out>, reset_sig_count=reset_sig_count@entry=2) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:413
 
#3  0x0000555555d5a815 in os_event_wait_time_low (event=<optimized out>, time_in_usec=<optimized out>, reset_sig_count=reset_sig_count@entry=2) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0event.cc:518
 
#4  0x0000555555e95824 in pc_sleep_if_needed (sig_count=2, next_loop_time=1498134561586) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/buf/buf0flu.cc:2722
 
#5  buf_flush_page_cleaner_coordinator (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/buf/buf0flu.cc:3223
 
#6  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd41c8700) at pthread_create.c:333
 
#7  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 12 (Thread 0x7fffd49c9700 (LWP 32752)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd49c8c80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd49c8c80, m1=m1@entry=0x7fffd49c8d20, m2=m2@entry=0x7fffd49c8d28, request=request@entry=0x7fffd49c8d30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd49c8d30, m2=0x7fffd49c8d28, m1=0x7fffd49c8d20, global_segment=9) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7fffd49c8d20, m2=m2@entry=0x7fffd49c8d28, request=request@entry=0x7fffd49c8d30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=9) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd49c9700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 11 (Thread 0x7fffd51ca700 (LWP 32751)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd51c9c80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd51c9c80, m1=m1@entry=0x7fffd51c9d20, m2=m2@entry=0x7fffd51c9d28, request=request@entry=0x7fffd51c9d30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd51c9d30, m2=0x7fffd51c9d28, m1=0x7fffd51c9d20, global_segment=8) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7fffd51c9d20, m2=m2@entry=0x7fffd51c9d28, request=request@entry=0x7fffd51c9d30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=8) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd51ca700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 10 (Thread 0x7fffd59cb700 (LWP 32750)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd59cac80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd59cac80, m1=m1@entry=0x7fffd59cad20, m2=m2@entry=0x7fffd59cad28, request=request@entry=0x7fffd59cad30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd59cad30, m2=0x7fffd59cad28, m1=0x7fffd59cad20, global_segment=7) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7fffd59cad20, m2=m2@entry=0x7fffd59cad28, request=request@entry=0x7fffd59cad30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=7) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd59cb700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 9 (Thread 0x7fffd61cc700 (LWP 32749)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd61cbc80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd61cbc80, m1=m1@entry=0x7fffd61cbd20, m2=m2@entry=0x7fffd61cbd28, request=request@entry=0x7fffd61cbd30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd61cbd30, m2=0x7fffd61cbd28, m1=0x7fffd61cbd20, global_segment=6) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7fffd61cbd20, m2=m2@entry=0x7fffd61cbd28, request=request@entry=0x7fffd61cbd30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=6) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
---Type <return> to continue, or q <return> to quit---
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd61cc700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 8 (Thread 0x7fffd69cd700 (LWP 32748)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd69ccc80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd69ccc80, m1=m1@entry=0x7fffd69ccd20, m2=m2@entry=0x7fffd69ccd28, request=request@entry=0x7fffd69ccd30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd69ccd30, m2=0x7fffd69ccd28, m1=0x7fffd69ccd20, global_segment=5) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fffd69ccd20, m2=m2@entry=0x7fffd69ccd28, request=request@entry=0x7fffd69ccd30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=5) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd69cd700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 7 (Thread 0x7fffd71ce700 (LWP 32747)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd71cdc80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd71cdc80, m1=m1@entry=0x7fffd71cdd20, m2=m2@entry=0x7fffd71cdd28, request=request@entry=0x7fffd71cdd30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd71cdd30, m2=0x7fffd71cdd28, m1=0x7fffd71cdd20, global_segment=4) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fffd71cdd20, m2=m2@entry=0x7fffd71cdd28, request=request@entry=0x7fffd71cdd30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=4) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd71ce700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 6 (Thread 0x7fffd79cf700 (LWP 32746)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd79cec80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd79cec80, m1=m1@entry=0x7fffd79ced20, m2=m2@entry=0x7fffd79ced28, request=request@entry=0x7fffd79ced30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd79ced30, m2=0x7fffd79ced28, m1=0x7fffd79ced20, global_segment=3) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fffd79ced20, m2=m2@entry=0x7fffd79ced28, request=request@entry=0x7fffd79ced30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=3) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd79cf700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 5 (Thread 0x7fffd81d0700 (LWP 32745)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd81cfc80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd81cfc80, m1=m1@entry=0x7fffd81cfd20, m2=m2@entry=0x7fffd81cfd28, request=request@entry=0x7fffd81cfd30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd81cfd30, m2=0x7fffd81cfd28, m1=0x7fffd81cfd20, global_segment=2) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fffd81cfd20, m2=m2@entry=0x7fffd81cfd28, request=request@entry=0x7fffd81cfd30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=2) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd81d0700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 4 (Thread 0x7fffd89d1700 (LWP 32744)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd89d0c80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd89d0c80, m1=m1@entry=0x7fffd89d0d20, m2=m2@entry=0x7fffd89d0d28, request=request@entry=0x7fffd89d0d30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd89d0d30, m2=0x7fffd89d0d28, m1=0x7fffd89d0d20, global_segment=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fffd89d0d20, m2=m2@entry=0x7fffd89d0d28, request=request@entry=0x7fffd89d0d30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=1) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd89d1700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 3 (Thread 0x7fffd91d2700 (LWP 32743)):
 
#0  0x00007ffff7bd564a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
 
#1  0x0000555555d50729 in LinuxAIOHandler::collect (this=this@entry=0x7fffd91d1c80) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:1886
 
#2  0x0000555555d55f48 in LinuxAIOHandler::poll (this=this@entry=0x7fffd91d1c80, m1=m1@entry=0x7fffd91d1d20, m2=m2@entry=0x7fffd91d1d28, request=request@entry=0x7fffd91d1d30)
 
    at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2031
 
#3  0x0000555555d5888b in os_aio_linux_handler (request=0x7fffd91d1d30, m2=0x7fffd91d1d28, m1=0x7fffd91d1d20, global_segment=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:2085
 
#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fffd91d1d20, m2=m2@entry=0x7fffd91d1d28, request=request@entry=0x7fffd91d1d30) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/os/os0file.cc:5683
 
#5  0x0000555555eeb486 in fil_aio_wait (segment=segment@entry=0) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/fil/fil0fil.cc:5436
 
#6  0x0000555555df2a58 in io_handler_thread (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/storage/innobase/srv/srv0start.cc:339
 
#7  0x00007ffff6cbb6ca in start_thread (arg=0x7fffd91d2700) at pthread_create.c:333
 
#8  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
 
 
Thread 2 (Thread 0x7ffff5804700 (LWP 32742)):
 
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
 
#1  0x0000555556078ce7 in inline_mysql_cond_timedwait (that=0x5555571ba660 <COND_timer>, mutex=0x5555571ba6a0 <LOCK_timer>, src_file=0x5555561c6ca8 "/home/a/test1/mariadb-environs/_depot/m-branch/10.2/mysys/thr_timer.c", src_line=292, 
    abstime=0x7ffff5803e90) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/include/mysql/psi/mysql_thread.h:1186
 
#2  timer_handler (arg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/mysys/thr_timer.c:292
 
#3  0x00007ffff6cbb6ca in start_thread (arg=0x7ffff5804700) at pthread_create.c:333
 
#4  0x00007ffff614dcaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
---Type <return> to continue, or q <return> to quit---
 
 
 
Thread 1 (Thread 0x7ffff7fd0740 (LWP 32738)):
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
 
#1  0x00007ffff607c3ea in __GI_abort () at abort.c:89
 
#2  0x00007ffff60be0d0 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff61d3f80 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
 
#3  0x00007ffff60c7c9a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7ffff61d3ff8 "free(): invalid next size (fast)", action=3) at malloc.c:5048
 
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3904
 
#5  0x00007ffff60cbd8c in __GI___libc_free (mem=<optimized out>) at malloc.c:2984
 
#6  0x0000555555a09040 in cleanup_variables (vars=vars@entry=0x5555579d2ca8) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_plugin.cc:3172
 
#7  0x0000555555a0be1b in plugin_thdvar_cleanup (thd=thd@entry=0x5555579d2678) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_plugin.cc:3198
 
#8  0x00005555559d1f28 in THD::free_connection (this=this@entry=0x5555579d2678) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1511
 
#9  0x00005555559d2470 in THD::~THD (this=0x5555579d2678, __in_chrg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1582
 
#10 0x00005555559d24f9 in THD::~THD (this=0x5555579d2678, __in_chrg=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_class.cc:1636
 
#11 0x0000555555abc87e in servers_init (dont_read_servers_table=dont_read_servers_table@entry=false) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/sql_servers.cc:175
 
#12 0x00005555559792b7 in init_server_components () at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/mysqld.cc:5469
 
#13 0x000055555597d311 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/a/test1/mariadb-environs/_depot/m-branch/10.2/sql/mysqld.cc:5843
 
#14 0x00007ffff60653f1 in __libc_start_main (main=0x55555595d860 <main(int, char**)>, argc=3, argv=0x7fffffffdec8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8)
 
    at ../csu/libc-start.c:291
 
#15 0x0000555555971c8a in _start ()


..*** Error in `/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld': malloc(): memory corruption: 0x0000560581e88410 ***
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x790cb)[0x7f5db1c330cb]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x84b2e)[0x7f5db1c3eb2e]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f5db1c409e4]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(my_malloc+0x35)[0x56057fe6a0f5]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(init_dynamic_array2+0x79)[0x56057fe51c09]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_ZN9hash_filo5clearEb+0xb6)[0x56057f7aeb96]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z10acl_reloadP3THD+0xb5)[0x56057f79bb65]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z8acl_initb+0x136)[0x56057f79dbb6]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z11mysqld_mainiPPc+0x2246)[0x56057f776946]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f5db1bda3f1]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_start+0x2a)[0x56057f76ac8a]


..*** Error in `/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld': malloc(): memory corruption: 0x000055d846ea3d90 ***
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x790cb)[0x7f205da260cb]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x84b2e)[0x7f205da31b2e]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f205da339e4]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(my_malloc+0x35)[0x55d8448870f5]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(init_io_cache+0x1e0)[0x55d844875940]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(+0xa9c5a3)[0x55d8448065a3]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(+0xa81b6a)[0x55d8447ebb6a]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z16init_read_recordP11READ_RECORDP3THDP5TABLEP10SQL_SELECTP9SORT_INFOibb+0x69e)[0x55d844489ece]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(+0x5670d6)[0x55d8442d10d6]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z14servers_reloadP3THD+0x1bf)[0x55d8442d271f]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z12servers_initb+0x106)[0x55d8442d2866]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(+0x4252b7)[0x55d84418f2b7]
 
/home/a/test1/mariadb-environs/m9-10.2/build/sql//mysqld(_Z11mysqld_mainiPPc+0x1c11)[0x55d844193311]

Comment by Dave Juntgen [ 2017-06-22 ]

Original attempt (mysql_install_db fails with AWS KMS + server audit enabled)

  1. Yes - /etc/my.cnf.d/enable_encryption.preset is present
  2. Even with encrypt-tmp-files commented out, mysql_install_db fails with corruption within InnoDB
  3. This appears to be realted, but a different problem than server_audit plugin load

[root@localhost mysql]# mysql_install_db --user=mysql
 
Installing MariaDB/MySQL system tables in '/var/lib/mysql/data' ...
 
2017-06-22 12:58:25 140433490081856 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/share/
 
2017-06-22 12:58:25 140433490081856 [Warning] Use --lc-messages-dir without language specific part instead.
 
2017-06-22 12:58:27 140433490081856 [ERROR] mysqld: AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : MissingAuthenticationTokenException - Missing Authentication Token
 
2017-06-22 12:58:27 140433490081856 [ERROR] InnoDB: cannot enable encryption, encryption plugin is not available
 
2017-06-22 12:58:27 140433490081856 [ERROR] Plugin 'InnoDB' init function returned error.
 
2017-06-22 12:58:27 140433490081856 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
170622 12:58:27 server_audit: server_audit_excl_users set to 'ehs_app'.
 
170622 12:58:27 server_audit: MariaDB Audit Plugin version 1.4.1-debug STARTED.
 
170622 12:58:27 server_audit: logging started to the file /var/lib/mysql/log-audit/audit.log.
 
2017-06-22 12:58:27 140433490081856 [ERROR] Unknown/unsupported storage engine: InnoDB
 
2017-06-22 12:58:27 140433490081856 [ERROR] Aborting
 
 
 
170622 12:58:27 server_audit: STOPPED

Attempt to execute mysql_install_db without server audit plugin

  • Unfortunately - even without the server audit plugin enabled, mysql_install_db fails to execute when the AWS KMS plugin enabled.

Attempt to execute mysql_install_db without server audit plugin and without innodb table and log encryption

Config:

[root@localhost mysql]# cat /etc/my.cnf.d/enable_encryption.preset
 
[mariadb]
 
aria-encrypt-tables
 
encrypt-binlog
 
encrypt-tmp-disk-tables
 
encrypt-tmp-files
 
#loose-innodb-encrypt-log
 
#loose-innodb-encrypt-tables

  • Partial progress - mysql_install_db creates innodb data and log files but mysql data is missing.

Summary

It appears the only way to have mysql_install_db install successfully is not to use AWS KMS plugin on install, and load it after and then encrypt. its worth mentioning that the server_audit plugin can be loaded dynamically after the server boots, but fails on restart of the server.

Comment by Andrii Nikitin (Inactive) [ 2017-06-22 ]

Note that now you had error '2017-06-22 12:58:27 140433490081856 [ERROR] mysqld: AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : MissingAuthenticationTokenException - Missing Authentication Token' which is related to aws credentials.
I was reliably succeeding with mysql_install_db with both plugins every time when encrypt-tmp-files was commented out.

Comment by Dave Juntgen [ 2017-06-22 ]

Confirmed - successful mysql_install_db, this was a permission issue to .aws/credentials, need to run as mysql user.

[root@localhost data]# sudo -u mysql mysql_install_db --user=mysql --datadir=/var/lib/mysql/data

Comment by Dave Juntgen [ 2017-06-22 ]

Question - is there a notable work around?

For example:

  1. Disable temp table encryption and then reenabled after boot using --init-file
Comment by Vladislav Vaintroub [ 2017-06-22 ]

you can try to disable audit, and then reenable after boot if that works.

Generated at Thu Feb 08 08:02:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.