Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.21, 10.2.4, 10.3.0
-
ovz container, 2.6.32-042stab120.6, CentOS release 6.7 (Final)
Description
Still having issues with startup with 10.2.13. MDEV-13869 did not fix my issue.
FYI, my 30 +/- minute start time is for a DB roughly 290GB and roughly 991K tables
2018-02-16 7:35:02 140326823504064 [Note] InnoDB: Highest supported file format is Barracuda.
|
2018-02-16 8:02:01 140326823504064 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
During the startup, gdb shows this:
Attaching to program: /usr/sbin/mysqld, process 26586
|
[New LWP 26587]
|
[New LWP 26588]
|
[New LWP 26598]
|
[New LWP 26599]
|
[New LWP 26600]
|
[New LWP 26601]
|
[New LWP 26602]
|
[New LWP 26603]
|
[New LWP 26604]
|
[New LWP 26605]
|
[New LWP 26606]
|
[New LWP 26607]
|
[New LWP 26608]
|
[New LWP 26609]
|
[New LWP 26610]
|
[New LWP 26611]
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
|
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
|
|
|
Thread 17 (Thread 0x7f08c2805700 (LWP 26611)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c2805700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 16 (Thread 0x7f08c3006700 (LWP 26610)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3006700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 15 (Thread 0x7f08c3807700 (LWP 26609)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3807700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 14 (Thread 0x7f08c4008700 (LWP 26608)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5587dbfab2e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6f42a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3179
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c4008700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 13 (Thread 0x7f08c4809700 (LWP 26607)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c4808ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c4808ae0, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c4808ba0, m2=0x7f08c4808b98, m1=0x7f08c4808b90, global_segment=9)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c4809700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 12 (Thread 0x7f08c500a700 (LWP 26606)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c5009ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c5009ae0, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c5009ba0, m2=0x7f08c5009b98, m1=0x7f08c5009b90, global_segment=8)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c500a700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 11 (Thread 0x7f08c580b700 (LWP 26605)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c580aae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c580aae0, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c580aba0, m2=0x7f08c580ab98, m1=0x7f08c580ab90, global_segment=7)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c580b700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 10 (Thread 0x7f08c600c700 (LWP 26604)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c600bae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c600bae0, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c600bba0, m2=0x7f08c600bb98, m1=0x7f08c600bb90, global_segment=6)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c600c700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 9 (Thread 0x7f08c680d700 (LWP 26603)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c680cae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c680cae0, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c680cba0, m2=0x7f08c680cb98, m1=0x7f08c680cb90, global_segment=5)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c680d700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 8 (Thread 0x7f08c700e700 (LWP 26602)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c700dae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c700dae0, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c700dba0, m2=0x7f08c700db98, m1=0x7f08c700db90, global_segment=4)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c700e700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 7 (Thread 0x7f08c780f700 (LWP 26601)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c780eae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c780eae0, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c780eba0, m2=0x7f08c780eb98, m1=0x7f08c780eb90, global_segment=3)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c780f700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 6 (Thread 0x7f08c8010700 (LWP 26600)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c800fae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c800fae0, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c800fba0, m2=0x7f08c800fb98, m1=0x7f08c800fb90, global_segment=2)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8010700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 5 (Thread 0x7f08c8811700 (LWP 26599)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c8810ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c8810ae0, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c8810ba0, m2=0x7f08c8810b98, m1=0x7f08c8810b90, global_segment=1)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8811700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 4 (Thread 0x7f08c9012700 (LWP 26598)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c9011ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c9011ae0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c9011ba0, m2=0x7f08c9011b98, m1=0x7f08c9011b90, global_segment=0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c9012700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 3 (Thread 0x7f1625d93700 (LWP 26588)):
|
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
|
#1 0x00005587d984e68f in inline_mysql_cond_timedwait (src_file=0x5587d9cf2158 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f1625d92c80,
|
mutex=0x5587dacb0860 <LOCK_checkpoint>, that=0x5587dacb0820 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186
|
#2 my_service_thread_sleep (control=control@entry=0x5587da3f46e0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c:115
|
#3 0x00005587d9846157 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_checkpoint.c:709
|
#4 0x00007f1636ce26ba in start_thread (arg=0x7f1625d93700) at pthread_create.c:333
|
#5 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 2 (Thread 0x7f1634a7e700 (LWP 26587)):
|
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
|
#1 0x00005587d9bd5762 in inline_mysql_cond_timedwait (that=0x5587dad0af60 <COND_timer>, mutex=0x5587dad0afa0 <LOCK_timer>,
|
src_file=0x5587d9d3a168 "/home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c", src_line=292, abstime=0x7f1634a7dd10)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186
|
#2 timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c:292
|
#3 0x00007f1636ce26ba in start_thread (arg=0x7f1634a7e700) at pthread_create.c:333
|
#4 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 1 (Thread 0x7f16383ac8c0 (LWP 26586)):
|
#0 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
|
#1 0x00005587d992bca5 in pread64 (__offset=0, __nbytes=16384, __buf=0x7f0ad43e8000, __fd=1817) at /usr/include/x86_64-linux-gnu/bits/unistd.h:99
|
#2 SyncFileIO::execute (request=..., this=<synthetic pointer>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1573
|
#3 os_file_io (in_type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, n=n@entry=16384, offset=offset@entry=0, err=err@entry=0x7ffd2786b44c)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:4877
|
#4 0x00005587d992c850 in os_file_pread (err=0x7ffd2786b44c, offset=0, n=16384, buf=0x7f0ad43e8000, file=1817, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5046
|
#5 os_file_read_page (type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, offset=offset@entry=0, n=n@entry=16384, o=o@entry=0x0, exit_on_err=true)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5083
|
#6 0x00005587d992e9f1 in os_file_read_func (n=16384, offset=0, buf=0x7f0ad43e8000, file=<optimized out>, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5493
|
#7 os_aio_func (type=..., mode=mode@entry=24, name=name@entry=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", file=..., buf=buf@entry=0x7f0ad43e8000, offset=0, n=16384, read_only=false,
|
m1=0x5588327ed1d0, m2=0x7f0acd19e5a0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:6683
|
#8 0x00005587d9acba01 in pfs_os_aio_func (src_file=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", src_line=5290, m2=0x7f0acd19e5a0, m1=0x5588327ed1d0,
|
read_only=<optimized out>, n=16384, offset=0, buf=0x7f0ad43e8000, file=..., name=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", mode=24, type=...)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/include/os0file.ic:256
|
#9 fil_io (type=..., sync=sync@entry=true, page_id=..., page_size=..., byte_offset=byte_offset@entry=0, len=16384, buf=0x7f0ad43e8000, message=0x7f0acd19e5a0, ignore_missing_space=false)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5285
|
#10 0x00005587d9a7e5fb in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffd2786b9e0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:192
|
#11 buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:460
|
#12 0x00005587d9a56e08 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10,
|
file=file@entry=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", line=4639, mtr=0x7ffd2786c050, err=0x0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0buf.cc:4335
|
#13 0x00005587d9ac97d4 in fsp_flags_try_adjust (space_id=space_id@entry=550663, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4637
|
#14 0x00005587d9aca0f4 in fil_ibd_open (validate=<optimized out>, validate@entry=false, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=550663, flags=<optimized out>,
|
space_name=<optimized out>, path_in=0x5588327e83f0 "./ddx_lab_802532/quickbooks_references.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4222
|
#15 0x00005587d9aa63bb in dict_check_sys_tables (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1480
|
#16 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1535
|
#17 0x00005587d99cbdc1 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:2544
|
#18 0x00005587d98afff1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/handler/ha_innodb.cc:4367
|
#19 0x00005587d966e084 in ha_initialize_handlerton (plugin=0x5587dbe68a80) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/handler.cc:520
|
#20 0x00005587d94e1b05 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffd27877640, plugin=plugin@entry=0x5587dbe68a80, argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=argv@entry=0x5587dbe22e40,
|
options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1416
|
#21 0x00005587d94e24da in plugin_init (argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=0x5587dbe22e40, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1697
|
#22 0x00005587d943c560 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5266
|
#23 0x00005587d9441437 in mysqld_main (argc=63, argv=0x5587dbe22e40) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5858
|
#24 0x00007f16362a6830 in __libc_start_main (main=0x5587d941fe00 <main(int, char**)>, argc=6, argv=0x7ffd27878238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
|
stack_end=0x7ffd27878228) at ../csu/libc-start.c:291
|
#25 0x00005587d9434779 in _start ()
|
Detaching from program: /usr/sbin/mysqld, process 26586
|
|
Of possible note, 'apt upgrade' from 10.2.12 to 10.2.13 reported that it found a mysql 5.6 data in /var/lib/mysql (been 10.2.x for a month)?
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- is caused by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
This DB has about 30GB and a lot of tables in many databases:
{noformat} [mysql]# find ./ -name "*.ibd" | wc -l 90219 {noformat} Start takes more than 5 minutes with heavy IO load made by mysql process: {noformat} 2017-09-22 10:07:27 140632756656096 [Note] InnoDB: Highest supported file format is Barracuda. 2017-09-22 10:13:05 140632756656096 [Note] InnoDB: 128 out of 128 rollback segments are active. {noformat} During the startup, strace shows this: {noformat} cat /root/strace | grep open [pid 29937] open("./deploy_test_b2c15/data_sources.ibd", O_RDONLY) = 1363 [pid 29937] open("./deploy_test_b2c15/data_sources.ibd", O_RDWR) = 1363 [pid 29937] open("./deploy_test_b2c15/data_sources_backup.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_backup.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_backup.ibd", O_RDONLY) = 1364 [pid 29937] open("./deploy_test_b2c15/data_sources_backup.ibd", O_RDWR) = 1364 [pid 29937] open("./deploy_test_b2c15/data_sources_brands.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_brands.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_brands.ibd", O_RDONLY) = 1365 [pid 29937] open("./deploy_test_b2c15/data_sources_brands.ibd", O_RDWR) = 1365 [pid 29937] open("./deploy_test_b2c15/data_sources_categories.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_categories.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_categories.ibd", O_RDONLY) = 1366 [pid 29937] open("./deploy_test_b2c15/data_sources_categories.ibd", O_RDWR) = 1366 [pid 29937] open("./deploy_test_b2c15/data_sources_categories_brands.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_categories_brands.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_categories_brands.ibd", O_RDONLY) = 1367 [pid 29937] open("./deploy_test_b2c15/data_sources_categories_brands.ibd", O_RDWR) = 1367 [pid 29937] open("./deploy_test_b2c15/data_sources_img_queue.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_img_queue.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_img_queue.ibd", O_RDONLY) = 1368 [pid 29937] open("./deploy_test_b2c15/data_sources_img_queue.ibd", O_RDWR) = 1368 [pid 29937] open("./deploy_test_b2c15/data_sources_methods.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/data_sources_methods.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/data_sources_methods.ibd", O_RDONLY) = 1369 [pid 29937] open("./deploy_test_b2c15/data_sources_methods.ibd", O_RDWR) = 1369 [pid 29937] open("./deploy_test_b2c15/default_delivery_address.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./deploy_test_b2c15/default_delivery_address.ibd", O_RDONLY) = 4108 [pid 29937] open("./deploy_test_b2c15/default_delivery_address.ibd", O_RDONLY) = 1370 [pid 29937] open("./deploy_test_b2c15/default_delivery_address.ibd", O_RDWR) = 1370?? {noformat} Full strace: {noformat} [pid 29937] open("./demo_b2c15/novinky4_kategorie_priv_write.ibd", O_RDONLY) = 3810 [pid 29937] lseek(3810, 0, SEEK_CUR) = 0 [pid 29937] lseek(3810, 0, SEEK_END) = 131072 [pid 29937] lseek(3810, 0, SEEK_SET) = 0 [pid 29937] pread(3810, "\343t\2[\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\31;\355\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] close(3810) = 0 [pid 29937] open("./demo_b2c15/novinky4_kategorie_priv_write.ibd", O_RDWR) = 3810 [pid 29937] fcntl(3810, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0 [pid 29937] pread(3810, "\343t\2[\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\31;\355\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] open("./demo_b2c15/novinky4_soubory.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./demo_b2c15/novinky4_soubory.ibd", O_RDONLY) = 4108 [pid 29937] fstat(4108, {st_mode=S_IFREG|0660, st_size=114688, ...}) = 0 [pid 29937] pread(4108, "W\352G\231\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\31`/\0\10\0\0\0\0\0\0"..., 65536, 0) = 65536 [pid 29937] close(4108) = 0 [pid 29937] close(3811) = 0 [pid 29937] open("./demo_b2c15/novinky4_soubory.ibd", O_RDONLY) = 3811 [pid 29937] lseek(3811, 0, SEEK_CUR) = 0 [pid 29937] lseek(3811, 0, SEEK_END) = 114688 [pid 29937] lseek(3811, 0, SEEK_SET) = 0 [pid 29937] pread(3811, "W\352G\231\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\31`/\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] close(3811) = 0 [pid 29937] open("./demo_b2c15/novinky4_soubory.ibd", O_RDWR) = 3811 [pid 29937] fcntl(3811, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0 [pid 29937] pread(3811, "W\352G\231\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\31`/\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] open("./demo_b2c15/novinky5.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./demo_b2c15/novinky5.ibd", O_RDONLY) = 4108 [pid 29937] fstat(4108, {st_mode=S_IFREG|0660, st_size=212992, ...}) = 0 [pid 29937] pread(4108, "\6\242(\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\32\10\352\0\10\0\0\0\0\0\0"..., 65536, 0) = 65536 [pid 29937] close(4108) = 0 [pid 29937] close(3812) = 0 [pid 29937] open("./demo_b2c15/novinky5.ibd", O_RDONLY) = 3812 [pid 29937] lseek(3812, 0, SEEK_CUR) = 0 [pid 29937] lseek(3812, 0, SEEK_END) = 212992 [pid 29937] lseek(3812, 0, SEEK_SET) = 0 [pid 29937] pread(3812, "\6\242(\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\32\10\352\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] close(3812) = 0 [pid 29937] open("./demo_b2c15/novinky5.ibd", O_RDWR) = 3812 [pid 29937] fcntl(3812, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0 [pid 29937] pread(3812, "\6\242(\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\32\10\352\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384 [pid 29937] open("./demo_b2c15/novinky5_formulare.isl", O_RDWR) = -1 ENOENT (No such file or directory) [pid 29937] open("./demo_b2c15/novinky5_formulare.ibd", O_RDONLY) = 4108 [pid 29937] fstat(4108, {st_mode=S_IFREG|0660, st_size=114688, ...}) = 0 [pid 29937] pread(4108, "\253\270\202\216\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\351\32\323)\0\10\0\0\0\0\0\0"..., 65536, 0) = 65536 [pid 29937] close(4108) = 0 [pid 29937] close(3813) = 0 {noformat} I found the very same problem (https://jira.mariadb.org/browse/MDEV-12610) closed as solved in 10.2.7. This is 10.2.8 and looks like problem appears again. This db was 10.0.xx upgraded to 10.1.xx and finally to 10.2.8. Can make this upgrade this issue? |
Still having issues with startup with 10.2.13. FYI, my 30 +/- minute start time is for a DB roughly 290GB and roughly 991K tables {noformat} 2018-02-16 7:35:02 140326823504064 [Note] InnoDB: Highest supported file format is Barracuda. 2018-02-16 8:02:01 140326823504064 [Note] InnoDB: 128 out of 128 rollback segments are active. {noformat} During the startup, gdb shows this: {noformat} Attaching to program: /usr/sbin/mysqld, process 26586 [New LWP 26587] [New LWP 26588] [New LWP 26598] [New LWP 26599] [New LWP 26600] [New LWP 26601] [New LWP 26602] [New LWP 26603] [New LWP 26604] [New LWP 26605] [New LWP 26606] [New LWP 26607] [New LWP 26608] [New LWP 26609] [New LWP 26610] [New LWP 26611] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. Thread 17 (Thread 0x7f08c2805700 (LWP 26611)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c2805700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 16 (Thread 0x7f08c3006700 (LWP 26610)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3006700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 15 (Thread 0x7f08c3807700 (LWP 26609)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3807700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 14 (Thread 0x7f08c4008700 (LWP 26608)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5587dbfab2e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6f42a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3179 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c4008700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 13 (Thread 0x7f08c4809700 (LWP 26607)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c4808ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c4808ae0, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c4808ba0, m2=0x7f08c4808b98, m1=0x7f08c4808b90, global_segment=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c4809700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 12 (Thread 0x7f08c500a700 (LWP 26606)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c5009ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c5009ae0, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c5009ba0, m2=0x7f08c5009b98, m1=0x7f08c5009b90, global_segment=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c500a700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 11 (Thread 0x7f08c580b700 (LWP 26605)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c580aae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c580aae0, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c580aba0, m2=0x7f08c580ab98, m1=0x7f08c580ab90, global_segment=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c580b700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 10 (Thread 0x7f08c600c700 (LWP 26604)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c600bae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c600bae0, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c600bba0, m2=0x7f08c600bb98, m1=0x7f08c600bb90, global_segment=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c600c700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 9 (Thread 0x7f08c680d700 (LWP 26603)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c680cae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c680cae0, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c680cba0, m2=0x7f08c680cb98, m1=0x7f08c680cb90, global_segment=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c680d700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 8 (Thread 0x7f08c700e700 (LWP 26602)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c700dae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c700dae0, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c700dba0, m2=0x7f08c700db98, m1=0x7f08c700db90, global_segment=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c700e700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 7 (Thread 0x7f08c780f700 (LWP 26601)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c780eae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c780eae0, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c780eba0, m2=0x7f08c780eb98, m1=0x7f08c780eb90, global_segment=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c780f700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 6 (Thread 0x7f08c8010700 (LWP 26600)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c800fae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c800fae0, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c800fba0, m2=0x7f08c800fb98, m1=0x7f08c800fb90, global_segment=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8010700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 5 (Thread 0x7f08c8811700 (LWP 26599)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c8810ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c8810ae0, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c8810ba0, m2=0x7f08c8810b98, m1=0x7f08c8810b90, global_segment=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8811700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 4 (Thread 0x7f08c9012700 (LWP 26598)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c9011ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c9011ae0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c9011ba0, m2=0x7f08c9011b98, m1=0x7f08c9011b90, global_segment=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c9012700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 3 (Thread 0x7f1625d93700 (LWP 26588)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d984e68f in inline_mysql_cond_timedwait (src_file=0x5587d9cf2158 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f1625d92c80, mutex=0x5587dacb0860 <LOCK_checkpoint>, that=0x5587dacb0820 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 my_service_thread_sleep (control=control@entry=0x5587da3f46e0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c:115 #3 0x00005587d9846157 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_checkpoint.c:709 #4 0x00007f1636ce26ba in start_thread (arg=0x7f1625d93700) at pthread_create.c:333 #5 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7f1634a7e700 (LWP 26587)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d9bd5762 in inline_mysql_cond_timedwait (that=0x5587dad0af60 <COND_timer>, mutex=0x5587dad0afa0 <LOCK_timer>, src_file=0x5587d9d3a168 "/home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c", src_line=292, abstime=0x7f1634a7dd10) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c:292 #3 0x00007f1636ce26ba in start_thread (arg=0x7f1634a7e700) at pthread_create.c:333 #4 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7f16383ac8c0 (LWP 26586)): #0 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 #1 0x00005587d992bca5 in pread64 (__offset=0, __nbytes=16384, __buf=0x7f0ad43e8000, __fd=1817) at /usr/include/x86_64-linux-gnu/bits/unistd.h:99 #2 SyncFileIO::execute (request=..., this=<synthetic pointer>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1573 #3 os_file_io (in_type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, n=n@entry=16384, offset=offset@entry=0, err=err@entry=0x7ffd2786b44c) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:4877 #4 0x00005587d992c850 in os_file_pread (err=0x7ffd2786b44c, offset=0, n=16384, buf=0x7f0ad43e8000, file=1817, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5046 #5 os_file_read_page (type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, offset=offset@entry=0, n=n@entry=16384, o=o@entry=0x0, exit_on_err=true) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5083 #6 0x00005587d992e9f1 in os_file_read_func (n=16384, offset=0, buf=0x7f0ad43e8000, file=<optimized out>, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5493 #7 os_aio_func (type=..., mode=mode@entry=24, name=name@entry=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", file=..., buf=buf@entry=0x7f0ad43e8000, offset=0, n=16384, read_only=false, m1=0x5588327ed1d0, m2=0x7f0acd19e5a0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:6683 #8 0x00005587d9acba01 in pfs_os_aio_func (src_file=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", src_line=5290, m2=0x7f0acd19e5a0, m1=0x5588327ed1d0, read_only=<optimized out>, n=16384, offset=0, buf=0x7f0ad43e8000, file=..., name=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", mode=24, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/include/os0file.ic:256 #9 fil_io (type=..., sync=sync@entry=true, page_id=..., page_size=..., byte_offset=byte_offset@entry=0, len=16384, buf=0x7f0ad43e8000, message=0x7f0acd19e5a0, ignore_missing_space=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5285 #10 0x00005587d9a7e5fb in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffd2786b9e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:192 #11 buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:460 #12 0x00005587d9a56e08 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", line=4639, mtr=0x7ffd2786c050, err=0x0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0buf.cc:4335 #13 0x00005587d9ac97d4 in fsp_flags_try_adjust (space_id=space_id@entry=550663, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4637 #14 0x00005587d9aca0f4 in fil_ibd_open (validate=<optimized out>, validate@entry=false, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=550663, flags=<optimized out>, space_name=<optimized out>, path_in=0x5588327e83f0 "./ddx_lab_802532/quickbooks_references.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4222 #15 0x00005587d9aa63bb in dict_check_sys_tables (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1480 #16 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1535 #17 0x00005587d99cbdc1 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:2544 #18 0x00005587d98afff1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/handler/ha_innodb.cc:4367 #19 0x00005587d966e084 in ha_initialize_handlerton (plugin=0x5587dbe68a80) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/handler.cc:520 #20 0x00005587d94e1b05 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffd27877640, plugin=plugin@entry=0x5587dbe68a80, argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=argv@entry=0x5587dbe22e40, options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1416 #21 0x00005587d94e24da in plugin_init (argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=0x5587dbe22e40, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1697 #22 0x00005587d943c560 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5266 #23 0x00005587d9441437 in mysqld_main (argc=63, argv=0x5587dbe22e40) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5858 #24 0x00007f16362a6830 in __libc_start_main (main=0x5587d941fe00 <main(int, char**)>, argc=6, argv=0x7ffd27878238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd27878228) at ../csu/libc-start.c:291 #25 0x00005587d9434779 in _start () Detaching from program: /usr/sbin/mysqld, process 26586 {noformat} Of possible note, 'apt upgrade' from 10.2.12 to 10.2.13 reported that it found a mysql 5.6 data in /var/lib/mysql (been 10.2.x for a month)? |
| Description |
Still having issues with startup with 10.2.13. FYI, my 30 +/- minute start time is for a DB roughly 290GB and roughly 991K tables {noformat} 2018-02-16 7:35:02 140326823504064 [Note] InnoDB: Highest supported file format is Barracuda. 2018-02-16 8:02:01 140326823504064 [Note] InnoDB: 128 out of 128 rollback segments are active. {noformat} During the startup, gdb shows this: {noformat} Attaching to program: /usr/sbin/mysqld, process 26586 [New LWP 26587] [New LWP 26588] [New LWP 26598] [New LWP 26599] [New LWP 26600] [New LWP 26601] [New LWP 26602] [New LWP 26603] [New LWP 26604] [New LWP 26605] [New LWP 26606] [New LWP 26607] [New LWP 26608] [New LWP 26609] [New LWP 26610] [New LWP 26611] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. Thread 17 (Thread 0x7f08c2805700 (LWP 26611)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c2805700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 16 (Thread 0x7f08c3006700 (LWP 26610)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3006700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 15 (Thread 0x7f08c3807700 (LWP 26609)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3807700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 14 (Thread 0x7f08c4008700 (LWP 26608)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5587dbfab2e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6f42a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3179 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c4008700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 13 (Thread 0x7f08c4809700 (LWP 26607)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c4808ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c4808ae0, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c4808ba0, m2=0x7f08c4808b98, m1=0x7f08c4808b90, global_segment=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c4809700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 12 (Thread 0x7f08c500a700 (LWP 26606)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c5009ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c5009ae0, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c5009ba0, m2=0x7f08c5009b98, m1=0x7f08c5009b90, global_segment=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c500a700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 11 (Thread 0x7f08c580b700 (LWP 26605)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c580aae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c580aae0, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c580aba0, m2=0x7f08c580ab98, m1=0x7f08c580ab90, global_segment=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c580b700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 10 (Thread 0x7f08c600c700 (LWP 26604)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c600bae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c600bae0, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c600bba0, m2=0x7f08c600bb98, m1=0x7f08c600bb90, global_segment=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c600c700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 9 (Thread 0x7f08c680d700 (LWP 26603)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c680cae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c680cae0, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c680cba0, m2=0x7f08c680cb98, m1=0x7f08c680cb90, global_segment=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c680d700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 8 (Thread 0x7f08c700e700 (LWP 26602)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c700dae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c700dae0, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c700dba0, m2=0x7f08c700db98, m1=0x7f08c700db90, global_segment=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c700e700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 7 (Thread 0x7f08c780f700 (LWP 26601)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c780eae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c780eae0, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c780eba0, m2=0x7f08c780eb98, m1=0x7f08c780eb90, global_segment=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c780f700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 6 (Thread 0x7f08c8010700 (LWP 26600)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c800fae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c800fae0, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c800fba0, m2=0x7f08c800fb98, m1=0x7f08c800fb90, global_segment=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8010700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 5 (Thread 0x7f08c8811700 (LWP 26599)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c8810ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c8810ae0, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c8810ba0, m2=0x7f08c8810b98, m1=0x7f08c8810b90, global_segment=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8811700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 4 (Thread 0x7f08c9012700 (LWP 26598)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c9011ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c9011ae0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c9011ba0, m2=0x7f08c9011b98, m1=0x7f08c9011b90, global_segment=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c9012700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 3 (Thread 0x7f1625d93700 (LWP 26588)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d984e68f in inline_mysql_cond_timedwait (src_file=0x5587d9cf2158 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f1625d92c80, mutex=0x5587dacb0860 <LOCK_checkpoint>, that=0x5587dacb0820 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 my_service_thread_sleep (control=control@entry=0x5587da3f46e0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c:115 #3 0x00005587d9846157 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_checkpoint.c:709 #4 0x00007f1636ce26ba in start_thread (arg=0x7f1625d93700) at pthread_create.c:333 #5 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7f1634a7e700 (LWP 26587)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d9bd5762 in inline_mysql_cond_timedwait (that=0x5587dad0af60 <COND_timer>, mutex=0x5587dad0afa0 <LOCK_timer>, src_file=0x5587d9d3a168 "/home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c", src_line=292, abstime=0x7f1634a7dd10) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c:292 #3 0x00007f1636ce26ba in start_thread (arg=0x7f1634a7e700) at pthread_create.c:333 #4 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7f16383ac8c0 (LWP 26586)): #0 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 #1 0x00005587d992bca5 in pread64 (__offset=0, __nbytes=16384, __buf=0x7f0ad43e8000, __fd=1817) at /usr/include/x86_64-linux-gnu/bits/unistd.h:99 #2 SyncFileIO::execute (request=..., this=<synthetic pointer>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1573 #3 os_file_io (in_type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, n=n@entry=16384, offset=offset@entry=0, err=err@entry=0x7ffd2786b44c) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:4877 #4 0x00005587d992c850 in os_file_pread (err=0x7ffd2786b44c, offset=0, n=16384, buf=0x7f0ad43e8000, file=1817, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5046 #5 os_file_read_page (type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, offset=offset@entry=0, n=n@entry=16384, o=o@entry=0x0, exit_on_err=true) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5083 #6 0x00005587d992e9f1 in os_file_read_func (n=16384, offset=0, buf=0x7f0ad43e8000, file=<optimized out>, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5493 #7 os_aio_func (type=..., mode=mode@entry=24, name=name@entry=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", file=..., buf=buf@entry=0x7f0ad43e8000, offset=0, n=16384, read_only=false, m1=0x5588327ed1d0, m2=0x7f0acd19e5a0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:6683 #8 0x00005587d9acba01 in pfs_os_aio_func (src_file=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", src_line=5290, m2=0x7f0acd19e5a0, m1=0x5588327ed1d0, read_only=<optimized out>, n=16384, offset=0, buf=0x7f0ad43e8000, file=..., name=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", mode=24, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/include/os0file.ic:256 #9 fil_io (type=..., sync=sync@entry=true, page_id=..., page_size=..., byte_offset=byte_offset@entry=0, len=16384, buf=0x7f0ad43e8000, message=0x7f0acd19e5a0, ignore_missing_space=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5285 #10 0x00005587d9a7e5fb in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffd2786b9e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:192 #11 buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:460 #12 0x00005587d9a56e08 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", line=4639, mtr=0x7ffd2786c050, err=0x0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0buf.cc:4335 #13 0x00005587d9ac97d4 in fsp_flags_try_adjust (space_id=space_id@entry=550663, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4637 #14 0x00005587d9aca0f4 in fil_ibd_open (validate=<optimized out>, validate@entry=false, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=550663, flags=<optimized out>, space_name=<optimized out>, path_in=0x5588327e83f0 "./ddx_lab_802532/quickbooks_references.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4222 #15 0x00005587d9aa63bb in dict_check_sys_tables (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1480 #16 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1535 #17 0x00005587d99cbdc1 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:2544 #18 0x00005587d98afff1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/handler/ha_innodb.cc:4367 #19 0x00005587d966e084 in ha_initialize_handlerton (plugin=0x5587dbe68a80) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/handler.cc:520 #20 0x00005587d94e1b05 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffd27877640, plugin=plugin@entry=0x5587dbe68a80, argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=argv@entry=0x5587dbe22e40, options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1416 #21 0x00005587d94e24da in plugin_init (argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=0x5587dbe22e40, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1697 #22 0x00005587d943c560 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5266 #23 0x00005587d9441437 in mysqld_main (argc=63, argv=0x5587dbe22e40) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5858 #24 0x00007f16362a6830 in __libc_start_main (main=0x5587d941fe00 <main(int, char**)>, argc=6, argv=0x7ffd27878238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd27878228) at ../csu/libc-start.c:291 #25 0x00005587d9434779 in _start () Detaching from program: /usr/sbin/mysqld, process 26586 {noformat} Of possible note, 'apt upgrade' from 10.2.12 to 10.2.13 reported that it found a mysql 5.6 data in /var/lib/mysql (been 10.2.x for a month)? |
Still having issues with startup with 10.2.13. FYI, my 30 +/- minute start time is for a DB roughly 290GB and roughly 991K tables {noformat} 2018-02-16 7:35:02 140326823504064 [Note] InnoDB: Highest supported file format is Barracuda. 2018-02-16 8:02:01 140326823504064 [Note] InnoDB: 128 out of 128 rollback segments are active. {noformat} During the startup, gdb shows this: {noformat} Attaching to program: /usr/sbin/mysqld, process 26586 [New LWP 26587] [New LWP 26588] [New LWP 26598] [New LWP 26599] [New LWP 26600] [New LWP 26601] [New LWP 26602] [New LWP 26603] [New LWP 26604] [New LWP 26605] [New LWP 26606] [New LWP 26607] [New LWP 26608] [New LWP 26609] [New LWP 26610] [New LWP 26611] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. Thread 17 (Thread 0x7f08c2805700 (LWP 26611)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c2805700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 16 (Thread 0x7f08c3006700 (LWP 26610)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3006700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 15 (Thread 0x7f08c3807700 (LWP 26609)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3807700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 14 (Thread 0x7f08c4008700 (LWP 26608)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00005587d9931f36 in os_event::wait (this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166 #2 os_event::wait_low (reset_sig_count=1, this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336 #3 os_event_wait_low (event=0x5587dbfab2e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535 #4 0x00005587d9a6f42a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3179 #5 0x00007f1636ce26ba in start_thread (arg=0x7f08c4008700) at pthread_create.c:333 #6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 13 (Thread 0x7f08c4809700 (LWP 26607)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c4808ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c4808ae0, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c4808ba0, m2=0x7f08c4808b98, m1=0x7f08c4808b90, global_segment=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c4809700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 12 (Thread 0x7f08c500a700 (LWP 26606)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c5009ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c5009ae0, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c5009ba0, m2=0x7f08c5009b98, m1=0x7f08c5009b90, global_segment=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c500a700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 11 (Thread 0x7f08c580b700 (LWP 26605)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c580aae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c580aae0, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c580aba0, m2=0x7f08c580ab98, m1=0x7f08c580ab90, global_segment=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c580b700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 10 (Thread 0x7f08c600c700 (LWP 26604)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c600bae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c600bae0, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c600bba0, m2=0x7f08c600bb98, m1=0x7f08c600bb90, global_segment=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c600c700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 9 (Thread 0x7f08c680d700 (LWP 26603)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c680cae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c680cae0, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c680cba0, m2=0x7f08c680cb98, m1=0x7f08c680cb90, global_segment=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c680d700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 8 (Thread 0x7f08c700e700 (LWP 26602)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c700dae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c700dae0, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c700dba0, m2=0x7f08c700db98, m1=0x7f08c700db90, global_segment=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c700e700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 7 (Thread 0x7f08c780f700 (LWP 26601)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c780eae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c780eae0, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c780eba0, m2=0x7f08c780eb98, m1=0x7f08c780eb90, global_segment=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c780f700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 6 (Thread 0x7f08c8010700 (LWP 26600)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c800fae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c800fae0, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c800fba0, m2=0x7f08c800fb98, m1=0x7f08c800fb90, global_segment=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8010700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 5 (Thread 0x7f08c8811700 (LWP 26599)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c8810ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c8810ae0, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c8810ba0, m2=0x7f08c8810b98, m1=0x7f08c8810b90, global_segment=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8811700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 4 (Thread 0x7f08c9012700 (LWP 26598)): #0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c9011ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893 #2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c9011ae0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038 #3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c9011ba0, m2=0x7f08c9011b98, m1=0x7f08c9011b90, global_segment=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092 #4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720 #5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330 #6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343 #7 0x00007f1636ce26ba in start_thread (arg=0x7f08c9012700) at pthread_create.c:333 #8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 3 (Thread 0x7f1625d93700 (LWP 26588)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d984e68f in inline_mysql_cond_timedwait (src_file=0x5587d9cf2158 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f1625d92c80, mutex=0x5587dacb0860 <LOCK_checkpoint>, that=0x5587dacb0820 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 my_service_thread_sleep (control=control@entry=0x5587da3f46e0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c:115 #3 0x00005587d9846157 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_checkpoint.c:709 #4 0x00007f1636ce26ba in start_thread (arg=0x7f1625d93700) at pthread_create.c:333 #5 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7f1634a7e700 (LWP 26587)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00005587d9bd5762 in inline_mysql_cond_timedwait (that=0x5587dad0af60 <COND_timer>, mutex=0x5587dad0afa0 <LOCK_timer>, src_file=0x5587d9d3a168 "/home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c", src_line=292, abstime=0x7f1634a7dd10) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186 #2 timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c:292 #3 0x00007f1636ce26ba in start_thread (arg=0x7f1634a7e700) at pthread_create.c:333 #4 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7f16383ac8c0 (LWP 26586)): #0 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84 #1 0x00005587d992bca5 in pread64 (__offset=0, __nbytes=16384, __buf=0x7f0ad43e8000, __fd=1817) at /usr/include/x86_64-linux-gnu/bits/unistd.h:99 #2 SyncFileIO::execute (request=..., this=<synthetic pointer>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1573 #3 os_file_io (in_type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, n=n@entry=16384, offset=offset@entry=0, err=err@entry=0x7ffd2786b44c) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:4877 #4 0x00005587d992c850 in os_file_pread (err=0x7ffd2786b44c, offset=0, n=16384, buf=0x7f0ad43e8000, file=1817, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5046 #5 os_file_read_page (type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, offset=offset@entry=0, n=n@entry=16384, o=o@entry=0x0, exit_on_err=true) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5083 #6 0x00005587d992e9f1 in os_file_read_func (n=16384, offset=0, buf=0x7f0ad43e8000, file=<optimized out>, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5493 #7 os_aio_func (type=..., mode=mode@entry=24, name=name@entry=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", file=..., buf=buf@entry=0x7f0ad43e8000, offset=0, n=16384, read_only=false, m1=0x5588327ed1d0, m2=0x7f0acd19e5a0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:6683 #8 0x00005587d9acba01 in pfs_os_aio_func (src_file=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", src_line=5290, m2=0x7f0acd19e5a0, m1=0x5588327ed1d0, read_only=<optimized out>, n=16384, offset=0, buf=0x7f0ad43e8000, file=..., name=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", mode=24, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/include/os0file.ic:256 #9 fil_io (type=..., sync=sync@entry=true, page_id=..., page_size=..., byte_offset=byte_offset@entry=0, len=16384, buf=0x7f0ad43e8000, message=0x7f0acd19e5a0, ignore_missing_space=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5285 #10 0x00005587d9a7e5fb in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffd2786b9e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:192 #11 buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:460 #12 0x00005587d9a56e08 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", line=4639, mtr=0x7ffd2786c050, err=0x0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0buf.cc:4335 #13 0x00005587d9ac97d4 in fsp_flags_try_adjust (space_id=space_id@entry=550663, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4637 #14 0x00005587d9aca0f4 in fil_ibd_open (validate=<optimized out>, validate@entry=false, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=550663, flags=<optimized out>, space_name=<optimized out>, path_in=0x5588327e83f0 "./ddx_lab_802532/quickbooks_references.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4222 #15 0x00005587d9aa63bb in dict_check_sys_tables (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1480 #16 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1535 #17 0x00005587d99cbdc1 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:2544 #18 0x00005587d98afff1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/handler/ha_innodb.cc:4367 #19 0x00005587d966e084 in ha_initialize_handlerton (plugin=0x5587dbe68a80) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/handler.cc:520 #20 0x00005587d94e1b05 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffd27877640, plugin=plugin@entry=0x5587dbe68a80, argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=argv@entry=0x5587dbe22e40, options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1416 #21 0x00005587d94e24da in plugin_init (argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=0x5587dbe22e40, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1697 #22 0x00005587d943c560 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5266 #23 0x00005587d9441437 in mysqld_main (argc=63, argv=0x5587dbe22e40) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5858 #24 0x00007f16362a6830 in __libc_start_main (main=0x5587d941fe00 <main(int, char**)>, argc=6, argv=0x7ffd27878238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd27878228) at ../csu/libc-start.c:291 #25 0x00005587d9434779 in _start () Detaching from program: /usr/sbin/mysqld, process 26586 {noformat} Of possible note, 'apt upgrade' from 10.2.12 to 10.2.13 reported that it found a mysql 5.6 data in /var/lib/mysql (been 10.2.x for a month)? |
| Affects Version/s | 10.2.13 [ 22910 ] | |
| Affects Version/s | 10.2.2 [ 22013 ] | |
| Affects Version/s | 10.3.0 [ 22127 ] |
| Fix Version/s | 10.2 [ 14601 ] | |
| Fix Version/s | 10.3 [ 22126 ] | |
| Fix Version/s | 10.3.5 [ 22905 ] | |
| Fix Version/s | 10.2.13 [ 22910 ] |
| Link |
This issue relates to |
| Link |
This issue is caused by |
| Link |
This issue duplicates |
| Status | Open [ 1 ] | Confirmed [ 10101 ] |
I will test the following patch to 10.2:
diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc
|
index f3bb127020a..0d80a7b7471 100644
|
--- a/storage/innobase/fil/fil0fil.cc
|
+++ b/storage/innobase/fil/fil0fil.cc
|
@@ -4215,7 +4215,12 @@ fil_ibd_open(
|
err = DB_ERROR;
|
}
|
|
- if (purpose != FIL_TYPE_IMPORT && !srv_read_only_mode) {
|
+ if (err == DB_SUCCESS
|
+ && purpose != FIL_TYPE_IMPORT && !srv_read_only_mode
|
+ && (validate
|
+ || (flags & ~(FSP_FLAGS_MASK_PAGE_COMPRESSION
|
+ | FSP_FLAGS_MEM_MASK))
|
+ >= (1U << FSP_FLAGS_POS_PAGE_SSIZE))) {
|
df_remote.close();
|
df_dict.close();
|
df_default.close(); |
In 10.1, I suspect that the .ibd files of any tables that use the attributes page_compressed=1 or DATA DIRECTORY will always be read on server startup. The above fix aims to avoid that.
Next, I will have to come up with a regression test for this.
After writing a test case that would catch opening of files on a restart following a normal shutdown, I see that there is a problem in the function dict_check_sys_tables(): it is opening the data file for every single table.
This problem appears to be present In MySQL 5.7 as well. My test case actually causes MySQL 5.7.21 to crash:
--source include/have_innodb.inc
|
--source include/not_embedded.inc
|
|
|
let $datadir= `select @@datadir`; |
|
|
SET GLOBAL innodb_file_per_table=1; |
CREATE TABLE tr(a INT)ENGINE=InnoDB ROW_FORMAT=REDUNDANT |
STATS_PERSISTENT=0;
|
CREATE TABLE tc(a INT)ENGINE=InnoDB ROW_FORMAT=COMPACT |
STATS_PERSISTENT=0 /* PAGE_COMPRESSED=1 PAGE_COMPRESSION_LEVEL=9 */; |
--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
|
eval CREATE TABLE td(a INT)ENGINE=InnoDB ROW_FORMAT=DYNAMIC |
STATS_PERSISTENT=0 DATA DIRECTORY='$MYSQL_TMP_DIR'; |
|
|
--source include/shutdown_mysqld.inc
|
|
|
--move_file $datadir/test/tr.ibd $datadir/test/tr0.ibd
|
--move_file $datadir/test/tc.ibd $datadir/test/tc0.ibd
|
--move_file $MYSQL_TMP_DIR/test/td.ibd $datadir/test/td0.ibd
|
--mkdir $datadir/test/tr.ibd
|
--mkdir $datadir/test/tc.ibd
|
--mkdir $MYSQL_TMP_DIR/test/td.ibd
|
|
|
--source include/start_mysqld.inc
|
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES |
WHERE engine = 'innodb' |
AND support IN ('YES', 'DEFAULT', 'ENABLED'); |
--source include/shutdown_mysqld.inc
|
|
|
--rmdir $datadir/test/tr.ibd
|
--rmdir $datadir/test/tc.ibd
|
--rmdir $MYSQL_TMP_DIR/test/td.ibd
|
|
|
--move_file $datadir/test/tr0.ibd $datadir/test/tr.ibd
|
--move_file $datadir/test/tc0.ibd $datadir/test/tc.ibd
|
--move_file $datadir/test/td0.ibd $MYSQL_TMP_DIR/test/td.ibd
|
|
|
--source include/start_mysqld.inc
|
SELECT * FROM tr; |
SELECT * FROM tc; |
SELECT * FROM td; |
DROP TABLE tr,tc,td; |
I had to comment out the PAGE_COMPRESSED and PAGE_COMPRESSION_LEVEL which I would want to test in MariaDB due to the 10.1 compatibility bug MDEV-11623.
The 5.7.21 server would crash like this:
|
5.7.21 |
2018-02-16T19:54:54.541992Z 0 [Note] Plugin 'FEDERATED' is disabled.
|
2018-02-16T19:54:54.546255Z 0 [Warning] unknown variable 'loose-mysqlx-port=13009'
|
2018-02-16T19:54:54.546270Z 0 [Warning] unknown variable 'loose-mysqlx-socket=/mariadb/5.7/build/mysql-test/var/tmp/mysqlx.1.sock'
|
2018-02-16T19:54:54.546274Z 0 [Warning] unknown variable 'loose-sha256_password_auto_generate_rsa_keys=0'
|
2018-02-16 22:54:54 0x7fa041665700 InnoDB: Assertion failure in thread 140326268720896 in file fil0fil.cc line 748
|
InnoDB: Failing assertion: size_bytes != (os_offset_t) -1
|
Interestingly, MySQL 5.7.21 would appear to happily open the directory as a file, and it would crash in a background thread:
|
5.7.21 stack trace from gdb |
#6 0x0000558a84109c09 in ut_dbg_assertion_failed (expr=0x558a847fe690 "size_bytes != (os_offset_t) -1", file=0x558a847fe360 "/mariadb/5.7/storage/innobase/fil/fil0fil.cc", line=748) at /mariadb/5.7/storage/innobase/ut/ut0dbg.cc:67
|
#7 0x0000558a841f708d in fil_node_open_file (node=0x558a8725f268) at /mariadb/5.7/storage/innobase/fil/fil0fil.cc:748
|
#8 0x0000558a84203e36 in fil_node_prepare_for_io (node=0x558a8725f268, system=0x558a87015778, space=0x558a8725eed8) at /mariadb/5.7/storage/innobase/fil/fil0fil.cc:5337
|
#9 0x0000558a84204ba4 in fil_io (type=..., sync=true, page_id=..., page_size=..., byte_offset=0, len=16384, buf=0x7fa06a334000, message=0x7fa069eb5cb0) at /mariadb/5.7/storage/innobase/fil/fil0fil.cc:5641
|
#10 0x0000558a8419e7ec in buf_read_page_low (err=0x7fa041663e04, sync=true, type=192, mode=132, page_id=..., page_size=..., unzip=false) at /mariadb/5.7/storage/innobase/buf/buf0rea.cc:193
|
#11 0x0000558a8419f239 in buf_read_page_background (page_id=..., page_size=..., sync=true) at /mariadb/5.7/storage/innobase/buf/buf0rea.cc:479
|
#12 0x0000558a84184569 in buf_load () at /mariadb/5.7/storage/innobase/buf/buf0dump.cc:677
|
#13 0x0000558a841848a1 in buf_dump_thread (arg=0x0) at /mariadb/5.7/storage/innobase/buf/buf0dump.cc:77
|
MariaDB does not crash, but it would attempt to open each .ibd file (opening directory as a file), spamming the error log with messages like this:
|
MariaDB 10.2.13 error log |
2018-02-16 21:06:02 140398864725952 [Warning] InnoDB: Ignoring tablespace for `test`.`td` because it could not be opened.
|
2018-02-16 21:06:02 140398864725952 [ERROR] InnoDB: The size of the file ./test/tr.ibd is only 40 bytes, should be at least 65536
|
Before MDEV-13869 the MariaDB debug build would probably crash when trying to access the table. In MDEV-13869 I specifically added a check that we will refuse to load a table definition if the size of the file is invalid.
| Labels | performance startup | performance startup upstream |
| Affects Version/s | 10.2.2 [ 22013 ] | |
| Affects Version/s | 10.2.13 [ 22910 ] |
I ported the test to MariaDB 5.5, and there the .ibd files were opened, but maybe not read, because there was no crash:
|
5.5 error log excerpt |
180216 22:39:49 InnoDB: Operating system error number 21 in a file operation.
|
InnoDB: Error number 21 means 'Is a directory'.
|
InnoDB: Some operating system error numbers are described at
|
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
|
180216 22:39:49 InnoDB: Error: trying to open a table, but could not
|
InnoDB: open the tablespace file './test/tc.ibd'!
|
InnoDB: Have you moved InnoDB .ibd files around without using the
|
InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE?
|
InnoDB: It is also possible that this is a temporary table #sql...,
|
InnoDB: and MySQL removed the .ibd file for this.
|
InnoDB: Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
|
InnoDB: for how to resolve the issue.
|
Because I do not remember anyone complaining about slow startup time with large number of tables on MySQL 5.5 or 5.6, I have to assume that there is no problem to open every file on startup, but that the files are actually being read.
So, we should test with regular files, not directories, with invalid content but possibly valid length. A revised test passes on 5.5, 10.0, 10.1 on both InnoDB and XtraDB:
--source include/have_innodb.inc
|
--source include/not_embedded.inc
|
|
|
let datadir= `select @@datadir`; |
|
|
SET GLOBAL innodb_file_format=Barracuda; |
SET GLOBAL innodb_file_per_table=1; |
CREATE TABLE tr(a INT)ENGINE=InnoDB ROW_FORMAT=REDUNDANT; |
CREATE TABLE tc(a INT)ENGINE=InnoDB ROW_FORMAT=COMPACT; |
|
|
--let $_server_id= `SELECT @@server_id`
|
--let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect
|
--exec echo "wait" > $_expect_file_name
|
shutdown_server;
|
|
|
--move_file $datadir/test/tr.ibd $datadir/test/tr0.ibd
|
--move_file $datadir/test/tc.ibd $datadir/test/tc0.ibd
|
perl;
|
die unless open OUT, ">", "$ENV{datadir}/test/tr.ibd"; |
print OUT "foo bar " x 2048; |
close OUT or die; |
die unless open OUT, ">", "$ENV{datadir}/test/tc.ibd"; |
print OUT "foo bar " x 2048; |
close OUT or die; |
EOF
|
|
|
--exec echo "restart" > $_expect_file_name
|
--enable_reconnect
|
--source include/wait_until_connected_again.inc
|
--disable_reconnect
|
|
|
SELECT COUNT(*) FROM INFORMATION_SCHEMA.ENGINES |
WHERE engine = 'innodb' |
AND support IN ('YES', 'DEFAULT', 'ENABLED'); |
--exec echo "wait" > $_expect_file_name
|
shutdown_server;
|
|
|
--remove_file $datadir/test/tr.ibd
|
--remove_file $datadir/test/tc.ibd
|
|
|
--move_file $datadir/test/tr0.ibd $datadir/test/tr.ibd
|
--move_file $datadir/test/tc0.ibd $datadir/test/tc.ibd
|
|
|
--exec echo "restart" > $_expect_file_name
|
--enable_reconnect
|
--source include/wait_until_connected_again.inc
|
--disable_reconnect
|
SELECT * FROM tr; |
SELECT * FROM tc; |
DROP TABLE tr,tc; |
On MySQL 5.7.21 this would crash due to the buffer pool load being enabled by default, and the pages of the tables being part of the buffer pool dump at shutdown. The test passes if I run
./mtr --mysqld=--skip-innodb-buffer-pool-load-at-startup testname
|
So, clearly this is a MariaDB specific problem. With my above patch, MariaDB would only emit the following types of messages:
|
MariaDB 10.2.13 with fix |
2018-02-16 22:54:46 140415539910400 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_B7tH/mysqld.1/data/ib_buffer_pool
|
2018-02-16 22:54:46 140415539910400 [ERROR] InnoDB: Space ID in fsp header is 1914725999, but in the page header it is 1864393313.
|
2018-02-16 22:54:46 140415539910400 [ERROR] InnoDB: Expected tablespace flags 0 but found 0x7220666f in the file ./test/tr.ibd
|
2018-02-16 22:54:46 140415539910400 [Note] InnoDB: trying to read page [page id: space=4, page number=0] in the background in a non-existing or being-dropped tablespace
|
Without innodb-buffer-pool-load-at-startup the above messages will vanish, and the test will pass.
Thus, the only problem is that I did not fully merge the MDEV-12610 fix to MariaDB 10.2, and the above patch fixes this. This bug is a regression from MDEV-11623, which is necessary to allow an upgrade from MariaDB versions 10.1.0 to 10.1.20.
I will push the fix tomorrow after cleaning up the test case.
| Labels | performance startup upstream | performance startup |
| Affects Version/s | 10.3.0 [ 22127 ] | |
| Affects Version/s | 10.2.4 [ 22116 ] | |
| Affects Version/s | 10.2.2 [ 22013 ] |
Actually my initial patch was wrong; it would cause every file to be reread on startup when using a non-default innodb_page_size. MariaDB 10.1 does suffer from the same problem; it would crash:
|
MariaDB 10.1 with innodb_page_size=4k |
InnoDB: Failing assertion: zip_size <= (1 << 14)
|
…
|
include/fsp0fsp.ic:83(fsp_flags_get_zip_size(unsigned long))[0x7f325454ab4d]
|
fil/fil0fil.cc:658(fil_node_open_file(fil_node_t*, fil_system_t*, fil_space_t*))[0x7f325454bdee]
|
fil/fil0fil.cc:5585(fil_node_prepare_for_io(fil_node_t*, fil_system_t*, fil_space_t*))[0x7f32545557e0]
|
fil/fil0fil.cc:1685(fil_space_get_space(unsigned long))[0x7f325454e5fe]
|
fil/fil0fil.cc:1801(fil_space_get_flags(unsigned long))[0x7f325454e8d3]
|
fil/fil0fil.cc:1828(fil_space_get_zip_size(unsigned long))[0x7f325454e92f]
|
buf/buf0buf.cc:3091(buf_page_get_gen(unsigned long, unsigned long, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned long, mtr_t*, dberr_t*))[0x7f32544d4c0b]
|
fil/fil0fil.cc:3860(fsp_flags_try_adjust(unsigned long, unsigned long))[0x7f325455257a]
|
fil/fil0fil.cc:4295(fil_open_single_table_tablespace(bool, bool, unsigned long, unsigned long, char const*, char const*))[0x7f3254553511]
|
This fix for 10.2 appears to work. I will have to backport it to 10.1 as well.
diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc
|
index f3bb127020a..e48e70c80d2 100644
|
--- a/storage/innobase/fil/fil0fil.cc
|
+++ b/storage/innobase/fil/fil0fil.cc
|
@@ -4215,7 +4215,12 @@ fil_ibd_open(
|
err = DB_ERROR;
|
}
|
|
- if (purpose != FIL_TYPE_IMPORT && !srv_read_only_mode) {
|
+ if (err == DB_SUCCESS
|
+ && purpose != FIL_TYPE_IMPORT && !srv_read_only_mode
|
+ && (validate
|
+ || (flags & ~(FSP_FLAGS_MASK_PAGE_COMPRESSION
|
+ | FSP_FLAGS_MEM_MASK))
|
+ >= (1U << FSP_FLAGS_POS_RESERVED))) {
|
df_remote.close();
|
df_dict.close();
|
df_default.close(); |
| Affects Version/s | 10.1.21 [ 22113 ] |
| Fix Version/s | 10.1 [ 16100 ] |
Hi Marko,
FYI, on MySQL 5.6, same number of tables, I'm around 5 minutes for startup:
2018-02-10 17:44:38 24846 [Note] InnoDB: Highest supported file format is Barracuda.
|
2018-02-10 17:49:32 24846 [Note] InnoDB: 128 rollback segment(s) are active.
|
michaelcaplan, I believe that every .ibd file is opened but not read by the InnoDB startup in MySQL 5.5 and later. This behaviour might date back all the way to MySQL 4.1, where Heikki introduced the innodb_file_per_table option and the .ibd files. It might also make sense, because opening a file would also lock the file, preventing multiple server instances from running on the same files.
The problem that every file is being read on normal startup is clearly unique to MariaDB, due to 2 different MariaDB specific problems: an incorrect code merge from MySQL 5.7.9 (which I fixed in MDEV-13869) and an upgrade procedure from faulty MariaDB 10.1 versions (a performance regression introduced in the fix of MDEV-11623 that was partly fixed in MariaDB 10.1.25 in MDEV-12610).
When it comes to my above patch, I believe that we should not need any condition on flags; we should always skip the adjustment if validate==false. The MariaDB 10.1.0‥10.1.20 bug was that the incorrect table flags were written into the FSP_SPACE_FLAGS in the first page of each .ibd file. At this point of execution, the flags should be dict_tf_to_fsp_flags(dict_sys_tables_type_to_tf(…)), derived from the TYPE and N_COLS columns of SYS_TABLES, which were not affected by MDEV-11623. There is no other caller that would pass validate=false to fil_ibd_open() or its 10.1 counterpart fil_open_single_table_tablespace().
MySQL 5.6 (as well as MariaDB 10.1) does attempt to read every .ibd file for which the DATA DIRECTORY attribute was used (or an .isl file exists):
|
MySQL 5.6.38 |
2018-02-17 14:52:29 9944 [ERROR] InnoDB: Error innodb-page-size mismatch in file '/mariadb/5.6/build/mysql-test/var/tmp/test/td.ibd',tablespace id=8, flags=1249. Please refer to http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
|
2018-02-17 14:52:29 9944 [ERROR] InnoDB: Could not find a valid tablespace file for 'test/td'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
|
2018-02-17 14:52:29 9944 [ERROR] InnoDB: Tablespace open failed for '"test"."td"', ignored.
|
Marko,
Thanks, as always, for your clear and detailed explanation of the issue. To be clear, I see no issue with the rational that you described to the start time I noted in 5.6. 5 min is much more manageable than 30 min.
Appreciate your work on this issue!
I pushed a fix to 10.1 and merged to 10.2. The new test innodb.restart should prevent this from happening in the future. It shuts down the server, then replaces some data files with garbage, and restarts.
For the table with DATA DIRECTORY some error messages will be issued (in 10.2 many more than in 10.1). Maybe some day, we should avoid the reads of these files. Maybe we should even avoid opening any files before the first actual access.
| issue.field.resolutiondate | 2018-02-17 13:44:52.0 | 2018-02-17 13:44:52.709 |
| Component/s | Storage Engine - XtraDB [ 10135 ] | |
| Fix Version/s | 10.3.5 [ 22905 ] | |
| Fix Version/s | 10.1.32 [ 22908 ] | |
| Fix Version/s | 10.2.14 [ 22911 ] | |
| Fix Version/s | 10.2 [ 14601 ] | |
| Fix Version/s | 10.1 [ 16100 ] | |
| Fix Version/s | 10.3 [ 22126 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Confirmed [ 10101 ] | Closed [ 6 ] |
michaelcaplan, could you try out one of the 10.2 packages with this fix and see if they work for you? Almost all the code changes since the MariaDB 10.2.13 release are InnoDB changes by me.
@marko I unfortunately only have access to a large database in production.
| Workflow | MariaDB v3 [ 85614 ] | MariaDB v4 [ 153809 ] |
| Link |
This issue relates to |
I am sorry for the trouble. It turns out that the MariaDB 10.1 fix for the
MDEV-12610slow startup afterMDEV-11623was not fully merged merged to MariaDB 10.2 by myself.This code path would be executed even if !validate.