Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.21, 10.2.4, 10.3.0
-
ovz container, 2.6.32-042stab120.6, CentOS release 6.7 (Final)
Description
Still having issues with startup with 10.2.13. MDEV-13869 did not fix my issue.
FYI, my 30 +/- minute start time is for a DB roughly 290GB and roughly 991K tables
2018-02-16 7:35:02 140326823504064 [Note] InnoDB: Highest supported file format is Barracuda.
|
2018-02-16 8:02:01 140326823504064 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
During the startup, gdb shows this:
Attaching to program: /usr/sbin/mysqld, process 26586
|
[New LWP 26587]
|
[New LWP 26588]
|
[New LWP 26598]
|
[New LWP 26599]
|
[New LWP 26600]
|
[New LWP 26601]
|
[New LWP 26602]
|
[New LWP 26603]
|
[New LWP 26604]
|
[New LWP 26605]
|
[New LWP 26606]
|
[New LWP 26607]
|
[New LWP 26608]
|
[New LWP 26609]
|
[New LWP 26610]
|
[New LWP 26611]
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
|
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
|
|
|
Thread 17 (Thread 0x7f08c2805700 (LWP 26611)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c2805700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 16 (Thread 0x7f08c3006700 (LWP 26610)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3006700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 15 (Thread 0x7f08c3807700 (LWP 26609)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5588274b9ca0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5588274b9ca0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6edab in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3507
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c3807700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 14 (Thread 0x7f08c4008700 (LWP 26608)):
|
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
|
#1 0x00005587d9931f36 in os_event::wait (this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:166
|
#2 os_event::wait_low (reset_sig_count=1, this=0x5587dbfab2e0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:336
|
#3 os_event_wait_low (event=0x5587dbfab2e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0event.cc:535
|
#4 0x00005587d9a6f42a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0flu.cc:3179
|
#5 0x00007f1636ce26ba in start_thread (arg=0x7f08c4008700) at pthread_create.c:333
|
#6 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 13 (Thread 0x7f08c4809700 (LWP 26607)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c4808ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c4808ae0, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c4808ba0, m2=0x7f08c4808b98, m1=0x7f08c4808b90, global_segment=9)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f08c4808b90, m2=m2@entry=0x7f08c4808b98, request=request@entry=0x7f08c4808ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c4809700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 12 (Thread 0x7f08c500a700 (LWP 26606)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c5009ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c5009ae0, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c5009ba0, m2=0x7f08c5009b98, m1=0x7f08c5009b90, global_segment=8)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f08c5009b90, m2=m2@entry=0x7f08c5009b98, request=request@entry=0x7f08c5009ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c500a700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 11 (Thread 0x7f08c580b700 (LWP 26605)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c580aae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c580aae0, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c580aba0, m2=0x7f08c580ab98, m1=0x7f08c580ab90, global_segment=7)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f08c580ab90, m2=m2@entry=0x7f08c580ab98, request=request@entry=0x7f08c580aba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c580b700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 10 (Thread 0x7f08c600c700 (LWP 26604)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c600bae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c600bae0, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c600bba0, m2=0x7f08c600bb98, m1=0x7f08c600bb90, global_segment=6)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f08c600bb90, m2=m2@entry=0x7f08c600bb98, request=request@entry=0x7f08c600bba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c600c700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 9 (Thread 0x7f08c680d700 (LWP 26603)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c680cae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c680cae0, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c680cba0, m2=0x7f08c680cb98, m1=0x7f08c680cb90, global_segment=5)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f08c680cb90, m2=m2@entry=0x7f08c680cb98, request=request@entry=0x7f08c680cba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c680d700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 8 (Thread 0x7f08c700e700 (LWP 26602)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c700dae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c700dae0, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c700dba0, m2=0x7f08c700db98, m1=0x7f08c700db90, global_segment=4)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f08c700db90, m2=m2@entry=0x7f08c700db98, request=request@entry=0x7f08c700dba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c700e700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 7 (Thread 0x7f08c780f700 (LWP 26601)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c780eae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c780eae0, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c780eba0, m2=0x7f08c780eb98, m1=0x7f08c780eb90, global_segment=3)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f08c780eb90, m2=m2@entry=0x7f08c780eb98, request=request@entry=0x7f08c780eba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c780f700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 6 (Thread 0x7f08c8010700 (LWP 26600)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c800fae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c800fae0, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c800fba0, m2=0x7f08c800fb98, m1=0x7f08c800fb90, global_segment=2)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f08c800fb90, m2=m2@entry=0x7f08c800fb98, request=request@entry=0x7f08c800fba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8010700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 5 (Thread 0x7f08c8811700 (LWP 26599)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c8810ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c8810ae0, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c8810ba0, m2=0x7f08c8810b98, m1=0x7f08c8810b90, global_segment=1)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f08c8810b90, m2=m2@entry=0x7f08c8810b98, request=request@entry=0x7f08c8810ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c8811700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 4 (Thread 0x7f08c9012700 (LWP 26598)):
|
#0 0x00007f1637e0664a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
|
#1 0x00005587d992b358 in LinuxAIOHandler::collect (this=this@entry=0x7f08c9011ae0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1893
|
#2 0x00005587d992b8a2 in LinuxAIOHandler::poll (this=this@entry=0x7f08c9011ae0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2038
|
#3 0x00005587d99307c9 in os_aio_linux_handler (request=0x7f08c9011ba0, m2=0x7f08c9011b98, m1=0x7f08c9011b90, global_segment=0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:2092
|
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f08c9011b90, m2=m2@entry=0x7f08c9011b98, request=request@entry=0x7f08c9011ba0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5720
|
#5 0x00005587d9ac4685 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5330
|
#6 0x00005587d99c6958 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:343
|
#7 0x00007f1636ce26ba in start_thread (arg=0x7f08c9012700) at pthread_create.c:333
|
#8 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 3 (Thread 0x7f1625d93700 (LWP 26588)):
|
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
|
#1 0x00005587d984e68f in inline_mysql_cond_timedwait (src_file=0x5587d9cf2158 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f1625d92c80,
|
mutex=0x5587dacb0860 <LOCK_checkpoint>, that=0x5587dacb0820 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186
|
#2 my_service_thread_sleep (control=control@entry=0x5587da3f46e0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_servicethread.c:115
|
#3 0x00005587d9846157 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/maria/ma_checkpoint.c:709
|
#4 0x00007f1636ce26ba in start_thread (arg=0x7f1625d93700) at pthread_create.c:333
|
#5 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 2 (Thread 0x7f1634a7e700 (LWP 26587)):
|
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
|
#1 0x00005587d9bd5762 in inline_mysql_cond_timedwait (that=0x5587dad0af60 <COND_timer>, mutex=0x5587dad0afa0 <LOCK_timer>,
|
src_file=0x5587d9d3a168 "/home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c", src_line=292, abstime=0x7f1634a7dd10)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/include/mysql/psi/mysql_thread.h:1186
|
#2 timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/mysys/thr_timer.c:292
|
#3 0x00007f1636ce26ba in start_thread (arg=0x7f1634a7e700) at pthread_create.c:333
|
#4 0x00007f163638d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
|
|
|
Thread 1 (Thread 0x7f16383ac8c0 (LWP 26586)):
|
#0 0x00007f1636cebd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
|
#1 0x00005587d992bca5 in pread64 (__offset=0, __nbytes=16384, __buf=0x7f0ad43e8000, __fd=1817) at /usr/include/x86_64-linux-gnu/bits/unistd.h:99
|
#2 SyncFileIO::execute (request=..., this=<synthetic pointer>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:1573
|
#3 os_file_io (in_type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, n=n@entry=16384, offset=offset@entry=0, err=err@entry=0x7ffd2786b44c)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:4877
|
#4 0x00005587d992c850 in os_file_pread (err=0x7ffd2786b44c, offset=0, n=16384, buf=0x7f0ad43e8000, file=1817, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5046
|
#5 os_file_read_page (type=..., file=file@entry=1817, buf=buf@entry=0x7f0ad43e8000, offset=offset@entry=0, n=n@entry=16384, o=o@entry=0x0, exit_on_err=true)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5083
|
#6 0x00005587d992e9f1 in os_file_read_func (n=16384, offset=0, buf=0x7f0ad43e8000, file=<optimized out>, type=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:5493
|
#7 os_aio_func (type=..., mode=mode@entry=24, name=name@entry=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", file=..., buf=buf@entry=0x7f0ad43e8000, offset=0, n=16384, read_only=false,
|
m1=0x5588327ed1d0, m2=0x7f0acd19e5a0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/os/os0file.cc:6683
|
#8 0x00005587d9acba01 in pfs_os_aio_func (src_file=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", src_line=5290, m2=0x7f0acd19e5a0, m1=0x5588327ed1d0,
|
read_only=<optimized out>, n=16384, offset=0, buf=0x7f0ad43e8000, file=..., name=0x5588327ed280 "./ddx_lab_802532/quickbooks_references.ibd", mode=24, type=...)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/include/os0file.ic:256
|
#9 fil_io (type=..., sync=sync@entry=true, page_id=..., page_size=..., byte_offset=byte_offset@entry=0, len=16384, buf=0x7f0ad43e8000, message=0x7f0acd19e5a0, ignore_missing_space=false)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:5285
|
#10 0x00005587d9a7e5fb in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffd2786b9e0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:192
|
#11 buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0rea.cc:460
|
#12 0x00005587d9a56e08 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10,
|
file=file@entry=0x5587d9d27f60 "/home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc", line=4639, mtr=0x7ffd2786c050, err=0x0)
|
at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/buf/buf0buf.cc:4335
|
#13 0x00005587d9ac97d4 in fsp_flags_try_adjust (space_id=space_id@entry=550663, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4637
|
#14 0x00005587d9aca0f4 in fil_ibd_open (validate=<optimized out>, validate@entry=false, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=550663, flags=<optimized out>,
|
space_name=<optimized out>, path_in=0x5588327e83f0 "./ddx_lab_802532/quickbooks_references.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/fil/fil0fil.cc:4222
|
#15 0x00005587d9aa63bb in dict_check_sys_tables (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1480
|
#16 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/dict/dict0load.cc:1535
|
#17 0x00005587d99cbdc1 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/srv/srv0start.cc:2544
|
#18 0x00005587d98afff1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.13/storage/innobase/handler/ha_innodb.cc:4367
|
#19 0x00005587d966e084 in ha_initialize_handlerton (plugin=0x5587dbe68a80) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/handler.cc:520
|
#20 0x00005587d94e1b05 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffd27877640, plugin=plugin@entry=0x5587dbe68a80, argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=argv@entry=0x5587dbe22e40,
|
options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1416
|
#21 0x00005587d94e24da in plugin_init (argc=argc@entry=0x5587da481c58 <remaining_argc>, argv=0x5587dbe22e40, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/sql_plugin.cc:1697
|
#22 0x00005587d943c560 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5266
|
#23 0x00005587d9441437 in mysqld_main (argc=63, argv=0x5587dbe22e40) at /home/buildbot/buildbot/build/mariadb-10.2.13/sql/mysqld.cc:5858
|
#24 0x00007f16362a6830 in __libc_start_main (main=0x5587d941fe00 <main(int, char**)>, argc=6, argv=0x7ffd27878238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
|
stack_end=0x7ffd27878228) at ../csu/libc-start.c:291
|
#25 0x00005587d9434779 in _start ()
|
Detaching from program: /usr/sbin/mysqld, process 26586
|
|
Of possible note, 'apt upgrade' from 10.2.12 to 10.2.13 reported that it found a mysql 5.6 data in /var/lib/mysql (been 10.2.x for a month)?
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- is caused by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Marko,
Thanks, as always, for your clear and detailed explanation of the issue. To be clear, I see no issue with the rational that you described to the start time I noted in 5.6. 5 min is much more manageable than 30 min.
Appreciate your work on this issue!
I pushed a fix to 10.1 and merged to 10.2. The new test innodb.restart should prevent this from happening in the future. It shuts down the server, then replaces some data files with garbage, and restarts.
For the table with DATA DIRECTORY some error messages will be issued (in 10.2 many more than in 10.1). Maybe some day, we should avoid the reads of these files. Maybe we should even avoid opening any files before the first actual access.
michaelcaplan, could you try out one of the 10.2 packages with this fix and see if they work for you? Almost all the code changes since the MariaDB 10.2.13 release are InnoDB changes by me.
@marko I unfortunately only have access to a large database in production.
MySQL 5.6 (as well as MariaDB 10.1) does attempt to read every .ibd file for which the DATA DIRECTORY attribute was used (or an .isl file exists):
MySQL 5.6.38
2018-02-17 14:52:29 9944 [ERROR] InnoDB: Error innodb-page-size mismatch in file '/mariadb/5.6/build/mysql-test/var/tmp/test/td.ibd',tablespace id=8, flags=1249. Please refer to http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue.2018-02-17 14:52:29 9944 [ERROR] InnoDB: Could not find a valid tablespace file for 'test/td'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue.2018-02-17 14:52:29 9944 [ERROR] InnoDB: Tablespace open failed for '"test"."td"', ignored.