|
Added gbd.txt trace when startup is seemingly locked up
|
|
Actually, disregard the above comment. What is actually going on in gbd.txt is the mariaDB upgrade to 10.2.22+maria~xenial process. It's been running for 4 hours now. Stuck here:
Setting up mariadb-client-core-10.2 (10.2.22+maria~xenial) ...
|
Setting up mariadb-client-10.2 (10.2.22+maria~xenial) ...
|
Setting up mariadb-server-core-10.2 (10.2.22+maria~xenial) ...
|
Setting up mariadb-server-10.2 (10.2.22+maria~xenial) ...
|
Installing new version of config file /etc/mysql/debian-start ...
|
mysqld is running at this point, but the error-log is has nothing logged since I shut it down before running apt upgrade.
|
|
If I understand right, the update process got stuck int he debian-start upgrade cycle which does not output to the log. Between these log lines, I captured gdb-startup.txt (pretty much identical to gbd.txt )
2019-03-07 20:38:38 140573848283328 [Note] InnoDB: Highest supported file format is Barracuda.
|
2019-03-07 21:31:01 140573848283328 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
|
|
Currently starting up our slave. 1.5 hours and still waiting. Acording to gbd, looks it is stuck in storage/innobase/fts/fts0fts.cc:6870 fts_check_and_drop_orphaned_tables. Not sure if I hit some infinite loop....
|
|
Currently around 3 hours into the statup cycle. At this point in the log:
2019-03-13 22:30:03 139816883783872 [Note] InnoDB: Highest supported file format is Barracuda.
|
ran gdb an hour and a half ago, and another time just now with thread apply all bt full to see what is going in. Diff of startup1.txt and startup3.txt seems to indicate it is slowly moving through fts_check_and_drop_orphaned_tables:
--- /home/michael/Downloads/mariadb/startup1.txt
|
+++ /home/michael/Downloads/mariadb/startup2.txt
|
@@ -557,11 +557,11 @@
|
mutex=0x560502458f00 <LOCK_checkpoint>, that=0x560502458ec0 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.22/include/mysql/psi/mysql_thread.h:1186
|
result = <optimized out>
|
#2 my_service_thread_sleep (control=control@entry=0x560501b9c8c0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.22/storage/maria/ma_servicethread.c:115
|
- abstime = {tv_sec = 1552536150, tv_nsec = 950850000}
|
+ abstime = {tv_sec = 1552541224, tv_nsec = 961758000}
|
res = 0 '\000'
|
#3 0x0000560500ff3277 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.22/storage/maria/ma_checkpoint.c:709
|
interval = 30
|
- sleeps = 1920
|
+ sleeps = 6960
|
sleep_time = 30
|
log_horizon_at_last_checkpoint = 4294976683
|
pagecache_flushes_at_last_checkpoint = 0
|
@@ -611,18 +611,18 @@
|
#0 __strcmp_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:32
|
No locals.
|
#1 0x00005605012a12eb in fts_check_and_drop_orphaned_tables (tables=<optimized out>, trx=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.22/storage/innobase/fts/fts0fts.cc:6870
|
- valid_aux = 0x5605e4bbcd90
|
- count = 96913
|
+ valid_aux = 0x560621fa4d90
|
+ count = 214123
|
fake_aux = <optimized out>
|
- aux_table = 0x7f1c841f0f80
|
- orig_parent_id = 2295311
|
+ aux_table = 0x7f1c84942940
|
+ orig_parent_id = 2426123
|
rename = false
|
parent_table = <optimized out>
|
drop = false
|
table = <optimized out>
|
next_aux_table = <optimized out>
|
orig_index_id = <optimized out>
|
- i = 391800
|
+ i = 631622
|
heap = <optimized out>
|
invalid_aux_tables = <optimized out>
|
aux_tables_to_rename = <optimized out>
|
|
|
|
The slowness in gdb-startup.txt is caused by the function fts_check_and_drop_orphaned_tables(). This function is needed until MDEV-18518 makes InnoDB support atomic transactions for creating multiple tables.
|
|
michaelcaplan, I am afraid that the startup problem affects anyone who creates a large number of InnoDB tables with FULLTEXT INDEXES. It could be an O(n²) or O(n³) algorithm that is being executed at startup.
Can you provide more information about the slow shutdown? Could it simply be a duplicate of MDEV-16865?
|
|
The shutdown could possibly be slow due to MDEV-18936.
|
|
My clients server takes over 7 hours to start:
2020-07-13 11:59:33 0 [Note] InnoDB: Completed initialization of buffer pool
2020-07-13 11:59:33 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-07-13 19:19:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-07-13 19:19:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables
My client has 13,000 databases with 1.8 million tables with about 65,000 Full Text Indexes.
There are 2,021,672 FTS_*.ibd files
|
|
thiru, can you rewrite fts_drop_orphaned_tables() to make it significantly faster? Is it necessary to scan SYS_TABLES at all? Do we really have to use the InnoDB SQL parser for the scanning?
I am thinking along the following lines:
- Traverse fil_system.space_list and construct a collection of (table_id,index_id) of all FTS_*.ibd tablespaces. Maybe std::set or std::unordered_set would do?
- If that collection is not empty, remove (TABLE_ID, ID) from the above collection for each record of SYS_INDEXES.
- For any surviving elements in the collection, construct the table name from (table_id,index_id) and invoke row_drop_table_for_mysql().
|
|
While working on this, thiru noticed that the upgrade logic for a Windows-specific bug that was introduced in MySQL 5.6 and fixed in 2013 in MySQL 5.6.16 is costing significant time. The only GA version of MariaDB that is affected by this bug is MariaDB Server 10.0.10. The fix was included in MariaDB Server 10.0.11 already. I think that it is extremely unlikely that someone would only now, after quite some time of 10.0 being at its end-of-life, upgrade straight from 10.0.10 to the newest 10.2, 10.3, 10.4, 10.5, or later release. Hence, it should be acceptable to remove some logic around DICT_TF2_FTS_AUX_HEX_NAME in an effort to speed up the startup.
The first step of my previous comment was not obvious even to me when I was re-reading it now. Yes, there is no dict_table_t::id or dict_index_t::id available in fil_space_t. But, those ID are encoded in the fil_space_t::name of the FTS_*.ibd tablespace! So, we can collect them from there. The last step of constructing the table name from the pair of (TABLE_ID, ID) should now be obvious as well (just invoke snprintf()).
|
|
The source tree origin/bb-10.2-MDEV-18867 ce05ecd7207e56dd7c260e7471873ef62b38259e 2020-08-24T15:54:43+05:30
behaved well during RQG testing.
|
|
I think that the fix would be cleaner if all new code were added in one compilation unit if possible. I sent a few review comments.
|
|
The revised version looks much better. I only have a couple minor requests:
- Please try to avoid adding code to outside the fts modules. Specifically, try to avoid any changes to storage/innobase/row/row0mysql.cc and storage/innobase/include/dict0mem.h.
- In fts_check_orphaned_tables(), add if (len!=8) continue for both occurrences.
|
|
In MariaDB Server 10.6, MDEV-18518 removes the function fts_drop_orphaned_tables() altogether, because the DDL operations will be atomic also with respect to the internal tables for the fulltext indexes.
|
gbd.txt
my.cnf