Bonus tip: Looks like is always happening approximately after 1 hour and 30 minutes after the service is started.

Novkovski, I would like to see the stack traces of all threads when this happens.

gdb -ex "set pagination 0" -ex "thread apply all backtrace" --batch -p $(pgrep -x mysqld)

I do not have any idea what could cause this to happen 1 hour and 30 minutes after the service has been started. Could it be due to some external monitoring or maintenance activity? In MDEV-13983 we have something similar, possibly related to InnoDB deadlock detection or LOCK_thread_count that it is being acquired in order to dump information about a conflicting transaction.

It happened again after approx. 1 hour and 30-40 minutes.
Here is the output: gdb

Has gdb been sanitized in some way, or are the debugging symbols missing? I do not see any parameters to function calls, or source code line numbers.

It would be helpful if you could save a core dump that you could analyze based on commands provided by me. We need to find out which thread is holding dict_sys->mutex. That should be relatively easy: print/x *dict_sys->mutex should reveal the thread identifier in hexadecimal. You can use thread find 0x… to find the thread. Finally, switch to that thread and issue backtrace. But, I would probably likely still need a complete backtrace of all threads.

Note that you can upload any confidential files to ftp.mariadb.com.

I`m not so Linux technical person so please guide me whatever I need to make.
It happened again and once MariaDB waits for some locks, the whole server does not work and mysql_error.log files get filled.
I have temporary set up a cron job to restart MariaDB every hour witch fixes the issue.

Isnt the latest change a fix to this issue?
https://github.com/MariaDB/server/commit/8a79fa0e4d0385818da056f7a4a39fde95d62fe3

Novkovski, thank you for noticing. Yes, your gdb output suggests that your report could be a duplicate of MDEV-19529.

There are many problems with the InnoDB fulltext search implementation, and there are not many useful regression tests. We also have some other fixes in the works that have not gone through stress tests (or code review) yet.

It is different issue from MDEV-19529 because it dealt with alter and fts_optimize_thread wait. But this one deals with srv_master_thread and fts_optimize_thread wait.
Will work on it. Thanks for raising the report Novkovski

This is a welcome step to the right direction, but I think that this needs some more work.

First of all, the in_queue should not be stored in a bit-field that is shared with other bit-fields that are protected by a different mutex.

I would suggest to use bool, and to document the possible state transitions carefully. We might consider using atomic memory access.

Second, in 10.1, fts_optimize_init() is not adding tables to the queue, while in 10.2 it is doing that. I’d like to see a 10.1 patch that does this. It should also avoid the unnecessary use of std::vector.

Third, fts_optimize_remove_table() should assert !table->fts->in_queue in the end.

At the end of fts_optimize_remove_table(), the fts_optimize_wq->mutex acquisition and release around the debug assertion should be inside ut_d(), to avoid unnecessary operations on the release build.

I saw a redundant sync_table = mem_heap_alloc(…) call whose result was immediately overwritten by {{sync_table=table;}

In fts_optimize_new_table() the assignment slot->running = false is redundant because of a preceding memset() call.

If fts_slots can be accessed by multiple threads, then we should extend some mutex hold time. It could be that it is only being accessed by a single thread.

Should we call fts_init_index() already on ha_innobase::open()? Otherwise, it seems that FTS-indexed columns could be updated before any fulltext search is performed (and ha_innobase::ft_init_ext() is called). Could that lead to some updates being missed by the fulltext indexes?

Finally, please check the following for differences in white-space or comments, and try to fix those:

diff -I^@@ <(git show origin/bb-10.1-thiru storage/innobase) <(git show origin/bb-10.1-thiru storage/xtradb/)

git show origin/bb-10.2-thiru|diff -^@@ - <(git show origin/bb-10.1-thiru storage/innobase)

Thanks, this looks OK. I made a suggestion to declare fts_optimize_wq) without static scope, to avoid having to add trivial non-inline accessor functions.