Ubuntu 20.10 minimal no gui

Do you have tables with indexed virtual columns?

I do not use virtual columns.

Can you run something like

gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mariadbd 12345 > /tmp/mariadbd.trace 

replacing 12345 with the actual mariadbd pid, of course

you might need to install mariadb*dbgsym* packages to get a meaningful output

I uploaded the trace file: mariadbd.trace here

sysprg, what do you take from it?

Is there any movement on this? This happens on CentOS 8 and Ubuntu 20.10

Usually at this point (judging by the log) we have a fully completed wsrep deinitialization and a transition to innodb deinitialization. However, here I do not see the line "[Note] InnoDB: FTS optimize thread exiting." in the log. Question to jason1430 - do I understand correctly that "mariadb.log" is the full log of the server? Or is it just a snippet that refers to wsrep, without other lines? Question to marko - can you tell me (based on "mariadbd.trace") if we got into innodb deinitialization in this case, or not? If not, then server probably got stuck in the wsrep deinitialization, which did not completed correctly.

Yes, it is the full log.

I had a node successfully stop. Here is the resulting log entry:

Normal Shutdown:
2021-06-10 18:08:51 0 [Note] WSREP: Flushing memory map to disk...
2021-06-10 18:08:51 0 [Note] InnoDB: FTS optimize thread exiting.
2021-06-10 18:08:51 0 [Note] InnoDB: Starting shutdown...
2021-06-10 18:08:51 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-06-10 18:08:51 0 [Note] InnoDB: Buffer pool(s) dump completed at 210610 18:08:51
2021-06-10 18:08:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-06-10 18:08:51 0 [Note] InnoDB: Shutdown completed; log sequence number 571364; transaction id 1173
2021-06-10 18:08:51 0 [Note] /usr/sbin/mariadbd: Shutdown complete

This is where it stops when it does not stop correctly:
Abnormal shutdown stops at:
WSREP: Flushing memory map to disk...

jason1430 thanks, now we must further study the issue to understand this hang in wsrep or wsrep managed to complete deinitialization, but then hangs occurs in the FTS processing (in innodb)

I do not see any occurrence of storage/innobase in mariadbd.trace. But, on a closer look, Thread 3 is executing in pthread_cond_timedwait(), with two ?? functions above it in the call stack. That might be the buf_flush_page_cleaner() thread. Also Thread 5 might be that thread, ?? is calling pthread_cond_wait(). Thread 4 appears to be the io_getevents() handler. So, clearly the InnoDB I/O subsystem is still up and running.

Thread 6 doesn’t look like InnoDB: some ?? is calling signal_hand(). Other threads seem to be an idle thread pool handler, and something Galera related.

It seems that InnoDB is at a very late phase of shutdown. Possibly Thread 4 is executing a sleep in logs_empty_and_mark_files_at_shutdown(). Where is the server error log? mariadb.log appears to be something else. The function logs_empty_and_mark_files_at_shutdown() should output the reason that is blocking the shutdown?

Could the parameter innodb_disallow_writes be a culprit for this? It is used by some Galera scripts, and the implementation is in my opinion misplaced: blocking the writes at the low level, instead of blocking them at the high level (blocking any operation that would generate redo log).

Finally, are there any InnoDB tables with FULLTEXT INDEX? I suspect that they cannot work correctly with Galera.

I can get you any log you need.

These servers have no additional databases on them. They have the default DBs. I'm not using this cluster because I need it to be stable.