[MDEV-32187] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch Created: 2023-09-16 Updated: 2023-11-04 |
|
| Status: | Open |
| Project: | MariaDB Server |
| Component/s: | Server |
| Affects Version/s: | 11.1.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Olivier LEVILLAIN | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | bug | ||
| Environment: |
Oracle Cloud Infra, VM.Standard.E2.1.Micro with Oracle Linux 8, MariaDB running in Docker compose with:
|
||
| Issue Links: |
|
||||||||
| Description |
|
I get this crash every once in a while (~4 to 5 days). I can't find find the core dump in my folder. Version: '11.1.2-MariaDB-1:11.1.2+maria~ubu2204' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help Server version: 11.1.2-MariaDB-1:11.1.2+maria~ubu2204 source revision: 9bc25d98209df6810f7a7d5e7dd3ae677a313ab5 Thread pointer: 0x0 Kernel version: Linux version 5.15.0-103.114.4.el8uek.x86_64 (mockbuild@host-100-100-224-28) (gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9.1.0.3), GNU ld version 2.36.1-2.0.1.el8) #2 SMP Mon Jun 26 10:13:01 |
| Comments |
| Comment by Marko Mäkelä [ 2023-09-16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can you please provide stack traces for all threads, following the instructions that the fatal error message linked to? You can see an example of that in | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-09-16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Ok @marko | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-09-21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Since I changed the image for the debug one, I have had no crash but the instance started correctly and for 3 days everything went fine but after 3 days it began restarting again and again (every minute or so) with each time almost the same sequence: 2023-09-21 09:03:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.2+maria~ubu2204 started. I don't find any dump in the image although I changed the settings. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2023-09-21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
danblack, can you please assist leolivier to provide stack traces from the hang? Based on the messages from the subsequent start-up, the server was not particularly busy when it hung: less than a megabyte of redo log written since the latest checkpoint, and only 40 pages had to be recovered. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Black [ 2023-09-22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
First:
This should make the current running kernel save a core file, maybe in a datadir volume. try running compose file like:
I'm not sure why you aren't seeing a shutdown. My usual thoughts with lack of output are OOM. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-09-23 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
My DB was totally crashed so I restarted with a backup I had and with the normal mariadb image and it worked (I didn't wait enoigh to see if I experimented a new crash as it may take several days)
In the database folder I have these files:
So, only the file aria_log.00000001 has been changed today... I still don't find any dump (or are they stored somewhere else?). EDIT: I just noticed I didn't use the user: mysql line in the compose file, does it change something? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-09-30 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
So my database worked well for several days and then on the 27th, it started again to crash. aria_log.00000001 backup-my.cnf eklectikDB ibtmp1 multi-master.info system_mysql_backup_unknown_version.sql.zst and as the container keeps restarting again and again, I can't connect to it and try to see if there are core files in other folders... I'm a bit desperate :/ In the logs, I can see a lot of lines like this: New Thread 0x7f75ed7fe640 (LWP 97778)] then at some point, new thraeds are created, but they never exits: and finally 2023-09-27 2:48:47 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections) and finally, these messages disappears and I get: [Thread 0x7f75d77fe640 (LWP 99849) exited] For help, type "help". For help, type "help". and after that it loops restarting with the gdb startup message I really don't know what to do. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-10-06 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I again reused my backup and ran the "normal" image without debug and experienced again some issues after a few days but the traces in the logs are different: 2023-09-30 10:02:19+00:00 [Note] [Entrypoint]: Temporary server stopped 2023-10-01 07:57:44+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.1.2+maria~ubu2204 started. ==> | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier LEVILLAIN [ 2023-11-04 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fyi, as I was totally unable to get the database working properly and I didn't find anyway to provide you with a log file, I gave up on the free OCI account and went back to my original install on a simple raspberry pi 4 (on which everything works very fine with exactly the same database files so this is platform related, not to the database content itself) |