Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.6.16
-
Redhat 4.18.0-553.16.1.el8_10.x86_64
Openstack VM
Description
The mariadb server hung when dumping data using mysqldump:
root 248699 248698 0 Sep23 ? 00:00:00 /opt/LU3P/bin/mysqldump --defaults-file=/etc/opt/LU3Pmysql/webnms/my.cnf --force --single-transaction WebNmsDB MediaPlaneElement UserPreferences pm_dashboard_data_table ngfm_AlarmSeverity pmchartconfigs ngfm_OssFilter pm_media_attributs_list_table ngfm_ProbableCauseText pm_media_collection_table ngfm_Comment OAMManagedElement pm_parse_history ngfm_AlarmFilter ngfm_HistoricalComment ngfm_AdvancedFilter pm_media_job_table -uEUbtiVBaRowtEYT1 -px xxxxxxxxxxxxxx
|
|
|
mysql 3725 2893 0 Sep23 ? 00:09:28 /opt/LU3P/sbin/mariadbd --defaults-file=/etc/opt/LU3Pmysql/webnms/my.cnf --basedir=/opt/LU3P --datadir=/var/lib/db/webnms --plugin-dir=/opt/LU3P/lib/mysql/plugin --user=mysql --log-error=/var/opt/log/mysql/webnms/mysql.err --pid-file=/var/opt/run/mysql/webnms/mysql.pid --socket=/var/opt/run/mysql/webnms/mysql.sock --port=3306
|
Any connection made to the MariaDB server just stuck there without return. Logging seems also blocked as no error log printed since the hang. And it still hangs as of now (for >24h).
Attached the config file my.cnf and the thread dump gcore_threads_dump.txt from the core file that is generated by gcore cmd against the hung MariaDB instance.
Let me know if more info is needed.
Attachments
Activity
Thank you.
The debugging package may not help here since the mariadbd binary of our product is different than the official binary release (it is built from source and stripped). We'll rebuild mariadbd with debugging info and try to reproduce to collect the full stack trace.
It is also noticed that an empty mysql-bin.rec file under datadir persisted for ~2 days. Not sure if this could provide any clue for the cause of hang.
Meanwhile, would you please advise how to detect mariadbd daemon hang and measures to recover with minimal impact to production data?
you can connect periodically and kill mariadbd if it fails, for example.
did you manage to get a more usable stack trace?
Thank you for the report. Unfortunately, the stack traces are incomplete, especially for Thread 16. Can you please follow https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ and install an additional package that contains the necessary debug information for producing complete stack traces, and extract the stack traces from the core dump again?