Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.3.15, 10.4.4, 10.4.5
-
Ubuntu 18.04
Description
Hello, each time when RocksDB not closed correctly or not enough space on system.
we go a message like : "RocksDB is unknow storage on server". and most of time have to clean everything manually and uninstall plugin and reinstall.
this time,(maybe a new improvement I got) :
2019-06-11 19:53:24 853537 [Warning] mysqld: Disk is full writing '/mysql/binlog/mariadb-bin.000117' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) |
2019-06-11 19:53:24 853537 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs |
2019-06-11 19:53:41 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:53:41 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 19:53:41 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:54:24 853540 [ERROR] MyRocks: aborting on WAL write error. |
190611 19:54:24 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
Server version: 10.4.5-MariaDB-1:10.4.5+maria~bionic-log |
key_buffer_size=134217728 |
read_buffer_size=2097152 |
max_used_connections=21 |
max_threads=102 |
thread_count=49 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760240 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x7f48891df008 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x7f4892978d68 thread_stack 0x49000 |
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5605648ba2ce] |
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x560564335275] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f48faedc890] |
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f48fa1ece97] |
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f48fa1ee801] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks19rdb_handle_io_errorEN7rocksdb6StatusENS_17RDB_IO_ERROR_TYPEE+0x9c)[0x7f48881e755c] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks20Rdb_transaction_impl16commit_no_binlogEv+0x189)[0x7f4888202669] |
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x1ec337)[0x7f48881e3337] |
/usr/sbin/mysqld(_ZN6TC_LOG18run_commit_orderedEP3THDb+0x3d)[0x560564416a7d] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x3a9)[0x56056441be89] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x21b)[0x56056441c3bb] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0x1d7)[0x56056441c7e7] |
/usr/sbin/mysqld(+0x9388bf)[0x56056441c8bf] |
/usr/sbin/mysqld(+0x9394a2)[0x56056441d4a2] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13log_and_orderEP3THDybbb+0x44)[0x56056441d654] |
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0xb72)[0x560564343632] |
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x3f)[0x56056421bb6f] |
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xaad)[0x56056412763d] |
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22a)[0x56056412fcca] |
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x166d)[0x56056413248d] |
/usr/sbin/mysqld(_Z10do_commandP3THD+0x148)[0x5605641338f8] |
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x2c2)[0x56056420d4c2] |
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x56056420d58d] |
/usr/sbin/mysqld(+0xd8323a)[0x56056486723a] |
nptl/pthread_create.c:463(start_thread)[0x7f48faed16db] |
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f48fa2cf88f] |
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f48892171d0): UPDATE `mysql_server` SET `error` ='unserialize(): Error at offset 20480 of 20480 bytes', `date_refresh` = '2019-06-11 19:53:24', `is_available` = 0 WHERE id =48 |
Connection ID (thread ID): 853540 |
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on |
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains |
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /mysql/data
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes |
Max core file size 0 unlimited bytes |
Max resident set unlimited unlimited bytes
|
Max processes 63340 63340 processes |
Max open files 16364 16364 files |
Max locked memory 16777216 16777216 bytes |
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 63340 63340 signals |
Max msgqueue size 819200 819200 bytes |
Max nice priority 0 0 |
Max realtime priority 0 0 |
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/share/apport/apport %p %s %c %d %P
|
|
2019-06-11 19:54:32 0 [Note] mysqld: Aria engine: starting recovery |
recovered pages: 0% 41% 57% 72% 86% 100% (0.0 seconds); tables to flush: 2 1 0 (0.0 seconds); |
2019-06-11 19:54:32 0 [Note] mysqld: Aria engine: recovery done |
2019-06-11 19:54:32 0 [Note] InnoDB: Using Linux native AIO |
2019-06-11 19:54:32 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2019-06-11 19:54:32 0 [Note] InnoDB: Uses event mutexes |
2019-06-11 19:54:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 |
2019-06-11 19:54:32 0 [Note] InnoDB: Number of pools: 1 |
2019-06-11 19:54:32 0 [Note] InnoDB: Using SSE2 crc32 instructions |
2019-06-11 19:54:32 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) |
2019-06-11 19:54:32 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M |
2019-06-11 19:54:32 0 [Note] InnoDB: Completed initialization of buffer pool |
2019-06-11 19:54:32 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
2019-06-11 19:54:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=15154203349 |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309429 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309430 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309431 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309440 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309426 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: Transaction 4309427 was in the XA prepared state. |
2019-06-11 19:54:32 0 [Note] InnoDB: 6 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo |
2019-06-11 19:54:32 0 [Note] InnoDB: Trx id counter is 4309447 |
2019-06-11 19:54:32 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. |
2019-06-11 19:54:33 0 [Note] InnoDB: Last binlog file '/mysql/binlog/mariadb-bin.000117', position 558788296 |
2019-06-11 19:54:33 0 [Note] InnoDB: 128 out of 128 rollback segments are active. |
2019-06-11 19:54:33 0 [Note] InnoDB: Starting in background the rollback of recovered transactions |
2019-06-11 19:54:33 0 [Note] InnoDB: Rollback of non-prepared transactions completed |
2019-06-11 19:54:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" |
2019-06-11 19:54:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables |
2019-06-11 19:54:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... |
2019-06-11 19:54:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. |
2019-06-11 19:54:33 0 [Note] InnoDB: Waiting for purge to start |
2019-06-11 19:54:33 0 [Note] InnoDB: 10.4.5 started; log sequence number 15154204735; transaction id 4309449 |
2019-06-11 19:54:33 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/data/ib_buffer_pool |
2019-06-11 19:54:33 0 [Note] Plugin 'FEEDBACK' is disabled. |
2019-06-11 19:54:33 0 [Note] RocksDB: 2 column families found |
2019-06-11 19:54:33 0 [Note] RocksDB: Column Families at start: |
2019-06-11 19:54:33 0 [Note] cf=default |
2019-06-11 19:54:33 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:54:33 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:54:33 0 [Note] cf=__system__ |
2019-06-11 19:54:33 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:54:33 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:54:34 0 [Note] InnoDB: Buffer pool(s) load completed at 190611 19:54:34 |
2019-06-11 19:54:46 0 [Note] RocksDB: Table_store: loaded DDL data for 30 tables |
2019-06-11 19:54:46 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
2019-06-11 19:54:46 0 [Note] MyRocks storage engine plugin has been successfully initialized. |
2019-06-11 19:54:46 0 [Note] Recovering after a crash using /mysql/binlog/mariadb-bin |
2019-06-11 19:54:46 0 [ERROR] Error in Log_event::read_log_event(): 'Event truncated', data_len: 12742, event_type: 23 |
2019-06-11 19:54:46 0 [Note] Starting crash recovery... |
2019-06-11 19:54:46 0 [Note] InnoDB: Starting recovery for XA transactions... |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309440 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309426 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309430 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309429 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 6 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309431 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction 4309427 in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:54:46 0 [Note] InnoDB: 6 transactions in prepared state after recovery |
2019-06-11 19:54:46 0 [Note] Found 6 prepared transaction(s) in InnoDB |
2019-06-11 19:54:46 0 [Note] Found 1 prepared transaction(s) in ROCKSDB |
2019-06-11 19:54:46 0 [Note] Crash recovery finished. |
2019-06-11 19:54:46 0 [Note] Server socket created on IP: '0.0.0.0'. |
2019-06-11 19:54:46 0 [Warning] 'proxies_priv' entry '@% root@aurelien-rdc' ignored in --skip-name-resolve mode. |
2019-06-11 19:54:46 27 [Note] Event Scheduler: scheduler thread started with id 27 |
2019-06-11 19:54:46 0 [Note] Reading of all Master_info entries succeded |
2019-06-11 19:54:46 0 [Note] Added new Master_info '' to hash table |
2019-06-11 19:54:46 0 [Note] /usr/sbin/mysqld: ready for connections. |
Version: '10.4.5-MariaDB-1:10.4.5+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2019-06-11 19:56:04 1469 [ERROR] mysqld: Error writing file '/mysql/binlog/mariadb-bin' (errno: 28 "No space left on device") |
2019-06-11 19:56:05 1495 [Warning] mysqld: Disk is full writing '/mysql/binlog/mariadb-bin.000118' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) |
2019-06-11 19:56:05 1495 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs |
2019-06-11 19:56:24 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008037.sst: No space left on device |
2019-06-11 19:56:24 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 19:56:24 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008037.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 19:57:05 1494 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008037.sst: No space left on device |
2019-06-11 19:57:05 1494 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 19:57:05 1494 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008037.sst: No space left on device |
2019-06-11 19:57:05 1494 [ERROR] MyRocks: aborting on WAL write error. |
190611 19:57:05 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
Server version: 10.4.5-MariaDB-1:10.4.5+maria~bionic-log |
key_buffer_size=134217728 |
read_buffer_size=2097152 |
max_used_connections=19 |
max_threads=102 |
thread_count=47 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760240 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x7f8523c9e008 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x7f856915fd68 thread_stack 0x49000 |
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56307e7042ce] |
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x56307e17f275] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f858571a890] |
linux/raise.c:51(__GI_raise)[0x7f8584a2ae97] |
stdlib/abort.c:81(__GI_abort)[0x7f8584a2c801] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks19rdb_handle_io_errorEN7rocksdb6StatusENS_17RDB_IO_ERROR_TYPEE+0x9c)[0x7f85125e755c] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks20Rdb_transaction_impl16commit_no_binlogEv+0x189)[0x7f8512602669] |
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x1ec337)[0x7f85125e3337] |
/usr/sbin/mysqld(_ZN6TC_LOG18run_commit_orderedEP3THDb+0x3d)[0x56307e260a7d] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x3a9)[0x56307e265e89] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x21b)[0x56307e2663bb] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0x1d7)[0x56307e2667e7] |
/usr/sbin/mysqld(+0x9388bf)[0x56307e2668bf] |
/usr/sbin/mysqld(+0x9394a2)[0x56307e2674a2] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13log_and_orderEP3THDybbb+0x44)[0x56307e267654] |
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0xb72)[0x56307e18d632] |
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x3f)[0x56307e065b6f] |
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xaad)[0x56307df7163d] |
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22a)[0x56307df79cca] |
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x166d)[0x56307df7c48d] |
/usr/sbin/mysqld(_Z10do_commandP3THD+0x148)[0x56307df7d8f8] |
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x2c2)[0x56307e0574c2] |
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x56307e05758d] |
/usr/sbin/mysqld(+0xd8323a)[0x56307e6b123a] |
nptl/pthread_create.c:463(start_thread)[0x7f858570f6db] |
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8584b0d88f] |
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f8523ccd1d0): UPDATE `mysql_server` SET `error` ='mysqli_connect(): (HY000/2002): No route to host', `date_refresh` = '2019-06-11 19:56:05', `is_available` = 0 WHERE id =27 |
Connection ID (thread ID): 1494 |
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on |
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains |
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /mysql/data
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes |
Max core file size 0 unlimited bytes |
Max resident set unlimited unlimited bytes
|
Max processes 63340 63340 processes |
Max open files 16364 16364 files |
Max locked memory 16777216 16777216 bytes |
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 63340 63340 signals |
Max msgqueue size 819200 819200 bytes |
Max nice priority 0 0 |
Max realtime priority 0 0 |
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/share/apport/apport %p %s %c %d %P
|
|
2019-06-11 19:57:12 0 [Note] InnoDB: Using Linux native AIO |
2019-06-11 19:57:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2019-06-11 19:57:12 0 [Note] InnoDB: Uses event mutexes |
2019-06-11 19:57:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 |
2019-06-11 19:57:12 0 [Note] InnoDB: Number of pools: 1 |
2019-06-11 19:57:12 0 [Note] InnoDB: Using SSE2 crc32 instructions |
2019-06-11 19:57:12 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) |
2019-06-11 19:57:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M |
2019-06-11 19:57:12 0 [Note] InnoDB: Completed initialization of buffer pool |
2019-06-11 19:57:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
2019-06-11 19:57:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=15157567605 |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310432 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310433 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310434 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310443 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310455 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: Transaction 4310456 was in the XA prepared state. |
2019-06-11 19:57:12 0 [Note] InnoDB: 6 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo |
2019-06-11 19:57:12 0 [Note] InnoDB: Trx id counter is 4310457 |
2019-06-11 19:57:12 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. |
2019-06-11 19:57:13 0 [Note] InnoDB: Last binlog file '/mysql/binlog/mariadb-bin.000118', position 44886862 |
2019-06-11 19:57:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. |
2019-06-11 19:57:13 0 [Note] InnoDB: Starting in background the rollback of recovered transactions |
2019-06-11 19:57:13 0 [Note] InnoDB: Rollback of non-prepared transactions completed |
2019-06-11 19:57:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" |
2019-06-11 19:57:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables |
2019-06-11 19:57:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... |
2019-06-11 19:57:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. |
2019-06-11 19:57:13 0 [Note] InnoDB: Waiting for purge to start |
2019-06-11 19:57:13 0 [Note] InnoDB: 10.4.5 started; log sequence number 15157569056; transaction id 4310459 |
2019-06-11 19:57:13 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/data/ib_buffer_pool |
2019-06-11 19:57:13 0 [Note] Plugin 'FEEDBACK' is disabled. |
2019-06-11 19:57:13 0 [Note] RocksDB: 2 column families found |
2019-06-11 19:57:13 0 [Note] RocksDB: Column Families at start: |
2019-06-11 19:57:13 0 [Note] cf=default |
2019-06-11 19:57:13 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:57:13 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:57:13 0 [Note] cf=__system__ |
2019-06-11 19:57:13 0 [Note] cf=__system__ |
2019-06-11 19:57:13 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:57:13 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:57:14 0 [Note] InnoDB: Buffer pool(s) load completed at 190611 19:57:14 |
2019-06-11 19:57:24 0 [Note] RocksDB: Table_store: loaded DDL data for 30 tables |
2019-06-11 19:57:24 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
2019-06-11 19:57:24 0 [Note] MyRocks storage engine plugin has been successfully initialized. |
2019-06-11 19:57:24 0 [Note] Recovering after a crash using /mysql/binlog/mariadb-bin |
2019-06-11 19:57:25 0 [Note] Starting crash recovery... |
2019-06-11 19:57:25 0 [Note] InnoDB: Starting recovery for XA transactions... |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310434 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310432 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310456 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310443 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 10 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310455 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction 4310433 in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:57:25 0 [Note] InnoDB: 6 transactions in prepared state after recovery |
2019-06-11 19:57:25 0 [Note] Found 6 prepared transaction(s) in InnoDB |
2019-06-11 19:57:25 0 [Note] Found 1 prepared transaction(s) in ROCKSDB |
2019-06-11 19:57:25 0 [Note] Crash recovery finished. |
2019-06-11 19:57:25 0 [Note] Server socket created on IP: '0.0.0.0'. |
2019-06-11 19:57:25 0 [Warning] 'proxies_priv' entry '@% root@aurelien-rdc' ignored in --skip-name-resolve mode. |
2019-06-11 19:57:25 27 [Note] Event Scheduler: scheduler thread started with id 27 |
2019-06-11 19:57:25 0 [Note] Reading of all Master_info entries succeded |
2019-06-11 19:57:25 0 [Note] Added new Master_info '' to hash table |
2019-06-11 19:57:25 0 [Note] /usr/sbin/mysqld: ready for connections. |
Version: '10.4.5-MariaDB-1:10.4.5+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2019-06-11 19:58:26 823 [Warning] mysqld: Disk is full writing '/mysql/binlog/mariadb-bin.000119' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) |
2019-06-11 19:58:26 823 [Warning] mysqld: Retry in 60 secs. Message reprinted2019-06-11 19:58:41 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008045.sst: No space left on device |
2019-06-11 19:58:41 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 19:58:41 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008045.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 19:59:26 824 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008045.sst: No space left on device |
2019-06-11 19:59:26 824 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 19:59:26 824 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008045.sst: No space left on device |
2019-06-11 19:59:26 824 [ERROR] MyRocks: aborting on WAL write error. |
190611 19:59:26 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
Server version: 10.4.5-MariaDB-1:10.4.5+maria~bionic-log |
key_buffer_size=134217728 |
read_buffer_size=2097152 |
max_used_connections=20 |
max_threads=102 |
thread_count=48 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760240 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x7f8ee3c96008 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x7f8ef6499d68 thread_stack 0x49000 |
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56092509e2ce] |
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x560924b19275] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f8f00729890] |
linux/raise.c:51(__GI_raise)[0x7f8effa39e97] |
stdlib/abort.c:81(__GI_abort)[0x7f8effa3b801] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks19rdb_handle_io_errorEN7rocksdb6StatusENS_17RDB_IO_ERROR_TYPEE+0x9c)[0x7f8e8d1e755c] |
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks20Rdb_transaction_impl16commit_no_binlogEv+0x189)[0x7f8e8d202669] |
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x1ec337)[0x7f8e8d1e3337] |
/usr/sbin/mysqld(_ZN6TC_LOG18run_commit_orderedEP3THDb+0x3d)[0x560924bfaa7d] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x3a9)[0x560924bffe89] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x21b)[0x560924c003bb] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0x1d7)[0x560924c007e7] |
/usr/sbin/mysqld(+0x9388bf)[0x560924c008bf] |
/usr/sbin/mysqld(+0x9394a2)[0x560924c014a2] |
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13log_and_orderEP3THDybbb+0x44)[0x560924c01654] |
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0xb72)[0x560924b27632] |
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x3f)[0x5609249ffb6f] |
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xaad)[0x56092490b63d] |
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22a)[0x560924913cca] |
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x166d)[0x56092491648d] |
/usr/sbin/mysqld(_Z10do_commandP3THD+0x148)[0x5609249178f8] |
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x2c2)[0x5609249f14c2] |
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x5609249f158d] |
/usr/sbin/mysqld(+0xd8323a)[0x56092504b23a] |
nptl/pthread_create.c:463(start_thread)[0x7f8f0071e6db] |
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8effb1c88f] |
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f8ee3cc51d0): UPDATE `mysql_server` SET `error` ='mysqli_connect(): (HY000/2002): No route to host', `date_refresh` = '2019-06-11 19:58:26', `is_available` = 0 WHERE id =57 |
Connection ID (thread ID): 824 |
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on |
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains |
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /mysql/data
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes |
Max core file size 0 unlimited bytes |
Max resident set unlimited unlimited bytes
|
Max processes 63340 63340 processes |
Max open files 16364 16364 files |
Max locked memory 16777216 16777216 bytes |
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 63340 63340 signals |
Max msgqueue size 819200 819200 bytes |
Max nice priority 0 0 |
Max realtime priority 0 0 |
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/share/apport/apport %p %s %c %d %P
|
|
2019-06-11 19:59:33 0 [Note] InnoDB: Using Linux native AIO |
2019-06-11 19:59:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2019-06-11 19:59:33 0 [Note] InnoDB: Uses event mutexes |
2019-06-11 19:59:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 |
2019-06-11 19:59:33 0 [Note] InnoDB: Number of pools: 1 |
2019-06-11 19:59:33 0 [Note] InnoDB: Using SSE2 crc32 instructions |
2019-06-11 19:59:33 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) |
2019-06-11 19:59:33 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M |
2019-06-11 19:59:33 0 [Note] InnoDB: Completed initialization of buffer pool |
2019-06-11 19:59:33 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
2019-06-11 19:59:33 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=15158991006 |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310882 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310883 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310884 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310885 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310886 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310887 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310888 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310889 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: Transaction 4310890 was in the XA prepared state. |
2019-06-11 19:59:33 0 [Note] InnoDB: 9 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo |
2019-06-11 19:59:33 0 [Note] InnoDB: Trx id counter is 4310905 |
2019-06-11 19:59:33 0 [Note] InnoDB: Last binlog file '/mysql/binlog/mariadb-bin.000119', position 22886892 |
2019-06-11 19:59:33 0 [Note] InnoDB: 128 out of 128 rollback segments are active. |
2019-06-11 19:59:33 0 [Note] InnoDB: Starting in background the rollback of recovered transactions |
2019-06-11 19:59:33 0 [Note] InnoDB: Rollback of non-prepared transactions completed |
2019-06-11 19:59:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" |
2019-06-11 19:59:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables |
2019-06-11 19:59:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... |
2019-06-11 19:59:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. |
2019-06-11 19:59:33 0 [Note] InnoDB: Waiting for purge to start |
2019-06-11 19:59:33 0 [Note] InnoDB: 10.4.5 started; log sequence number 15158991217; transaction id 4310907 |
2019-06-11 19:59:33 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/data/ib_buffer_pool |
2019-06-11 19:59:33 0 [Note] Plugin 'FEEDBACK' is disabled. |
2019-06-11 19:59:33 0 [Note] RocksDB: 2 column families found |
2019-06-11 19:59:33 0 [Note] RocksDB: Column Families at start: |
2019-06-11 19:59:33 0 [Note] cf=default |
2019-06-11 19:59:33 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:59:33 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:59:33 0 [Note] cf=__system__ |
2019-06-11 19:59:33 0 [Note] write_buffer_size=67108864 |
2019-06-11 19:59:33 0 [Note] target_file_size_base=67108864 |
2019-06-11 19:59:34 0 [Note] InnoDB: Buffer pool(s) load completed at 190611 19:59:34 |
2019-06-11 19:59:43 0 [Note] RocksDB: Table_store: loaded DDL data for 30 tables |
2019-06-11 19:59:43 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
2019-06-11 19:59:43 0 [Note] MyRocks storage engine plugin has been successfully initialized. |
2019-06-11 19:59:43 0 [Note] Recovering after a crash using /mysql/binlog/mariadb-bin |
2019-06-11 19:59:43 0 [Note] Starting crash recovery... |
2019-06-11 19:59:43 0 [Note] InnoDB: Starting recovery for XA transactions... |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310884 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 10 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310886 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310890 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310882 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310885 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310889 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310883 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310888 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction 4310887 in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] InnoDB: Transaction contains changes to 1 rows |
2019-06-11 19:59:43 0 [Note] InnoDB: 9 transactions in prepared state after recovery |
2019-06-11 19:59:43 0 [Note] Found 9 prepared transaction(s) in InnoDB |
2019-06-11 19:59:43 0 [Note] Found 1 prepared transaction(s) in ROCKSDB |
2019-06-11 19:59:43 0 [Note] Crash recovery finished. |
2019-06-11 19:59:43 0 [Note] Server socket created on IP: '0.0.0.0'. |
2019-06-11 19:59:43 0 [Warning] 'proxies_priv' entry '@% root@aurelien-rdc' ignored in --skip-name-resolve mode. |
2019-06-11 19:59:43 27 [Note] Event Scheduler: scheduler thread started with id 27 |
2019-06-11 19:59:43 0 [Note] Reading of all Master_info entries succeded |
2019-06-11 19:59:43 0 [Note] Added new Master_info '' to hash table |
2019-06-11 19:59:43 0 [Note] /usr/sbin/mysqld: ready for connections. |
Version: '10.4.5-MariaDB-1:10.4.5+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2019-06-11 20:00:46 1281 [Warning] mysqld: Disk is full writing '/mysql/binlog/mariadb-bin.000120' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) |
2019-06-11 20:00:46 1281 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs |
2019-06-11 20:01:07 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008054.sst: No space left on device |
2019-06-11 20:01:07 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 20:01:07 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008054.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 20:01:08 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008054.sst: No space left on device, Accumulated background error counts: 2 |
2019-06-11 20:01:46 1281 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008054.sst: No space left on device |
2019-06-11 20:01:46 1281 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 20:01:46 1281 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008054.sst: No space left on device |
[...]
|
2019-06-11 20:47:09 0 [Note] InnoDB: Using Linux native AIO |
2019-06-11 20:47:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2019-06-11 20:47:09 0 [Note] InnoDB: Uses event mutexes |
2019-06-11 20:47:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 |
2019-06-11 20:47:09 0 [Note] InnoDB: Number of pools: 1 |
2019-06-11 20:47:09 0 [Note] InnoDB: Using SSE2 crc32 instructions |
2019-06-11 20:47:09 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) |
2019-06-11 20:47:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M |
2019-06-11 20:47:09 0 [Note] InnoDB: Completed initialization of buffer pool |
2019-06-11 20:47:09 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
2019-06-11 20:47:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=15240401077 |
2019-06-11 20:47:09 0 [Note] InnoDB: Starting final batch to recover 26 pages from redo log. |
2019-06-11 20:47:09 0 [Note] InnoDB: Last binlog file '/mysql/binlog/mariadb-bin.000133', position 280935 |
2019-06-11 20:47:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active. |
2019-06-11 20:47:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" |
2019-06-11 20:47:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables |
2019-06-11 20:47:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... |
2019-06-11 20:47:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. |
2019-06-11 20:47:10 0 [Note] InnoDB: Waiting for purge to start |
2019-06-11 20:47:10 0 [Note] InnoDB: 10.4.5 started; log sequence number 15240530718; transaction id 4335162 |
2019-06-11 20:47:10 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/data/ib_buffer_pool |
2019-06-11 20:47:10 0 [Note] Plugin 'FEEDBACK' is disabled. |
2019-06-11 20:47:10 0 [Note] RocksDB: 2 column families found |
2019-06-11 20:47:10 0 [Note] RocksDB: Column Families at start: |
2019-06-11 20:47:10 0 [Note] cf=default |
2019-06-11 20:47:10 0 [Note] write_buffer_size=67108864 |
2019-06-11 20:47:10 0 [Note] target_file_size_base=67108864 |
2019-06-11 20:47:10 0 [Note] cf=__system__ |
2019-06-11 20:47:10 0 [Note] write_buffer_size=67108864 |
2019-06-11 20:47:10 0 [Note] target_file_size_base=67108864 |
2019-06-11 20:47:10 0 [Note] InnoDB: Buffer pool(s) load completed at 190611 20:47:10 |
2019-06-11 20:47:11 0 [Note] RocksDB: Table_store: loaded DDL data for 30 tables |
2019-06-11 20:47:11 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
2019-06-11 20:47:11 0 [Note] MyRocks storage engine plugin has been successfully initialized. |
|
2019-06-11 20:47:11 0 [Note] Recovering after a crash using /mysql/binlog/mariadb-bin |
2019-06-11 20:47:11 0 [Note] Starting crash recovery... |
2019-06-11 20:47:11 0 [Note] Found 2 prepared transaction(s) in ROCKSDB |
2019-06-11 20:47:11 0 [Note] Crash recovery finished. |
2019-06-11 20:47:11 0 [Note] Server socket created on IP: '0.0.0.0'. |
2019-06-11 20:47:11 0 [Warning] 'proxies_priv' entry '@% root@aurelien-rdc' ignored in --skip-name-resolve mode. |
2019-06-11 20:47:11 27 [Note] Event Scheduler: scheduler thread started with id 27 |
2019-06-11 20:47:11 0 [Note] Reading of all Master_info entries succeded |
2019-06-11 20:47:11 0 [Note] Added new Master_info '' to hash table |
2019-06-11 20:47:11 0 [Note] /usr/sbin/mysqld: ready for connections. |
Version: '10.4.5-MariaDB-1:10.4.5+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2019-06-11 20:47:20 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008373.sst: No space left on device |
2019-06-11 20:47:20 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 20:47:20 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008373.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 20:47:21 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008373.sst: No space left on device, Accumulated background error counts: 2 |
2019-06-11 20:48:02 212 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008373.sst: No space left on device |
2019-06-11 20:48:02 212 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 20:48:02 212 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008373.sst: No space left on device |
2019-06-11 20:48:02 212 [ERROR] MyRocks: aborting on WAL write error. |
190611 20:48:02 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
Server version: 10.4.5-MariaDB-1:10.4.5+maria~bionic-log |
key_buffer_size=134217728 |
read_buffer_size=2097152 |
max_used_connections=15 |
max_threads=102 |
thread_count=43 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760240 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x7f26e71b3008 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x7f274e46cd68 thread_stack 0x49000 |
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561e5d5672ce] |
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x561e5cfe2275] |
[...]
|
2019-06-11 10:17:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
2019-06-11 10:17:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active. |
2019-06-11 10:17:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables |
2019-06-11 10:17:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... |
2019-06-11 10:17:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. |
2019-06-11 10:17:21 0 [Note] InnoDB: Waiting for purge to start |
2019-06-11 10:17:21 0 [Note] InnoDB: 10.4.5 started; log sequence number 13266777338; transaction id 3794185 |
2019-06-11 10:17:21 0 [Note] Plugin 'FEEDBACK' is disabled. |
2019-06-11 10:17:21 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/data/ib_buffer_pool |
2019-06-11 10:17:21 0 [Note] RocksDB: 2 column families found |
2019-06-11 10:17:21 0 [Note] RocksDB: Column Families at start: |
2019-06-11 10:17:21 0 [Note] cf=default |
2019-06-11 10:17:21 0 [Note] write_buffer_size=67108864 |
2019-06-11 10:17:21 0 [Note] target_file_size_base=67108864 |
2019-06-11 10:17:21 0 [Note] cf=__system__ |
2019-06-11 10:17:21 0 [Note] write_buffer_size=67108864 |
2019-06-11 10:17:21 0 [Note] target_file_size_base=67108864 |
2019-06-11 10:17:21 0 [Note] RocksDB: Table_store: loaded DDL data for 23 tables |
2019-06-11 10:17:21 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
2019-06-11 10:17:21 0 [Note] MyRocks storage engine plugin has been successfully initialized. |
2019-06-11 10:17:21 0 [Note] Server socket created on IP: '0.0.0.0'. |
2019-06-11 10:17:21 0 [Warning] 'proxies_priv' entry '@% root@aurelien-rdc' ignored in --skip-name-resolve mode. |
2019-06-11 10:17:21 27 [Note] Event Scheduler: scheduler thread started with id 27 |
2019-06-11 10:17:21 0 [Note] Reading of all Master_info entries succeded |
2019-06-11 10:17:21 0 [Note] Added new Master_info '' to hash table |
2019-06-11 10:17:21 0 [Note] /usr/sbin/mysqld: ready for connections. |
Version: '10.4.5-MariaDB-1:10.4.5+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2019-06-11 10:17:22 0 [Note] InnoDB: Buffer pool(s) load completed at 190611 10:17:22 |
2019-06-11 15:09:43 437562 [Warning] Aborted connection 437562 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437567 [Warning] Aborted connection 437567 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437565 [Warning] Aborted connection 437565 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437570 [Warning] Aborted connection 437570 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437569 [Warning] Aborted connection 437569 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437564 [Warning] Aborted connection 437564 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437566 [Warning] Aborted connection 437566 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437571 [Warning] Aborted connection 437571 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 15:09:43 437578 [Warning] Aborted connection 437578 to db: 'pmacontrol' user: 'root' host: '127.0.0.1' (Got an error reading communication packets) |
2019-06-11 19:53:24 853537 [Warning] mysqld: Disk is full writing '/mysql/binlog/mariadb-bin.000117' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) |
2019-06-11 19:53:24 853537 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs |
2019-06-11 19:53:41 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:53:41 0 [ERROR] RocksDB: BackgroundErrorReason: 1 |
2019-06-11 19:53:41 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/mariadb-10.4.5/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device, Accumulated background error counts: 1 |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: BackgroundErrorReason: 2 |
2019-06-11 19:54:24 853540 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/008017.sst: No space left on device |
2019-06-11 19:54:24 853540 [ERROR] MyRocks: aborting on WAL write error. |
190611 19:54:24 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
|
|