Can you shed any light on the naming issue before we start digging into everything else, e.g. crash reasons or correlation with Galera?
I couldn't find the exact crash reason! The only thing I can say, is that we were running a very large insert when the db crashed. The insert was meant to migrate from innodb to tokudb. We're not using Galera!
After the crash, the error "No such file or directory" started showing every time.
The file name is auto generated by tokudb, so nothing we could have done there.
_MYDB_MYTAB_P_day_20140101_key_idx1_9a60c_2_1b_B_3.tokudb
Everything about the name was right, except the post, everything after idx1.
Okay, thanks, I'll remove Galera from the equation (you put it into the Affected version/s list, that's why I added it as a possible factor).
Regarding the file naming, prohaska7 is a TokuDB expert, we'll wait for his response.
The problem reappeared for some reason!
No crash involved this time.
TokuDB is looking for a file that belonged to a table that don't exist anymore.
Here is what seems to be the problem:
Select * FROM information_schema.tokudb_file_map z
WHERE SUBSTRING_INDEX(table_name, '#',1) NOT IN (SELECT table_name FROM information_schema.tables WHERE ENGINE='Tokudb');
It returns a table_name.
Anyway to clean this up?
Looks like the crash left the state of the tokudb.directory to be inconsistent with the tokudb file names. We would be interested in any way to reproduce this problem.
The tokudb.directory is a meta-data fractal tree that contains a mapping from data names (like database/table-type) to the file name that contains the fractal tree. There is no current way to delete these names. However, since the tokudb.directory is a fractal tree, a program could be written that deletes the extraneous rows from it. If you supply your mysql data directory, i can develop such a program.
Here are my dir:
datadir /var/lib/mysql/
tokudb_data_dir /data/mysql/tokudb_data
tokudb_log_dir /data/mysql/tokudb_log
tokudb_tmp_dir /data/mysql/tokudb_tmp
The crash happened when trying a very large insert from innodb->tokudb.
We're talking 600M rows. I take it that this was a very large transaction to be tried in a single command, but it shouldn't have crashed MariaDB/TokuDB.
After MariaDB came back, we started getting the : Got error 2 "No such file or directory"
To work around that, I dropped the TokuDB table, which didn't solve the problem, but seemed to have dropped from mysql metadata.
This may explain why I have the metadata issue.
I can probably reproduce the issue! I did before, but I though it was due to a faulty drive. I can try again, but I'd like to be able to clean it afterward
The mysql error log may have some stack trace information in it when mysqld crashed. Would like to see it.
It had nothing! MySQL process died fast enough that it didn't write anything in it.
And unfortunately, I rotated it already. So I can't send it.
I'll see if I can spawn an AWS instance to reproduce the problem.
I have seen mysqld killed by the OOM killer. When this happens, the error log does NOT contain any crash information. Is there anything in the linux system log about mysqld being killed?
And that was it:
Mar 25 08:38:32 hqm-db-test02 kernel: mysqld invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Mar 25 08:38:32 hqm-db-test02 kernel: mysqld cpuset=/ mems_allowed=0
Mar 25 08:38:32 hqm-db-test02 kernel: Pid: 7850, comm: mysqld Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Mar 25 08:38:32 hqm-db-test02 kernel: Call Trace:
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff81160d6a>] ? alloc_pages_vma+0x9a/0x150
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff81143f0b>] ? handle_pte_fault+0x76b/0xb50
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff810aa43e>] ? futex_wake+0x10e/0x120
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8114452a>] ? handle_mm_fault+0x23a/0x310
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8100bbce>] ? invalidate_interrupt0+0xe/0x20
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8113e017>] ? sys_madvise+0x2d7/0x7b0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8113e090>] ? sys_madvise+0x350/0x7b0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff8100bbce>] ? invalidate_interrupt0+0xe/0x20
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff81513bfe>] ? do_page_fault+0x3e/0xa0
Mar 25 08:38:32 hqm-db-test02 kernel: [<ffffffff81510fb5>] ? page_fault+0x25/0x30
.....
Mar 25 08:38:32 hqm-db-test02 kernel: Out of memory: Kill process 2560 (mysqld) score 968 or sacrifice child
Mar 25 08:38:32 hqm-db-test02 kernel: Killed process 2560, UID 497, (mysqld) total-vm:43565700kB, anon-rss:3568500kB, file-rss:4756kB
Mar 25 08:39:25 hqm-db-test02 kernel: __ratelimit: 3 callbacks suppressed
I had so much messages from selinux, which I just disabled, I missed the obvious!
Now we know what killed mysql. I will need to adjust my memory settings for mysql.
So, this type of crashed, corrupts the tokudb catalog.
Thanks for the info. To summarize, a mysqld crash while an innodb table was being altered to tokudb can result in an inconsistency in the tokudb files. If you have the original mysql data directory after the crash, I would like to play with it.
It wasn't an alter, but an "insert into tk_table select * from innodb_table.
What do you mean by the data directory? The innodb/tokudb files? Or the tokudb.directory?
Interesting. So tk_table was already created, the 'insert select' was executed, and the system crashed while the 'insert select' was still running?
BTW, only create table, drop table, rename table, and alter table change the contents of the tokudb.directory.
The best case for me is to get the entire MySQL data directory.
If I can't get that, then all of the tokudb meta data files (log*.tokulog*, tokudb.), some of the tokudb files (.tokudb), and the output from "find $MYSQL_DATA_DIR".
Yes DB crashed during the Insert! Does a large insert changes the tokudb file name?
As stated before, the file it was expecting was:
/data/mysql/tokudb_data/_MYDB_MYTAB_P_day_20140101_key_idx1_9a60c_2_1b_B_3.tokudb
But the file was named:
/data/mysql/tokudb_data/_MYDB_MYTAB_P_day_20140101_key_idx1_99906_5_1b.tokudb
If the target table is empty, tokudb will use a bulk loader which loads into new tokudb files and then does a big rename at the end. Was tk_table empty when the insert select was started?
Yes it was... We're getting closer
I have not been able to reproduce this problem of crashing while inserting into an empty partitioned tokudb table.
Is the original mysql data directory after the crash available?
Yes the dir is available... I cant post the content as it is sensible information!
Anything special in the dir you need?
Did you simulate the crash by killing the mysql process? Like a kill -9?
i created a table s with a bunch of rows in it. currently tokudb but it does not matter (IMO).
i created a partitioned table t empty with tokudb.
i ran insert into t select * from s.
killed mysqld at various points while the insert select was running.
in all cases, the table t was restored to a valid empty table pointing to fractal tree files that contain no rows.
why do i want the mysql dir? i want to run tokudb recovery and see where things go wrong.
if i cant get the entire dir, can i get the tokudb recovery log files (log*.tokulog*) and a listing of the mysql data dir files (the output from find MYSQL_DATA_DIR)?
I sent you all the info by email.
Anything special you would like me to monitor or log to make sure we have a full view if this happens again?
y
Hello. Need the tokudb recovery log files (log*.tokulog*).
I followed you advise and created a table with the same structure, renamed the files to reflect the files in the dictionary, then dropped the old table again.
It worked nicely.
As for reproducing the bug, I will see what I can do.
Thanks for your help.
The original bug of getting an error when querying the tokudb_fractal_tree_info table is fixed in tokudb 7.5.7. See https://tokutek.atlassian.net/browse/DB-829.
Thanks.
We still have 7.5.6 in our 5.5 and 10.0 trees, so it should be fixed with the next merge.