[MDEV-6165] TokuDB fails to load, even when following instructions on website Created: 2014-04-24  Updated: 2014-10-05  Resolved: 2014-10-05

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 5.5.37, 10.0.10
Fix Version/s: 5.5.40, 10.0.14

Type: Bug Priority: Major
Reporter: Andrew Sun (Inactive) Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: tokudb, upstream

Issue Links:
Blocks
is blocked by MDEV-6173 TokuDB cannot start after upgrade if ... Closed

 Description   

I have disabled transparent hugepages, yet when I run
install soname 'ha_tokudb';
I get

ERROR 1123 (HY000): Can't initialize function 'TokuDB'; Plugin initialization function failed.

This issue broke my website when I upgraded from Ubuntu 13.10 with MariaDB 5.5.34 to Ubuntu 14.04 with MariaDB 10.0.10. Did I miss anything? Is there another cause for this issue?



 Comments   
Comment by Elena Stepanova [ 2014-04-24 ]

Hi,

Please find your error log file and see what error messages it contains. If you can't decide which ones are relevant, please attach it.

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Here is the error log when I start the server:
140424 12:39:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140424 12:39:02 [Note] InnoDB: The InnoDB memory heap is disabled
140424 12:39:02 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140424 12:39:02 [Note] InnoDB: Compressed tables use zlib 1.2.8
140424 12:39:02 [Note] InnoDB: Using Linux native AIO
140424 12:39:02 [Note] InnoDB: Not using CPU crc32 instructions
140424 12:39:02 [Note] InnoDB: Initializing buffer pool, size = 128.0M
140424 12:39:02 [Note] InnoDB: Completed initialization of buffer pool
140424 12:39:02 [Note] InnoDB: Highest supported file format is Barracuda.
140424 12:39:02 [Note] InnoDB: 128 rollback segment(s) are active.
140424 12:39:03 [Note] InnoDB: Waiting for purge to start
140424 12:39:03 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.15-63.0 started; log sequence number 5079025365
140424 12:39:03 [Note] Plugin 'FEEDBACK' is disabled.
140424 12:39:03 [ERROR] Function 'TokuDB' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_trx' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_trx' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_lock_waits' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_lock_waits' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_locks' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_locks' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_file_map' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_fractal_tree_info' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140424 12:39:03 [ERROR] Function 'TokuDB_fractal_tree_block_map' already exists
140424 12:39:03 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
Cannot upgrade TokuDB version 24 database. Previous improper shutdown detected.
140424 12:39:03 [ERROR] Plugin 'TokuDB' init function returned error.
140424 12:39:03 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
140424 12:39:03 [Note] Server socket created on IP: '127.0.0.1'.
140424 12:39:03 [Note] Event Scheduler: Loaded 0 events
140424 12:39:03 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.10-MariaDB-1~trusty' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution

And here is the error log when I do install soname 'ha_tokudb';
Cannot upgrade TokuDB version 24 database. Previous improper shutdown detected.
140424 12:42:04 [ERROR] Plugin 'TokuDB' init function returned error.
140424 12:42:04 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.

I see that there are some improperly closed databases? How can I repair them? I think these were caused by the service mysql stop failing when doing the release upgrade (removing MariaDB).

Comment by Elena Stepanova [ 2014-04-24 ]

Cannot upgrade TokuDB version 24 database. Previous improper shutdown detected.

This is a serious problem.
Currently TokuDB is unable to load the recovery log from a previous version if it wasn't shutdown properly. Here are the options from the TokuDB expert, as posted on #maria IRC channel:

<prohaska7�>�� bring up the old version (probably with clients locked out) and shut it down. this will write a nice shutdown entry into the tokudb recovery log. then bring up the new version (which should see the old log with the nice shutdown and proceed)
<prohaska7> one could create a new instance of the old version, put the tokudb files in its data directory, bring it up and shut it down. then copy the tokudb log files to the new version and bring it up.
<prohaska7> clean shutdown is required.

Ideally, if you have a backup from before the upgrade, just restore it, rollback to the 5.5 version, perform the clean shutdown, and then upgrade normally.
If you don't have a backup, create one before you downgrade, then if you have issues after you have upgraded again, you can use it and only replace the TokuDB log.
If you can bring up a separate instance as the 2nd suggestion above says, it will be safer.

We will discuss the issue with the TokuDB team to see if we can come up with a proper solution in future.
Thanks for reporting.

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

When I downgraded to 5.5, ha_tokudb.so was missing. I thought it was supposed to be there!

Comment by Elena Stepanova [ 2014-04-24 ]

Which 5.5 version did you use, and which package?

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

I used 5.5.36, from package mariadb-server-5.5

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Oh, never mind. I accidentally used the one from the Ubuntu repos

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

After using the repo deb http://mariadb.mirror.iweb.com/repo/5.5/ubuntu trusty main and installing mariadb-server-5.5, I still get the "Cannot upgrade TokuDB version 24 database. Previous improper shutdown detected" error. Possibly I could try 5.5.34, which is what I was running?

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Actually, could you point me to where I could obtain the ha_tokudb.so file for 5.5.34? I think if I use an older version, it might work.

Comment by Elena Stepanova [ 2014-04-24 ]

You can use this repo for example: http://ftp.osuosl.org/pub/mariadb/mariadb-5.5.34/repo/ubuntu/
There should be 5.5.34 version on some other mirrors, but probably not on all of them.

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

The DEB file mariadb-server-5.5_5.5.34+maria-1~saucy_i386 does not contain ha_tokudb.so...I thought it had TokuDB.

Comment by Elena Stepanova [ 2014-04-24 ]

TokuDB is present only in 64-bit packages. Is there a reason you are installing i386? If you had TokuDB before, you must be running it on a 64-bit machine.

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Oh, oops! I downloaded the 32-bit version! I will see if I can swap out the file...

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Agh! The old ha_tokudb is not compatible! What else can I do? I had trouble installing the old version of MariaDB in a virtual machine. Is there another way?

Comment by Elena Stepanova [ 2014-04-24 ]

The old ha_tokudb is not compatible with its own log? Does it throw the same error?
Are you certain you had 5.5.34 before, and that ha_tokudb.so has been actually downgraded now?

So far the two options above is all that TokuDB experts could suggest: either downgrade or install the old version separately, copy TokuDB files there, run it, shut it down, copy the log from there.
For the time being, we probably have a better chance to solve the problem of installing the old version than to find another way to fix the TokuDB problem, since the latter doesn't really depend on us, that's the engine specifics.

What kind of trouble have you had installing the old version of MariaDB?

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Here is the log when I use the old .so:
140424 16:07:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140424 16:07:10 InnoDB: The InnoDB memory heap is disabled
140424 16:07:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140424 16:07:10 InnoDB: Compressed tables use zlib 1.2.8
140424 16:07:10 InnoDB: Using Linux native AIO
140424 16:07:10 InnoDB: Initializing buffer pool, size = 128.0M
140424 16:07:10 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
140424 16:07:10 [ERROR] Plugin 'InnoDB' init function returned error.
140424 16:07:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140424 16:07:10 [Note] Plugin 'FEEDBACK' is disabled.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for STORAGE ENGINE plugin TokuDB not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_trx not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_lock_waits not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_locks not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_file_map not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_fractal_tree_info not supported by this version of the server)
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_fractal_tree_block_map not supported by this version of the server)
140424 16:07:10 [ERROR] Couldn't load plugins from 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for STORAGE ENGINE plugin TokuDB not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_trx not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_trx' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_lock_waits not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_lock_waits' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_locks not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_locks' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_file_map not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_file_map' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_fractal_tree_info not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_info' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for INFORMATION SCHEMA plugin TokuDB_fractal_tree_block_map not supported by this version of the server)
140424 16:07:10 [Warning] Couldn't load plugin named 'TokuDB_fractal_tree_block_map' with soname 'ha_tokudb.so'.
140424 16:07:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140424 16:07:10 [ERROR] Aborting

140424 16:07:10 [Note] /usr/sbin/mysqld: Shutdown complete

140424 16:07:10 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

I will actually try the .so from 5.5.36, because there is a chance that I had that version (my brain does not remember things very well...)

When I tried to install the old version through apt-get and that repo, something went wrong with the dependencies and apt-get did not want to install them (possibly because they were not available on the repo?)

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

I still get a similar error log when I use the .so from 5.5.36, which is the latest version I could have used before the upgrade.

Comment by Elena Stepanova [ 2014-04-24 ]

There are two problems here.
The server doesn't start because InnoDB fails to start. It happens because of this:

InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!

It's caused by different defaults in 5.5 and 10.0.
Try to set in your cnf file (in 'mysqld' section) innodb_log_file_size=50331648, it should solve the problem.

The second problem is

140424 16:07:10 [ERROR] Can't open shared library 'ha_tokudb.so' (errno: 8, API version for STORAGE ENGINE plugin TokuDB not supported by this version of the server)

It's normal if you are trying to use TokuDB from 5.5 on a 10.0 server or vice versa. Unfortunately, you need to downgrade the whole server (or to install the 5.5 server with 5.5 TokuDB).

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

I'm using 5.5 server with 5.5 TokuDB. I'm confused.

Comment by Andrew Sun (Inactive) [ 2014-04-24 ]

Okay, the server actually does not fail to start when the config option is set. Yay.

However, TokuDB still fails to load, with the could not upgrade version 24 database error. I will try to swap out the .so again.

Comment by Andrew Sun (Inactive) [ 2014-04-25 ]

I have 5.5.37 installed, and when using the ha_tokudb.so from 5.5.36, I get the same not supported errors.

Comment by Andrew Sun (Inactive) [ 2014-04-25 ]

I fixed it!

I installed 5.5.36 from the Ubuntu repos, and then copied the tokudb library from the 5.5.36 deb I downloaded from the "official" source. Now I can upgrade!

Thanks for your patience in helping me resolve this issue.

Comment by Elena Stepanova [ 2014-04-25 ]

Great!
My apologies for the unpleasant experience. I hope eventually we will be able to get TokuDB be more friendly in terms of upgrade.

Comment by Elena Stepanova [ 2014-04-25 ]

I have created an explicit request MDEV-6173 for improving the upgrade logic, and MDEV-6174 for documenting the current situation. Technically MDEV-6173 is a duplicate of MDEV-6165, but I wanted to make a point that it's a general problem, not a user error as it might seem from the current issue. If it gets fixed in TokuDB, or if we find some other way to minimize the risk, this issue will be closed along with MDEV-6173. Meanwhile, I will keep it open.

Generated at Thu Feb 08 07:09:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.