mariadb version 10.0 and mariadb version 10.1 for linux

Which packages are you trying to install, exactly? Please paste the full file name for the server package, for example.
And of course please check whether you actually have libjemalloc.so.1.

i just installed mariadb 10.1 on my ppc64 server, but while installing there some errors appear which is jemalloc errors (see file attached)

i already analyzed it and seems mariadb cannot create system tables because of it (jemalloc error). but on process install, there's jemalloc on the package and seems to be ok with packages install

The attachment does not show anything useful, only the warning that the key is not installed, which in itself doesn't mean much.
Please attach the whole installation log and run ldd <path to mysqld>.

Hai Elena

Heres the error files

dev-hcms-appdb.err

Jimmy is my friend and we tried to install MariaDB 10.1 on PPC64 IBM. jemalloc already installed but the error seems around in jemalloc. and the effect was MariaDB cannot create system tables on default directory (/var/lib/mysql). why this happened?

danblack,
You are installing ppc64 packages regularly, right? Could you please take a look if you know anything about it?

In the buildbot config, we have this comment which might be relevant:

1. There is no epel repo for Red Hat 7.1 P8 LE (Little Endian), so there are no
2. jemalloc packages available, so that is disabled below – dbart (2015-01-22)

And so we build packages for Red Hat 7 with jemalloc, but for Red Hat 7.1 – without. However, we don't seem to provide 7.1 packages currently.

On the other hand, pramudyapp says that it does not work even on a machine where jemalloc is installed – I guess either it's a wrong jemalloc, or it's installed in a wrong location?

svoj, also FYI.

@Elena Stepanova I already installed again (MariaDB 10.1.20) from fresh OS installation and seems not working again. The error showing caused by jemalloc again. I was looking for error log on /var/lib/mysql and that was same error clue again on the attachment before (dev-hcms.appdb.err). I have no clue at all right now.

our machine is RHEL 7.2 (Maipo) and here its when i ran ldd <path to mysqld>

linux-vdso64.so.1 => (0x00003fff98400000)
libpthread.so.0 => /opt/at8.0/lib64/power8/libpthread.so.0 (0x00003fff967c0000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00003fff96750000)
libaio.so.1 => /lib64/libaio.so.1 (0x00003fff96720000)
libz.so.1 => /opt/at8.0/lib64/power8/libz.so.1 (0x00003fff966f0000)
libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00003fff96680000)
libcrypt.so.1 => /opt/at8.0/lib64/power8/libcrypt.so.1 (0x00003fff96630000)
libssl.so.1.0.0 => /opt/at8.0/lib64/power8/libssl.so.1.0.0 (0x00003fff965b0000)
libcrypto.so.1.0.0 => /opt/at8.0/lib64/power8/libcrypto.so.1.0.0 (0x00003fff96330000)
libdl.so.2 => /opt/at8.0/lib64/libdl.so.2 (0x00003fff96300000)
libstdc+.so.6 => /opt/at8.0/lib64/power8/libstdc+.so.6 (0x00003fff96150000)
libm.so.6 => /opt/at8.0/lib64/power8/libm.so.6 (0x00003fff96060000)
libc.so.6 => /opt/at8.0/lib64/power8/libc.so.6 (0x00003fff95e50000)
/opt/at8.0/lib64/ld64.so.1 => /lib64/ld64.so.1 (0x000000005b3c0000)
libgcc_s.so.1 => /opt/at8.0/lib64/power8/libgcc_s.so.1 (0x00003fff95e20000)

i tried to execute command => lscpu | grep -y byte. And the result is : Big Endian

(fyi, check attachment for full error log while installation)

Hai finally MariaDB already installed with a 'little bit tricky'. I saw jemalloc.so.1 cannot be load by mysqld service so i tried running mysqld_safe with parameter malloc-lib and then pointing to /usr/lib64/libjemalloc.so.1

But seems not finish already, i must install system database which is mysql db itself. So i tried to copy mysql db folder on another mariadb server to my ppc server. Then mysqld_safe is running now

But i know that way was 'naive' way, my question is why libjemalloc.so.1 cannot be loaded by mysqld script (which is mariadb service) on PPC? It doesnt supposed to be like that

I'm not usually doing packages (but I probably should). I'll look into the cause of this tomorrow after I get a RHEL7.2 BE instance going.

Notes so far:

As M4t and pramudyapp indicate, this is a PPC64 (big endian) install. I'm curious as to why a BE install was chosen over a LE. FYI I haven't enabled CRC32 performance related for Big endian so setting innodb_checksum_algorithm=none may give you performance benefit, especially i you choose to stay on a BE platform.

picture.JPG seem to indicate that the jemalloc was installed from EPEL however error1_installation.jpg seem to indicate retrieval from yum.mariadb.org.

There might be some library version incompatibility of the libraries needed by jemalloc.

As RHEL7.2 is using systemd I assume mysqld_safe shouldn't actually be used. https://mariadb.com/kb/en/mariadb/systemd/ has no direct equivalent documented however it should be possible to set LD_PRELOAD environment variable in systemd to the jemalloc path.

The last question as to why any special attention is needed for libjemalloc is still most pertinent and I'll attempt to resolve this.

Ok problem confirmed.

Apologies - I was wrong about systemd - seems the RHEL builds don't install/use the systemd service file (MDEV-10797).

On adding strace to the mysqld_safe in the init script the following was found:

30386 open("/opt/at8.0/lib64/power8/libjemalloc.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

30386 open("/opt/at8.0/lib64/libjemalloc.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

30386 open("/opt/at8.0/lib64/power8/libjemalloc.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

30386 open("/opt/at8.0/lib64/libjemalloc.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

30386 writev(2, [{"/usr/sbin/mysqld", 16}, {": ", 2}, {"error while loading shared libraries", 36}, {": ", 2}, {"libjemalloc.so.1", 16}, {": ", 2}, {"cannot open shared object file", 30}, {": ", 2}, {"No such file or directory", 25}, {"\n", 1}], 10) = 132

This can be corrected with the following:

$ sudo /opt/at8.0/sbin/ldconfig

$ ldconfig -C /opt/at8.0/etc/ld.so.cache --print-cache | grep -i  jemalloc

        libjemalloc.so.1 (libc6,64bit) => /lib64/libjemalloc.so.1

To create system tables the following is suggested (rather than copying from another instance):

sudo mysql_install_db

So for final workarounds:

In the install log it seems advance-toolchain-at8.0-runtime.ppc64 is installed before jemalloc was.

The advance-toolchain installs /etc/cron.d/at8.0-7_ldconfig which has a helper that seems to watch inotify_add_watch(3, "/etc/ld.so.cache", IN_MODIFY) and triggers its own cache rebuild at any time the /etc/ld.so.cache is rebuilt. Unfortunately this isn't started on the install of advance-toolchain-at8.0-runtime. I'll raise this with the advance-toolchain developers for a potential solution there.

As an interim workaround is the following acceptable?

--- a/support-files/rpm/server-postin.sh

+++ b/support-files/rpm/server-postin.sh

@@ -49,6 +49,13 @@ if [ $1 = 1 ] ; then

   # owns all database files.

   chown -R %{mysqld_user}:%{mysqld_group} $datadir

+  # Workaround for MDEV-11386

+  if compgen -G "/opt/at*/sbin/ldconfig" > /dev/null  ; then

+    for ldconfig in /opt/at*/sbin/ldconfig; do

+      $ldconfig

+    done

+  fi

+

   if [ ! -e $datadir/mysql ]; then

     # Create data directory

     mkdir -p $datadir/{mysql,test}

Hi @Daniel Black

Let me try as you suggested above, the server location is on another office and can't remote from outside. I will try again in next week.

Thank you!!!

The above workaround is for creating new packages.

In the meantime for new installs with the existing packages the following should work:

as root:

(configure mariadb repo)

yum install advance-toolchain-at8.0-runtime

/opt/at8.0/bin/watch_ldconfig &

yum install MariaDB-server

I'd like some feedback as to why you chose a BE install for PPC64 on a new machine too. Not a bad thing but LE overall is more tested. The same error as above will probably occur with the LE so the same workaound will apply if you choose a ppc64le install. Email me off bug report danielgb@au1.ibm.com if you don't want it to put it here.

Hi @Daniel Black

Have you check your personal email?. I sent you email before.
I followed your instruction above and MariaDB installation working perfectly (10.1.21) and i added innodb_checksum_algorithm=none to configuration file

Maybe this issue could be fix in new updates?

Thank you anyway

Test using RPM build with the changes from https://github.com/MariaDB/server/pull/291

This was build with at10.0 however the same applies with at8.0.

[root@rhel73 ~]# yum remove jemalloc

[root@rhel73 ~]# ls -la /lib64/libjemalloc.so.1

ls: cannot access /lib64/libjemalloc.so.1: No such file or directory

[root@rhel73 ~]#  ldconfig -C /opt/at10.0/etc/ld.so.cache --print-cache | grep -i  jemalloc 

[root@rhel73 ~]# ps -ef | grep watch

root        10     2  0 Jan20 ?        00:00:00 [watchdog/0]

root        11     2  0 Jan20 ?        00:00:00 [watchdog/1]

root        16     2  0 Jan20 ?        00:00:00 [watchdog/2]

root        21     2  0 Jan20 ?        00:00:00 [watchdog/3]

root        26     2  0 Jan20 ?        00:00:00 [watchdog/4]

root        31     2  0 Jan20 ?        00:00:00 [watchdog/5]

root        36     2  0 Jan20 ?        00:00:00 [watchdog/6]

root        41     2  0 Jan20 ?        00:00:00 [watchdog/7]

root        46     2  0 Jan20 ?        00:00:00 [watchdog/8]

root        51     2  0 Jan20 ?        00:00:00 [watchdog/9]

root     16872 12465  0 10:40 pts/1    00:00:00 grep --color=auto watch

[root@rhel73 ~]# yum install jemalloc

nstalled:

  jemalloc.ppc64 0:3.6.0-1.el7                                                                                                                               

Complete!

[root@rhel73 ~]# ldconfig -C /opt/at10.0/etc/ld.so.cache --print-cache | grep -i  jemalloc

[root@rhel73 ~]# rpm -i --nodeps  ~ozlabs/build_mariadb/MariaDB-10.0.30-rhel-ppc64-server.rpm

[root@rhel73 ~]# ldconfig -C /opt/at10.0/etc/ld.so.cache --print-cache | grep -i  jemalloc

        libjemalloc.so.1 (libc6,64bit) => /lib64/libjemalloc.so.1

The targets of the Advance Toolchain development are AT 10 release (10.0-3) that will be available [in] March and the next AT 8.0 will be available only [in] April.

Looking at https://github.com/MariaDB/server/releases it seems 10.0.29 was 9 days ago and 10.1.21 was 5 days ago making the targets of 10.0.30 and 10.1.22 on https://jira.mariadb.org/projects/MDEV?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page& are going to be extended.

As such 10.2.4 seems like a good goal for the pull request (is MDEV-10807 under consideration?) and potentially the next 10.0 release. After that it should be possible to revert the change in the PR after updating the versions in yum.mariadb.org

serg, easy workaround in pull request. Acceptable for 10.2.4?

1. I'd rather not use bash-isms, this could be done like

for ldconfig in /opt/at*/sbin/ldconfig; do

   test -x $ldconfig &&  $ldconfig

done

2. You wrote "Temporary Workaround ....will be corrected in Advance Toolchain 10.0-3 and 8.0-8". When, do you think, it could happen?

ack.

The targets of the Advance Toolchain development are AT 10 release (10.0-3) that will be available in March and the next AT 8.0 will be available in April.

MDEV-10807 - AT10.0 for MariaDB-10.2 - I haven't seen any indication as to what is happening.

The advance-toolchain source/build is now public https://github.com/advancetoolchain/advance-toolchain - I have an internal IBM bug report that references this bug. I've noticed that the release candidate hasn't fixed this yet.

Pushed amended version of Daniel's fix.

danblack, do you still believe that use of ATC is justified? It gives some performance benefits at cost of painful maintenance issues (both for us and our users).

Also how ATC plans to fix that? The only viable solution is to substitute ldconfig with some script that would run both system wide and ATC ldconfig. Cron job is not reasonable solution.

The better solution is for a ld.so that reads both cache files. I'll convey the maintenance burden and see what the fix plan is. I'll look at AT performance in the near future.