[#MDEV-12254] MariaRocks: check whether it gets into packages

Comment by Sergei Petrunia [ 2017-03-14 ]

Running this to check tarbake packages:

PREFIX=http://hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499

FILES="kvm-tarbake-jaunty-x86/mariadb-10.2.5.tar.gz \

kvm-bintar-centos5-amd64/mariadb-10.2.5-linux-x86_64.tar.gz \

kvm-bintar-centos5-x86/mariadb-10.2.5-linux-i686.tar.gz \

kvm-bintar-trusty-amd64/mariadb-10.2.5-linux-x86_64.tar.gz \

kvm-bintar-trusty-x86/mariadb-10.2.5-linux-i686.tar.gz \

for  i in $FILES ;  do

  wget "$PREFIX/$i";

  fname=`basename $i`

  tar tf $fname | grep rocksdb

  rm $fname

  echo '---------------------------'

done

I find that

http://hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-tarbake-jaunty-x86/mariadb-10.2.5.tar.gz

includes MyRocks and RocksDB sources and test suite.

http://hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-bintar-trusty-amd64/mariadb-10.2.5-linux-x86_64.tar.gz

includes mariadb-10.2.5-linux-x86_64/lib/plugin/ha_rocksdb.so
includes MyRocks testsuite in mariadb-10.2.5-linux-x86_64/mysql-test/plugin/rocksdb

Other packages do not seem to include anything related to MyRocks (which is expected for at least some of them).

Comment by Sergei Petrunia [ 2017-03-14 ]

Debian package doesn't include MyRocks :

find . -name '*deb'

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-connect_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-client_10.2.5+maria~xenial_all.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-gssapi-client_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmysqlclient18_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-cracklib-password-check_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-server_10.2.5+maria~xenial_all.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmariadb3_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-oqgraph_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-server-core-10.2_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-spider_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-test-data_10.2.5+maria~xenial_all.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-common_10.2.5+maria~xenial_all.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-server-10.2_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-client-10.2_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-mroonga_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-client-core-10.2_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-test_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmariadbd-dev_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmariadbd19_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mysql-common_10.2.5+maria~xenial_all.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmariadbclient18_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/mariadb-plugin-gssapi-server_10.2.5+maria~xenial_i386.deb

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-deb-xenial-x86/debs/binary/libmariadb-dev_10.2.5+maria~xenial_i386.deb

psergey@hasky:~/tmp10$ find . -name '*deb' | while read a ; do less $a | grep rocksdb ; done

psergey@hasky:~/tmp10$

Comment by Sergei Petrunia [ 2017-03-14 ]

psergey@hasky:~/tmp11$ find . -name '*.rpm'

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-cassandra-engine.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-server.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-connect-engine.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-compat.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-shared.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-common.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-client.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-test.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-gssapi-server.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-devel.rpm

./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-oqgraph-engine.rpm

find . -name '*.rpm' |  while read a ; do echo "===========LOOKING INTO $a" ;  less $a | grep rocksdb ; done

shows

===========LOOKING INTO ./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-server.rpm

/usr/lib64/mysql/plugin/ha_rocksdb.so

...

===========LOOKING INTO ./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-test.rpm

/usr/share/mysql-test/plugin/rocksdb

/usr/share/mysql-test/plugin/rocksdb/rocksdb

... (lots of tests)

So, Fedora25 RPM package includes ha_rocksdb.so and the tests.

Comment by Sergei Petrunia [ 2017-03-14 ]

Some of MyRocks tools are in the "client" package:

===========LOOKING INTO ./hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13499/kvm-rpm-fedora25-amd64/rpms/MariaDB-10.2.5-fedora25-x86_64-client.rpm

/usr/bin/ldb

/usr/bin/mysql_ldb

/usr/bin/sst_dump

which I think is not entirely correct as these are tools for examining server's files.

Comment by Sergei Petrunia [ 2017-03-14 ]

kvm-rpm-centos7-amd64 and kvm-zyp-opensuse13-amd64 have the same situation like Fedora25 RPMs.

Comment by Sergei Petrunia [ 2017-03-14 ]

Checked Windows builds, mariadb-10.2.5-win32.zip and mariadb-10.2.5-winx64.zip. Both of them have the needed files:

mariadb-10.2.5-winx64/bin/ldb.exe

mariadb-10.2.5-winx64/bin/ldb.pdb

mariadb-10.2.5-winx64/bin/sst_dump.exe

mariadb-10.2.5-winx64/bin/sst_dump.pdb

mariadb-10.2.5-winx64/bin/mysql_ldb.exe

mariadb-10.2.5-winx64/bin/mysql_ldb.pdb

mariadb-10.2.5-winx64/lib/plugin/ha_rocksdb.dll

mariadb-10.2.5-winx64/lib/plugin/ha_rocksdb.pdb

mariadb-10.2.5-winx64/mysql-test/plugin/rocksdb/ *

I didn't check .msi.

Comment by Sergei Petrunia [ 2017-03-14 ]

Questions open so far:

Which compression libraries is ha_rocksdb.so linked with? (one way to do this is to start the server and look at @@rocksdb_supported_compression_types)
Should we have it the server RPM or move to a special RocksDB-SE package? (not sure about what the name should be)
* Other storage engines are in their own RPM packages
Why are server-side tools in the *-client package? They should be in the "server tools", or "storage engine" package.

Comment by Sergei Petrunia [ 2017-03-17 ]

Looking at the Windows x64 tarball, build 13577:

MariaDB [test]> show variables like '%rocksdb_supported_compression%';

+-------------------------------------+-------+

| Variable_name                       | Value |

+-------------------------------------+-------+

| rocksdb_supported_compression_types | Zlib  |

+-------------------------------------+-------+

1 row in set (0.00 sec)

MariaDB [test]> show variables like '%version_compile_os%';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| version_compile_os | Win64 |

+--------------------+-------+

1 row in set (0.00 sec)

Same with the MSI package.

Comment by Vicențiu Ciorbaru [ 2017-03-25 ]

Debian packages seem to be in order:
ldd usr/lib/mysql/plugin/ha_rocksdb.so
linux-vdso.so.1 => (0x00007ffe8bb09000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb2094ec000)
libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007fb2092e4000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb2090c9000)
libstdc+.so.6 => /usr/lib/x86_64-linux-gnu/libstdc+.so.6 (0x00007fb208d41000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb208a38000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb20866e000)
/lib64/ld-linux-x86-64.so.2 (0x000055889521e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb208457000)

We just need to make sure that our builders have libsnappy and zlib installed.

Comment by Vicențiu Ciorbaru [ 2017-03-26 ]

RPM packages now add mysql_ldb and sst_dump into the rocksdb package. Dependencies are added correctly (if they exist).

Comment by Sergei Petrunia [ 2017-03-27 ]

Looking at the bintars produced by a build from yesterday: http://hasky.askmonty.org/archive/bb-10.2-mariarocks/build-13696:

kvm-tarbake-jaunty-x86/mariadb-10.2.5.tar.gz

Source code in storage/rocksdb

kvm-bintar-centos5-amd64/mariadb-10.2.5-linux-x86_64.tar.gz

no MyRocks

kvm-bintar-trusty-amd64/mariadb-10.2.5-linux-x86_64.tar.gz

lib/plugin/ha_rocksdb.so
mysql-test/plugin/rocksdb - has tests

kvm-bintar-quantal-amd64/mariadb-10.2.5-linux-x86_64.tar.gz

no MyRocks

kvm-bintar-centos5-x86/mariadb-10.2.5-linux-i686.tar.gz
kvm-bintar-trusty-x86/mariadb-10.2.5-linux-i686.tar.gz
kvm-bintar-quantal-x86/mariadb-10.2.5-linux-i686.tar.gz

No MyRocks (this is expected for 32-bit systems)

kvm-bintar-trusty-amd64/mariadb-10.2.5-linux-x86_64.tar.gz

Due to ~~MDEV-12370~~ has a limited use (non MyRocks-specific issue)

mysqld is still linked to libsnappy.so :

ldd mariadb-10.2.5-linux-x86_64/bin/mysqld

        linux-vdso.so.1 =>  (0x00007ffed2bb4000)

        libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f4209cfe000)

        libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f4209afc000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f42098f0000)

        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f42096b8000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f42094b4000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4209296000)

        libsystemd-daemon.so.0 => not found

        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4208f14000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4208c0a000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4208841000)

        /lib64/ld-linux-x86-64.so.2 (0x0000558bd800b000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f420862b000)

ha_rocksdb.so is not:

ldd mariadb-10.2.5-linux-x86_64/lib/plugin/ha_rocksdb.so

        linux-vdso.so.1 =>  (0x00007ffdeffa6000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd2f583c000)

        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd2f54ba000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd2f51b0000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2f4de7000)

        /lib64/ld-linux-x86-64.so.2 (0x000056524c44a000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd2f4bd1000)

Looking at what ha_rocksdb.so supports I see:

MariaDB [test]> show variables like 'rocksdb%compre%';

+-------------------------------------+-------------+

| Variable_name                       | Value       |

+-------------------------------------+-------------+

| rocksdb_supported_compression_types | Snappy,Zlib |

+-------------------------------------+-------------+

so I assume that libsnappy is linked statically.

Comment by Sergei Petrunia [ 2017-03-27 ]

Using the same build as above,
Checked mariadb-plugin-rocksdb_10.2.5+maria~trusty_amd64.deb :

ldd  ./usr/lib/mysql/plugin/ha_rocksdb.so

        linux-vdso.so.1 =>  (0x00007ffc46b0e000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5d663b1000)

        libsnappy.so.1 => /usr/lib/libsnappy.so.1 (0x00007f5d661ab000)

        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5d65f90000)

        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5d65c81000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5d65979000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5d655af000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f5d66c15000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5d65399000)

snappy and libz are there, which confirms cvicentiu's result.

Now, checking the MariaDB-10.2.5-fedora25-x86_64-rocksdb-engine.rpm

ldd ./usr/lib64/mysql/plugin/ha_rocksdb.so

./usr/lib64/mysql/plugin/ha_rocksdb.so: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by ./usr/lib64/mysql/plugin/ha_rocksdb.so)

./usr/lib64/mysql/plugin/ha_rocksdb.so: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./usr/lib64/mysql/plugin/ha_rocksdb.so)

        linux-vdso.so.1 =>  (0x00007ffc98fee000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f53a3d65000)

        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f53a3b4a000)

        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f53a383b000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f53a3533000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f53a3169000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f53a4604000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f53a2f53000)

It depends on libz but not on libsnappy?

Comment by Vicențiu Ciorbaru [ 2017-03-27 ]

psergey Libsnappy is not installed on that builder when the build was performed. When libsnappy gets installed (Daniel has been notified), it should work.

Comment by Sergei Golubchik [ 2017-05-24 ]

This is done by now, right?

[MDEV-12254] MariaRocks: check whether it gets into packages Created: 2017-03-14 Updated: 2017-05-24 Resolved: 2017-05-24
Status:	Closed
Project:	MariaDB Server
Component/s:	Packaging, Storage Engine - RocksDB
Fix Version/s:	10.2.6

[MDEV-12254] MariaRocks: check whether it gets into packages Created: 2017-03-14 Updated: 2017-05-24 Resolved: 2017-05-24