[MDEV-23090] Sid builds of ha_mroonga lost liblz4.so dependency Created: 2020-06-29  Updated: 2023-04-27

Status: Stalled
Project: MariaDB Server
Component/s: Platform Debian, Storage Engine - Mroonga
Affects Version/s: 10.3, 10.4, 10.5
Fix Version/s: 10.4, 10.5

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Otto Kekäläinen
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Since the last releases, ha_mroonga.so has lost one of dependencies on Sid, both x86 and amd64:

=== /usr/lib/mysql/plugin/ha_mroonga.so
 
 	/lib64/ld-linux-x86-64.so.2 
 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
 	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
-	liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 
 	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
 	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 
 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
 	linux-vdso.so.1

As I understand serg, it may be a bug, a loss of the library. If so, it was probably caused by the recent rebuild of VMs (TODO-2441).

 Comments   
Comment by Daniel Bartholomew [ 2020-06-30 ]

The rebuilt sid build VMs have liblz4-dev installed. It was actually automatically installed when I ran 'sudo apt build-dep mariadb-server' when creating the build VMs. So I'm not sure why it disappeared from the dep list for ha_mroonga.so.

Comment by Elena Stepanova [ 2020-07-03 ]

Indeed, the library is there, and moreover, other components still link with it, it's only mroonga that doesn't.
And it's not caused by a code change, even a bulid from the source released as 10.3.23 on the current Sid VMs has the same result, even though in 10.3.23 release binaries ha_mroonga.so was linked with liblz4.so.
Also, on other Debians it's still is.

So I suppose the difference was still caused by the VM upgrade, but not by the lack of library. Maybe otto will be interested to investigate.
Meanwhile I'll just make buildbot ignore it, we can't guarantee stable dependencies on the unstable Debian anyway.

Comment by Otto Kekäläinen [ 2020-07-04 ]

Thanks for reporting elenst! Is the snippet above from a command you ran manually or do we have some linked libraries listing done automatically in buildbot?

I was wondering how to test this automatically and/or search from old logs.

It is good that we catch these failures during development building against latest upstream dependencies, otherwise it is much more difficult to fix if they are detected only after official releases. Eventually what is in Debian unstable is what is in Debian stable and Ubuntu stable in 1-2 years.

The latest upload of a new liblz4-dev to Debian unstable was in November 2019: https://tracker.debian.org/pkg/lz4

Would this match with the builder image update frequency?

The liblz4 was not an explicit dependency until it was added in April 2020 in 5cdf245d7e2ab339ad3dba0dbbb591ab80e0dad0.

The git history of storage/mroonga does not have any commits related to this to what I can see.

A more likely culprint is in the lz4 detection code recently changed by serg:

± git log -p cmake/FindLZ4.cmake
 
commit 6d3747a294d75ab8153f82c5b2d61c3bf17b04de
 
Author: Sergei Golubchik <serg@mariadb.org>
 
Date:   Sun Jun 28 16:31:55 2020 +0200
 
 
 
    make rocksdb cmake checks less verbose on repeat
 
    
    * fix FindLZ4 to follow convention (LIBRARIES, not LIBRARY)
 
    * remove redundant checks from rocksdb/CMakeLists.txt
 
    * put all checks through the same macro that uniformly
 
      checks for a package, prints the message, adds definition
 
 
 
diff --git a/cmake/FindLZ4.cmake b/cmake/FindLZ4.cmake
 
index e97dd63e2b0..46c1fdc1b9f 100644
 
--- a/cmake/FindLZ4.cmake
 
+++ b/cmake/FindLZ4.cmake
 
@@ -1,9 +1,9 @@
 
 find_path(LZ4_INCLUDE_DIR NAMES lz4.h)
 
-find_library(LZ4_LIBRARY NAMES lz4)
 
+find_library(LZ4_LIBRARIES NAMES lz4)
 
 
 include(FindPackageHandleStandardArgs)
 
 FIND_PACKAGE_HANDLE_STANDARD_ARGS(
 
     LZ4 DEFAULT_MSG
 
-    LZ4_LIBRARY LZ4_INCLUDE_DIR)
 
+    LZ4_LIBRARIES LZ4_INCLUDE_DIR)
 
 
-mark_as_advanced(LZ4_INCLUDE_DIR LZ4_LIBRARY)
 
+mark_as_advanced(LZ4_INCLUDE_DIR LZ4_LIBRARIES)

Comment by Otto Kekäläinen [ 2020-07-04 ]

Related:

grep -rFi liblz4 storage/mroonga/
 
storage/mroonga/vendor/groonga/configure.ac:    PKG_CHECK_MODULES([LIBLZ4],
 
storage/mroonga/vendor/groonga/configure.ac:                      [liblz4],
 
storage/mroonga/vendor/groonga/configure.ac:      AC_MSG_ERROR("No liblz4 found")
 
storage/mroonga/vendor/groonga/CMakeLists.txt:    set(LIBLZ4_INCLUDE_DIRS
 
storage/mroonga/vendor/groonga/CMakeLists.txt:    set(LZ4_LIBS liblz4)
 
storage/mroonga/vendor/groonga/CMakeLists.txt:    if(NOT DEFINED LIBLZ4_FOUND)
 
storage/mroonga/vendor/groonga/CMakeLists.txt:      pkg_check_modules(LIBLZ4 liblz4)
 
storage/mroonga/vendor/groonga/CMakeLists.txt:    if(LIBLZ4_FOUND)
 
storage/mroonga/vendor/groonga/CMakeLists.txt:      # pkg_check_modules does not guarantee that LIBLZ4_LIBRARY_DIRS will be
 
storage/mroonga/vendor/groonga/CMakeLists.txt:        NAMES ${LIBLZ4_LIBRARIES}
 
storage/mroonga/vendor/groonga/CMakeLists.txt:        PATHS ${LIBLZ4_LIBRARY_DIRS}
 
storage/mroonga/vendor/groonga/CMakeLists.txt:        NAMES ${LIBLZ4_LIBRARIES})
 
storage/mroonga/vendor/groonga/lib/CMakeLists.txt:  ${LIBLZ4_INCLUDE_DIRS}
 
storage/mroonga/vendor/groonga/lib/Makefile.am:	$(LIBLZ4_CFLAGS)			\
 
storage/mroonga/vendor/groonga/lib/Makefile.am:	$(LIBLZ4_LIBS)				\
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:  set(LIBLZ4_SOURCES
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:    ${LIBLZ4_SOURCES})
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:  set_source_files_properties(${LIBLZ4_SOURCES}
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:    add_library(liblz4 STATIC ${LIBLZ4_SOURCES})
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:      liblz4
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:    add_library(liblz4 SHARED ${LIBLZ4_SOURCES})
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:    set_target_properties(liblz4 PROPERTIES OUTPUT_NAME "lz4")
 
storage/mroonga/vendor/groonga/vendor/lz4/CMakeLists.txt:    install(TARGETS liblz4

Comment by Elena Stepanova [ 2020-07-04 ]

otto

Is the snippet above from a command you ran manually or do we have some linked libraries listing done automatically in buildbot?

We do now, recently added upon Sergei's request.

was wondering how to test this automatically and/or search from old logs.

The mismatches are shown as a failure of minor-upgrade-all test, where at the end of the test you will see the output similar to the snippet. Here is an example:
https://buildbot.askmonty.org/buildbot/builders/kvm-deb-sid-x86/builds/10737/steps/minor-upgrade-all/logs/stdio
(the link will expire with time, as buildbot links always do).
The test compares ldd output with a baseline corresponding to the version/distribution/architecture, if there is such a baseline stored on github, or with the previously released version, as it happens in the case above.
The full output of ldd is shown among buildbot logs as ldd.new, and the output from the previous version (regardless whether it was used as a baseline or not) is shown as ldd.old:
https://buildbot.askmonty.org/buildbot/builders/kvm-deb-sid-x86/builds/10737/steps/minor-upgrade-all

However, this particular problem won't be visible anymore in the new builds, because I have updated the baseline to match the new status.

A more likely culprint is in the lz4 detection code recently changed by Sergei Golubchik

I don't see how it is possible. This change hasn't been released yet, and as I said in the earlier comment, even when mroonga is built on the new Sid images from the already released 10.3.23, it doesn't link anymore with liblz4, even though it did when it was built from the same sources before the release. At least it didn't when I tried to build it on the current Sid image, using the same steps as buildbot does.
I leave it to you and Sergei to decide on the reasons, though.

Comment by Otto Kekäläinen [ 2020-07-04 ]

Thanks for the pointers! As reference, status in Debian.org repositories:

# head -n 1 /etc/os-release
 
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
 
root@cb221730ef20:/build# ldd /usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_mroonga.so | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1 
 
 
# head -n 1 /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
 
root@5af281eedfba:/build# ldd /usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_mroonga.so | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1

History for 10.5.x releases on mariadb.org for Debian Sid:

root@cb221730ef20:/build/mgroonga# URL=http://mirror.netinch.com/pub/mariadb//mariadb-10.5.4/repo/debian/pool/main/m/mariadb-10.5/mariadb-plugin-mroonga_10.5.4+maria~sid_amd64.deb
 
root@cb221730ef20:/build/mgroonga# curl -s $URL -o temp.deb; dpkg -x temp.deb .; find . -name *.so | xargs ldd | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1 
 
 
root@cb221730ef20:/build/mgroonga# URL=http://mirror.netinch.com/pub/mariadb//mariadb-10.5.4/repo/debian/pool/main/m/mariadb-10.5/mariadb-plugin-mroonga_10.5.3+maria~sid_amd64.deb
 
root@cb221730ef20:/build/mgroonga# curl -s $URL -o temp.deb; dpkg -x temp.deb .; find . -name *.so | xargs ldd | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1 
 
 
root@cb221730ef20:/build/mgroonga# URL=http://mirror.netinch.com/pub/mariadb//mariadb-10.5.4/repo/debian/pool/main/m/mariadb-10.5/mariadb-plugin-mroonga_10.5.2+maria~sid_amd64.deb
 
root@cb221730ef20:/build/mgroonga# curl -s $URL -o temp.deb; dpkg -x temp.deb .; find . -name *.so | xargs ldd | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1

Indeed, regression in the 10.5.4 sid release.

No regression on the Buster and Stretch releases:

root@cb221730ef20:/build/mgroonga# URL=http://mirror.netinch.com/pub/mariadb//mariadb-10.5.4/repo/debian/pool/main/m/mariadb-10.5/mariadb-plugin-mroonga_10.5.4+maria~buster_amd64.deb
 
root@cb221730ef20:/build/mgroonga# curl -s $URL -o temp.deb; dpkg -x temp.deb .; find . -name *.so | xargs ldd | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1 
root@cb221730ef20:/build/mgroonga# URL=http://mirror.netinch.com/pub/mariadb//mariadb-10.5.4/repo/debian/pool/main/m/mariadb-10.5/mariadb-plugin-mroonga_10.5.4+maria~stretch_amd64.deb
 
 
 
root@cb221730ef20:/build/mgroonga# curl -s $URL -o temp.deb; dpkg -x temp.deb .; find . -name *.so | xargs ldd | sort | sed 's/(.*)//'
 
	/lib64/ld-linux-x86-64.so.2 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
	liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
	linux-vdso.so.1

Comment by Otto Kekäläinen [ 2020-07-05 ]

I did not find what the regression is and I cannot put more time on this right now as there are so many open PRs I need to champion/refresh/refine to fix urgent bugs. I'll return to this later, or maybe faust can try hunting what the regression in Sid was?

Comment by Otto Kekäläinen [ 2020-07-05 ]

This is maybe related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887655

Other notes:

  • I was not able to find in my Debian.org builds that it would have even tried to check for module liblz4 in any mariadb-10.3 or mariadb-10.4 build.
  • In my Debian.org builds of mariadb-10.1 is at least tried to check for liblz4 but to no avail: 

    $ grep -F -A 1 "Checking for module 'liblz4'" *build*.log
    		 
    latest.build-1434c69810.log:-- Checking for module 'liblz4'
    		 
    latest.build-1434c69810.log---   No package 'liblz4' found
    		 
    --
    		 
    latest.build-1488ebda00.log:-- Checking for module 'liblz4'
    		 
    latest.build-1488ebda00.log---   No package 'liblz4' found
    		 
    --
    		 
    latest.build-205544c809.log:-- Checking for module 'liblz4'
    		 
    latest.build-205544c809.log---   No package 'liblz4' found
    		 
    --
    		 
    latest.build-22e620c534.log:-- Checking for module 'liblz4'
    		 
    latest.build-22e620c534.log---   No package 'liblz4' found
    		 
    ...

  • In my local builds of MariaDB.org sources it has found liblz4 at least once, but normally it does not. Latest build today did not find.

$ grep -F -A 1 "Checking for module 'liblz4'" *build*.log
 
5f2df64086a.build-56fd4a2f27a.log:-- Checking for module 'liblz4'
 
5f2df64086a.build-56fd4a2f27a.log---   No package 'liblz4' found
 
--
 
aafb305d7eb.build-56fd4a2f27a.log:-- Checking for module 'liblz4'
 
aafb305d7eb.build-56fd4a2f27a.log---   No package 'liblz4' found
 
--
 
autobake-deb.build-06e5f28f9f1.log:-- Checking for module 'liblz4'
 
autobake-deb.build-06e5f28f9f1.log---   No package 'liblz4' found
 
--
 
build-af661b49ba5-ok-10.5-cs.log:-- Checking for module 'liblz4'
 
build-af661b49ba5-ok-10.5-cs.log---   Found liblz4, version 131
 
--
 
e1e442a327b.build-56fd4a2f27a.log:-- Checking for module 'liblz4'
 
e1e442a327b.build-56fd4a2f27a.log---   No package 'liblz4' found
 
--
 
ecb0e0ade47.build-56fd4a2f27a.log:-- Checking for module 'liblz4'
 
ecb0e0ade47.build-56fd4a2f27a.log---   No package 'liblz4' found

  • Log were it was found is dated June 14, 2020.
  • Noted also that Mroonga has it's own test for lz4 in `mroonga/check_libgroonga_support_lz4.inc`

I am confused on that conclusions to draw here. Need to dig up more in the CMake files on how Mroonga builds and how it includes lz4...

Generated at Thu Feb 08 09:19:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.