Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16525

MyRocks linking fails with: Undefined reference to `ZDICT_trainFromBuffer'

Details

    Description

      https://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/4213/steps/compile/logs/stdio

      librocksdblib.a(compaction_job.cc.o): In function `rocksdb::ZSTD_TrainDictionary(std::__cxx11::basic_string
      		 
      <char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long,
      		 
       std::allocator<unsigned long> > const&, unsigned long)':
      		 
      ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h:812: undefined reference to `ZDICT_trainFromBuffer'
      		 
      ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h:813: undefined reference to `ZDICT_isError'
      		 
      collect2: error: ld returned 1 exit status
      		 
      storage/rocksdb/CMakeFiles/sst_dump.dir/build.make:98: recipe for target 'storage/rocksdb/sst_dump' failed

      It actually attempts to link with libzstd:

      /usr/bin/x86_64-linux-gnu-g++   -g -O2 
      -fdebug-prefix-map=/home/buildbot/buildbot/build/mariadb-10.2.16=. 
      -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security 
      -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -fno-rtti -O3 -g -static-libgcc 
      -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF
      		 
        -specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now 
      CMakeFiles/sst_dump.dir/rocksdb/tools/sst_dump.cc.o  
      -o sst_dump  -lpthread librocksdblib.a -lsnappy -lzstd -lz -lpthread

      but fails for some reason

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MDEV-28011 Debian autobake cleanup to remove out of support distros

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 5 older comments
            psergei Sergei Petrunia added a comment -

            I don't see any bugs about this against the debian package: https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=zstd.

            To check further, I've tried installing a newer RocksDB from stretch-backports as described here: https://backports.debian.org/Instructions/ 

            admin@ip-172-31-17-107:~$ sudo apt-get -t stretch-backports install "libzstd1" 
            ...
            		 
            Get:1 http://ftp.debian.org/debian stretch-backports/main amd64 libzstd-dev amd64 1.3.3+dfsg-1~bpo9+1 [231 kB]
            		 
            Get:2 http://ftp.debian.org/debian stretch-backports/main amd64 libzstd1 amd64 1.3.3+dfsg-1~bpo9+1 [188 kB]

            and after that both .a and .so have the function:

            admin@ip-172-31-17-107:~$ nm  /usr/lib/x86_64-linux-gnu/libzstd.a | grep ZDICT_trainFromBuffer
            		 
            0000000000001110 T ZDICT_trainFromBuffer_cover
            		 
            00000000000026b0 T ZDICT_trainFromBuffer
            		 
            00000000000025a0 T ZDICT_trainFromBuffer_legacy
            		 
            0000000000001990 T ZDICT_trainFromBuffer_unsafe_legacy


            admin@ip-172-31-17-107:~$ objdump -T  /usr/lib/x86_64-linux-gnu/libzstd.so | grep ZDICT_trainFromBuffer
            		 
            00000000000537b0 g    DF .text  000000000000010c  Base        ZDICT_trainFromBuffer_legacy
            		 
            000000000004a790 g    DF .text  0000000000000328  Base        ZDICT_trainFromBuffer_cover
            		 
            00000000000538c0 g    DF .text  0000000000000070  Base        ZDICT_trainFromBuffer

            psergei Sergei Petrunia added a comment - I don't see any bugs about this against the debian package: https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=zstd . To check further, I've tried installing a newer RocksDB from stretch-backports as described here: https://backports.debian.org/Instructions/ admin@ip-172-31-17-107:~$ sudo apt-get -t stretch-backports install "libzstd1" ... Get:1 http://ftp.debian.org/debian stretch-backports/main amd64 libzstd-dev amd64 1.3.3+dfsg-1~bpo9+1 [231 kB] Get:2 http://ftp.debian.org/debian stretch-backports/main amd64 libzstd1 amd64 1.3.3+dfsg-1~bpo9+1 [188 kB] and after that both .a and .so have the function: admin@ip-172-31-17-107:~$ nm /usr/lib/x86_64-linux-gnu/libzstd.a | grep ZDICT_trainFromBuffer 0000000000001110 T ZDICT_trainFromBuffer_cover 00000000000026b0 T ZDICT_trainFromBuffer 00000000000025a0 T ZDICT_trainFromBuffer_legacy 0000000000001990 T ZDICT_trainFromBuffer_unsafe_legacy admin@ip-172-31-17-107:~$ objdump -T /usr/lib/x86_64-linux-gnu/libzstd.so | grep ZDICT_trainFromBuffer 00000000000537b0 g DF .text 000000000000010c Base ZDICT_trainFromBuffer_legacy 000000000004a790 g DF .text 0000000000000328 Base ZDICT_trainFromBuffer_cover 00000000000538c0 g DF .text 0000000000000070 Base ZDICT_trainFromBuffer
            psergei Sergei Petrunia added a comment -

            cvicentiu otto any thoughts? Should this be reported as a bug against Debian?

            The only solution that I see right now is:

            • Disable linking against libzstd on Debian Stretch. (We cannot link against a package from -Backports, can we ?)

            How do we achieve the above? Remove the libzstd-dev package from the Debian Stretch builder?

            psergei Sergei Petrunia added a comment - cvicentiu otto any thoughts? Should this be reported as a bug against Debian? The only solution that I see right now is: Disable linking against libzstd on Debian Stretch. (We cannot link against a package from -Backports , can we ?) How do we achieve the above? Remove the libzstd-dev package from the Debian Stretch builder?
            psergei Sergei Petrunia added a comment -

            Thanks to wlad for taking part in the above investigation

            psergei Sergei Petrunia added a comment - Thanks to wlad for taking part in the above investigation
            otto Otto Kekäläinen added a comment -

            You can file a bug against Debian and try to persuade the maintainer/release manager to push this backported version into a stable release update. In the mean time probably needs to be disabled for Debian Stretch in debian/autobake-deb.sh that checks if version 1.3.3 or newer is available and if not, it will simply sed out the library from debian/control.

            otto Otto Kekäläinen added a comment - You can file a bug against Debian and try to persuade the maintainer/release manager to push this backported version into a stable release update. In the mean time probably needs to be disabled for Debian Stretch in debian/autobake-deb.sh that checks if version 1.3.3 or newer is available and if not, it will simply sed out the library from debian/control.
            serg Sergei Golubchik added a comment -

            Right. It turns out that stretch has libzstd1-1.1.2-1. And the visibility of ZDICT_trainFromBuffer was changed in this commit — note how it changed

            #  define ZSTDLIB_API

            to

            #  define ZDICTLIB_VISIBILITY __attribute__ ((visibility ("default"))) 
            ...
            		 
            #  define ZDICTLIB_API ZDICTLIB_VISIBILITY

            And that commit is present in tags starting from v1.1.3.

            The proper fix our side is to only use libzstd if it's usable, where "usable" in this context means "exports ZDICT_trainFromBuffer function"

            serg Sergei Golubchik added a comment - Right. It turns out that stretch has libzstd1- 1.1.2 -1. And the visibility of ZDICT_trainFromBuffer was changed in this commit — note how it changed # define ZSTDLIB_API to # define ZDICTLIB_VISIBILITY __attribute__ ((visibility ("default"))) ... # define ZDICTLIB_API ZDICTLIB_VISIBILITY And that commit is present in tags starting from v1.1.3 . The proper fix our side is to only use libzstd if it's usable, where "usable" in this context means "exports ZDICT_trainFromBuffer function"

            People

              cvicentiu Vicențiu Ciorbaru
              psergei Sergei Petrunia
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.