[MXS-1757] Problem while linking libavrorouter.so Created: 2018-04-04  Updated: 2019-02-11  Resolved: 2019-02-11

Status: Closed
Project: MariaDB MaxScale
Component/s: avrorouter, build
Affects Version/s: 2.2.5, 2.2.15
Fix Version/s: 2.3.4

Type: Bug Priority: Major
Reporter: Valerii Kravchuk Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

Ubuntu 14.04.5 LTS, Fedora 27


 Description   

I hit a problem while building current MaxScale 2.2 from GitHub source on Ubuntu 14.04. Every time I get linking error like this:

...
 
[ 41%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_file.c.o
 
[ 42%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_index.c.o
 
[ 42%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x28c): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x329): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x437): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [all] Error 2

I've executed the following according to the history:

openxs@ao756:~/git/MaxScale/build$ fc -l
 
...
 
1986     sudo apt-get update
 
1987     sudo apt-get upgrade
 
1988     cd git/MaxScale/
 
1989     git pull
 
1990     cd build/
 
...
 
1994     sudo rm -rf *
 
1995     ../BUILD/install_build_deps.sh
 
1996     cmake .. -DCMAKE_INSTALL_PREFIX=/usr
 
1997     time make

Some details about the environment:

openxs@ao756:~/git/MaxScale/build$ cd ..
 
openxs@ao756:~/git/MaxScale$ git branch
 
  2.0
 
* 2.2
 
openxs@ao756:~/git/MaxScale$ git log -1
 
commit 9291815bcd5081bbc1a055f2aa97475e19c363f8
 
Author: Markus Mцєkelцє <markus.makela@mariadb.com>
 
Date:   Sat Mar 31 20:21:24 2018 +0300
 
 
 
    Wait for monitor to start in mxs1719
 
 
 
    The test starts MaxScale and immediately connects to it. As the monitoring
 
    is asynchronous, we need to give it a few seconds to stabilize.
 
openxs@ao756:~/git/MaxScale$ cat /etc/issue
 
Ubuntu 14.04.5 LTS \n \l
 
 
 
openxs@ao756:~/git/MaxScale$ uname -a
 
Linux ao756 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
 
openxs@ao756:~/git/MaxScale/build$ dpkg -l | grep snappy
 
ii  libsnappy-dev                                         1.1.0-1ubuntu1                                      amd64        fast compression/decompression library (development files)
 
ii  libsnappy1                                            1.1.0-1ubuntu1                                      amd64        fast compression/decompression library
 
openxs@ao756:~/git/MaxScale/build$

I tried to follow https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-building-mariadb-maxscale-from-source-code/

What am I doing wrong?

 Comments   
Comment by markus makela [ 2018-04-04 ]

Seems like the Avro C API links against snappy when MaxScale doesn't. I think that the script needs to be updated to build the API library without snappy.

Comment by markus makela [ 2018-04-04 ]

Fixed the script.

Comment by Valerii Kravchuk [ 2018-09-21 ]

I am not sure what you fixed and how to proceed, but I get exactly the same failure on the same system when using current GitHub code:

...
 
[ 42%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_index.c.o
 
[ 42%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x28c): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x329): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x437): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [all] Error 2
 
openxs@ao756:~/git/MaxScale/build$ cd ..
 
openxs@ao756:~/git/MaxScale$ git log -1
 
commit d55c07dc2ef1364413a40f9a53eb2876612e21ad
 
Author: Markus Mцєkelцє <markus.makela@mariadb.com>
 
Date:   Fri Sep 21 10:26:30 2018 +0300
 
 
 
    MXS-2066: Reset resultset collection by default
 
 
 
    The collection of resultsets needs to be disabled by default when a
 
    response is received to cover the cases where an error is returned.
 
 
 
    The collection of results should also not be set for queries that do not
 
    generate any responses.
 
openxs@ao756:~/git/MaxScale$ cat /etc/issue
 
Ubuntu 14.04.5 LTS \n \l
 
 
 
openxs@ao756:~/git/MaxScale$

Comment by markus makela [ 2018-09-21 ]

There appear to still be some problems.

Comment by Valerii Kravchuk [ 2018-09-26 ]

I have basically same problem on Fedora 27:

...
 
[ 44%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x295): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x332): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x44f): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/build.make:260: server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [CMakeFiles/Makefile2:2217: server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [Makefile:139: all] Error 2
 
178.14user 35.31system 3:48.31elapsed 93%CPU (0avgtext+0avgdata 354760maxresident)k
 
156584inputs+313984outputs (488major+6978299minor)pagefaults 0swaps

It seems problem happens on systems where older MaxScale (2.2.4) was once built from source. I see no (easy) way to "clean up" the system from the remainings of previous .sh script run.

Comment by markus makela [ 2018-10-01 ]

I believe removing the /usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a file and installing the library again should fix it. Make sure to run the installation script in a clean directly (/tmp should be OK).

Comment by Valerii Kravchuk [ 2018-10-01 ]

Does not seem to help in my case:

...
 
[ 43%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_schema.c.o
 
[ 43%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_rbr.c.o
 
[ 43%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_file.c.o
 
[ 44%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_index.c.o
 
[ 44%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x295): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x332): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x44f): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/build.make:260: server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [CMakeFiles/Makefile2:2217: server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [Makefile:139: all] Error 2
 
[openxs@fc23 build]$ fc -l
 
...
 
1035     sudo rm -rf *
 
1036     ls
 
1037     sudo rm /usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a
 
1038     ../BUILD/install_build_deps.sh
 
1039     cmake .. -DCMAKE_INSTALL_PREFIX=/usr
 
1040     make
 
[openxs@fc23 build]$

Comment by markus makela [ 2018-10-01 ]

The relevant part of the installation script:

# Avro C API                                                                                                                                                                                  
wget -q -r -l1 -nH --cut-dirs=2 --no-parent -A.tar.gz --no-directories http://mirror.netinch.com/pub/apache/avro/stable/c
 
if [ $? != 0 ]
 
then
 
    echo "Error getting avro-c"
 
    exit 1
 
fi
 
avro_filename=`ls -1 avro*.tar.gz`
 
avro_dir=`echo "$avro_filename" | sed "s/.tar.gz//"`
 
tar -axf $avro_filename
 
mkdir $avro_dir/build
 
pushd $avro_dir/build
 
# The -DSNAPPY_FOUND=N is used to prevent the library from linking against libsnappy (MaxScale doesn't link against it)                                                                       
cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_FLAGS=-fPIC -DCMAKE_CXX_FLAGS=-fPIC -DSNAPPY_FOUND=N
 
make
 
sudo make install
 
popd

Can you try to run those commands manually and see whether it finds the snappy package and configures the library to use it?

Comment by Valerii Kravchuk [ 2019-01-16 ]

So, when I get this error:

...
 
[ 44%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x295): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x332): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x44f): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/build.make:260: server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [CMakeFiles/Makefile2:2217: server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [Makefile:139: all] Error 2

what exactly should I do? I tried commands above literally after that:

[openxs@fc23 build]$ wget -q -r -l1 -nH --cut-dirs=2 --no-parent -A.tar.gz --no-directories http://mirror.netinch.com/pub/apache/avro/stable/c
 
[openxs@fc23 build]$ if [ $? != 0 ]
 
>
 
> then
 
>
 
>     echo "Error getting avro-c"
 
>
 
>     exit 1
 
>
 
> fi
 
[openxs@fc23 build]$ avro_filename=`ls -1 avro*.tar.gz`
 
[openxs@fc23 build]$
 
[openxs@fc23 build]$ avro_dir=`echo "$avro_filename" | sed "s/.tar.gz//"`
 
[openxs@fc23 build]$
 
[openxs@fc23 build]$ tar -axf $avro_filename
 
[openxs@fc23 build]$ ls $avro_dir
 
AUTHORS             CPackConfig.txt   README
 
build               docs              README.maintaining_win32.txt
 
build.sh            examples          src
 
ChangeLog           FindSnappy.cmake  tests
 
cmake_avrolib.bat   INSTALL           version.sh
 
cmake_avrolib.sh    LICENSE           VERSION.txt
 
CMakeLists.txt      NEWS
 
cmake_pretty.cmake  NOTICE
 
[openxs@fc23 build]$ ls $avro_dir/build
 
CMakeCache.txt       CPackConfig.cmake        docs                  Makefile
 
CMakeFiles           CPackSourceConfig.cmake  examples              src
 
cmake_install.cmake  CTestTestfile.cmake      install_manifest.txt  tests
 
[openxs@fc23 build]$ rm -rf $avro_dir/build
 
[openxs@fc23 build]$ mkdir $avro_dir/build
 
[openxs@fc23 build]$ pushd $avro_dir/build
 
~/git/MaxScale/build/avro-c-1.8.2/build ~/git/MaxScale/build
 
[openxs@fc23 build]$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_FLAGS=-fPIC -DCMAKE_CXX_FLAGS=-fPIC -DSNAPPY_FOUND=N
 
-- The C compiler identification is GNU 7.3.1
 
-- The CXX compiler identification is GNU 7.3.1
 
-- Check for working C compiler: /usr/bin/cc
 
-- Check for working C compiler: /usr/bin/cc -- works
 
-- Detecting C compiler ABI info
 
-- Detecting C compiler ABI info - done
 
-- Detecting C compile features
 
-- Detecting C compile features - done
 
-- Check for working CXX compiler: /usr/bin/c++
 
-- Check for working CXX compiler: /usr/bin/c++ -- works
 
-- Detecting CXX compiler ABI info
 
-- Detecting CXX compiler ABI info - done
 
-- Detecting CXX compile features
 
-- Detecting CXX compile features - done
 
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
 
Enabled deflate codec
 
-- Found Snappy: /lib64/libsnappy.so
 
Enabled snappy codec
 
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.3.12")
 
-- Checking for module 'liblzma'
 
--   Found liblzma, version 5.2.3
 
Enabled lzma codec
 
CMake Warning (dev) at CMakeLists.txt:169 (set):
 
  Policy CMP0053 is not set: Simplify variable reference and escape sequence
 
  evaluation.  Run "cmake --help-policy CMP0053" for policy details.  Use the
 
  cmake_policy command to set the policy and suppress this warning.
 
 
 
  For input:
 
 
 
    '@ZLIB_PKG@ @LZMA_PKG@ @SNAPPY_PKG@'
 
 
 
  the old evaluation rules produce:
 
 
 
    'zlib liblzma libsnappy'
 
 
 
  but the new evaluation rules produce:
 
 
 
    '@ZLIB_PKG@ @LZMA_PKG@ @SNAPPY_PKG@'
 
 
 
  Using the old result for compatibility since the policy is not set.
 
This warning is for project developers.  Use -Wno-dev to suppress it.
 
 
 
-- Checking for module 'jansson>=2.3'
 
--   Found jansson, version 2.9
 
-- Searching for asciidoc...
 
CMake Warning at docs/CMakeLists.txt:51 (message):
 
  asciidoc not found.  HTML documentation will *NOT* be built.
 
 
 
 
 
-- Configuring done
 
-- Generating done
 
-- Build files have been written to: /home/openxs/git/MaxScale/build/avro-c-1.8.2/build
 
[openxs@fc23 build]$ make
 
...
 
[100%] Built target test_cpp
 
[openxs@fc23 build]$ sudo make install
 
...
 
-- Installing: /usr/bin/avromod
 
-- Set runtime path of "/usr/bin/avromod" to ""
 
[openxs@fc23 build]$ popd
 
~/git/MaxScale/build

Then I tried make again:

[openxs@fc23 build]$ make
 
[  2%] Built target libmicrohttpd
 
[  4%] Built target jansson
 
[  6%] Built target connector-c
 
[  8%] Built target pcre2
 
[  8%] Built target inih
 
[ 20%] Built target maxscale-common
 
[ 21%] Built target maxavro
 
[ 22%] Built target maxavrocheck
 
[ 22%] Built target test_values
 
[ 24%] Built target maxscale_sqlite
 
[ 25%] Built target qc_sqlite
 
[ 25%] Built target canonizer
 
[ 26%] Built target maxpasswd
 
[ 26%] Built target maxkeys
 
[ 27%] Built target maxscale
 
[ 28%] Built target test_trxtracking
 
[ 28%] Built target test_trxcompare
 
[ 29%] Built target test_thread
 
[ 30%] Built target test_json
 
[ 30%] Built target test_logorder
 
[ 31%] Built target test_adminusers
 
[ 32%] Built target test_hint
 
[ 32%] Built target test_hash
 
[ 33%] Built target test_log
 
[ 33%] Built target test_spinlock
 
[ 34%] Built target test_config
 
[ 35%] Built target profile_trxboundaryparser
 
[ 36%] Built target test_maxscalepcre2
 
[ 37%] Built target test_filter
 
[ 37%] Built target test_dcb
 
[ 37%] Built target test_atomic
 
[ 38%] Built target test_utils
 
[ 39%] Built target test_buffer
 
[ 39%] Built target test_service
 
[ 39%] Built target test_local_address
 
[ 39%] Built target test_logthrottling
 
[ 39%] Built target test_modulecmd
 
[ 39%] Built target test_users
 
[ 39%] Built target test_http
 
[ 40%] Built target test_modutil
 
[ 41%] Built target test_poll
 
[ 41%] Built target test_semaphore
 
[ 42%] Built target test_server
 
[ 42%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x295): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x332): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x44f): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/build.make:260: server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [CMakeFiles/Makefile2:2217: server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [Makefile:139: all] Error 2

After getting the same error, what shell I do next?

"make clean; make" ended up with the same error:

...
 
[ 44%] Building C object server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/avro_index.c.o
 
[ 44%] Linking C shared library libavrorouter.so
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `encode_snappy':
 
codec.c:(.text+0x18f): undefined reference to `snappy_max_compressed_length'
 
codec.c:(.text+0x295): undefined reference to `snappy_compress'
 
/usr/lib/gcc/x86_64-redhat-linux/7/../../../libavro.a(codec.o): In function `decode_snappy':
 
codec.c:(.text+0x332): undefined reference to `snappy_uncompressed_length'
 
codec.c:(.text+0x44f): undefined reference to `snappy_uncompress'
 
collect2: error: ld returned 1 exit status
 
make[2]: *** [server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/build.make:260: server/modules/routing/avrorouter/libavrorouter.so.1.0.0] Error 1
 
make[1]: *** [CMakeFiles/Makefile2:2217: server/modules/routing/avrorouter/CMakeFiles/avrorouter.dir/all] Error 2
 
make: *** [Makefile:139: all] Error 2
 
[openxs@fc23 build]$ cd ..
 
[openxs@fc23 MaxScale]$ git branch
 
  2.0
 
* 2.2
 
  2.2.2
 
  develop
 
  release-1.4.2
 
  release-1.4.4
 
[openxs@fc23 MaxScale]$ git log -1
 
commit 1e1836354be5b55b69ada1b3c961a8ff14ce66c5 (HEAD -> 2.2, origin/2.2)
 
Author: Markus Mцєkelцє <markus.makela@mariadb.com>
 
Date:   Sat Jan 12 04:55:35 2019 +0200
 
 
 
    MXS-2257: Extend password encryption tutorial
 
 
 
    Fixed the documentation on the arguments to maxkeys, which is a directory,
 
    and added a short paragraph about alternative key file locations. Also
 
    documented that keys are read from the directory where the `datadir`
 
    parameter points to.

Comment by markus makela [ 2019-02-11 ]

Seems that the avro library is linked against snappy even though it should not be. With the current code it seems that this is only fixable by preventing the snappy libraries from being found which in practice means removing them before configuring and compiling the library. Given that the code in question isn't a part of MaxScale, a patch step would be required in the installation script that completely prevents the use of the snappy library.

Comment by markus makela [ 2019-02-11 ]

Added code that patches the avro source code to never link against snappy.

Generated at Thu Feb 08 04:09:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.