Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24357

Connect REST API is not working for ubuntu when installed from package and WITH_SSL=bundled

Details

    Description

      CONNECT SE is a powerful SE which has a feature of using REST queries for creating/querying the tables as can be found in KB - CONNECT rest queries.
      This feature is achieved by using library which can be created as described in KB-getrest-library which is linked with libcpprest.
      To use the REST feature one should enable CONNECT_WITH_REST:BOOL=ON argument which is by default ON for the source configuration.
      After that one can manually create GetRest.so library as explained in article and dynamically link it for the usage or one can have libcpprest statically linked into the ha_connect.so plugin library.
      Now the problem here is for the later (when the libcpprest is installed and in case when using ubuntu OS. Below is the procedure shown:

      $ lsb_release -a
      No LSB modules are available.
      Distributor ID:	Ubuntu
      Description:	Ubuntu 18.04.5 LTS
      Release:	18.04
      Codename:	bionic
       
      $ dpkg -S libcpprest
      ....
      libcpprest-dev: /usr/lib/x86_64-linux-gnu/libcpprest.so
      

      Building the ha_connect.so results in no statically linked libcpprest.
      After googling found that the problem is stated here and indeed in my case:

      $ ls -la /usr/lib/x86_64-linux-gnu/cmake/ |grep cpp
      -rw-r--r--   1 root root    393 Apr 23  2018 cpprestsdk-config.cmake
      -rw-r--r--   1 root root   5268 Apr 23  2018 cpprestsdk-targets.cmake
      -rw-r--r--   1 root root    886 Apr 23  2018 cpprestsdk-targets-none.cmake
      

      Based on search procedure of cmake find_package as stated here we are ending in the result found in CMakeCache.txt in builddir:

      //The directory containing a CMake configuration file for cpprestsdk.
      cpprestsdk_DIR:PATH=cpprestsdk_DIR-NOTFOUND
      

      One way to do so is to find distro version ( I couldn't find right regex from cmake --system-information|grep CMAKE_SYSTEM ) and use lsb_release. That is suggested in the patch.

      However another problem is that this will not work when WITH_SSL=bundled (or if we are using wolfssl.
      The errors are attached in this ticket (probably will be since cannot attach the file).

      $ cmake --build . --target connect
       
      -- Ubuntu
      -- -lcpprest
      -- Configuring OQGraph
      -- Boost version: 1.65.1
      -- Performance Schema is required by TokuDB
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /home/anel/mariadb/builds/10.4
      [1/2] Building CXX object storage/connect/CMakeFiles/connect.dir/restget.cpp.o
      FAILED: storage/connect/CMakeFiles/connect.dir/restget.cpp.o 
      /usr/bin/c++  
      -DDBUG_TRACE 
      -DFORCE_INIT_OF_VARS 
      -DGZ_SUPPORT 
      -DHAVE_CONFIG_H 
      -DHUGE_SUPPORT 
      -DLIBXML2_SUPPORT 
      -DLINUX 
      -DMARIADB 
      -DMYSQL_DYNAMIC_PLUGIN 
      -DNOCRYPT 
      -DODBC_SUPPORT 
      -DREST_SOURCE 
      -DREST_SUPPORT 
      -DUBUNTU 
      -DUNIX 
      -DVCT_SUPPORT 
      -DXMAP 
      -DZIP_SUPPORT 
      -D_FILE_OFFSET_BITS=64 
      -Dconnect_EXPORTS 
      -I/home/anel/mariadb/10.4/wsrep-lib/include 
      -I/home/anel/mariadb/10.4/wsrep-lib/wsrep-API/v26 
      -Iinclude 
      -I/home/anel/mariadb/10.4/include 
      -I/home/anel/mariadb/10.4/sql 
      -Iextra/wolfssl 
      -I/home/anel/mariadb/10.4/extra/wolfssl/wolfssl 
      -I/home/anel/mariadb/10.4/extra/wolfssl/wolfssl/wolfssl 
      -I/usr/include/libxml2 
      -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -fPIC -Wall -Wmissing-declarations -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=unused-value -Wno-error=parentheses -Wno-error=strict-aliasing -Wno-error=misleading-indentation -Wno-error=type-limits -fpermissive -fexceptions -fPIC  -g -DENABLED_DEBUG_SYNC -ggdb3 -DSAFE_MUTEX -DTRASH_FREED_MEMORY -Wall -Wextra -Wformat-security -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Woverloaded-virtual -Wnon-virtual-dtor -Wvla -Wwrite-strings -Werror -Wno-unused-function -Wno-unused-variable -Wno-unused-value -Wno-parentheses -Wno-strict-aliasing -Wno-misleading-indentation -Wno-type-limits -fPIC   -std=gnu++11 -MD -MT storage/connect/CMakeFiles/connect.dir/restget.cpp.o -MF storage/connect/CMakeFiles/connect.dir/restget.cpp.o.d -o storage/connect/CMakeFiles/connect.dir/restget.cpp.o -c /home/anel/mariadb/10.4/storage/connect/restget.cpp
      In file included from /home/anel/mariadb/10.4/extra/wolfssl/wolfssl/wolfssl/ssl.h:33:0,
                       from /home/anel/mariadb/10.4/extra/wolfssl/wolfssl/wolfssl/openssl/ssl.h:35,
                       from /usr/include/boost/asio/ssl/detail/openssl_types.hpp:21,
                       from /usr/include/boost/asio/ssl/context_base.hpp:19,
                       from /usr/include/boost/asio/ssl/context.hpp:27,
                       from /usr/include/boost/asio/ssl.hpp:19,
                       from /usr/include/cpprest/http_client.h:53,
                       from /home/anel/mariadb/10.4/storage/connect/restget.cpp:6:
      /home/anel/mariadb/10.4/extra/wolfssl/wolfssl/wolfssl/wolfcrypt/settings.h:2060:14: error: #warning "For timing resistance / side-channel attack prevention consider using harden options" [-Werror=cpp]
                   #warning "For timing resistance / side-channel attack prevention consider using harden options"
                    ^~~~~~~
      In file included from /usr/include/boost/config.hpp:61:0,
                       from /usr/include/boost/algorithm/string/std_containers_traits.hpp:18,
                       from /usr/include/boost/algorithm/string.hpp:18,
                       from /usr/include/cpprest/asyncrt_utils.h:31,
                       from /usr/include/cpprest/astreambuf.h:23,
                       from /usr/include/cpprest/filestream.h:19,
                       from /home/anel/mariadb/10.4/storage/connect/restget.cpp:5:
      /usr/include/boost/asio/ssl/context_base.hpp:127:3: error: ‘SSL_OP_ALL’ was not declared in this scop....
      

      In order to solve this easiest solution is to use WITH_SSL=system.
      When applying the patch and building the binaries correct result is obtained:

      $ ldd ./storage/connect/ha_connect.so |grep cpp
      	libcpprest.so.2.10 => /usr/lib/x86_64-linux-gnu/libcpprest.so.2.10 (0x00007f5f8f760000)
      

      Attachments

        1. tabrest.cpp
          11 kB
        2. tabrest.h
          2 kB
        3. wolfssl_error.txt
          41 kB

        Issue Links

          Activity

            Thank you for working on this. I am sure it works like you describe.

            Nevertheless, I will take the opposite decision and completely remove the possibility of compiling a statically linked to cpprestsdk version of Connect. The new version of CMakeLists.txt will just contain:

            #
            # REST
            #
             
            OPTION(CONNECT_WITH_REST "Compile CONNECT storage engine with REST support" ON)
             
            IF(CONNECT_WITH_REST)
            # MESSAGE(STATUS "=====> REST support is ON")
              SET(CONNECT_SOURCES ${CONNECT_SOURCES} tabrest.cpp tabrest.h)
              add_definitions(-DREST_SUPPORT)
            ENDIF(CONNECT_WITH_REST)
            

            The reason why?

            Allowing to compile a version of Connect statically linked to cpprestsdk was actually a temporary tool allowing to test REST on early versions of connect having no other mean to do it and should have been already removed.

            Indeed, an engine module, such as Connect, must be loadable by all means and therefore, never be dependent on an external product. This was already true for Java and JBDC and must be also true for REST.

            Don't forget that compiling a version is not only to use it locally but more generally to make a distribution (see Creating a Debian Repository) Even your module will work locally, not only it won't if you export it to another machine without cpprestsdk, but it will not load at all, making it useless even for those not using REST.

            Furthermore, cpprestsdk is not mandatory anymore, most future versions will use curl instead and curl seems to be available on almost all systems.

            So what you should do now (the version you have is not able to use curl) is to use REST the way it is described in appendix 2 and see if you have any trouble when compiling the external GetRest.so lib on Ubuntu.

            Thanks for your understanding.

            bertrandop Olivier Bertrand added a comment - Thank you for working on this. I am sure it works like you describe. Nevertheless, I will take the opposite decision and completely remove the possibility of compiling a statically linked to cpprestsdk version of Connect. The new version of CMakeLists.txt will just contain: # # REST # OPTION(CONNECT_WITH_REST "Compile CONNECT storage engine with REST support" ON) IF(CONNECT_WITH_REST) # MESSAGE(STATUS "=====> REST support is ON") SET(CONNECT_SOURCES ${CONNECT_SOURCES} tabrest.cpp tabrest.h) add_definitions(-DREST_SUPPORT) ENDIF(CONNECT_WITH_REST) The reason why? Allowing to compile a version of Connect statically linked to cpprestsdk was actually a temporary tool allowing to test REST on early versions of connect having no other mean to do it and should have been already removed. Indeed, an engine module, such as Connect, must be loadable by all means and therefore, never be dependent on an external product. This was already true for Java and JBDC and must be also true for REST. Don't forget that compiling a version is not only to use it locally but more generally to make a distribution (see Creating a Debian Repository) Even your module will work locally, not only it won't if you export it to another machine without cpprestsdk, but it will not load at all, making it useless even for those not using REST. Furthermore, cpprestsdk is not mandatory anymore, most future versions will use curl instead and curl seems to be available on almost all systems. So what you should do now (the version you have is not able to use curl) is to use REST the way it is described in appendix 2 and see if you have any trouble when compiling the external GetRest.so lib on Ubuntu. Thanks for your understanding.

            Ok, I see what you mean, I just lost a lot of time and found out this problem related to my environment. I don't know what do you mean by appendix 2 - I gues this link?
            https://mariadb.com/kb/en/connect-making-the-getrest-library/
            I already have used GetRest.so for dynamically linking and it works, showed in example above also.

            Please take a look other part of the patch and let me know is it acceptable, not related to static linkage, but rather to WITH_SSL option:

            IF(CONNECT_WITH_REST AND NOT WITH_SSL MATCHES "bundled")
            

            Do you want me to help regarding cURL development?
            I will be working more on CONNECT in next period .
            Thanks

            anel Anel Husakovic added a comment - Ok, I see what you mean, I just lost a lot of time and found out this problem related to my environment. I don't know what do you mean by appendix 2 - I gues this link? https://mariadb.com/kb/en/connect-making-the-getrest-library/ I already have used GetRest.so for dynamically linking and it works, showed in example above also. Please take a look other part of the patch and let me know is it acceptable, not related to static linkage, but rather to WITH_SSL option: IF(CONNECT_WITH_REST AND NOT WITH_SSL MATCHES "bundled") Do you want me to help regarding cURL development? I will be working more on CONNECT in next period . Thanks

            I don't anything about that SSL stuff. Do you think it is important? Perhaps it could be wrong by preventing Connect to generate the _tabrest _ table type that could work anyway with CURL, and is harmless when not using REST.

            I have attached the new _tabrest _files. Maybe you can look at the way I call CURL in the Xcurl function. I did not know how to execute an external program and found this looking in the NET but there maybe a more appropriate way to do it. Perhaps also, replacing manually them in the source, you can test using REST via CURL.

            bertrandop Olivier Bertrand added a comment - I don't anything about that SSL stuff. Do you think it is important? Perhaps it could be wrong by preventing Connect to generate the _tabrest _ table type that could work anyway with CURL, and is harmless when not using REST. I have attached the new _tabrest _files. Maybe you can look at the way I call CURL in the Xcurl function. I did not know how to execute an external program and found this looking in the NET but there maybe a more appropriate way to do it. Perhaps also, replacing manually them in the source, you can test using REST via CURL.

            When we use with_ssl=bundled we got compilation error. We have to disable this.
            Will take a look this week to see cURL implementation and let you know . Thanks.
            So regarding this MDEV, just let me know what you decided regarding WITH_SSL patch so we can close it and start working on different MDEV (like cURL for example).

            anel Anel Husakovic added a comment - When we use with_ssl=bundled we got compilation error. We have to disable this. Will take a look this week to see cURL implementation and let you know . Thanks. So regarding this MDEV, just let me know what you decided regarding WITH_SSL patch so we can close it and start working on different MDEV (like cURL for example).

            There is nothing specific in the four files added when REST is ON that can explain a compilation error related to SSL. Unless something is found showing the contrary, and in this case that is the file itself that must be fixed to avoid it, no patch should be applied here.
            If people have problem with SSL they just have to compile WITH REST OFF.
            So this MDEV is closed.

            bertrandop Olivier Bertrand added a comment - There is nothing specific in the four files added when REST is ON that can explain a compilation error related to SSL. Unless something is found showing the contrary, and in this case that is the file itself that must be fixed to avoid it, no patch should be applied here. If people have problem with SSL they just have to compile WITH REST OFF. So this MDEV is closed.

            People

              bertrandop Olivier Bertrand
              anel Anel Husakovic
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.