Uploaded image for project: 'MariaDB Connector/C'
  1. MariaDB Connector/C
  2. CONC-102

Core dump while doing ssl connect using mariadb client library

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 2.0.0
    • 2.1
    • None
    • None
    • libmariadb.so.2

    Description

      We have mariadb db client which does connect to mysql using ssl. We started seeing this coredump during ssl connect to mysql server.

      Please let me know if you need any other info

      Core was generated by `/opt/adc/adc/bin/monitord'.
      Program terminated with signal 11, Segmentation fault.
      #0 0x00007f08bdb85056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6
      Missing separate debuginfos, use: debuginfo-install ulcm-dscinstance-5.0-0.x86_64
      (gdb) bt
      #0 0x00007f08bdb85056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6
      #1 0x00007f08bde92b25 in ?? () from /usr/lib64/libssl.so.6
      #2 0x00007f08bde92e26 in SSL_CTX_use_PrivateKey_file () from /usr/lib64/libssl.so.6
      #3 0x00007f08be0d3af6 in my_ssl_set_certs (mysql=0x7f08985a89e0) at /home/buildbot/src/build/libmariadb/ma_secure.c:229
      #4 my_ssl_init (mysql=0x7f08985a89e0) at /home/buildbot/src/build/libmariadb/ma_secure.c:336
      #5 0x00007f08be0d2eee in send_client_reply_packet (mpv=0x7f08b4de0e20, pkt=0x7f08b4de0dd0 "\235\360\216\305@\265\206,@\312k\354ķ\363c\027", pkt_len=<value optimized out>)
      at /home/buildbot/src/build/libmariadb/my_auth.c:315
      #6 client_mpvio_write_packet (mpv=0x7f08b4de0e20, pkt=0x7f08b4de0dd0 "\235\360\216\305@\265\206,@\312k\354ķ\363c\027", pkt_len=<value optimized out>)
      at /home/buildbot/src/build/libmariadb/my_auth.c:474
      #7 0x00007f08be0d25bc in native_password_auth_client (vio=0x7f08b4de0e20, mysql=0x7f08985a89e0) at /home/buildbot/src/build/libmariadb/my_auth.c:114
      #8 0x00007f08be0d27f9 in run_plugin_auth (mysql=0x7f08985a89e0, data=0x7f08985aef13 "", data_len=21, data_plugin=0x7f08be0e188b "mysql_native_password",
      db=0x7f08ac01c0d8 "dscmondb") at /home/buildbot/src/build/libmariadb/my_auth.c:606
      #9 0x00007f08be0c5e1f in mthd_my_real_connect (mysql=0x7f08985a89e0, host=0x7f08ac01bff8 "r710t01-v03", user=0x7f08ac0309f8 "dscwriterinst",
      passwd=0x7f08ac01b798 "8e7e92a0-165e-4f01-82c1-28a5e4fa7b7b", db=0x7f08ac01c0d8 "dscmondb", port=3306, unix_socket=0x0, client_flag=128)
      at /home/buildbot/src/build/libmariadb/libmariadb.c:1888
      #10 0x00007f08be2fd903 in MySqlClient::connect (this=0x13a9ed8, server="r710t01-v03", user="dscwriterinst", password="8e7e92a0-165e-4f01-82c1-28a5e4fa7b7b", database="dscmondb",
      ssl_key="/opt/adc/adc/etc/dscweb.pem", ssl_cert="/opt/adc/adc/etc/dscweb.pem", ssl_ca="/opt/adc/adc/etc/dscweb.pem", err_string="") at MySqlClient.cpp:47
      #11 0x000000000040e155 in connect (this=0x13a97c0) at /vob/adc/router/db_client/DataBaseClient.h:142
      #12 UlticomMonitorD::MonitorThread::connectToMySql (this=0x13a97c0) at MonitorThread.cpp:403
      #13 0x0000000000410e8c in UlticomMonitorD::MonitorThread::processProtoMessage (this=0x13a97c0, proto_buf=...,
      recv_buf=0x7f08b4de16a0 "\n\027\b\001\022\005INST1\032\nDSC_CLSTR1 \003\032]\n\vr710t01-v03\022\rdscwriterinst\032$8e7e92a0-165e-4f01-82c1-28a5e4fa7b7b\"\bdscmondb*\vr710t01-v030\b8\001", message_size=120) at MonitorThread.cpp:557
      #14 0x00000000004116e5 in UlticomMonitorD::MonitorThread::threadMain (this=0x13a97c0, pArguments=<value optimized out>) at MonitorThread.cpp:331
      #15 0x00007f08be80096b in UlticomUtils::_thread_internal_main_routine (arg=0x13a97c0) at Thread.cpp:108
      #16 0x0000003da8e07851 in start_thread () from /lib64/libpthread.so.0
      #17 0x0000003da8ae811d in clone () from /lib64/libc.so.6

      Attachments

        Activity

          mpatil Mangala added a comment -

          Hello Georg Richter,

          Any news about this??

          We also saw another core dump in another box. This looks very similar to the one I sent earlier. Little different at the end. Thought it may also help you for further debugging.

          Core was generated by `/opt/adc/adc/bin/monitord'.
          Program terminated with signal 11, Segmentation fault.
          #0 0x00007fdec3b43203 in X509_get_issuer_name () from /usr/lib64/libcrypto.so.6
          Missing separate debuginfos, use: debuginfo-install ulcm-dscinstance-5.0-0.x86_64
          (gdb) bt
          #0 0x00007fdec3b43203 in X509_get_issuer_name () from /usr/lib64/libcrypto.so.6
          #1 0x00007fdec3e1de58 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.6
          #2 0x00007fdec3e16741 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.6
          #3 0x00007fdec3e194ab in ssl3_connect () from /usr/lib64/libssl.so.6
          #4 0x00007fdec407254f in my_ssl_connect (ssl=0x7fdea801a840)
          at /home/buildbot/src/build/libmariadb/ma_secure.c:384
          #5 0x00007fdec4071f03 in send_client_reply_packet (mpv=0x7fdec0ffce20,
          pkt=0x7fdec0ffcdd0 "\022\351\235\310\v\275\066\361\240StG;MS\232s\005", <incomplete sequence \354\200>, pkt_len=<value optimized out>) at /home/buildbot/src/build/libmariadb/my_auth.c:319
          #6 client_mpvio_write_packet (mpv=0x7fdec0ffce20,
          pkt=0x7fdec0ffcdd0 "\022\351\235\310\v\275\066\361\240StG;MS\232s\005", <incomplete sequence \354\200>, pkt_len=<value optimized out>) at /home/buildbot/src/build/libmariadb/my_auth.c:474
          #7 0x00007fdec40715bc in native_password_auth_client (vio=0x7fdec0ffce20, mysql=0x7fdea8011fa0)
          at /home/buildbot/src/build/libmariadb/my_auth.c:114
          #8 0x00007fdec40717f9 in run_plugin_auth (mysql=0x7fdea8011fa0, data=0x7fdea8018563 "",
          data_len=21, data_plugin=0x7fdec408088b "mysql_native_password", db=0x7fdea4011eb8 "dscmondb")
          at /home/buildbot/src/build/libmariadb/my_auth.c:606
          #9 0x00007fdec4064e1f in mthd_my_real_connect (mysql=0x7fdea8011fa0, host=0x7fdea4011e58 "beijing",
          user=0x7fdea4011e88 "dscwriterinst",
          passwd=0x7fdea4011d98 "a58a68f6-8ec1-4e02-b478-d7003c020322", db=0x7fdea4011eb8 "dscmondb",
          port=3306, unix_socket=0x0, client_flag=128)
          at /home/buildbot/src/build/libmariadb/libmariadb.c:1888
          #10 0x00007fdec429c903 in MySqlClient::connect (this=0x1d8d898, server="beijing", user=
          "dscwriterinst", password="a58a68f6-8ec1-4e02-b478-d7003c020322", database="dscmondb", ssl_key=
          "/opt/adc/adc/etc/dscweb.pem", ssl_cert="/opt/adc/adc/etc/dscweb.pem", ssl_ca=
          "/opt/adc/adc/etc/dscweb.pem", err_string="") at MySqlClient.cpp:47
          #11 0x000000000040e155 in connect (this=0x1d8d320) at /vob/adc/router/db_client/DataBaseClient.h:142
          #12 UlticomMonitorD::MonitorThread::connectToMySql (this=0x1d8d320) at MonitorThread.cpp:403
          #13 0x0000000000410e8c in UlticomMonitorD::MonitorThread::processProtoMessage (this=0x1d8d320,
          proto_buf=...,
          recv_buf=0x7fdec0ffd6a0 "\n\025\b\001\022\005inst2\032\bcluster1 \003\032U\n\abeijing\022\rdscwriterinst\032$a58a68f6-8ec1-4e02-b478-d7003c020322\"\bdscmondb*\abeijing0\b8\001", message_size=110)
          at MonitorThread.cpp:557
          #14 0x00000000004116e5 in UlticomMonitorD::MonitorThread::threadMain (this=0x1d8d320,
          pArguments=<value optimized out>) at MonitorThread.cpp:331
          #15 0x00007fdec479f96b in UlticomUtils::_thread_internal_main_routine (arg=0x1d8d320)
          at Thread.cpp:108
          #16 0x000000340ec07851 in start_thread () from /lib64/libpthread.so.0
          #17 0x000000340e8e890d in clone () from /lib64/libc.so.6

          mpatil Mangala added a comment - Hello Georg Richter, Any news about this?? We also saw another core dump in another box. This looks very similar to the one I sent earlier. Little different at the end. Thought it may also help you for further debugging. Core was generated by `/opt/adc/adc/bin/monitord'. Program terminated with signal 11, Segmentation fault. #0 0x00007fdec3b43203 in X509_get_issuer_name () from /usr/lib64/libcrypto.so.6 Missing separate debuginfos, use: debuginfo-install ulcm-dscinstance-5.0-0.x86_64 (gdb) bt #0 0x00007fdec3b43203 in X509_get_issuer_name () from /usr/lib64/libcrypto.so.6 #1 0x00007fdec3e1de58 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.6 #2 0x00007fdec3e16741 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.6 #3 0x00007fdec3e194ab in ssl3_connect () from /usr/lib64/libssl.so.6 #4 0x00007fdec407254f in my_ssl_connect (ssl=0x7fdea801a840) at /home/buildbot/src/build/libmariadb/ma_secure.c:384 #5 0x00007fdec4071f03 in send_client_reply_packet (mpv=0x7fdec0ffce20, pkt=0x7fdec0ffcdd0 "\022\351\235\310\v\275\066\361\240StG;MS\232s\005", <incomplete sequence \354\200>, pkt_len=<value optimized out>) at /home/buildbot/src/build/libmariadb/my_auth.c:319 #6 client_mpvio_write_packet (mpv=0x7fdec0ffce20, pkt=0x7fdec0ffcdd0 "\022\351\235\310\v\275\066\361\240StG;MS\232s\005", <incomplete sequence \354\200>, pkt_len=<value optimized out>) at /home/buildbot/src/build/libmariadb/my_auth.c:474 #7 0x00007fdec40715bc in native_password_auth_client (vio=0x7fdec0ffce20, mysql=0x7fdea8011fa0) at /home/buildbot/src/build/libmariadb/my_auth.c:114 #8 0x00007fdec40717f9 in run_plugin_auth (mysql=0x7fdea8011fa0, data=0x7fdea8018563 "", data_len=21, data_plugin=0x7fdec408088b "mysql_native_password", db=0x7fdea4011eb8 "dscmondb") at /home/buildbot/src/build/libmariadb/my_auth.c:606 #9 0x00007fdec4064e1f in mthd_my_real_connect (mysql=0x7fdea8011fa0, host=0x7fdea4011e58 "beijing", user=0x7fdea4011e88 "dscwriterinst", passwd=0x7fdea4011d98 "a58a68f6-8ec1-4e02-b478-d7003c020322", db=0x7fdea4011eb8 "dscmondb", port=3306, unix_socket=0x0, client_flag=128) at /home/buildbot/src/build/libmariadb/libmariadb.c:1888 #10 0x00007fdec429c903 in MySqlClient::connect (this=0x1d8d898, server="beijing", user= "dscwriterinst", password="a58a68f6-8ec1-4e02-b478-d7003c020322", database="dscmondb", ssl_key= "/opt/adc/adc/etc/dscweb.pem", ssl_cert="/opt/adc/adc/etc/dscweb.pem", ssl_ca= "/opt/adc/adc/etc/dscweb.pem", err_string="") at MySqlClient.cpp:47 #11 0x000000000040e155 in connect (this=0x1d8d320) at /vob/adc/router/db_client/DataBaseClient.h:142 #12 UlticomMonitorD::MonitorThread::connectToMySql (this=0x1d8d320) at MonitorThread.cpp:403 #13 0x0000000000410e8c in UlticomMonitorD::MonitorThread::processProtoMessage (this=0x1d8d320, proto_buf=..., recv_buf=0x7fdec0ffd6a0 "\n\025\b\001\022\005inst2\032\bcluster1 \003\032U\n\abeijing\022\rdscwriterinst\032$a58a68f6-8ec1-4e02-b478-d7003c020322\"\bdscmondb*\abeijing0\b8\001", message_size=110) at MonitorThread.cpp:557 #14 0x00000000004116e5 in UlticomMonitorD::MonitorThread::threadMain (this=0x1d8d320, pArguments=<value optimized out>) at MonitorThread.cpp:331 #15 0x00007fdec479f96b in UlticomUtils::_thread_internal_main_routine (arg=0x1d8d320) at Thread.cpp:108 #16 0x000000340ec07851 in start_thread () from /lib64/libpthread.so.0 #17 0x000000340e8e890d in clone () from /lib64/libc.so.6
          georg Georg Richter added a comment -

          Hi,

          can you please try with latest revision 148 from launchpad lp:mariadb-native-client ?

          georg Georg Richter added a comment - Hi, can you please try with latest revision 148 from launchpad lp:mariadb-native-client ?
          mpatil Mangala added a comment -

          I tried the latest revision 148

          Same issue may be core is slightly different.

          #0 0x00007feaea958056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6
          Missing separate debuginfos, use: debuginfo-install ulcm-dscinstance-5.0-0.x86_64
          (gdb) bt
          #0 0x00007feaea958056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6
          #1 0x00007feaeac65b25 in ?? () from /usr/lib64/libssl.so.6
          #2 0x00007feaeac65e26 in SSL_CTX_use_PrivateKey_file () from /usr/lib64/libssl.so.6
          #3 0x00007feaeaeaca0b in my_ssl_set_certs (mysql=0x7feac4011fa0)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/ma_secure.c:231
          #4 my_ssl_init (mysql=0x7feac4011fa0)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/ma_secure.c:321
          #5 0x00007feaeaeabfee in send_client_reply_packet (mpv=0x7feae21e2e20,
          pkt=0x7feae21e2dd0 "\361\351C5\272=\267:\232*\261\254M8h\204TY\026\060",
          pkt_len=<value optimized out>)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:315
          #6 client_mpvio_write_packet (mpv=0x7feae21e2e20,
          pkt=0x7feae21e2dd0 "\361\351C5\272=\267:\232*\261\254M8h\204TY\026\060",
          pkt_len=<value optimized out>)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:474
          #7 0x00007feaeaeab6bc in native_password_auth_client (vio=0x7feae21e2e20, mysql=0x7feac4011fa0)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:114
          #8 0x00007feaeaeab8f9 in run_plugin_auth (mysql=0x7feac4011fa0, data=0x7feac4018563 "",
          data_len=21, data_plugin=0x7feaeaeba90f "mysql_native_password", db=0x7fead4011eb8 "dscmondb")
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:606
          #9 0x00007feaeae9a421 in mthd_my_real_connect (mysql=0x7feac4011fa0, host=0x7fead4011e58 "beijing",
          user=0x7fead4011e88 "dscwriterinst",
          passwd=0x7fead4011d98 "a58a68f6-8ec1-4e02-b478-d7003c020322", db=0x7fead4011eb8 "dscmondb",
          port=3306, unix_socket=0x0, client_flag=128)
          at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/libmariadb.c:1890
          #10 0x00007feaeb0d7903 in MySqlClient::connect (this=0x1737758, server="beijing", user=
          "dscwriterinst", password="a58a68f6-8ec1-4e02-b478-d7003c020322", database="dscmondb", ssl_key=
          "/opt/adc/adc/etc/dscweb.pem", ssl_cert="/opt/adc/adc/etc/dscweb.pem", ssl_ca=
          "/opt/adc/adc/etc/dscweb.pem", err_string="") at MySqlClient.cpp:47
          #11 0x000000000040f1c5 in connect (this=0x1737040) at /vob/adc/router/db_client/DataBaseClient.h:142
          #12 UlticomMonitorD::MonitorThread::connectToMySql (this=0x1737040) at MonitorThread.cpp:403
          #13 0x0000000000410eb5 in UlticomMonitorD::MonitorThread::processProtoMessage (this=0x1737040,
          proto_buf=...,
          recv_buf=0x7feae21e36a0 "\n\025\b\001\022\005inst2\032\bcluster1 \003\032U\n\abeijing\022\rdscwriterinst\032$a58a68f6-8ec1-4e02-b478-d7003c020322\"\bdscmondb*\abeijing0\b8\001", message_size=110)
          at MonitorThread.cpp:568
          #14 0x00000000004116b5 in UlticomMonitorD::MonitorThread::threadMain (this=0x1737040,
          pArguments=<value optimized out>) at MonitorThread.cpp:331
          #15 0x00007feaeb5da96b in UlticomUtils::_thread_internal_main_routine (arg=0x1737040)
          at Thread.cpp:108
          #16 0x000000340ec07851 in start_thread () from /lib64/libpthread.so.0
          #17 0x000000340e8e890d in clone () from /lib64/libc.so.6

          mpatil Mangala added a comment - I tried the latest revision 148 Same issue may be core is slightly different. #0 0x00007feaea958056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6 Missing separate debuginfos, use: debuginfo-install ulcm-dscinstance-5.0-0.x86_64 (gdb) bt #0 0x00007feaea958056 in EVP_PKEY_copy_parameters () from /usr/lib64/libcrypto.so.6 #1 0x00007feaeac65b25 in ?? () from /usr/lib64/libssl.so.6 #2 0x00007feaeac65e26 in SSL_CTX_use_PrivateKey_file () from /usr/lib64/libssl.so.6 #3 0x00007feaeaeaca0b in my_ssl_set_certs (mysql=0x7feac4011fa0) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/ma_secure.c:231 #4 my_ssl_init (mysql=0x7feac4011fa0) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/ma_secure.c:321 #5 0x00007feaeaeabfee in send_client_reply_packet (mpv=0x7feae21e2e20, pkt=0x7feae21e2dd0 "\361\351C5\272=\267:\232*\261\254M8h\204TY\026\060", pkt_len=<value optimized out>) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:315 #6 client_mpvio_write_packet (mpv=0x7feae21e2e20, pkt=0x7feae21e2dd0 "\361\351C5\272=\267:\232*\261\254M8h\204TY\026\060", pkt_len=<value optimized out>) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:474 #7 0x00007feaeaeab6bc in native_password_auth_client (vio=0x7feae21e2e20, mysql=0x7feac4011fa0) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:114 #8 0x00007feaeaeab8f9 in run_plugin_auth (mysql=0x7feac4011fa0, data=0x7feac4018563 "", data_len=21, data_plugin=0x7feaeaeba90f "mysql_native_password", db=0x7fead4011eb8 "dscmondb") at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/my_auth.c:606 #9 0x00007feaeae9a421 in mthd_my_real_connect (mysql=0x7feac4011fa0, host=0x7fead4011e58 "beijing", user=0x7fead4011e88 "dscwriterinst", passwd=0x7fead4011d98 "a58a68f6-8ec1-4e02-b478-d7003c020322", db=0x7fead4011eb8 "dscmondb", port=3306, unix_socket=0x0, client_flag=128) at /u/cm/~maria-captains/mariadb-native-client/trunk/libmariadb/libmariadb.c:1890 #10 0x00007feaeb0d7903 in MySqlClient::connect (this=0x1737758, server="beijing", user= "dscwriterinst", password="a58a68f6-8ec1-4e02-b478-d7003c020322", database="dscmondb", ssl_key= "/opt/adc/adc/etc/dscweb.pem", ssl_cert="/opt/adc/adc/etc/dscweb.pem", ssl_ca= "/opt/adc/adc/etc/dscweb.pem", err_string="") at MySqlClient.cpp:47 #11 0x000000000040f1c5 in connect (this=0x1737040) at /vob/adc/router/db_client/DataBaseClient.h:142 #12 UlticomMonitorD::MonitorThread::connectToMySql (this=0x1737040) at MonitorThread.cpp:403 #13 0x0000000000410eb5 in UlticomMonitorD::MonitorThread::processProtoMessage (this=0x1737040, proto_buf=..., recv_buf=0x7feae21e36a0 "\n\025\b\001\022\005inst2\032\bcluster1 \003\032U\n\abeijing\022\rdscwriterinst\032$a58a68f6-8ec1-4e02-b478-d7003c020322\"\bdscmondb*\abeijing0\b8\001", message_size=110) at MonitorThread.cpp:568 #14 0x00000000004116b5 in UlticomMonitorD::MonitorThread::threadMain (this=0x1737040, pArguments=<value optimized out>) at MonitorThread.cpp:331 #15 0x00007feaeb5da96b in UlticomUtils::_thread_internal_main_routine (arg=0x1737040) at Thread.cpp:108 #16 0x000000340ec07851 in start_thread () from /lib64/libpthread.so.0 #17 0x000000340e8e890d in clone () from /lib64/libc.so.6
          mpatil Mangala added a comment -

          This may give some more hints...When we saw these core dumps in the client connection... we also saw these errors in client
          But with the same credentials and keys subsequent connections are ok...

          Error Code: 2026 Error String: SSL connection error: key type mismatch
          Error Code: 2003 Error String: Can't connect to MySQL server on 'beijing' (107)
          Error Code: 2013 Error String: Lost connection to MySQL server during query
          Error Code: 1049 Error String: Unknown database 'dscmondb'

          mpatil Mangala added a comment - This may give some more hints...When we saw these core dumps in the client connection... we also saw these errors in client But with the same credentials and keys subsequent connections are ok... Error Code: 2026 Error String: SSL connection error: key type mismatch Error Code: 2003 Error String: Can't connect to MySQL server on 'beijing' (107) Error Code: 2013 Error String: Lost connection to MySQL server during query Error Code: 1049 Error String: Unknown database 'dscmondb'
          georg Georg Richter added a comment -

          Fixed in rev. 151

          georg Georg Richter added a comment - Fixed in rev. 151

          People

            georg Georg Richter
            mpatil Mangala
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.