[MDEV-27403] Crash in evp_md_ctx_clear_digest/my_md5 while running mysql_install_db on macOS Created: 2022-01-02  Updated: 2022-01-22  Resolved: 2022-01-20

Status: Closed
Project: MariaDB Server
Component/s: Compiling, SSL
Affects Version/s: 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.2.42, 10.3.33, 10.4.23, 10.5.14, 10.6.6, 10.7.2, 10.8.1

Type: Bug Priority: Major
Reporter: Valerii Kravchuk Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: macos, mysql_install_db, openssl
Environment:

macOS 10.13.6 High Sierra, OpenSSL 3.0.0


Issue Links:
PartOf
is part of MDEV-25785 Add support for OpenSSL 3.0 Closed

 Description   

I get crashes like this:

Yuliyas-Air:maria10.7 Valerii$ scripts/mysql_install_db --no-defaults
Installing MariaDB/MySQL system tables in './data' ...
2022-01-02 18:59:54 0 [Warning] InnoDB: innodb_open_files 300 should not be greater than the open_files_limit 256
220102 18:59:55 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.7.2-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467998 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fa73808d218
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ffeebd34960 thread_stack 0x49000
0   mariadbd                            0x00000001047c0b6c my_print_stacktrace + 60
Printing to addr2line failed
0   mariadbd                            0x0000000103f61fdd handle_fatal_signal + 733
0   libsystem_platform.dylib            0x00007fff56436f5a _sigtramp + 26
0   ???                                 0x00007ffeebd313a8 0x0 + 140732854899624
0   libcrypto.3.dylib                   0x00000001060f6a1c evp_md_ctx_clear_digest + 220
0   libcrypto.3.dylib                   0x00000001060f6a8c EVP_MD_CTX_reset + 58
0   mariadbd                            0x00000001044985b1 my_md5 + 65
0   mariadbd                            0x00000001042d0d1a _ZN10TABLE_LIST8calc_md5EPc + 58
0   mariadbd                            0x00000001042c05a9 _Z17mysql_create_viewP3THDP10TABLE_LIST21enum_view_create_mode + 2601
0   mariadbd                            0x00000001041dc36f _Z21mysql_execute_commandP3THDb + 6911
0   mariadbd                            0x00000001041d66fb _Z11mysql_parseP3THDPcjP12Parser_state + 731
0   mariadbd                            0x00000001041d6123 _Z9bootstrapP13st_mysql_file + 1043
0   mariadbd                            0x00000001040b2bf3 _Z11mysqld_mainiPPc + 3779
0   libdyld.dylib                       0x00007fff56128015 start + 1
0   ???                                 0x000000000000000c 0x0 + 12
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fa7380a7630): CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT
  Host,
  User,
  IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password,
  IF(JSON_VALUE(Priv, '$.access') &         1, 'Y', 'N') AS Select_priv,
  IF(JSON_VALUE(Priv, '$.access') &         2, 'Y', 'N') AS Insert_priv,
  IF(JSON_VALUE(Priv, '$.access') &         4, 'Y', 'N') AS Update_priv,
  IF(JSON_VALUE(Priv, '$.access') &         8, 'Y', 'N') AS Delete_priv,
  IF(JSON_VALUE(Priv, '$.access') &        16, 'Y', 'N') AS Create_priv,
  IF(JSON_VALUE(Priv, '$.access') &        32, 'Y', 'N') AS Drop_priv,
  IF(JSON_VALUE(Priv, '$.access') &        64, 'Y', 'N') AS Reload_priv,
  IF(JSON_VALUE(Priv, '$.access') &       128, 'Y', 'N') AS Shutdown_priv,
  IF(JSON_VALUE(Priv, '$.access') &       256, 'Y', 'N') AS Process_priv,
  IF(JSON_VALUE(Priv, '$.access') &       512, 'Y', 'N') AS File_priv,
  IF(JSON_VALUE(Priv, '$.access') &      1024, 'Y', 'N') AS Grant_priv,
  IF(JSON_VALUE(Priv, '$.access') &      2048, 'Y', 'N') AS References_priv,
  IF(JSON_VALUE(Priv, '$.access') &      4096, 'Y', 'N') AS Index_priv,
  IF(JSON_VALUE(Priv, '$.access') &      8192, 'Y', 'N') AS Alter_priv,
  IF(JSON_VALUE(Priv, '$.access') &     16384, 'Y', 'N') AS Show_db_priv,
  IF(JSON_VALUE(Priv, '$.access') &     32768, 'Y', 'N') AS Super_priv,
  IF(JSON_VALUE(Priv, '$.access') &     65536, 'Y', 'N') AS Create_tmp_table_priv,
  IF(JSON_VALUE(Priv, '$.access') &    131072, 'Y', 'N') AS Lock_tables_priv,
  IF(JSON_VALUE(Priv, '$.access') &    262144, 'Y', 'N') AS Execute_priv,
  IF(JSON_VALUE(Priv, '$.access') &    524288, 'Y', 'N') AS Repl_slave_priv,
  IF(JSON_VALUE(Priv, '$.access') &   1048576, 'Y', 'N') AS Repl_client_priv,
  IF(JSON_VALUE(Priv, '$.access') &   2097152, 'Y', 'N') AS Create_view_priv,
  IF(JSON_VALUE(Priv, '$.access') &   4194304, 'Y', 'N') AS Show_view_priv,
  IF(JSON_VALUE(Priv, '$.access') &   8388608, 'Y', 'N') AS Create_routine_priv,
  IF(JSON_VALUE(Priv, '$.access') &  16777216, 'Y', 'N') AS Alter_routine_priv,
  IF(JSON_VALUE(Priv, '$.access') &  33554432, 'Y', 'N') AS Create_user_priv,
  IF(JSON_VALUE(Priv, '$.access') &  67108864, 'Y', 'N') AS Event_priv,
  IF(JSON_VALUE(Priv, '$.access') & 134217728, 'Y', 'N') AS Trigger_priv,
  IF(JSON_VALUE(Priv, '$.access') & 268435456, 'Y', 'N') AS Create_tablespace_priv,
  IF(JSON_VALUE(Priv, '$.access') & 536870912, 'Y', 'N') AS Delete_history_priv,
  ELT(IFNULL(JSON_VALUE(Priv, '$.ssl_type'), 0) + 1, '', 'ANY','X509', 'SPECIFIED') AS ssl_type,
  IFNULL(JSON_VALUE(Priv, '$.ssl_cipher'), '') AS ssl_cipher,
  IFNULL(JSON_VALUE(Priv, '$.x509_issuer'), '') AS x509_issuer,
  IFNULL(JSON_VALUE(Priv, '$.x509_subject'), '') AS x509_subject,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_questions'), 0) AS UNSIGNED) AS max_questions,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_updates'), 0) AS UNSIGNED) AS max_updates,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_connections'), 0) AS UNSIGNED) AS max_connections,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_user_connections'), 0) AS SIGNED) AS max_user_connections,
  IFNULL(JSON_VALUE(Priv, '$.plugin'), '') AS plugin,
  IFNULL(JSON_VALUE(Priv, '$.authentication_string'), '') AS authentication_string,
  IF(IFNULL(JSON_VALUE(Priv, '$.password_last_changed'), 1) = 0, 'Y', 'N') AS password_expired,
  ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role,
  IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time
  FROM global_priv;
 
Connection ID (thread ID): 1
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: /cores/core.%P
scripts/mysql_install_db: line 525: 35533 Segmentation fault: 11  "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K
 
Installation of system tables failed!  Examine the logs in
./data for more information.
 
The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:
 
    shell> scripts/mysql_install_db --defaults-file=~/.my.cnf
 
You can also try to start the mysqld daemon with:
 
    shell> ./bin/mariadbd --skip-grant-tables --general-log &
 
and use the command line tool ./bin/mariadb
to connect to the mysql database and look at the grant tables:
 
    shell> ./bin/mysql -u root mysql
    mysql> show tables;
 
Try 'mysqld --help' if you have problems with paths.  Using
--general-log gives you a log in ./data that may be helpful.
 
The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss
 
Please check all of the above before submitting a bug report
at https://mariadb.org/jira
 
Yuliyas-Air:maria10.7 Valerii$ openssl version
OpenSSL 3.0.0 7 sep 2021 (Library: OpenSSL 3.0.0 7 sep 2021)

while installing MariaDB Server 10.4+ built from GitHub source on maxOS 10.13.6. This leaves me with semi-installed and semi-functional server.

Build was configured as usual:

cmake .. -DCMAKE_INSTALL_PREFIX=/Users/Valerii/dbs/maria10.7 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DWITH_EMBEDDED_SERVER=OFF -DPLUGIN_TOKUDB=NO -DWITH_SSL=system -DENABLE_DTRACE=1



 Comments   
Comment by Sergei Golubchik [ 2022-01-02 ]

OpenSSL 3.0 is not supported yet. See MDEV-25785, but also MDEV-26950, MDEV-27236, and MDEV-26302.

Comment by Valerii Kravchuk [ 2022-01-22 ]

Why this is closed as "Fixed" in 10.4.23, if a build from today's GitHub source still crashes the same way:

Yuliyas-Air:~ Valerii$ cd dbs/maria10.4
Yuliyas-Air:maria10.4 Valerii$ bin/mysqld --version
bin/mysqld  Ver 10.4.23-MariaDB for osx10.13 on x86_64 (Source distribution)
Yuliyas-Air:maria10.4 Valerii$ rm -rf data/*
Yuliyas-Air:maria10.4 Valerii$ scripts/mysql_install_db --no-defaults
Installing MariaDB/MySQL system tables in './data' ...
2022-01-22 19:54:32 0 [Warning] InnoDB: innodb_open_files 300 should not be greater than the open_files_limit 256
220122 19:54:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.4.23-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467769 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fc714840608
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ffeea378b38 thread_stack 0x49000
0   mysqld                              0x00000001060a2cac my_print_stacktrace + 60
0   mysqld                              0x00000001058b4166 handle_fatal_signal + 742
0   libsystem_platform.dylib            0x00007fff659f8f5a _sigtramp + 26
0   ???                                 0x000000010b809314 0x0 + 4487942932
0   libcrypto.3.dylib                   0x000000010782aea0 evp_md_ctx_clear_digest + 220
0   libcrypto.3.dylib                   0x000000010782af10 EVP_MD_CTX_reset + 58
0   mysqld                              0x0000000105db93a1 my_md5 + 65
0   mysqld                              0x0000000105bc155a _ZN10TABLE_LIST8calc_md5EPc + 58
0   mysqld                              0x0000000105bb26ef _Z17mysql_create_viewP3THDP10TABLE_LIST21enum_view_create_mode + 2335
0   mysqld                              0x0000000105aeae18 _Z21mysql_execute_commandP3THD + 12008
0   mysqld                              0x0000000105ae44bd _Z11mysql_parseP3THDPcjP12Parser_statebb + 525
0   mysqld                              0x0000000105ae3e04 _Z9bootstrapP13st_mysql_file + 628
0   mysqld                              0x00000001059dc60d _Z11mysqld_mainiPPc + 3677
0   libdyld.dylib                       0x00007fff656ea015 start + 1
0   ???                                 0x000000000000000c 0x0 + 12
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fc7148ff620): CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT
  Host,
  User,
  IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password,
  IF(JSON_VALUE(Priv, '$.access') &         1, 'Y', 'N') AS Select_priv,
  IF(JSON_VALUE(Priv, '$.access') &         2, 'Y', 'N') AS Insert_priv,
  IF(JSON_VALUE(Priv, '$.access') &         4, 'Y', 'N') AS Update_priv,
  IF(JSON_VALUE(Priv, '$.access') &         8, 'Y', 'N') AS Delete_priv,
  IF(JSON_VALUE(Priv, '$.access') &        16, 'Y', 'N') AS Create_priv,
  IF(JSON_VALUE(Priv, '$.access') &        32, 'Y', 'N') AS Drop_priv,
  IF(JSON_VALUE(Priv, '$.access') &        64, 'Y', 'N') AS Reload_priv,
  IF(JSON_VALUE(Priv, '$.access') &       128, 'Y', 'N') AS Shutdown_priv,
  IF(JSON_VALUE(Priv, '$.access') &       256, 'Y', 'N') AS Process_priv,
  IF(JSON_VALUE(Priv, '$.access') &       512, 'Y', 'N') AS File_priv,
  IF(JSON_VALUE(Priv, '$.access') &      1024, 'Y', 'N') AS Grant_priv,
  IF(JSON_VALUE(Priv, '$.access') &      2048, 'Y', 'N') AS References_priv,
  IF(JSON_VALUE(Priv, '$.access') &      4096, 'Y', 'N') AS Index_priv,
  IF(JSON_VALUE(Priv, '$.access') &      8192, 'Y', 'N') AS Alter_priv,
  IF(JSON_VALUE(Priv, '$.access') &     16384, 'Y', 'N') AS Show_db_priv,
  IF(JSON_VALUE(Priv, '$.access') &     32768, 'Y', 'N') AS Super_priv,
  IF(JSON_VALUE(Priv, '$.access') &     65536, 'Y', 'N') AS Create_tmp_table_priv,
  IF(JSON_VALUE(Priv, '$.access') &    131072, 'Y', 'N') AS Lock_tables_priv,
  IF(JSON_VALUE(Priv, '$.access') &    262144, 'Y', 'N') AS Execute_priv,
  IF(JSON_VALUE(Priv, '$.access') &    524288, 'Y', 'N') AS Repl_slave_priv,
  IF(JSON_VALUE(Priv, '$.access') &   1048576, 'Y', 'N') AS Repl_client_priv,
  IF(JSON_VALUE(Priv, '$.access') &   2097152, 'Y', 'N') AS Create_view_priv,
  IF(JSON_VALUE(Priv, '$.access') &   4194304, 'Y', 'N') AS Show_view_priv,
  IF(JSON_VALUE(Priv, '$.access') &   8388608, 'Y', 'N') AS Create_routine_priv,
  IF(JSON_VALUE(Priv, '$.access') &  16777216, 'Y', 'N') AS Alter_routine_priv,
  IF(JSON_VALUE(Priv, '$.access') &  33554432, 'Y', 'N') AS Create_user_priv,
  IF(JSON_VALUE(Priv, '$.access') &  67108864, 'Y', 'N') AS Event_priv,
  IF(JSON_VALUE(Priv, '$.access') & 134217728, 'Y', 'N') AS Trigger_priv,
  IF(JSON_VALUE(Priv, '$.access') & 268435456, 'Y', 'N') AS Create_tablespace_priv,
  IF(JSON_VALUE(Priv, '$.access') & 536870912, 'Y', 'N') AS Delete_history_priv,
  ELT(IFNULL(JSON_VALUE(Priv, '$.ssl_type'), 0) + 1, '', 'ANY','X509', 'SPECIFIED') AS ssl_type,
  IFNULL(JSON_VALUE(Priv, '$.ssl_cipher'), '') AS ssl_cipher,
  IFNULL(JSON_VALUE(Priv, '$.x509_issuer'), '') AS x509_issuer,
  IFNULL(JSON_VALUE(Priv, '$.x509_subject'), '') AS x509_subject,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_questions'), 0) AS UNSIGNED) AS max_questions,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_updates'), 0) AS UNSIGNED) AS max_updates,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_connections'), 0) AS UNSIGNED) AS max_connections,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_user_connections'), 0) AS SIGNED) AS max_user_connections,
  IFNULL(JSON_VALUE(Priv, '$.plugin'), '') AS plugin,
  IFNULL(JSON_VALUE(Priv, '$.authentication_string'), '') AS authentication_string,
  IF(IFNULL(JSON_VALUE(Priv, '$.password_last_changed'), 1) = 0, 'Y', 'N') AS password_expired,
  ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role,
  IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role,
  CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time
  FROM global_priv;
 
Connection ID (thread ID): 6
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: /cores/core.%P
scripts/mysql_install_db: line 524: 65033 Segmentation fault: 11  "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K
 
Installation of system tables failed!  Examine the logs in
./data for more information.
 
The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:
 
    shell> scripts/mysql_install_db --defaults-file=~/.my.cnf
 
You can also try to start the mysqld daemon with:
 
    shell> ./bin/mysqld --skip-grant-tables --general-log &
 
and use the command line tool ./bin/mysql
to connect to the mysql database and look at the grant tables:
 
    shell> ./bin/mysql -u root mysql
    mysql> show tables;
 
Try 'mysqld --help' if you have problems with paths.  Using
--general-log gives you a log in ./data that may be helpful.
 
The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss
 
Please check all of the above before submitting a bug report
at http://mariadb.org/jira

I see no related commits and related task to add OpenSSL 3.0 support is closed for 10.8.1+ only. Please, clarify.

Comment by Sergei Golubchik [ 2022-01-22 ]

it says "fixed in 10.4.23", there's no guarantee it's fixed in an earlier commit.

technically, it's not merged yet in 10.4, the fix is in 10.2 branch, commit c9beef4315.

Generated at Thu Feb 08 09:52:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.