Daniel Black added a comment - Yesterday - edited I tried on this "SELECT * FROM sometable WHERE something=1 AND (SELECT 1 FROM information_schema.users) = 1" without preproduction, so it looks like this is part of the query and the rest of it is significant enough to cause the bug trigger. Do you have the rest of the query?

Matteo Golinelli added a comment - Yesterday Here are some instructions and code to reproduce the bug in a (semi-)minimal setup. You can find a README.md with all the instructions to run this using Docker and Docker Compose. https://github.com/Golim/mariadb-reproduce-crash

Daniel Black added a comment - Yesterday Confirmed. Thanks for example. With --skip-grant-tables there isn't the full initialization to make this possible. crash Thread 2 received signal SIGSEGV, Segmentation fault. [Switching to Thread 87635.87680] 0x0000000000bb5f97 in pfs_unlock_mutex_v1 (mutex=0xa676f6c2e6374) at /home/dan/repos/mariadb-server-main/storage/perfschema/pfs.cc:3667 3667 pfs_mutex->m_last_locked= 0; (rr) bt #0 0x0000000000bb5f97 in pfs_unlock_mutex_v1 (mutex=0xa676f6c2e6374) at /home/dan/repos/mariadb-server-main/storage/perfschema/pfs.cc:3667 #1 0x00000000006e868d in inline_mysql_mutex_unlock (that=<optimized out>) at /home/dan/repos/mariadb-server-main/include/mysql/psi/mysql_thread.h:786 #2 0x00000000006e868d in fill_users_schema_table (thd=<optimized out>, tables=<optimized out>, cond=<optimized out>) #3 0x00000000007eff21 in get_schema_tables_result (join=join@entry=0x7fbcdc030d90, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /home/dan/repos/mariadb-server-main/sql/sql_show.cc:9716 #4 0x00000000007b0fb7 in JOIN::exec_inner (this=this@entry=0x7fbcdc030d90) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:5020 #5 0x00000000007982c8 in JOIN::exec (this=0x7fbcdc030d90) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:4842 #6 mysql_select (thd=thd@entry=0x7fbcdc000fd8, tables=<optimized out>, fields=@0x7fbcdc02faa8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x7fbcdc010ef0, last = 0x7fbcdc010ef0, elements = 1}, <No data fields>}, conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x7fbcdc0110f0, unit=0x7fbcdc005330, select_lex=0x7fbcdc02f7f0) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:5375 #7 0x0000000000797f89 in handle_select (thd=thd@entry=0x7fbcdc000fd8, lex=lex@entry=0x7fbcdc005250, result=result@entry=0x7fbcdc0110f0, setup_tables_done_option=setup_tables_done_option@entry=0) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:633 #8 0x00000000007689bd in execute_sqlcom_select (thd=thd@entry=0x7fbcdc000fd8, all_tables=0x7fbcdc02fca8) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:6191 #9 0x000000000076416a in mysql_execute_command (thd=thd@entry=0x7fbcdc000fd8, is_called_from_prepared_stmt=false) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:3979 #10 0x000000000075fd2c in mysql_parse (thd=thd@entry=0x7fbcdc000fd8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7fbd037fd280) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:7915 #11 0x000000000075e476 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7fbcdc000fd8, packet=packet@entry=0x7fbcdc008c19 "SELECT 1 FROM information_schema.users", packet_length=packet_length@entry=38, blocking=true) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:1902 #12 0x0000000000760182 in do_command (thd=thd@entry=0x7fbcdc000fd8, blocking=true) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:1415 #13 0x000000000087d24d in do_handle_one_connection (connect=<optimized out>, connect@entry=0x12096f58, put_in_cache=<optimized out>) at /home/dan/repos/mariadb-server-main/sql/sql_connect.cc:1415 #14 0x000000000087d08a in handle_one_connection (arg=arg@entry=0x12096f58) at /home/dan/repos/mariadb-server-main/sql/sql_connect.cc:1327 #15 0x0000000000bb4809 in pfs_spawn_thread (arg=0x120156d8) at /home/dan/repos/mariadb-server-main/storage/perfschema/pfs.cc:2198 #16 0x00007fbd1da7dba8 in start_thread () from /lib64/libc.so.6 #17 0x00007fbd1db01994 in clone () from /lib64/libc.so.6 debug mode - stalls at lock Thread 2 (Thread 0x7fd4b80946c0 (LWP 87822) "one_connection"): #0 0x00007fd4bf480d72 in __pthread_mutex_lock_full () from /lib64/libc.so.6 No symbol table info available. #1 0x00000000016f966f in safe_mutex_lock (mp=0x2bc37ae8, my_flags=0, file=0x1829c35 "/home/dan/repos/mariadb-server-main/sql/sql_acl.cc", line=13153) at /home/dan/repos/mariadb-server-main/mysys/thr_mutex.c:234 error = 25336885 #2 0x00000000016f4057 in psi_mutex_lock (that=0x2bc37ae8, file=0x1829c35 "/home/dan/repos/mariadb-server-main/sql/sql_acl.cc", line=13153) at /home/dan/repos/mariadb-server-main/mysys/my_thr_init.c:471 state = {m_flags = 25516214, m_operation = PSI_MUTEX_LOCK, m_mutex = 0x1854ae4, m_thread = 0x1a3380000009, m_timer_start = 140551597390456, m_timer = 0xffffffffff, m_wait = 0x19e8562} locker = 0x0 result = 0 #3 0x00000000008a2a2c in inline_mysql_mutex_lock (that=0x2bc37ae8, src_file=0x1829c35 "/home/dan/repos/mariadb-server-main/sql/sql_acl.cc", src_line=13153) at /home/dan/repos/mariadb-server-main/include/mysql/psi/mysql_thread.h:746 No locals. #4 0x00000000008c1268 in fill_users_schema_table (thd=0x7fd46c000dc8, tables=0x7fd46c035068, cond=0x0) at /home/dan/repos/mariadb-server-main/sql/sql_acl.cc:13153 res = 0 see_whole_table = true table = 0x7fd46c03eaf0 #5 0x0000000000aa47cb in get_schema_tables_result (join=0x7fd46c0362f8, executed_place=PROCESSED_BY_JOIN_EXEC) at /home/dan/repos/mariadb-server-main/sql/sql_show.cc:9716 is_subselect = false cond = 0x0 backup_ctx = {m_thd = 0x7fd46c000dc8, m_sctx = 0x7fd46c002fa8} check_level_save = {m_thd = 0x7fd46c000dc8, m_check_level = CHECK_FIELD_IGNORE} table_list = 0x7fd46c035068 thd = 0x7fd46c000dc8 lex = 0x7fd46c005210 result = false org_stage = {m_key = 0, m_name = 0x1812156 "Executing", m_flags = 25460997} _db_stack_frame_ = {func = 0x184ca45 "JOIN::exec_inner", file = 0x184bc5b "/home/dan/repos/mariadb-server-main/sql/sql_select.cc", level = 2147483656, line = -1, prev = 0x7fd4b808fc48} err_handler = {<Internal_error_handler> = {_vptr$Internal_error_handler = 0x2074000 <vtable for Warnings_only_error_handler+16>, m_prev_internal_handler = 0x0}, <No data fields>} tab = 0x7fd46c037cb0 #6 0x0000000000a40576 in JOIN::exec_inner (this=0x7fd46c0362f8) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:5020 columns_list = 0x7fd46c034d50 _db_stack_frame_ = {func = 0x184cb07 "mysql_select", file = 0x184bc5b "/home/dan/repos/mariadb-server-main/sql/sql_select.cc", level = 2147483655, line = -1, prev = 0x7fd4b8090130} trace_wrapper = {<Json_writer_struct> = {_vptr$Json_writer_struct = 0x2062060 <vtable for Json_writer_object+16>, my_writer = 0x0, context = {writer = 0x0}, closed = false}, <No data fields>} trace_exec = {<Json_writer_struct> = {_vptr$Json_writer_struct = 0x2062060 <vtable for Json_writer_object+16>, my_writer = 0x0, context = {writer = 0x0}, closed = false}, <No data fields>} trace_steps = {<Json_writer_struct> = {_vptr$Json_writer_struct = 0x2061ff8 <vtable for Json_writer_array+16>, my_writer = 0x0, context = {writer = 0x0}, closed = false}, <No data fields>} #7 0x0000000000a3f829 in JOIN::exec (this=0x7fd46c0362f8) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:4842 res = 32724 #8 0x0000000000a19573 in mysql_select (thd=0x7fd46c000dc8, tables=0x7fd46c035068, fields=@0x7fd46c034d50: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x7fd46c034fe8, last = 0x7fd46c034fe8, elements = 1}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x7fd46c0362d0, unit=0x7fd46c0052f0, select_lex=0x7fd46c034a98) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:5375 err = 0 free_join = true exec_error = false _db_stack_frame_ = {func = 0x184bc91 "handle_select", file = 0x184bc5b "/home/dan/repos/mariadb-server-main/sql/sql_select.cc", level = 2147483654, line = -1, prev = 0x7fd4b8090298} join = 0x7fd46c0362f8 #9 0x0000000000a18e11 in handle_select (thd=0x7fd46c000dc8, lex=0x7fd46c005210, result=0x7fd46c0362d0, setup_tables_done_option=0) at /home/dan/repos/mariadb-server-main/sql/sql_select.cc:633 unit = 0x7fd46c0052f0 res = false select_lex = 0x7fd46c034a98 _db_stack_frame_ = {func = 0x1840a74 "mysql_execute_command", file = 0x1840046 "/home/dan/repos/mariadb-server-main/sql/sql_parse.cc", level = 2147483653, line = -1, prev = 0x7fd4b80914c0} #10 0x00000000009b889a in execute_sqlcom_select (thd=0x7fd46c000dc8, all_tables=0x7fd46c035068) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:6191 save_protocol = 0x0 lex = 0x7fd46c005210 result = 0x7fd46c0362d0 res = false #11 0x00000000009ae636 in mysql_execute_command (thd=0x7fd46c000dc8, is_called_from_prepared_stmt=false) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:3979 privileges_requested = SELECT_ACL res = 0 lex = 0x7fd46c005210 select_lex = 0x7fd46c034a98 first_table = 0x7fd46c035068 all_tables = 0x7fd46c035068 unit = 0x7fd46c0052f0 _db_stack_frame_ = {func = 0x18415f5 "mysql_parse", file = 0x1840046 "/home/dan/repos/mariadb-server-main/sql/sql_parse.cc", level = 2147483652, line = -1, prev = 0x7fd4b8091e40} ots = {ctx = 0x7fd46c004f70, traceable = false} orig_binlog_format = BINLOG_FORMAT_MIXED orig_current_stmt_binlog_format = BINLOG_FORMAT_STMT #12 0x00000000009a5f97 in mysql_parse (thd=0x7fd46c000dc8, rawbuf=0x7fd46c0349f0 "select 1 from information_schema.users", length=38, parser_state=0x7fd4b80925b0) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:7915 found_semicolon = 0x0 error = 32724 lex = 0x7fd46c005210 err = false _db_stack_frame_ = {func = 0x18404bc "dispatch_command", file = 0x1840046 "/home/dan/repos/mariadb-server-main/sql/sql_parse.cc", level = 2147483651, line = -1, prev = 0x7fd4b8092770} #13 0x00000000009a2b70 in dispatch_command (command=COM_QUERY, thd=0x7fd46c000dc8, packet=0x7fd46c00bca9 "select 1 from information_schema.users", packet_length=38, blocking=true) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:1902 packet_end = 0x7fd46c034a16 "" parser_state = {m_lip = {lookahead_token = -1, lookahead_yylval = 0x0, m_thd = 0x7fd46c000dc8, m_ptr = 0x7fd46c034a17 "", m_tok_start = 0x7fd46c034a17 "", m_tok_end = 0x7fd46c034a17 "", m_end_of_query = 0x7fd46c034a16 "", m_tok_start_prev = 0x7fd46c034a16 "", m_buf = 0x7fd46c0349f0 "select 1 from information_schema.users", m_buf_length = 38, m_echo = true, m_echo_saved = false, m_cpp_buf = 0x7fd46c034a70 "select 1 from information_schema.users", m_cpp_ptr = 0x7fd46c034a96 "", m_cpp_tok_start = 0x7fd46c034a96 "", m_cpp_tok_start_prev = 0x7fd46c034a96 "", m_cpp_tok_end = 0x7fd46c034a96 "", m_body_utf8 = 0x0, m_body_utf8_ptr = 0x0, m_cpp_utf8_processed_ptr = 0x0, next_state = MY_LEX_END, found_semicolon = 0x0, ignore_space = false, stmt_prepare_mode = false, multi_statements = true, yylineno = 1, m_digest = 0x0, in_comment = NO_COMMENT, in_comment_saved = NO_COMMENT, m_cpp_text_start = 0x7fd46c034a91 "users", m_cpp_text_end = 0x7fd46c034a96 "", m_underscore_cs = 0x0}, m_yacc = {yacc_yyss = 0x0, yacc_yyvs = 0x0, m_set_signal_info = {m_item = {0x0 <repeats 13 times>}}, m_lock_type = TL_READ_DEFAULT, m_mdl_type = MDL_SHARED_READ}, m_digest_psi = 0x0} net = 0x7fd46c001110 error = false do_end_of_statement = true _db_stack_frame_ = {func = 0x18403b0 "do_command", file = 0x1840046 "/home/dan/repos/mariadb-server-main/sql/sql_parse.cc", level = 2147483650, line = -1, prev = 0x7fd4b8093848} drop_more_results = false #14 0x00000000009a6fb8 in do_command (thd=0x7fd46c000dc8, blocking=true) at /home/dan/repos/mariadb-server-main/sql/sql_parse.cc:1415 return_value = (unknown: 0x6c001110) packet = 0x7fd46c00bca8 "\003select 1 from information_schema.users" packet_length = 39 net = 0x7fd46c001110 command = COM_QUERY _db_stack_frame_ = {func = 0x19f499a "?func", file = 0x19f49a0 "?file", level = 2147483649, line = -1, prev = 0x0} #15 0x0000000000ba3d9e in do_handle_one_connection (connect=0x2bc16ff8, put_in_cache=true) at /home/dan/repos/mariadb-server-main/sql/sql_connect.cc:1415 create_user = true thr_create_utime = 58881449345 thd = 0x7fd46c000dc8 #16 0x0000000000ba3b2c in handle_one_connection (arg=0x2bc16ff8) at /home/dan/repos/mariadb-server-main/sql/sql_connect.cc:1327 connect = 0x2bc16ff8 #17 0x000000000112620c in pfs_spawn_thread (arg=0x2bc33068) at /home/dan/repos/mariadb-server-main/storage/perfschema/pfs.cc:2198 typed_arg = 0x2bc33068 user_arg = 0x2bc16ff8 user_start_routine = 0xba3ad0 <handle_one_connection(void*)> pfs = 0x0 klass = 0x0 #18 0x00007fd4bf47dba8 in start_thread () from /lib64/libc.so.6 No symbol table info available. #19 0x00007fd4bf501b8c in __clone3 () from /lib64/libc.so.6 No symbol table info available.   (gdb) p *acl_cache $2 = {<hash_filo> = {m_psi_key = 0, key_offset = 0, key_length = 0, get_key = 0x8a1ee0 <acl_entry_get_key(void const*, unsigned long*, char)>, m_size = 256, free_element = 0x16ef9e0 <my_free>, init = false, hash_charset = 0x2409458 <my_charset_utf8mb3_bin>, first_link = 0x0, last_link = 0x0, lock = {m_mutex = {global = {__data = {__lock = -2139199899, __count = 0, __owner = 0, __nusers = 0, __kind = 65552, __spins = -28785, __elision = -28785, __list = {__prev = 0x1, __next = 0x8f8f8f8f8f8f8f8f}}, __size = "ef~\200", '\000' <repeats 12 times>, "\020\000\001\000\217\217\217\217\001\000\000\000\000\000\000\000\217\217\217\217\217\217\217\217", __align = 2155767397}, mutex = {__data = { __lock = -1886417009, __count = 2408550287, __owner = -1886417009, __nusers = 2408550287, __kind = -1886417009, __spins = -28785, __elision = -28785, __list = {__prev = 0x8f8f8f8f8f8f8f8f, __next = 0x8f8f8f8f8f8f8f8f}}, __size = '\217' <repeats 40 times>, __align = -8102099357864587377}, file = 0x8f8f8f8f8f8f8f8f <error: Cannot access memory at address 0x8f8f8f8f8f8f8f8f>, name = 0x8f8f8f8f8f8f8f8f <error: Cannot access memory at address 0x8f8f8f8f8f8f8f8f>, line = 2408550287, count = 2408550287, create_flags = 10344644715844964239, active_flags = 10344644715844964239, id = 10344644715844964239, thread = 10344644715844964239, locked_mutex = 0x8f8f8f8f8f8f8f8f, used_mutex = 0x8f8f8f8f8f8f8f8f, prev = 0x8f8f8f8f8f8f8f8f, next = 0x8f8f8f8f8f8f8f8f}, m_psi = 0x8f8f8f8f8f8f8f8f}, cache = {key_offset = 0, key_length = 0, blength = 0, records = 0, flags = 0, array = {buffer = 0x0, elements = 0, max_element = 0, alloc_increment = 0, size_of_element = 0, m_psi_key = 0, malloc_flags = 0}, get_key = 0x0, hash_function = 0x0, free = 0x0, charset = 0x0}}, <No data fields>} (gdb)

Daniel Black added a comment - 8 hours ago Aside - even with k3s - use multiple containers - one for db and one for app and let the mariadb container handle its intialization (and create database via MARIADB_DATABASE ) like: apiVersion: v1 kind: Namespace metadata: name: myapp --- apiVersion: apps/v1 kind: Deployment metadata: name: mariadb-deployment namespace: myapp spec: replicas: 1 selector: matchLabels: app: mariadb template: metadata: labels: app: mariadb spec: containers: - name: mariadb image: mariadb:latest env: - name: MARIADB_ROOT_PASSWORD value: "rootpassword" - name: MARIADB_DATABASE value: "mydatabase" ports: - containerPort: 3306 volumeMounts: - name: mariadb-data mountPath: /var/lib/mysql readinessProbe: exec: command: - /usr/local/bin/healthcheck.sh - --connect - --innodb_initialized initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 5 livenessProbe: exec: command: - /usr/local/bin/healthcheck.sh - --connect - --innodb_initialized initialDelaySeconds: 20 periodSeconds: 10 timeoutSeconds: 5 volumes: - name: mariadb-data persistentVolumeClaim: claimName: mariadb-pvc --- apiVersion: v1 kind: Service metadata: name: mariadb namespace: myapp spec: selector: app: mariadb ports: - protocol: TCP port: 3306 targetPort: 3306 --- apiVersion: apps/v1 kind: Deployment metadata: name: python-app-deployment namespace: myapp spec: replicas: 1 selector: matchLabels: app: python-app template: metadata: labels: app: python-app spec: containers: - name: python-app image: my-python-app:latest env: - name: DB_HOST value: mariadb.myapp.svc.cluster.local - name: DB_PORT value: "3306" - name: DB_USER value: "root" - name: DB_PASSWORD value: "rootpassword" ports: - containerPort: 5000 readinessProbe: httpGet: path: /health port: 5000 initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 3 command: ["python", "app.py"] initContainers: - name: wait-for-mariadb image: busybox - sh - -c - "until nc -z mariadb.myapp.svc.cluster.local 3306; do echo 'Waiting for MariaDB...'; sleep 2; done;"