Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35213

Server crash or assertion failure upon query with high value of mhnsw_min_limit

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • N/A
    • 11.7.1
    • Vector search
    • None

    Description

      The test case is essentially this (this is a template, not the actual test, don't try to execute it, use the attachment instead):

      SET @cache.save= @@mhnsw_cache_size;
      		 
      SET GLOBAL mhnsw_cache_size= 8*1024*1024*1024;
      		 
      CREATE OR REPLACE TABLE t (pk BIGINT AUTO_INCREMENT PRIMARY KEY, v VECTOR(96) NOT NULL) ENGINE=MyISAM;
      		 
      # Inserting data...
      		 
      SELECT COUNT(*) FROM t;
      		 
      COUNT(*)
      		 
      20000
      		 
      ALTER TABLE t ADD VECTOR (v);
      		 
      SET mhnsw_min_limit=16*1024;
      		 
      SELECT v into @x FROM t WHERE pk=1;
      		 
      SELECT pk FROM t ORDER BY VEC_DISTANCE_EUCLIDEAN(@x, v) LIMIT 5;

      So far i couldn't reproduce it on random data, so the attached test case is big, it contains a 20K rows subset of Deep Image data.

      bb-11.6-MDEV-32887-vector 4dfa605f405df3bee7142be6bcdc364f5a5e0db3 debug

      		 
      mariadbd: /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/mysys/queues.c:218: queue_insert: Assertion `queue->elements < queue->max_elements' failed.
      		 
      241019 20:16:07 [ERROR] mysqld got signal 6 ;
      		 
       
      		 
      #10 0x00007fd62e253e32 in __GI___assert_fail (assertion=assertion@entry=0x558a2a94fe10 "queue->elements < queue->max_elements", file=file@entry=0x558a2a94fdc8 "/data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/mysys/queues.c", line=line@entry=218, function=function@entry=0x558a2a94fe90 <__PRETTY_FUNCTION__.1> "queue_insert") at ./assert/assert.c:101
      		 
      #11 0x0000558a2a3733d5 in queue_insert (queue=0x7fd6221f8130, element=0x7fd60c5322f0 "\3609\032\r\326\177") at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/mysys/queues.c:218
      		 
      #12 0x0000558a295cf2d6 in Queue<Visited, void>::push (element=0x7fd60c5322f0, this=0x7fd6221f8130) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_queue.h:46
      		 
      #13 search_layer (ctx=0x7fd60c12ce60, graph=graph@entry=0x7fd60c21ba00, target=target@entry=0x7fd60c1fb998, threshold=threshold@entry=-1, result_size=result_size@entry=5, layer=layer@entry=0, inout=inout@entry=0x7fd6221f81e0, construction=false) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/vector_mhnsw.cc:1079
      		 
      #14 0x0000558a295d410a in mhnsw_read_first (table=table@entry=0x7fd60c11c448, keyinfo=keyinfo@entry=0x7fd60c00d808, dist=dist@entry=0x7fd60c0183e0, limit=limit@entry=5) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/vector_mhnsw.cc:1282
      		 
      #15 0x0000558a27f7f0c7 in TABLE::hlindex_read_first (this=this@entry=0x7fd60c11c448, nr=<optimized out>, item=0x7fd60c0183e0, limit=limit@entry=5) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_base.cc:9968
      		 
      #16 0x0000558a28387c8f in join_read_first (tab=0x7fd60c1f8e60) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:24765
      		 
      #17 0x0000558a282f1d02 in sub_select (join=0x7fd60c018630, join_tab=0x7fd60c1f8e60, end_of_records=false) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:23651
      		 
      #18 0x0000558a2835917c in do_select (join=join@entry=0x7fd60c018630, procedure=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:23165
      		 
      #19 0x0000558a283d9c0a in JOIN::exec_inner (this=this@entry=0x7fd60c018630) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:5026
      		 
      #20 0x0000558a283da094 in JOIN::exec (this=this@entry=0x7fd60c018630) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:4809
      		 
      #21 0x0000558a283d3f8d in mysql_select (thd=thd@entry=0x7fd60c000d58, tables=0x7fd60c017208, fields=..., conds=conds@entry=0x0, og_num=og_num@entry=1, order=order@entry=0x7fd60c0184a8, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x7fd60c018608, unit=0x7fd60c005250, select_lex=0x7fd60c016bc8) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:5342
      		 
      #22 0x0000558a283d4ab2 in handle_select (thd=thd@entry=0x7fd60c000d58, lex=lex@entry=0x7fd60c005170, result=result@entry=0x7fd60c018608, setup_tables_done_option=setup_tables_done_option@entry=0) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_select.cc:624
      		 
      #23 0x0000558a281aac74 in execute_sqlcom_select (thd=thd@entry=0x7fd60c000d58, all_tables=0x7fd60c017208) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_parse.cc:6151
      		 
      #24 0x0000558a281d3862 in mysql_execute_command (thd=thd@entry=0x7fd60c000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_parse.cc:3953
      		 
      #25 0x0000558a281e8828 in mysql_parse (thd=thd@entry=0x7fd60c000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7fd6221f9440) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_parse.cc:7873
      		 
      #26 0x0000558a281ef1fc in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7fd60c000d58, packet=packet@entry=0x7fd60c232e09 "", packet_length=packet_length@entry=63, blocking=blocking@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_parse.cc:1892
      		 
      #27 0x0000558a281f79ed in do_command (thd=0x7fd60c000d58, blocking=blocking@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_parse.cc:1405
      		 
      #28 0x0000558a286f2898 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x558a2ff62438, put_in_cache=put_in_cache@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_connect.cc:1448
      		 
      #29 0x0000558a286f374e in handle_one_connection (arg=0x558a2ff62438) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/sql/sql_connect.cc:1350
      		 
      #30 0x0000558a298c1397 in pfs_spawn_thread (arg=0x558a2ff2bc48) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-ubsan/storage/perfschema/pfs.cc:2198
      		 
      #31 0x00007fd62e2a8044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
      		 
      #32 0x00007fd62e32861c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

      non-debug

      		 
      #2  <signal handler called>
      		 
      #3  _mm256_madd_epi16(long long __vector(4), long long __vector(4)) (__B=..., __A=...) at /usr/lib/gcc/x86_64-linux-gnu/12/include/avx2intrin.h:343
      		 
      #4  FVector::dot_product(short const*, short const*, unsigned long) [clone .avx_avx2_fma] (v1=0x56320c4e8768 <MHNSW_Share::~MHNSW_Share()+8>, v2=0x7ff5e81a1340, len=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:136
      		 
      #5  0x000056320c4e4d94 in FVector::distance_to (vec_len=<optimized out>, other=<optimized out>, this=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:212
      		 
      #6  FVectorNode::distance_to (this=this@entry=0x56320d3e2638 <vtable for MHNSW_Share+16>, other=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:705
      		 
      #7  0x000056320c4e5a0e in VisitedSet::create (node=0x56320d3e2638 <vtable for MHNSW_Share+16>, this=0x7ff5f80ff0f0) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:841
      		 
      #8  search_layer (ctx=<optimized out>, graph=graph@entry=0x7ff5e8008880, target=target@entry=0x7ff5e81a1338, threshold=threshold@entry=-1, result_size=result_size@entry=5, layer=layer@entry=0, inout=inout@entry=0x7ff5f80ff2a0, construction=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:1073
      		 
      #9  0x000056320c4e7387 in mhnsw_read_first (table=0x7ff5e81a3718, keyinfo=<optimized out>, dist=<optimized out>, limit=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/vector_mhnsw.cc:1282
      		 
      #10 0x000056320c09518e in TABLE::hlindex_read_first (this=<optimized out>, nr=<optimized out>, item=<optimized out>, limit=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_base.cc:9968
      		 
      #11 0x000056320c162982 in join_read_first (tab=0x7ff5e819e960) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:24765
      		 
      #12 0x000056320c1524e1 in sub_select (join=0x7ff5e8014950, join_tab=0x7ff5e819e960, end_of_records=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:23651
      		 
      #13 0x000056320c186d86 in do_select (procedure=<optimized out>, join=0x7ff5e8014950) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:23165
      		 
      #14 JOIN::exec_inner (this=this@entry=0x7ff5e8014950) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:5026
      		 
      #15 0x000056320c187175 in JOIN::exec (this=this@entry=0x7ff5e8014950) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:4809
      		 
      #16 0x000056320c185188 in mysql_select (thd=thd@entry=0x7ff5e8000c68, tables=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:5342
      		 
      #17 0x000056320c185984 in handle_select (thd=thd@entry=0x7ff5e8000c68, lex=lex@entry=0x7ff5e8004eb8, result=result@entry=0x7ff5e8014928, setup_tables_done_option=setup_tables_done_option@entry=0) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_select.cc:624
      		 
      #18 0x000056320c0f6ac0 in execute_sqlcom_select (thd=thd@entry=0x7ff5e8000c68, all_tables=0x7ff5e8013528) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_parse.cc:6151
      		 
      #19 0x000056320c105ba0 in mysql_execute_command (thd=thd@entry=0x7ff5e8000c68, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_parse.cc:3953
      		 
      #20 0x000056320c1070c6 in mysql_parse (thd=0x7ff5e8000c68, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_parse.cc:7873
      		 
      #21 0x000056320c10993d in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7ff5e8000c68, packet=packet@entry=0x7ff5e821dfd9 "", packet_length=packet_length@entry=63, blocking=blocking@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_parse.cc:1991
      		 
      #22 0x000056320c10b874 in do_command (thd=0x7ff5e8000c68, blocking=blocking@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_parse.cc:1405
      		 
      #23 0x000056320c23ae87 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x56320ec5e448, put_in_cache=put_in_cache@entry=true) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_connect.cc:1448
      		 
      #24 0x000056320c23b1fd in handle_one_connection (arg=arg@entry=0x56320ec5e448) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/sql/sql_connect.cc:1350
      		 
      #25 0x000056320c5f1c4c in pfs_spawn_thread (arg=0x56320ec16ca8) at /data/bld/preview-11.7-bb-11.6-MDEV-32887-vector-rel/storage/perfschema/pfs.cc:2198
      		 
      #26 0x00007ff5feaa8044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
      		 
      #27 0x00007ff5feb2861c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

      Note: The test case takes 5-15 seconds on my machine in shm before it crashes. However, every once in a while, it froze upon SELECT, so if it takes much longer, it may be the case, then either retry it or check it for a hang.

      Attachments

        1. vec.test.gz
          7.95 MB
          Elena Stepanova

        Issue Links

          is caused by

          New Feature - A new feature of the product, which has yet to be developed. MDEV-34939 vector search in 11.7

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.