Details

    Description

      I've been updating one of my applications from OQGraphv2 to v3 and while initial tests showed everything working, I'm now getting troublesome crashes when I put some load on the OQGraph table.

      The application is basically a webservice (in PHP) that queries the OQGraph table (schema attached) and returns the result. The query in itself should be fairly simple:

      SELECT db.* FROM db_history AS db INNER JOIN version_history AS v ON db.nodeID = v.linkid WHERE origid = 1 AND destid = 3 AND latch = 'dijkstras';

      When doing single sequential requests in the browser, everything works fine. But once I started to put some load on the webservice (using siege as load testing tool), MariaDB crashes quickly after 1-2 requests (crash dump attached), always with the same crash dump.

      I should note that the database setup is a master-slave replication setup. I'm not sure that has anything to do with the crashes I'm seeing though.

      I can reproduce this fairly easily and reliably in my test environment, on both the slave and the master node, but have not been successful so far in producing a test case that does not involve running siege.

      Attachments

        1. backtrace_2.log
          6 kB
        2. backtrace.log
          8 kB
        3. carbon.log
          3 kB
        4. crash.log
          4 kB
        5. data.sql
          2 kB
        6. debug.log
          410 kB
        7. explain.log
          3 kB
        8. oqgraph_crash_without_latch.log
          863 kB
        9. oqgraph_load.core.tar.xz
          1.12 MB
        10. schema.sql
          0.8 kB
        11. show_variables.log
          385 kB
        12. threaddump_2.log
          61 kB
        13. threaddump.log
          88 kB
        14. valgrind.log
          10 kB
        15. variables
          69 kB

        Issue Links

          Activity

            pprkut Heinz Wiesinger created issue -
            arjen Arjen Lentz made changes -
            Field Original Value New Value
            Assignee Arjen Lentz [ arjen ]
            arjen Arjen Lentz made changes -
            Assignee Arjen Lentz [ arjen ] Andrew McDonnell [ andymc73 ]
            andymc73 Andrew McDonnell added a comment - - edited

            Hello Heinz

            Is it possible you can rebuild MariaDB with debugging turned on and try again? Having had a very quick look at the code, a bunch of inline functions seem to be optimised out of the stack trace between oqgraph::cursor::seek_to() and the function where the crash seems to happen which makes it hard to see what could be happening

            FWIW I am guessing something somewhere has not been done with proper thread-safety. In that vein, is there any form of logging with timing information produced by your web service that might be available?

            The other thing is turning on mariadb logging, if you are able to get mysqld to run with -#d,oq-debug,info on the command line and attach those logs as well...

            in the meantime I'll see if I can repeat this on my system, although I suspect it may take a little while

            andymc73 Andrew McDonnell added a comment - - edited Hello Heinz Is it possible you can rebuild MariaDB with debugging turned on and try again? Having had a very quick look at the code, a bunch of inline functions seem to be optimised out of the stack trace between oqgraph::cursor::seek_to() and the function where the crash seems to happen which makes it hard to see what could be happening FWIW I am guessing something somewhere has not been done with proper thread-safety. In that vein, is there any form of logging with timing information produced by your web service that might be available? The other thing is turning on mariadb logging, if you are able to get mysqld to run with -#d,oq-debug,info on the command line and attach those logs as well... in the meantime I'll see if I can repeat this on my system, although I suspect it may take a little while

            Actually I may be wrong about the optimised inline, but the debug stack trace will still be useful

            andymc73 Andrew McDonnell added a comment - Actually I may be wrong about the optimised inline, but the debug stack trace will still be useful

            I recompiled MariaDB with debugging turned on, but for whatever reason I don't seem to get a backtrace now anymore. I'll attach the debugging log for now and see what I can do about the backtrace.

            pprkut Heinz Wiesinger added a comment - I recompiled MariaDB with debugging turned on, but for whatever reason I don't seem to get a backtrace now anymore. I'll attach the debugging log for now and see what I can do about the backtrace.
            pprkut Heinz Wiesinger made changes -
            Attachment debug.log [ 30505 ]

            Managed to get something with --core-file. Let's see if it helps

            pprkut Heinz Wiesinger added a comment - Managed to get something with --core-file. Let's see if it helps
            pprkut Heinz Wiesinger made changes -
            Attachment backtrace.log [ 30506 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.0.13 [ 16000 ]
            andymc73 Andrew McDonnell added a comment - - edited

            Thanks Heinz - Hopefully you kept that core file around!

            Can you please report the value of lex in frame #0

            (If you haven't worked with gdb much, do the following: f 0, print lex )

            Can you further dump this at that same place: print *this

            (This may be a bit of a slow process to start with...)

            andymc73 Andrew McDonnell added a comment - - edited Thanks Heinz - Hopefully you kept that core file around! Can you please report the value of lex in frame #0 (If you haven't worked with gdb much, do the following: f 0 , print lex ) Can you further dump this at that same place: print *this (This may be a bit of a slow process to start with...)

            I would also be curious to get a complete thread dump:

            thread apply all bt full

            andymc73 Andrew McDonnell added a comment - I would also be curious to get a complete thread dump: thread apply all bt full

            Jep, still have the core dump. Output of the variables is in the "variables" file, since the value of *this is rather big. Thread dump attached as well.

            pprkut Heinz Wiesinger added a comment - Jep, still have the core dump. Output of the variables is in the "variables" file, since the value of *this is rather big. Thread dump attached as well.
            pprkut Heinz Wiesinger made changes -
            Attachment threaddump.log [ 30507 ]
            Attachment variables [ 30508 ]

            Ah, just saw something in the thread dump that I totally forgot about. There's actually two more queries in play here. The first one only queries the backing table for a nodeid for a given value. It doesn't touch the OQGraph table so I don't think that's an issue here. However, the second one queries the OQGraph table and I think this is where things race with each other.

            SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1

            Naturally, when I do one request after the other, those two queries would never run concurrently, but they would with siege simulating multiple simultaneous requests.

            pprkut Heinz Wiesinger added a comment - Ah, just saw something in the thread dump that I totally forgot about. There's actually two more queries in play here. The first one only queries the backing table for a nodeid for a given value. It doesn't touch the OQGraph table so I don't think that's an issue here. However, the second one queries the OQGraph table and I think this is where things race with each other. SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1 Naturally, when I do one request after the other, those two queries would never run concurrently, but they would with siege simulating multiple simultaneous requests.
            arjen Arjen Lentz added a comment -

            Awesome catch Heinz, I reckon that should be very useful.

            How big is the core file? Perhaps we can find a way to get it to Andrew... might be easier to work with then.

            arjen Arjen Lentz added a comment - Awesome catch Heinz, I reckon that should be very useful. How big is the core file? Perhaps we can find a way to get it to Andrew... might be easier to work with then.

            I added a compressed version of the core dump. Beware when uncompressing, it's nearly 1GB in size.

            pprkut Heinz Wiesinger added a comment - I added a compressed version of the core dump. Beware when uncompressing, it's nearly 1GB in size.
            pprkut Heinz Wiesinger made changes -
            Attachment oqgraph_load.core.tar.xz [ 30600 ]

            The immediate cause of the crash is that `lex` is NULL inside MariaDB engine code. It remains to be seen if we have uncovered a latent defect in MariaDB or if this is due to OQGraph doing something wrong.

            Hienz, are you able to tell me how the queries are being constructed? Specifically, does the web service use 'a prepared statement' of any kind? Sorry I can't be more specific at this point.

            Perhaps you could also humour me and patch your mariadb instance as follows:

            there is a file called sql/sql_class.h, on approx. line # 2476 is the following function:

              void check_limit_rows_examined()
              {
                if (++accessed_rows_and_keys > lex->limit_rows_examined_cnt)
                  killed= ABORT_QUERY;
              }

            If you could change the code to look like the following, rebuild and see if your problem is fixed or not?

              void check_limit_rows_examined()
              {
                accessed_rows_and_keys++;
                if (lex && (accessed_rows_and_keys > lex->limit_rows_examined_cnt)) 
                  killed= ABORT_QUERY;
              }
             

            andymc73 Andrew McDonnell added a comment - The immediate cause of the crash is that `lex` is NULL inside MariaDB engine code. It remains to be seen if we have uncovered a latent defect in MariaDB or if this is due to OQGraph doing something wrong. Hienz, are you able to tell me how the queries are being constructed? Specifically, does the web service use 'a prepared statement' of any kind? Sorry I can't be more specific at this point. Perhaps you could also humour me and patch your mariadb instance as follows: there is a file called sql/sql_class.h , on approx. line # 2476 is the following function: void check_limit_rows_examined() { if (++accessed_rows_and_keys > lex->limit_rows_examined_cnt) killed= ABORT_QUERY; } If you could change the code to look like the following, rebuild and see if your problem is fixed or not? void check_limit_rows_examined() { accessed_rows_and_keys++; if (lex && (accessed_rows_and_keys > lex->limit_rows_examined_cnt)) killed= ABORT_QUERY; }  

            I'm not doing anything special during the query construction I think. The query is constructed as a string before being sent as a whole to the database server, so no prepared statements. I do make use of string escaping, I doubt that's relevant here though.

            I patched the sources, build is running, but since I'm doing this all in a VM it takes a while to finish. I let you know as soon as I have something.

            pprkut Heinz Wiesinger added a comment - I'm not doing anything special during the query construction I think. The query is constructed as a string before being sent as a whole to the database server, so no prepared statements. I do make use of string escaping, I doubt that's relevant here though. I patched the sources, build is running, but since I'm doing this all in a VM it takes a while to finish. I let you know as soon as I have something.

            I'm still getting the crash. New crash log and thread dump attached.

            pprkut Heinz Wiesinger added a comment - I'm still getting the crash. New crash log and thread dump attached.
            pprkut Heinz Wiesinger made changes -
            Attachment backtrace_2.log [ 30601 ]
            Attachment threaddump_2.log [ 30602 ]

            Sorry, for the moment I am stumped.

            One thing, the core dump as is is not very useful unless accompanied by exact binary that produced it.

            It would seem to be thread related obviously, anyway.

            Hienz, if you get a chance, are you able to run mysqld using valgrind ? This will make it run slow and probably produce a tonne of logging, but it might pick up the memory corruption at an earlier point.

            andymc73 Andrew McDonnell added a comment - Sorry, for the moment I am stumped. One thing, the core dump as is is not very useful unless accompanied by exact binary that produced it. It would seem to be thread related obviously, anyway. Hienz, if you get a chance, are you able to run mysqld using valgrind ? This will make it run slow and probably produce a tonne of logging, but it might pick up the memory corruption at an earlier point.

            Arjen, do you know how to write an mtr test that would cause queries to be sent in parallel?

            andymc73 Andrew McDonnell added a comment - Arjen, do you know how to write an mtr test that would cause queries to be sent in parallel?

            Hey Andrew. I tried running mysqld with valgrind, but my valgrind experience is limited. I used:

            valgrind /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/run/mysql/mysql.pid --core-file --skip-networking

            Then causing the crash I still see the segfault and get the core dump, but valgrind doesn't show any memory errors.

            pprkut Heinz Wiesinger added a comment - Hey Andrew. I tried running mysqld with valgrind, but my valgrind experience is limited. I used: valgrind /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/run/mysql/mysql.pid --core-file --skip-networking Then causing the crash I still see the segfault and get the core dump, but valgrind doesn't show any memory errors.

            Ok, after a pointer from Elena, I got something. I'll work on a few more things she pointed me too, in the meantime I hope this helps already.

            pprkut Heinz Wiesinger added a comment - Ok, after a pointer from Elena, I got something. I'll work on a few more things she pointed me too, in the meantime I hope this helps already.
            pprkut Heinz Wiesinger made changes -
            Attachment valgrind.log [ 31007 ]

            Attaching the general log and the data in the backing table. Hopefully this can help in generating a test case

            pprkut Heinz Wiesinger added a comment - Attaching the general log and the data in the backing table. Hopefully this can help in generating a test case
            pprkut Heinz Wiesinger made changes -
            Attachment data.sql [ 31008 ]
            Attachment carbon.log [ 31009 ]
            pprkut Heinz Wiesinger made changes -
            Attachment show_variables.log [ 31010 ]
            Attachment explain.log [ 31011 ]
            elenst Elena Stepanova added a comment - - edited

            Please try the following test case:

            INSTALL SONAME 'ha_oqgraph';
             
            --connect (con1,localhost,root,,test)
            	CREATE TABLE `db_history` ( `version` VARCHAR(10) NOT NULL, `updateJSON` MEDIUMTEXT, `prevVersion` VARCHAR(10) NOT NULL, `nodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', `prevNodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', PRIMARY KEY (`prevNodeID`,`nodeID`), KEY `version` (`version`) USING BTREE, KEY `prevVersion` (`prevVersion`) USING BTREE, KEY `nodeID` (`nodeID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
            	INSERT INTO `db_history` VALUES ('0.0.1','{}','0.0.0',1,0),('0.0.2','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.1',2,1),('0.0.3','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.2',3,2);
            	CREATE TABLE IF NOT EXISTS version_history ( latch VARCHAR(32) NULL, origid BIGINT UNSIGNED NULL, destid BIGINT UNSIGNED NULL, weight DOUBLE NULL, seq BIGINT UNSIGNED NULL, linkid BIGINT UNSIGNED NULL, KEY (latch, origid, destid) USING HASH, KEY (latch, destid, origid) USING HASH ) ENGINE=OQGRAPH DATA_TABLE='db_history' ORIGID='prevNodeID' DESTID='nodeID';
            	SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;
            --disconnect con1
             
            --connect (con2,localhost,root,,test)
            	SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;

            Stack trace from 10.0 revno 4209 (tag mariadb-10.0.11):

            #3  <signal handler called>
            #4  0x00000000006edf15 in THD::check_limit_rows_examined (this=0x7f948c791008) at 10.0.11/sql/sql_class.h:2478
            #5  0x00000000006edf7e in handler::increment_statistics (this=0x7f948b437e20, offset=&system_status_var::ha_read_key_count) at 10.0.11/sql/sql_class.h:4866
            #6  0x00000000008738bb in handler::ha_index_read_map (this=0x7f948b437e20, buf=0x7f948b483620 "\376\005\060.0.2", key=0x7f948b01e7a8 "\001", keypart_map=1, find_flag=HA_READ_KEY_OR_NEXT) at 10.0.11/sql/handler.cc:2587
            #7  0x00007f9493532307 in oqgraph3::cursor::seek_to (this=0x7f948b029470, origid=..., destid=...) at 10.0.11/storage/oqgraph/oqgraph_thunk.cc:503
            #8  0x00007f949351b5e9 in boost::find_vertex (id=1, g=...) at 10.0.11/storage/oqgraph/oqgraph_shim.h:480
            #9  0x00007f9493515e05 in open_query::oqgraph_share::find_vertex (this=0x7f948b433380, id=1) at 10.0.11/storage/oqgraph/graphcore.cc:485
            #10 0x00007f9493516131 in open_query::oqgraph::search (this=0x7f948b414260, latch=0x7f9495665f7c, orig_id=0x7f9495665f88, dest_id=0x0) at 10.0.11/storage/oqgraph/graphcore.cc:726
            #11 0x00007f9493512e45 in ha_oqgraph::index_read_idx (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", index=0, key=0x7f948b029260 "", key_len=44, find_flag=HA_READ_KEY_EXACT) at 10.0.11/storage/oqgraph/ha_oqgraph.cc:966
            #12 0x00007f949351266c in ha_oqgraph::index_read (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", key_len=44, find_flag=HA_READ_KEY_EXACT) at 10.0.11/storage/oqgraph/ha_oqgraph.cc:815
            #13 0x00007f949351494a in handler::index_read_map (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", keypart_map=3, find_flag=HA_READ_KEY_EXACT) at 10.0.11/sql/handler.h:2888
            #14 0x0000000000873852 in handler::ha_index_read_map (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", keypart_map=3, find_flag=HA_READ_KEY_EXACT) at 10.0.11/sql/handler.cc:2585
            #15 0x0000000000879461 in handler::read_range_first (this=0x7f948b508020, start_key=0x7f948b508110, end_key=0x7f948b508130, eq_range_arg=true, sorted=true) at 10.0.11/sql/handler.cc:5362
            #16 0x00000000007da6c8 in handler::multi_range_read_next (this=0x7f948b508020, range_info=0x7f9495666200) at 10.0.11/sql/multi_range_read.cc:296
            #17 0x000000000099507c in QUICK_RANGE_SELECT::get_next (this=0x7f948b024dc0) at 10.0.11/sql/opt_range.cc:11673
            #18 0x00000000009a4576 in rr_quick (info=0x7f948b02c9c8) at 10.0.11/sql/records.cc:346
            #19 0x00000000006da929 in join_init_read_record (tab=0x7f948b02c920) at 10.0.11/sql/sql_select.cc:18468
            #20 0x00000000006d8898 in sub_select (join=0x7f948b049420, join_tab=0x7f948b02c920, end_of_records=false) at 10.0.11/sql/sql_select.cc:17575
            #21 0x00000000006d815f in do_select (join=0x7f948b049420, fields=0x0, table=0x7f948b0b2020, procedure=0x0) at 10.0.11/sql/sql_select.cc:17240
            #22 0x00000000006b3cbe in JOIN::exec_inner (this=0x7f948b049420) at 10.0.11/sql/sql_select.cc:2619
            #23 0x00000000006b2d66 in JOIN::exec (this=0x7f948b049420) at 10.0.11/sql/sql_select.cc:2365
            #24 0x00000000006b622b in mysql_select (thd=0x7f948c79c008, rref_pointer_array=0x7f948c7a06e0, tables=0x7f948b02a620, wild_num=0, fields=..., conds=0x7f948b0248e0, og_num=1, order=0x7f948b0470a0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f948b029210, unit=0x7f948c79fd10, select_lex=0x7f948c7a0418) at 10.0.11/sql/sql_select.cc:3303
            #25 0x00000000006ac5c5 in handle_select (thd=0x7f948c79c008, lex=0x7f948c79fc48, result=0x7f948b029210, setup_tables_done_option=0) at 10.0.11/sql/sql_select.cc:372
            #26 0x00000000006805d9 in execute_sqlcom_select (thd=0x7f948c79c008, all_tables=0x7f948b02a620) at 10.0.11/sql/sql_parse.cc:5263
            #27 0x00000000006789c6 in mysql_execute_command (thd=0x7f948c79c008) at 10.0.11/sql/sql_parse.cc:2554
            #28 0x0000000000682d63 in mysql_parse (thd=0x7f948c79c008, rawbuf=0x7f948b00e320 "SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` "..., length=212, parser_state=0x7f9495667600) at 10.0.11/sql/sql_parse.cc:6409
            #29 0x0000000000675c67 in dispatch_command (command=COM_QUERY, thd=0x7f948c79c008, packet=0x7f948c7a2009 "SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` "..., packet_length=212) at 10.0.11/sql/sql_parse.cc:1309
            #30 0x000000000067500c in do_command (thd=0x7f948c79c008) at 10.0.11/sql/sql_parse.cc:1006
            #31 0x000000000079608f in do_handle_one_connection (thd_arg=0x7f948c79c008) at 10.0.11/sql/sql_connect.cc:1379
            #32 0x0000000000795de2 in handle_one_connection (arg=0x7f948c79c008) at 10.0.11/sql/sql_connect.cc:1293
            #33 0x0000000000ceaa98 in pfs_spawn_thread (arg=0x7f948c7f4808) at 10.0.11/storage/perfschema/pfs.cc:1860
            #34 0x00007f949532db50 in start_thread (arg=<optimized out>) at pthread_create.c:304
            #35 0x00007f9493825a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

            Also repeatable on the current tree.

            elenst Elena Stepanova added a comment - - edited Please try the following test case: INSTALL SONAME 'ha_oqgraph' ;   --connect (con1,localhost,root,,test) CREATE TABLE `db_history` ( `version` VARCHAR (10) NOT NULL , `updateJSON` MEDIUMTEXT, `prevVersion` VARCHAR (10) NOT NULL , `nodeID` BIGINT (20) UNSIGNED NOT NULL DEFAULT '0' , `prevNodeID` BIGINT (20) UNSIGNED NOT NULL DEFAULT '0' , PRIMARY KEY (`prevNodeID`,`nodeID`), KEY `version` (`version`) USING BTREE, KEY `prevVersion` (`prevVersion`) USING BTREE, KEY `nodeID` (`nodeID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; INSERT INTO `db_history` VALUES ( '0.0.1' , '{}' , '0.0.0' ,1,0),( '0.0.2' , '{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}' , '0.0.1' ,2,1),( '0.0.3' , '{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}' , '0.0.2' ,3,2); CREATE TABLE IF NOT EXISTS version_history ( latch VARCHAR (32) NULL , origid BIGINT UNSIGNED NULL , destid BIGINT UNSIGNED NULL , weight DOUBLE NULL , seq BIGINT UNSIGNED NULL , linkid BIGINT UNSIGNED NULL , KEY (latch, origid, destid) USING HASH, KEY (latch, destid, origid) USING HASH ) ENGINE=OQGRAPH DATA_TABLE= 'db_history' ORIGID= 'prevNodeID' DESTID= 'nodeID' ; SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; --disconnect con1   --connect (con2,localhost,root,,test) SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; Stack trace from 10.0 revno 4209 (tag mariadb-10.0.11): #3 <signal handler called> #4 0x00000000006edf15 in THD::check_limit_rows_examined (this=0x7f948c791008) at 10.0.11/sql/sql_class.h:2478 #5 0x00000000006edf7e in handler::increment_statistics (this=0x7f948b437e20, offset=&system_status_var::ha_read_key_count) at 10.0.11/sql/sql_class.h:4866 #6 0x00000000008738bb in handler::ha_index_read_map (this=0x7f948b437e20, buf=0x7f948b483620 "\376\005\060.0.2", key=0x7f948b01e7a8 "\001", keypart_map=1, find_flag=HA_READ_KEY_OR_NEXT) at 10.0.11/sql/handler.cc:2587 #7 0x00007f9493532307 in oqgraph3::cursor::seek_to (this=0x7f948b029470, origid=..., destid=...) at 10.0.11/storage/oqgraph/oqgraph_thunk.cc:503 #8 0x00007f949351b5e9 in boost::find_vertex (id=1, g=...) at 10.0.11/storage/oqgraph/oqgraph_shim.h:480 #9 0x00007f9493515e05 in open_query::oqgraph_share::find_vertex (this=0x7f948b433380, id=1) at 10.0.11/storage/oqgraph/graphcore.cc:485 #10 0x00007f9493516131 in open_query::oqgraph::search (this=0x7f948b414260, latch=0x7f9495665f7c, orig_id=0x7f9495665f88, dest_id=0x0) at 10.0.11/storage/oqgraph/graphcore.cc:726 #11 0x00007f9493512e45 in ha_oqgraph::index_read_idx (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", index=0, key=0x7f948b029260 "", key_len=44, find_flag=HA_READ_KEY_EXACT) at 10.0.11/storage/oqgraph/ha_oqgraph.cc:966 #12 0x00007f949351266c in ha_oqgraph::index_read (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", key_len=44, find_flag=HA_READ_KEY_EXACT) at 10.0.11/storage/oqgraph/ha_oqgraph.cc:815 #13 0x00007f949351494a in handler::index_read_map (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", keypart_map=3, find_flag=HA_READ_KEY_EXACT) at 10.0.11/sql/handler.h:2888 #14 0x0000000000873852 in handler::ha_index_read_map (this=0x7f948b508020, buf=0x7f948b445320 "\374\rbreadth_first", key=0x7f948b029260 "", keypart_map=3, find_flag=HA_READ_KEY_EXACT) at 10.0.11/sql/handler.cc:2585 #15 0x0000000000879461 in handler::read_range_first (this=0x7f948b508020, start_key=0x7f948b508110, end_key=0x7f948b508130, eq_range_arg=true, sorted=true) at 10.0.11/sql/handler.cc:5362 #16 0x00000000007da6c8 in handler::multi_range_read_next (this=0x7f948b508020, range_info=0x7f9495666200) at 10.0.11/sql/multi_range_read.cc:296 #17 0x000000000099507c in QUICK_RANGE_SELECT::get_next (this=0x7f948b024dc0) at 10.0.11/sql/opt_range.cc:11673 #18 0x00000000009a4576 in rr_quick (info=0x7f948b02c9c8) at 10.0.11/sql/records.cc:346 #19 0x00000000006da929 in join_init_read_record (tab=0x7f948b02c920) at 10.0.11/sql/sql_select.cc:18468 #20 0x00000000006d8898 in sub_select (join=0x7f948b049420, join_tab=0x7f948b02c920, end_of_records=false) at 10.0.11/sql/sql_select.cc:17575 #21 0x00000000006d815f in do_select (join=0x7f948b049420, fields=0x0, table=0x7f948b0b2020, procedure=0x0) at 10.0.11/sql/sql_select.cc:17240 #22 0x00000000006b3cbe in JOIN::exec_inner (this=0x7f948b049420) at 10.0.11/sql/sql_select.cc:2619 #23 0x00000000006b2d66 in JOIN::exec (this=0x7f948b049420) at 10.0.11/sql/sql_select.cc:2365 #24 0x00000000006b622b in mysql_select (thd=0x7f948c79c008, rref_pointer_array=0x7f948c7a06e0, tables=0x7f948b02a620, wild_num=0, fields=..., conds=0x7f948b0248e0, og_num=1, order=0x7f948b0470a0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f948b029210, unit=0x7f948c79fd10, select_lex=0x7f948c7a0418) at 10.0.11/sql/sql_select.cc:3303 #25 0x00000000006ac5c5 in handle_select (thd=0x7f948c79c008, lex=0x7f948c79fc48, result=0x7f948b029210, setup_tables_done_option=0) at 10.0.11/sql/sql_select.cc:372 #26 0x00000000006805d9 in execute_sqlcom_select (thd=0x7f948c79c008, all_tables=0x7f948b02a620) at 10.0.11/sql/sql_parse.cc:5263 #27 0x00000000006789c6 in mysql_execute_command (thd=0x7f948c79c008) at 10.0.11/sql/sql_parse.cc:2554 #28 0x0000000000682d63 in mysql_parse (thd=0x7f948c79c008, rawbuf=0x7f948b00e320 "SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` "..., length=212, parser_state=0x7f9495667600) at 10.0.11/sql/sql_parse.cc:6409 #29 0x0000000000675c67 in dispatch_command (command=COM_QUERY, thd=0x7f948c79c008, packet=0x7f948c7a2009 "SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` "..., packet_length=212) at 10.0.11/sql/sql_parse.cc:1309 #30 0x000000000067500c in do_command (thd=0x7f948c79c008) at 10.0.11/sql/sql_parse.cc:1006 #31 0x000000000079608f in do_handle_one_connection (thd_arg=0x7f948c79c008) at 10.0.11/sql/sql_connect.cc:1379 #32 0x0000000000795de2 in handle_one_connection (arg=0x7f948c79c008) at 10.0.11/sql/sql_connect.cc:1293 #33 0x0000000000ceaa98 in pfs_spawn_thread (arg=0x7f948c7f4808) at 10.0.11/storage/perfschema/pfs.cc:1860 #34 0x00007f949532db50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #35 0x00007f9493825a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 Also repeatable on the current tree.
            serg Sergei Golubchik made changes -
            Workflow defaullt [ 41619 ] MariaDB v2 [ 43238 ]
            andymc73 Andrew McDonnell added a comment - - edited

            Well thats interesting, the above test doesn't crash for me.

            Using latest github (10.1.0)

            I also tried re-ordering the --disconnect con1 to after the second query.

            andymc73 Andrew McDonnell added a comment - - edited Well thats interesting, the above test doesn't crash for me. Using latest github (10.1.0) I also tried re-ordering the --disconnect con1 to after the second query.

            I am running on a quad core AMD phenom with Debian Wheezy & gcc 4.7.2 & boost 1.49.0

            Heinz, what is your version for those?

            andymc73 Andrew McDonnell added a comment - I am running on a quad core AMD phenom with Debian Wheezy & gcc 4.7.2 & boost 1.49.0 Heinz, what is your version for those?
            andymc73 Andrew McDonnell added a comment - - edited

            I just triggered a crash, but not in an expected manner:

            I put DROP TABLE statements at the end to stop MTR complaining about the state. I first had my DROP inside a third --connect, then I took the third connect away and had the drop just standalone.

            Now I need to work out if this is a manifestation of the same thing or something different

            EDIT actually scratch that. I cant find a trace or anything, so must be my interpretation of MTR - once you use a --connect, all statements need to be inside one?

            andymc73 Andrew McDonnell added a comment - - edited I just triggered a crash, but not in an expected manner: I put DROP TABLE statements at the end to stop MTR complaining about the state. I first had my DROP inside a third --connect , then I took the third connect away and had the drop just standalone. Now I need to work out if this is a manifestation of the same thing or something different EDIT actually scratch that. I cant find a trace or anything, so must be my interpretation of MTR - once you use a --connect , all statements need to be inside one?

            I triggered the bug on multiple systems. Originally on a virtual server with a dual core Intel Xeon, CentOS 6, glibc 2.12, boost 1.41.0, with the packages from the mariadb repo.
            My debugging system is a VirtualBox VM, single core Intel Core i7 with Slackware-current gcc 4.8.2, glibc 2.19, boost 1.55.0

            pprkut Heinz Wiesinger added a comment - I triggered the bug on multiple systems. Originally on a virtual server with a dual core Intel Xeon, CentOS 6, glibc 2.12, boost 1.41.0, with the packages from the mariadb repo. My debugging system is a VirtualBox VM, single core Intel Core i7 with Slackware-current gcc 4.8.2, glibc 2.19, boost 1.55.0
            elenst Elena Stepanova added a comment - - edited

            andymc73,

            Did you try to run the same test with valgrind?
            It crashed on two machines for me, but a it's often a gamble on luck, valgrind might help when the luck doesn't.

            And please don't use 10.1.0, use 10.0, we don't need another variable in here.

            elenst Elena Stepanova added a comment - - edited andymc73 , Did you try to run the same test with valgrind? It crashed on two machines for me, but a it's often a gamble on luck, valgrind might help when the luck doesn't. And please don't use 10.1.0, use 10.0, we don't need another variable in here.
            andymc73 Andrew McDonnell added a comment - - edited

            Which means I still havent repeated this

            [EDIT] - this was typed before I refreshed the page and saw your comments Elena, sorry

            andymc73 Andrew McDonnell added a comment - - edited Which means I still havent repeated this [EDIT] - this was typed before I refreshed the page and saw your comments Elena, sorry
            andymc73 Andrew McDonnell added a comment - - edited My version of the MTR test is at https://github.com/pastcompute/server/blob/oqgraph_mdev_6282/storage/oqgraph/mysql-test/oqgraph/regression_mdev6282.test

            10.0 trunk or last release?

            andymc73 Andrew McDonnell added a comment - 10.0 trunk or last release?

            It must be 10+ years since I used Slackware Maybe I'll have to create a virtual machine to try it out for this... is 'Slackware-current' 14.1?

            andymc73 Andrew McDonnell added a comment - It must be 10+ years since I used Slackware Maybe I'll have to create a virtual machine to try it out for this... is 'Slackware-current' 14.1?
            elenst Elena Stepanova added a comment - - edited

            I had a crash on both, stack trace is from the last release. I think for certainty it's better to use the last release (tag mariadb-10.0.11); when you get a reliable test, you can always switch to the trunk.

            Your test looks okay from MTR perspective. What exactly does it say?

            DROP only matters if the SELECT does not crash; if MTR complains about leftovers, it means the test did not do the trick.
            But in my experiments it was very important, on whatever reason, to disconnect con1 before connecting with con2.
            You might also try to put a sleep between my disconnect and connect, at some point I did that, but somehow it stopped being relevant later.

            And btw my machines were simple Wheezy and Precise, so Slackware should not matter.

            UPD: and your test case doesn't crash for me either. Mine still does (on 10.0 revno 4241 built with cmake . -DCMAKE_BUILD_TYPE=Debug && make)

            elenst Elena Stepanova added a comment - - edited I had a crash on both, stack trace is from the last release. I think for certainty it's better to use the last release (tag mariadb-10.0.11); when you get a reliable test, you can always switch to the trunk. Your test looks okay from MTR perspective. What exactly does it say? DROP only matters if the SELECT does not crash; if MTR complains about leftovers, it means the test did not do the trick. But in my experiments it was very important, on whatever reason, to disconnect con1 before connecting with con2. You might also try to put a sleep between my disconnect and connect, at some point I did that, but somehow it stopped being relevant later. And btw my machines were simple Wheezy and Precise, so Slackware should not matter. UPD: and your test case doesn't crash for me either. Mine still does (on 10.0 revno 4241 built with cmake . -DCMAKE_BUILD_TYPE=Debug && make)

            OK, I just built against my bzr working branch, after merged from lp:maria

            http://bazaar.launchpad.net/~andymc73/maria/oqgraph-maintenance

            Here is the relevant MTR output, after adjusting to ensure con1 is closed first and then using a con3 to do the drop.

            I will next revert back to the tag mariadb-10.0.11 and try again

            TEST                                      RESULT   TIME (ms) or COMMENT
            --------------------------------------------------------------------------
             
            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            INSTALL SONAME 'ha_oqgraph';
            CREATE TABLE `db_history` (
            `version` VARCHAR(10) NOT NULL,
            `updateJSON` MEDIUMTEXT,
            `prevVersion` VARCHAR(10) NOT NULL,
            `nodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0',
            `prevNodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0',
            PRIMARY KEY (`prevNodeID`,`nodeID`),
            KEY `version` (`version`) USING BTREE,
            KEY `prevVersion` (`prevVersion`) USING BTREE,
            KEY `nodeID` (`nodeID`) 
            ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
            INSERT INTO `db_history` VALUES 
            ('0.0.1','{}','0.0.0',1,0),
            ('0.0.2','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.1',2,1),
            ('0.0.3','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.2',3,2);
            CREATE TABLE IF NOT EXISTS version_history (
            latch VARCHAR(32) NULL,
            origid BIGINT UNSIGNED NULL,
            destid BIGINT UNSIGNED NULL,
            weight DOUBLE NULL,
            seq BIGINT UNSIGNED NULL,
            linkid BIGINT UNSIGNED NULL,
            KEY (latch, origid, destid) USING HASH,
            KEY (latch, destid, origid) USING HASH 
            ) ENGINE=OQGRAPH DATA_TABLE='db_history' ORIGID='prevNodeID' DESTID='nodeID';
            SELECT `db`.`version`, `db`.`nodeID` 
            FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` 
            WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;
            version	nodeID
            0.0.3	3
            SELECT `db`.`version`, `db`.`nodeID` 
            FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` 
            WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;
            version	nodeID
            0.0.3	3
            DROP TABLE db_history;
            DROP TABLE version_history;
            oqgraph.regression_mdev6282              [ pass ]     18
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Spent 0.018 of 5 seconds executing testcases

            As an aside, I have had trouble with this branch showing properly on launchpad. I can push and pull from it but it doesnt update or show files or update karma, I posted a problem report on launchpad answers about three months ago and got nothing but crickets... So you can see why I am really glab we are moving over to github...

            andymc73 Andrew McDonnell added a comment - OK, I just built against my bzr working branch, after merged from lp:maria http://bazaar.launchpad.net/~andymc73/maria/oqgraph-maintenance Here is the relevant MTR output, after adjusting to ensure con1 is closed first and then using a con3 to do the drop. I will next revert back to the tag mariadb-10.0.11 and try again TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 INSTALL SONAME 'ha_oqgraph'; CREATE TABLE `db_history` ( `version` VARCHAR(10) NOT NULL, `updateJSON` MEDIUMTEXT, `prevVersion` VARCHAR(10) NOT NULL, `nodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', `prevNodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', PRIMARY KEY (`prevNodeID`,`nodeID`), KEY `version` (`version`) USING BTREE, KEY `prevVersion` (`prevVersion`) USING BTREE, KEY `nodeID` (`nodeID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; INSERT INTO `db_history` VALUES ('0.0.1','{}','0.0.0',1,0), ('0.0.2','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.1',2,1), ('0.0.3','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.2',3,2); CREATE TABLE IF NOT EXISTS version_history ( latch VARCHAR(32) NULL, origid BIGINT UNSIGNED NULL, destid BIGINT UNSIGNED NULL, weight DOUBLE NULL, seq BIGINT UNSIGNED NULL, linkid BIGINT UNSIGNED NULL, KEY (latch, origid, destid) USING HASH, KEY (latch, destid, origid) USING HASH ) ENGINE=OQGRAPH DATA_TABLE='db_history' ORIGID='prevNodeID' DESTID='nodeID'; SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; version nodeID 0.0.3 3 SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; version nodeID 0.0.3 3 DROP TABLE db_history; DROP TABLE version_history; oqgraph.regression_mdev6282 [ pass ] 18 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 0.018 of 5 seconds executing testcases As an aside, I have had trouble with this branch showing properly on launchpad. I can push and pull from it but it doesnt update or show files or update karma, I posted a problem report on launchpad answers about three months ago and got nothing but crickets... So you can see why I am really glab we are moving over to github...
            elenst Elena Stepanova added a comment - - edited

            What about running with valgrind, did you try it?

            It's also interesting that MTR is happy at the end. I mean, you install the plugin but never uninstall it, MTR's post-test check should complain about it. Are you using the stock MariaDB, or is it a modified version?

            elenst Elena Stepanova added a comment - - edited What about running with valgrind, did you try it? It's also interesting that MTR is happy at the end. I mean, you install the plugin but never uninstall it, MTR's post-test check should complain about it. Are you using the stock MariaDB, or is it a modified version?
            andymc73 Andrew McDonnell added a comment - - edited

            Here are my build settings:

            cmake .. -DWITH_EXTRA_CHARSETS=complex -DWITH_PLUGIN_ARIA=1 -DWITH_READLINE=1 -DWITH_SSL=bundled -DWITH_MAX=1 -DWITH_EMBEDDED_SERVER=1

            I'll try again with yours

            andymc73 Andrew McDonnell added a comment - - edited Here are my build settings: cmake .. -DWITH_EXTRA_CHARSETS=complex -DWITH_PLUGIN_ARIA=1 -DWITH_READLINE=1 -DWITH_SSL=bundled -DWITH_MAX=1 -DWITH_EMBEDDED_SERVER=1 I'll try again with yours

            I copied that plugin line from yours, but in retrospect it is probably redundant, because I dont have it in any of the other tests for oqgraph.
            So I presume the build somehow "knows" that files under storage/oqgraph/mysql-test have that plugin already

            I still have to try with valgrind , waiting for r 4209 (mariadb-10.0.11) to compile

            andymc73 Andrew McDonnell added a comment - I copied that plugin line from yours, but in retrospect it is probably redundant, because I dont have it in any of the other tests for oqgraph. So I presume the build somehow "knows" that files under storage/oqgraph/mysql-test have that plugin already I still have to try with valgrind , waiting for r 4209 (mariadb-10.0.11) to compile
            elenst Elena Stepanova added a comment - - edited

            Actually, it is very interesting, try valgrind on yours first.
            I just ran my test on 10.0.10 official release build (that's what I had handy as non-debug), and it didn't crash, but valgrind shows invalid read just where the debug build crashes.

            Valgrind complaints from 10.0.10 release build:

            ==30192== Invalid read of size 8
            ==30192==    at 0x7146A3: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:4857)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486)
            ==30192==    by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8e9070 is 3,584 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146AF: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486)
            ==30192==    by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146B6: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486)
            ==30192==    by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid write of size 8
            ==30192==    at 0x7146BE: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486)
            ==30192==    by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146C5: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486)
            ==30192==    by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146A3: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:4857)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296)
            ==30192==    by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==  Address 0xc8e9070 is 3,584 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146AF: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296)
            ==30192==    by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146B6: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296)
            ==30192==    by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==  Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid write of size 8
            ==30192==    at 0x7146BE: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296)
            ==30192==    by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x7146C5: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473)
            ==30192==    by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503)
            ==30192==    by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296)
            ==30192==    by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==  Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x714560: handler::ha_index_next(unsigned char*) (sql_class.h:4857)
            ==30192==    by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306)
            ==30192==    by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8e9078 is 3,592 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x71456C: handler::ha_index_next(unsigned char*) (sql_class.h:2473)
            ==30192==    by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306)
            ==30192==    by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x714573: handler::ha_index_next(unsigned char*) (sql_class.h:2473)
            ==30192==    by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306)
            ==30192==    by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid write of size 8
            ==30192==    at 0x71457B: handler::ha_index_next(unsigned char*) (sql_class.h:2473)
            ==30192==    by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306)
            ==30192==    by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)
            ==30192== Invalid read of size 8
            ==30192==    at 0x714582: handler::ha_index_next(unsigned char*) (sql_class.h:2473)
            ==30192==    by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306)
            ==30192==    by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi
            ==30192==    by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877)
            ==30192==    by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965)
            ==30192==    by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549)
            ==30192==    by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329)
            ==30192==    by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296)
            ==30192==    by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620)
            ==30192==    by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346)
            ==30192==    by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464)
            ==30192==    by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129)
            ==30192==    by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609)
            ==30192==    by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355)
            ==30192==    by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372)
            ==30192==    by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306)
            ==30192==  Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd
            ==30192==    at 0x4C27BF4: free (vg_replace_malloc.c:469)
            ==30192==    by 0x56AEDC: THD::~THD() (sql_list.h:633)
            ==30192==    by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730)
            ==30192==    by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841)
            ==30192==    by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390)
            ==30192==    by 0x663401: handle_one_connection (sql_connect.cc:1293)
            ==30192==    by 0x8B190A: pfs_spawn_thread (pfs.cc:1853)
            ==30192==    by 0x4E35B4F: start_thread (pthread_create.c:304)
            ==30192==    by 0x5EF3A7C: clone (clone.S:112)

            elenst Elena Stepanova added a comment - - edited Actually, it is very interesting, try valgrind on yours first. I just ran my test on 10.0.10 official release build (that's what I had handy as non-debug), and it didn't crash, but valgrind shows invalid read just where the debug build crashes. Valgrind complaints from 10.0.10 release build: ==30192== Invalid read of size 8 ==30192== at 0x7146A3: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:4857) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486) ==30192== by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8e9070 is 3,584 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146AF: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486) ==30192== by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146B6: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486) ==30192== by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid write of size 8 ==30192== at 0x7146BE: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486) ==30192== by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146C5: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xC9F8434: open_query::oqgraph_share::find_vertex(unsigned long long) const (oqgraph_shim.h:486) ==30192== by 0xC9FB3FC: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:726) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146A3: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:4857) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296) ==30192== by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== Address 0xc8e9070 is 3,584 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146AF: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296) ==30192== by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146B6: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296) ==30192== by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid write of size 8 ==30192== at 0x7146BE: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296) ==30192== by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x7146C5: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (sql_class.h:2473) ==30192== by 0xCA089E1: oqgraph3::cursor::seek_to(boost::optional<unsigned long long>, boost::optional<unsigned long long>) (oqgraph_thunk.cc:503) ==30192== by 0xCA01604: boost::out_edges(unsigned long long, oqgraph3::graph const&) (oqgraph_shim.h:296) ==30192== by 0xCA03E5A: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x714560: handler::ha_index_next(unsigned char*) (sql_class.h:4857) ==30192== by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306) ==30192== by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8e9078 is 3,592 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x71456C: handler::ha_index_next(unsigned char*) (sql_class.h:2473) ==30192== by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306) ==30192== by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x714573: handler::ha_index_next(unsigned char*) (sql_class.h:2473) ==30192== by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306) ==30192== by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8e82d0 is 96 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid write of size 8 ==30192== at 0x71457B: handler::ha_index_next(unsigned char*) (sql_class.h:2473) ==30192== by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306) ==30192== by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8eb488 is 12,824 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112) ==30192== Invalid read of size 8 ==30192== at 0x714582: handler::ha_index_next(unsigned char*) (sql_class.h:2473) ==30192== by 0xCA09188: oqgraph3::cursor::seek_next() (oqgraph_thunk.cc:306) ==30192== by 0xCA03F03: void boost::breadth_first_visit<oqgraph3::graph, boost::queue<unsigned long long, std::deque<unsigned long long, std::allocator<unsigned long long> > >, boost::bfs_visitor<std::pair<boost::predecessor_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, unsigned long long> > > >, boost::on_tree_edge>, std::pair<boost::distance_recorder<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, double, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, double> > > >, boost::on_tree_edge>, open_query::oqgraph_visit_dist<boost::associative_property_map<boost::unordered::unordered_map<unsigned long long, unsigned long long, boost::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsi ==30192== by 0xC9FCC7D: open_query::oqgraph::search(int*, unsigned long long*, unsigned long long*) (graphcore.cc:877) ==30192== by 0xC9F6D40: ha_oqgraph::index_read_idx(unsigned char*, unsigned int, unsigned char const*, unsigned int, ha_rkey_function) (ha_oqgraph.cc:965) ==30192== by 0x71467C: handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) (handler.cc:2549) ==30192== by 0x71880A: handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) (handler.cc:5329) ==30192== by 0x69BED4: handler::multi_range_read_next(void**) (multi_range_read.cc:296) ==30192== by 0x815EB1: QUICK_RANGE_SELECT::get_next() (opt_range.cc:11620) ==30192== by 0x8342E4: rr_quick(READ_RECORD*) (records.cc:346) ==30192== by 0x5D3B23: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:17464) ==30192== by 0x5D424C: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:17129) ==30192== by 0x5E6E3F: JOIN::exec_inner() (sql_select.cc:2609) ==30192== by 0x5E5A19: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:2355) ==30192== by 0x5E963C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:372) ==30192== by 0x58FAEF: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:5306) ==30192== Address 0xc8ed170 is 20,224 bytes inside a block of size 21,216 free'd ==30192== at 0x4C27BF4: free (vg_replace_malloc.c:469) ==30192== by 0x56AEDC: THD::~THD() (sql_list.h:633) ==30192== by 0x507D3A: unlink_thd(THD*) (mysqld.cc:2730) ==30192== by 0x507D91: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2841) ==30192== by 0x66323A: do_handle_one_connection(THD*) (sql_connect.cc:1390) ==30192== by 0x663401: handle_one_connection (sql_connect.cc:1293) ==30192== by 0x8B190A: pfs_spawn_thread (pfs.cc:1853) ==30192== by 0x4E35B4F: start_thread (pthread_create.c:304) ==30192== by 0x5EF3A7C: clone (clone.S:112)

            4209 doesnt crash either.

            Currently trying with valgrind on 4209 with my build settings

            andymc73 Andrew McDonnell added a comment - 4209 doesnt crash either. Currently trying with valgrind on 4209 with my build settings
            andymc73 Andrew McDonnell added a comment - - edited

            Valgrind gives me this:

            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 23396, winpid: 23396, exit: 256] died after mysql-test-run waited 0.2 seconds for /scratch/develop/maria/bzr/maria-oqgraph-maintenance/build/mysql-test/var/run/mysqld.1.pid to be created.
            oqgraph.regression_mdev6282              [ fail ]
                    Test ended at 2014-06-15 22:51:01
             
            CURRENT_TEST: oqgraph.regression_mdev6282
             
             
            Failed to start mysqld.1
            mysqltest failed but provided no output

            I am going to have to stop for the night shortly, sorry.

            But that trace you just posted could prove helpful, thanks

            andymc73 Andrew McDonnell added a comment - - edited Valgrind gives me this: worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 23396, winpid: 23396, exit: 256] died after mysql-test-run waited 0.2 seconds for /scratch/develop/maria/bzr/maria-oqgraph-maintenance/build/mysql-test/var/run/mysqld.1.pid to be created. oqgraph.regression_mdev6282 [ fail ] Test ended at 2014-06-15 22:51:01   CURRENT_TEST: oqgraph.regression_mdev6282     Failed to start mysqld.1 mysqltest failed but provided no output I am going to have to stop for the night shortly, sorry. But that trace you just posted could prove helpful, thanks

            Bingo - just got a core against the github build (I still had open in another window so I tried it with your build settings)

            So it seems that the cmake settings are important for some reason - probably compiler optimisations or something

            In increment statistics again...

            Thread 1 (Thread 0x7fefebfdf700 (LWP 27670)):
            #0  __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
            #1  0x0000000000e45db7 in my_write_core (sig=11) at /scratch/develop/maria/github/server/mysys/stacktrace.c:457
            #2  0x0000000000861c9d in handle_fatal_signal (sig=11) at /scratch/develop/maria/github/server/sql/signal_handler.cc:262
            #3  <signal handler called>
            #4  0x00000000006eb77d in THD::check_limit_rows_examined (this=0x7fefe3798070) at /scratch/develop/maria/github/server/sql/sql_class.h:2478
            #5  0x00000000006eb7e6 in handler::increment_statistics (this=0x7fefe1c6c088, offset=&system_status_var::ha_read_key_count) at /scratch/develop/maria/github/server/sql/sql_class.h:4866
            #6  0x0000000000867c71 in handler::ha_index_read_map (this=0x7fefe1c6c088, buf=0x7fefe1c5dd88 "\376\005\060.0.2", key=0x7fefe1c1dc88 "\001", keypart_map=1, find_flag=HA_READ_KEY_OR_NEXT) at /scratch/develop/maria/github/server/sql/handler.cc:2587
            #7  0x00007fefe29e6e4e in oqgraph3::cursor::seek_to (this=0x7fefe1c9b410, origid=..., destid=...) at /scratch/develop/maria/github/server/storage/oqgraph/oqgraph_thunk.cc:503
            #8  0x00007fefe29d02f9 in boost::find_vertex (id=1, g=...) at /scratch/develop/maria/github/server/storage/oqgraph/oqgraph_shim.h:480
            #9  0x00007fefe29cab15 in open_query::oqgraph_share::find_vertex (this=0x7fefe1c1d4f0, id=1) at /scratch/develop/maria/github/server/storage/oqgraph/graphcore.cc:485

            andymc73 Andrew McDonnell added a comment - Bingo - just got a core against the github build (I still had open in another window so I tried it with your build settings) So it seems that the cmake settings are important for some reason - probably compiler optimisations or something In increment statistics again... Thread 1 (Thread 0x7fefebfdf700 (LWP 27670)): #0 __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63 #1 0x0000000000e45db7 in my_write_core (sig=11) at /scratch/develop/maria/github/server/mysys/stacktrace.c:457 #2 0x0000000000861c9d in handle_fatal_signal (sig=11) at /scratch/develop/maria/github/server/sql/signal_handler.cc:262 #3 <signal handler called> #4 0x00000000006eb77d in THD::check_limit_rows_examined (this=0x7fefe3798070) at /scratch/develop/maria/github/server/sql/sql_class.h:2478 #5 0x00000000006eb7e6 in handler::increment_statistics (this=0x7fefe1c6c088, offset=&system_status_var::ha_read_key_count) at /scratch/develop/maria/github/server/sql/sql_class.h:4866 #6 0x0000000000867c71 in handler::ha_index_read_map (this=0x7fefe1c6c088, buf=0x7fefe1c5dd88 "\376\005\060.0.2", key=0x7fefe1c1dc88 "\001", keypart_map=1, find_flag=HA_READ_KEY_OR_NEXT) at /scratch/develop/maria/github/server/sql/handler.cc:2587 #7 0x00007fefe29e6e4e in oqgraph3::cursor::seek_to (this=0x7fefe1c9b410, origid=..., destid=...) at /scratch/develop/maria/github/server/storage/oqgraph/oqgraph_thunk.cc:503 #8 0x00007fefe29d02f9 in boost::find_vertex (id=1, g=...) at /scratch/develop/maria/github/server/storage/oqgraph/oqgraph_shim.h:480 #9 0x00007fefe29cab15 in open_query::oqgraph_share::find_vertex (this=0x7fefe1c1d4f0, id=1) at /scratch/develop/maria/github/server/storage/oqgraph/graphcore.cc:485

            Check the error logs, possibly you have old version of valgrind, it will say something about an unknown option.
            Also, please paste again the full text of the test that you are running, I'll check if it fails for me (that nothing has dramatically changed in the test)

            elenst Elena Stepanova added a comment - Check the error logs, possibly you have old version of valgrind, it will say something about an unknown option. Also, please paste again the full text of the test that you are running, I'll check if it fails for me (that nothing has dramatically changed in the test)

            This triggered it - all i did was reformat, and add the MTR cleanup

            --connect (con1,localhost,root,,test)
             
              CREATE TABLE `db_history` (
                `version` VARCHAR(10) NOT NULL,
                `updateJSON` MEDIUMTEXT,
                `prevVersion` VARCHAR(10) NOT NULL,
                `nodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0',
                `prevNodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0',
                PRIMARY KEY (`prevNodeID`,`nodeID`),
                KEY `version` (`version`) USING BTREE,
                KEY `prevVersion` (`prevVersion`) USING BTREE,
                KEY `nodeID` (`nodeID`) 
              ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
             
              INSERT INTO `db_history` VALUES 
                ('0.0.1','{}','0.0.0',1,0),
                ('0.0.2','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.1',2,1),
                ('0.0.3','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.2',3,2);
            	
              CREATE TABLE IF NOT EXISTS version_history (
                latch VARCHAR(32) NULL,
                origid BIGINT UNSIGNED NULL,
                destid BIGINT UNSIGNED NULL,
                weight DOUBLE NULL,
                seq BIGINT UNSIGNED NULL,
                linkid BIGINT UNSIGNED NULL,
                KEY (latch, origid, destid) USING HASH,
                KEY (latch, destid, origid) USING HASH 
              ) ENGINE=OQGRAPH DATA_TABLE='db_history' ORIGID='prevNodeID' DESTID='nodeID';
             
              SELECT `db`.`version`, `db`.`nodeID` 
              FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` 
              WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;
            --disconnect con1
             
            --connect (con2,localhost,root,,test)
              SELECT `db`.`version`, `db`.`nodeID` 
              FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` 
              WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1;
            --disconnect con2
             
            --connect (con3,localhost,root,,test)
            DROP TABLE db_history;
            DROP TABLE version_history;
            --disconnect con3
             

            andymc73 Andrew McDonnell added a comment - This triggered it - all i did was reformat, and add the MTR cleanup --connect (con1,localhost,root,,test)   CREATE TABLE `db_history` ( `version` VARCHAR(10) NOT NULL, `updateJSON` MEDIUMTEXT, `prevVersion` VARCHAR(10) NOT NULL, `nodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', `prevNodeID` BIGINT(20) UNSIGNED NOT NULL DEFAULT '0', PRIMARY KEY (`prevNodeID`,`nodeID`), KEY `version` (`version`) USING BTREE, KEY `prevVersion` (`prevVersion`) USING BTREE, KEY `nodeID` (`nodeID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8;   INSERT INTO `db_history` VALUES ('0.0.1','{}','0.0.0',1,0), ('0.0.2','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.1',2,1), ('0.0.3','{\"data\":{\"add\":{},\"update\":{},\"delete\":{}}}','0.0.2',3,2); CREATE TABLE IF NOT EXISTS version_history ( latch VARCHAR(32) NULL, origid BIGINT UNSIGNED NULL, destid BIGINT UNSIGNED NULL, weight DOUBLE NULL, seq BIGINT UNSIGNED NULL, linkid BIGINT UNSIGNED NULL, KEY (latch, origid, destid) USING HASH, KEY (latch, destid, origid) USING HASH ) ENGINE=OQGRAPH DATA_TABLE='db_history' ORIGID='prevNodeID' DESTID='nodeID';   SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; --disconnect con1   --connect (con2,localhost,root,,test) SELECT `db`.`version`, `db`.`nodeID` FROM `version_history` AS `v` INNER JOIN `db_history` AS `db` ON `db`.`nodeID` = `v`.`linkid` WHERE `latch` = 'breadth_first' AND `origid` = '1' ORDER BY `weight` DESC LIMIT 1; --disconnect con2   --connect (con3,localhost,root,,test) DROP TABLE db_history; DROP TABLE version_history; --disconnect con3  
            elenst Elena Stepanova added a comment - - edited

            Okay, great, whatever works. Surely debug and non-debug builds are very different, it happens often that what crashes on one doesn't crash on another.
            Valgrind usually shows something anyway, but since you've already got the crash, valgrind doesn't matter.

            elenst Elena Stepanova added a comment - - edited Okay, great, whatever works. Surely debug and non-debug builds are very different, it happens often that what crashes on one doesn't crash on another. Valgrind usually shows something anyway, but since you've already got the crash, valgrind doesn't matter.

            So this appears to be a use after free - I got it with the following gdb sequence

            bre oqgraph_thunk.cc:499
            runtable->in_use->status_var.ha_read_key_count
            cont
            cont
            cont
            bre handler.cc:2587
            cont
            print table->in_use->status_var.ha_read_key_count
            p  (int*)(&table->in_use->status_var.ha_read_key_count)
            watch *(result of prior command)  #  <-- for some reason gdb was being annoying and would not let me put this on one line

            After table->in_use->status_var.ha_read_key_count is incremented to 5, the underlying data gets free()'d

            Old value = 5
            New value = -1886417009
            __memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:1285
            1285    ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
            (gdb) bt
            #0  __memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:1285
            #1  0x0000000000f48de3 in free_memory (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/safemalloc.c:275
            #2  0x0000000000f48ab3 in sf_free (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/safemalloc.c:194
            #3  0x0000000000f37db6 in my_free (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/my_malloc.c:216
            #4  0x00000000005c23a6 in ilink::operator delete (ptr_arg=0x7fffefb6f070, size=21624) at /scratch/develop/maria/github/server/sql/sql_list.h:633
            #5  0x00000000006364d1 in THD::~THD (this=0x7fffefb6f070, __in_chrg=<optimized out>) at /scratch/develop/maria/github/server/sql/sql_class.cc:1638
            #6  0x00000000005b74ad in unlink_thd (thd=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/mysqld.cc:2745
            #7  0x00000000005b780d in one_thread_per_connection_end (thd=0x7fffefb6f070, put_in_cache=true) at /scratch/develop/maria/github/server/sql/mysqld.cc:2856
            #8  0x0000000000789e53 in do_handle_one_connection (thd_arg=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/sql_c
            onnect.cc:1390
            #9  0x0000000000789b00 in handle_one_connection (arg=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/sql_connect.cc:1293
            #10 0x0000000000b6d57a in pfs_spawn_thread (arg=0x7fffefa690f0) at /scratch/develop/maria/github/server/storage/perfschema/pfs.cc:1860
            #11 0x00007ffff7bc7b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
            #12 0x00007ffff6918a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
            #13 0x0000000000000000 in ?? ()

            A new thread is then created, presumably the next connections query.

            But when it comes back around to increment_statistics(), for some reason the value of table->in_use is pointing to the wrong (old) data.

            andymc73 Andrew McDonnell added a comment - So this appears to be a use after free - I got it with the following gdb sequence bre oqgraph_thunk.cc:499 runtable->in_use->status_var.ha_read_key_count cont cont cont bre handler.cc:2587 cont print table->in_use->status_var.ha_read_key_count p (int*)(&table->in_use->status_var.ha_read_key_count) watch *(result of prior command) # <-- for some reason gdb was being annoying and would not let me put this on one line After table->in_use->status_var.ha_read_key_count is incremented to 5, the underlying data gets free()'d Old value = 5 New value = -1886417009 __memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:1285 1285 ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory. (gdb) bt #0 __memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:1285 #1 0x0000000000f48de3 in free_memory (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/safemalloc.c:275 #2 0x0000000000f48ab3 in sf_free (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/safemalloc.c:194 #3 0x0000000000f37db6 in my_free (ptr=0x7fffefb6f070) at /scratch/develop/maria/github/server/mysys/my_malloc.c:216 #4 0x00000000005c23a6 in ilink::operator delete (ptr_arg=0x7fffefb6f070, size=21624) at /scratch/develop/maria/github/server/sql/sql_list.h:633 #5 0x00000000006364d1 in THD::~THD (this=0x7fffefb6f070, __in_chrg=<optimized out>) at /scratch/develop/maria/github/server/sql/sql_class.cc:1638 #6 0x00000000005b74ad in unlink_thd (thd=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/mysqld.cc:2745 #7 0x00000000005b780d in one_thread_per_connection_end (thd=0x7fffefb6f070, put_in_cache=true) at /scratch/develop/maria/github/server/sql/mysqld.cc:2856 #8 0x0000000000789e53 in do_handle_one_connection (thd_arg=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/sql_c onnect.cc:1390 #9 0x0000000000789b00 in handle_one_connection (arg=0x7fffefb6f070) at /scratch/develop/maria/github/server/sql/sql_connect.cc:1293 #10 0x0000000000b6d57a in pfs_spawn_thread (arg=0x7fffefa690f0) at /scratch/develop/maria/github/server/storage/perfschema/pfs.cc:1860 #11 0x00007ffff7bc7b50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #12 0x00007ffff6918a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #13 0x0000000000000000 in ?? () A new thread is then created, presumably the next connections query. But when it comes back around to increment_statistics() , for some reason the value of table->in_use is pointing to the wrong (old) data.

            When the first connection exits, there is a THD object that gets free()'d.
            This THD* is linked to a table object as table.in_use
            This table object is referenced by a _graph object retained by OQGraph that is constructed by ha_oqgraph implementation of handler::open()
            When the second connection starts, nothing is done that resets the _graph object table object to point to a different in_use

            Actually this is probably only a first approximation as to what is going on. But essentially this is a use-after-free bug.

            andymc73 Andrew McDonnell added a comment - When the first connection exits, there is a THD object that gets free()'d. This THD* is linked to a table object as table.in_use This table object is referenced by a _graph object retained by OQGraph that is constructed by ha_oqgraph implementation of handler::open() When the second connection starts, nothing is done that resets the _graph object table object to point to a different in_use Actually this is probably only a first approximation as to what is going on. But essentially this is a use-after-free bug.
            andymc73 Andrew McDonnell added a comment - - edited

            Message posted to oqgraph-dev list: hopefully Antony can help, as he wrote the original code.

            Hi all,
             
            https://mariadb.atlassian.net/browse/MDEV-6282 is a use after free bug that
            happens when one connection is opened, oqgraph queried, then the connection
            closed, then a second connection opened and oqgraph queried again in the same
            server session.
             
            Essentially, our handler in open() eventually creates a graph share object
            that keeps a reference to the underlying TABLE object, this table object has
            an in_use field that is a THD*.
             
            But when the connection goes away, mysqld free()'d that THD.  Except the
            handler never processes close() at this point, and we never get a chance to
            cause that table object to 'refresh' for want of a better word.
             
            Which means when the next query comes along, eventually it calls index_read()
            on our handler which calls seek_to() in the graph code which calls back into
            mysql using ha_index_read_map on the backing table, which then crashes because
            it eventually calls increment_statistics which first increments free()'d
            memory via &SSV and then accesses a bogus member pointer in the long dead THD.
             
            I dont yet know enough about the internals of mysqld to know how to handle this...
             
            cheers,
            Andrew

            andymc73 Andrew McDonnell added a comment - - edited Message posted to oqgraph-dev list: hopefully Antony can help, as he wrote the original code. Hi all,   https://mariadb.atlassian.net/browse/MDEV-6282 is a use after free bug that happens when one connection is opened, oqgraph queried, then the connection closed, then a second connection opened and oqgraph queried again in the same server session.   Essentially, our handler in open() eventually creates a graph share object that keeps a reference to the underlying TABLE object, this table object has an in_use field that is a THD*.   But when the connection goes away, mysqld free()'d that THD. Except the handler never processes close() at this point, and we never get a chance to cause that table object to 'refresh' for want of a better word.   Which means when the next query comes along, eventually it calls index_read() on our handler which calls seek_to() in the graph code which calls back into mysql using ha_index_read_map on the backing table, which then crashes because it eventually calls increment_statistics which first increments free()'d memory via &SSV and then accesses a bogus member pointer in the long dead THD.   I dont yet know enough about the internals of mysqld to know how to handle this...   cheers, Andrew
            andymc73 Andrew McDonnell made changes -
            andymc73 Andrew McDonnell made changes -

            Doh. I merged with the latest 10.1 HEAD and now this test case is not triggering the bug for me (at least, with my current CONFIGURE settings.) So need to spend some time narrowing things down more (unless this bug is by some luck fixed by other changes in 10.1 and is still different to MDEV-6784)

            andymc73 Andrew McDonnell added a comment - Doh. I merged with the latest 10.1 HEAD and now this test case is not triggering the bug for me (at least, with my current CONFIGURE settings.) So need to spend some time narrowing things down more (unless this bug is by some luck fixed by other changes in 10.1 and is still different to MDEV-6784 )

            Triggered using `--parallel` with MTR

            andymc73 Andrew McDonnell added a comment - Triggered using `--parallel` with MTR

            --parallel=N option does a completely different thing. It runs several independent instances of test environment in parallel – N servers (or N*X servers, if the test requires several mysqld servers), on different ports, with separate datadirs, the corresponding number of instances of the tests, etc.

            You can run mysqlslap from MTR, see mysql-test/t/mysqlslap.test for an example of how it is done.

            elenst Elena Stepanova added a comment - --parallel=N option does a completely different thing. It runs several independent instances of test environment in parallel – N servers (or N*X servers, if the test requires several mysqld servers), on different ports, with separate datadirs, the corresponding number of instances of the tests, etc. You can run mysqlslap from MTR, see mysql-test/t/mysqlslap.test for an example of how it is done.

            OK, thanks. I'll have to remember that for MDEV-6345

            andymc73 Andrew McDonnell added a comment - OK, thanks. I'll have to remember that for MDEV-6345

            OK, did some more digging.

            OQGRAPH seems to allocate some memory in handler::open() using init_tmp_table_share().
            And then stores references to the backing store in that memory.
            Except that when the THD goes away that memory vanishes with it. And even though the backing store table gets told what the THD is, there is no way to reverse lookup such that we can get called back when the THD associated with that table is being deleted. And bang.

            I looked through the THD unlink code and there were possibilities related to LOCK TABLES and things called user level locks but I dont think they were quite what we need here.

            I think the solution is to find a better way to persist our memory outside of the THD when our table gets not closed when the THD that opens it goes away. Which is probably just restating what we already know, in a different way. Except made more concrete by singling out probably bad use of per-thread tables.

            andymc73 Andrew McDonnell added a comment - OK, did some more digging. OQGRAPH seems to allocate some memory in handler::open() using init_tmp_table_share(). And then stores references to the backing store in that memory. Except that when the THD goes away that memory vanishes with it. And even though the backing store table gets told what the THD is, there is no way to reverse lookup such that we can get called back when the THD associated with that table is being deleted. And bang . I looked through the THD unlink code and there were possibilities related to LOCK TABLES and things called user level locks but I dont think they were quite what we need here. I think the solution is to find a better way to persist our memory outside of the THD when our table gets not closed when the THD that opens it goes away. Which is probably just restating what we already know, in a different way. Except made more concrete by singling out probably bad use of per-thread tables.

            Can you please try testing this building MariaDB using the branch:

            https://github.com/pastcompute/server/tree/oqgraph_mdev_6282_6345_5748

            git clone https://github.com/pastcompute/server.git
            git checkout oqgraph_mdev_6282_6345_5748
            ... build and test as normal ...

            andymc73 Andrew McDonnell added a comment - Can you please try testing this building MariaDB using the branch: https://github.com/pastcompute/server/tree/oqgraph_mdev_6282_6345_5748 git clone https://github.com/pastcompute/server.git git checkout oqgraph_mdev_6282_6345_5748 ... build and test as normal ...

            This is fantastic news
            I'll have a look at it at the weekend and let you know how things go

            pprkut Heinz Wiesinger added a comment - This is fantastic news I'll have a look at it at the weekend and let you know how things go

            I built the patched tree and after putting it through its paces I have good and bad news.
            So, the bad news is that I still get a crash. The good news is that my test using the siege benchmark tool now works without problems. No crashes even with 100 concurrent requests. The crash that happens seems to really hit a corner case.
            I have the debugging log attached. It looks like everything works fine as long as the query has a "latch" parameter specified. If I issue a "latch" query from one connection and follow up with a simple "SELECT * FROM graph_table" (so no "latch" parameter in the query) on another connection, it crashes.

            pprkut Heinz Wiesinger added a comment - I built the patched tree and after putting it through its paces I have good and bad news. So, the bad news is that I still get a crash. The good news is that my test using the siege benchmark tool now works without problems. No crashes even with 100 concurrent requests. The crash that happens seems to really hit a corner case. I have the debugging log attached. It looks like everything works fine as long as the query has a "latch" parameter specified. If I issue a "latch" query from one connection and follow up with a simple "SELECT * FROM graph_table" (so no "latch" parameter in the query) on another connection, it crashes.
            pprkut Heinz Wiesinger made changes -
            Attachment oqgraph_crash_without_latch.log [ 35800 ]

            OK, that still needs to be fixed, but as a test, do you get the same crash with explicitly setting latch = '' ?

            andymc73 Andrew McDonnell added a comment - OK, that still needs to be fixed, but as a test, do you get the same crash with explicitly setting latch = '' ?

            I missed a couple of spots that maybe needed fixing, can you please pull the latest on that branch and try again?

            Note, this is actually MariaDB 10.1.* that you are testing, I will need to apply the same fix to 10.0.* if it works

            andymc73 Andrew McDonnell added a comment - I missed a couple of spots that maybe needed fixing, can you please pull the latest on that branch and try again? Note, this is actually MariaDB 10.1.* that you are testing, I will need to apply the same fix to 10.0.* if it works

            Alright, I will update to the new version. Will take a bit though as I compile this in a VM. In the meantime, setting latch = '' doesn't crash, but also doesn't produce the same output as the plain 'SELECT * FROM graph_table"

            pprkut Heinz Wiesinger added a comment - Alright, I will update to the new version. Will take a bit though as I compile this in a VM. In the meantime, setting latch = '' doesn't crash, but also doesn't produce the same output as the plain 'SELECT * FROM graph_table"

            With the latest version all crashes seem to be gone

            pprkut Heinz Wiesinger added a comment - With the latest version all crashes seem to be gone

            I am hoping to get the fix for this backported to 10.0.x (the launchpad-BZR tree) in the next month or so, I may have found an assistant to help me

            serg As well as the liekly fix for these three bugs, I have a bunch of minor structural improvements in my github 10.1 fork, which includes MDEV-5871 as well, would it be worth me submitting a pull for that?

            (i.e. I am assuming that development of 10.1 is now somewhat independent of 10.0 ??)

            andymc73 Andrew McDonnell added a comment - I am hoping to get the fix for this backported to 10.0.x (the launchpad-BZR tree) in the next month or so, I may have found an assistant to help me serg As well as the liekly fix for these three bugs, I have a bunch of minor structural improvements in my github 10.1 fork, which includes MDEV-5871 as well, would it be worth me submitting a pull for that? (i.e. I am assuming that development of 10.1 is now somewhat independent of 10.0 ??)

            yes, sure, submit a pull request.

            As I've seen no activity on MDEV-5871 for quite a while, I've already applied MDEV-5871 patch myself, in my tree.
            But it is not pushed yet, so I can remove that commit from my tree. Depends on when you'll submit your pull request

            serg Sergei Golubchik added a comment - yes, sure, submit a pull request. As I've seen no activity on MDEV-5871 for quite a while, I've already applied MDEV-5871 patch myself, in my tree. But it is not pushed yet, so I can remove that commit from my tree. Depends on when you'll submit your pull request

            I just rebased my work on latest 10.1 and intending to submit pull request after re-testing

            andymc73 Andrew McDonnell added a comment - I just rebased my work on latest 10.1 and intending to submit pull request after re-testing
            andymc73 Andrew McDonnell made changes -
            Affects Version/s 10.1.0 [ 12200 ]

            10.1 version pull request: https://github.com/MariaDB/server/pull/16

            I'll be trying to get 10.0 sorted soon

            andymc73 Andrew McDonnell added a comment - 10.1 version pull request: https://github.com/MariaDB/server/pull/16 I'll be trying to get 10.0 sorted soon
            serg Sergei Golubchik added a comment - - edited

            Just one question before I reassign it to myself for merging.

            "get 10.0 sorted soon" — does it mean fixing this bug in 10.0? Then I don't want to merge your 10.1 fix, we merge 10.0 to 10.1 and if I merge your 10.0 fix into 10.0 and your 10.1 fix into 10.1, we'll end up having two your fixes in 10.1.

            Generally, a bug should be fixed only once in the lowest applicable version, then the fix will be merged upwards.

            So, shall I merge your 10.1 fix into 10.1? Or I should rather wait for your 10.0 fix and them merge it into 10.0 together with all other our 10.0 fixes?

            serg Sergei Golubchik added a comment - - edited Just one question before I reassign it to myself for merging. "get 10.0 sorted soon" — does it mean fixing this bug in 10.0? Then I don't want to merge your 10.1 fix, we merge 10.0 to 10.1 and if I merge your 10.0 fix into 10.0 and your 10.1 fix into 10.1, we'll end up having two your fixes in 10.1. Generally, a bug should be fixed only once in the lowest applicable version, then the fix will be merged upwards. So, shall I merge your 10.1 fix into 10.1? Or I should rather wait for your 10.0 fix and them merge it into 10.0 together with all other our 10.0 fixes?

            OK, please wait then - I was able to merge 10.1 back to 10.0 and all the tests passed... I didn't realise that but it makes sense.

            I'll cancel the existing pull request.

            From reading what you prefer, I'll make a new branch off 10.0 and cherry-pick everything and then do a new pull request against 10.0 instead.

            andymc73 Andrew McDonnell added a comment - OK, please wait then - I was able to merge 10.1 back to 10.0 and all the tests passed... I didn't realise that but it makes sense. I'll cancel the existing pull request. From reading what you prefer, I'll make a new branch off 10.0 and cherry-pick everything and then do a new pull request against 10.0 instead.

            Yes. If you plan to fix the bug (or do some change, whatever) in both 10.0 anf 10.1, then you only need to do it and provide a pull request for 10.0 — I'll take care of propagating it to 10.1. Of course, if you want something to be done in 10.1 only, then you just create a pull request for 10.1.

            serg Sergei Golubchik added a comment - Yes. If you plan to fix the bug (or do some change, whatever) in both 10.0 anf 10.1, then you only need to do it and provide a pull request for 10.0 — I'll take care of propagating it to 10.1. Of course, if you want something to be done in 10.1 only, then you just create a pull request for 10.1.
            andymc73 Andrew McDonnell added a comment - https://github.com/MariaDB/server/pull/17
            serg Sergei Golubchik made changes -
            Assignee Andrew McDonnell [ andymc73 ] Sergei Golubchik [ serg ]

            I did a forward merge test to 10.1, there are a couple of instances where there is a merge conflict that needs to be resolved C,B or similar

            I pushed a branch that has the correct result if you need something to compare against

            https://github.com/pastcompute/server/tree/10.1-oqgraph-6282-6345-6784-cherrypick-test

            andymc73 Andrew McDonnell added a comment - I did a forward merge test to 10.1, there are a couple of instances where there is a merge conflict that needs to be resolved C,B or similar I pushed a branch that has the correct result if you need something to compare against https://github.com/pastcompute/server/tree/10.1-oqgraph-6282-6345-6784-cherrypick-test

            I checked pull/16 and pull/17. Both fix the problem and result in a crash-free load test here

            Thank you very much!

            pprkut Heinz Wiesinger added a comment - I checked pull/16 and pull/17. Both fix the problem and result in a crash-free load test here Thank you very much!

            No worries!

            BTW pull #16 has been cancelled although it is a good test -

            My understanding is that #17 will get merged into 10.0 branch first, then Sergei will at some point merge 10.0 -> 10.1

            andymc73 Andrew McDonnell added a comment - No worries! BTW pull #16 has been cancelled although it is a good test - My understanding is that #17 will get merged into 10.0 branch first, then Sergei will at some point merge 10.0 -> 10.1
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Component/s Storage Engine - OQGRAPH [ 10130 ]
            Fix Version/s 10.0.18 [ 18702 ]
            Fix Version/s 10.0 [ 16000 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Workflow MariaDB v2 [ 43238 ] MariaDB v3 [ 66108 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 66108 ] MariaDB v4 [ 147919 ]

            People

              serg Sergei Golubchik
              pprkut Heinz Wiesinger
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.