Debian Wheezy x86_64

Linux greeneggs.lentz.com.au 3.9.3-x86_64-linode33 #1 SMP Mon May 20 10:22:57 EDT 2013 x86_64 GNU/Linux

Hi Daniel,

Given that it's 5.5 (hence no multi-source replication there), what exactly do you mean by multi-master? Could you please specify the replication topology you are using?

And another question, for better understanding – how come are you using a Wheezy package on Squeeze?

Thanks.

> Could you please specify the replication topology you are using?

Two servers, both masters to each other, both slaves of each other.

> how come are you using a Wheezy package on Squeeze?

Was in error. Am using a wheezy package on wheezy.

valgrind output of mysqld running under load for about an hour.

If you want me to run with other valgrind options let me know.

Hi Daniel,

Can you share any information in regard to SELECTs that you're using? Even better if there's also information about the data structures.

Thanks.

Elena, I don't see relevance for that request. And it's unpractical.
First of all, it could be any query (including INSERT, UPDATE, DELETE as well as DDL) or any other activity in the server that leaks - so in that sense, only a full general query log output could possibly be of any help. However, it's more likely to just provide a large pile of relatively useless data.
We're not discussing a single instance where some queries cause a problem, this system is merely where we've specifically put instrumentation in place to track down the problem that we've seen in a couple of systems. These systems have pretty much nothing in common in terms of applications that run on them, so from that angle also I don't think it's relevant.

I would think that the Valgrind output should provide a good lead when it comes to memory leaks?

A related question, can you please confirm whether the build team runs the testsuite on a Valgrind build for each release? This used to be done but given the issue with the memory leak in 5.5.31 I'd just like to make sure. Thanks

Arjen,

There is obvious relevance though.
The valgrind errors that Daniel provided indicate that the memory loss happens on SELECTs, apparently on SELECTs with subqueries and/or joins. Usually when it happens, it happens in some specific circumstances (the query structure, the data structure). I am running a big general set of tests with valgrind over the last couple of days, and it might (or might not) catch the problem at the end, but that's impractical for solving this particular problem in the limited time before 5.5.33 release. It would be much faster and more efficient if Daniel was able to provide any information in regard to the queries that he actually uses that might fit the valgrind errors he caught, so I could narrow down the test and hit the error faster.

For the question about valgrind and MTR, yes, I confirm that the team runs the test suite on a Valgrind build for each release (for each push, for that matter). You can see it as "work-amd64-valgrind" builder results on the buildbot page.

The db is doing about 60 selects per second across 40 databases with a fair diversity of code on it. The last trackback identified 20 in ~40mins so is still a large dataset to filter.

I was playing with the following in an init script and having the general log running. Was hoping that --vgdb=yes --vgdb-error=1 would freeze activity (waiting for debugger) in the general log leaving an offending query somewhere near the end of the general log. Any other ideas?

/usr/bin/valgrind --leak-check=full --read-var-info=yes --undef-value-errors=no --trace-children=yes --trace-children-skip='/cut,/wc,/grep,/date,/dirname,/expr,/logger,/rm,/my_print_defaults,/sed' --log-file=/tmp/valgrind.mysqld.%p --vgdb=yes --vgdb-error=1 \
/usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &

note: --undef-value-errors=no is a big cpu saver

> Was hoping that --vgdb=yes --vgdb-error=1

Didn't work. Compiling valgrind from source with the --error-limit at 1 (./coregrind/m_errormgr.c M_COLLECT_NO_ERRORS_AFTER_FOUND=1 (not 10,000,000)).

Hi Elena - tnx for that info, good to hear.
As Dan indicated, we're trying to narrow things down as it's too many schemas and queries on this server.

At this point I'd take whatever I could get. If I understand correctly, Daniel's run with valgrind also had general log enabled, is it so? If it is, would it be possible for me to see that general log? 60 selects per second in an hour-long test run is not that bad, I could actually go through the log to see what it has that my tests don't and proceed from there.

Elena - k can do. might have to make that a private upload.
Will leave the practical side of that to Dan to work out.

Hi Daniel,

Could you please try another thing: turn off index_merge* switches in the optimizer_switch (globally) and see if it makes the problem go away? It might or might not, depending on which queries cause the leak. According to Sergei Petrunia's analysis, it's either queries that use index_merge, or queries with GROUP BY and/or ORDER BY (and join, of course). The latter is not very helpful, because there are a huge number of queries with ORDER BY in the flow; so it's easier to check the index_merge theory first.

Ok. Have set index_merge*=off on the non-valgrind running server to see if there's a leak globally. Will run this under valgrind if there proves to be a leak or tomorrow.

before:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

after:
MariaDB [(none)]> set global optimizer_switch='index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off';
Query OK, 0 rows affected (0.01 sec)

Good news and bad. Its not index merge. Still leaking.

Thanks, Daniel.

That's bad news, since my next attempts to reproduce it over the weekend (now by populating your schema with some random values and re-running the entire general log) also didn't hit the problem.

I'll think what else to do.

My thoughts are to get this arcane patch compiled and try to view the datastructure ( https://bugs.kde.org/show_bug.cgi?id=108905 ) to give a clue or write a patch on valgrind to record the allocation time of memory and try to correlate it that way.

Hi Daniel,

To make sure I build server the same way you do (I mean the valgrind one) – how do you build it?
Also, was there anything suspicious at all in the error log?

Maybe you could use valgrind massif?
http://www.mysqlperformanceblog.com/2013/01/09/profiling-mysql-memory-usage-with-valgrind-massif/

The valgrind I've used so far has just been the debian packaged valgrind. I did end up compling a valgrind-3.8.1 with a basic ./configure && make && sudo make install

Mariadb was fairly standard:
apt-get build-dep mariadb-5.5
apt-get install cmake libaio1 libaio-dev fakeroot valgrind
apt-get source mariadb-5.5
cd mariadb-5.5*

edited debian/control and debian/rules as per https://kb.askmonty.org/en/how-to-produce-a-full-stack-trace-for-mysqld-the-mariadb-server/#comment_359 (except using 5.5 as version number)
edit debian/rules added -DCMAKE_BUILD_TYPE=Debug to cmake as per https://kb.askmonty.org/en/compiling-mariadb-for-debugging/

dpkg-buildpackage -nc -us -uc (http://www.debian.org/doc/manuals/maint-guide/build.en.html)
dpkg -i

edit /etc/init.d/mysql to contain
/usr/local/bin/valgrind --leak-check=full --read-var-info=yes --undef-value-errors=no --trace-children=yes --trace-children-skip='/cut,/wc,/grep,/date,/dirname,/expr,/logger,/rm,/my_print_defaults,/sed' --log-file=/tmp/valgrind.mysqld.%p --time-stamp=yes \

before the mysqld_safe line

Thanks for the link Patryk - I had seen this before but dismissed it as not enough. Given what I have currently I'll take another look.

massiff output pushed to MDEV-4974_2.tar.gz ftp://ftp.askmonty.org/private/

I've yet to look at it myself. On irc if you have any requests

MDEV-4974_3.tar.gz added - threshold lowered and configured to do millisecond offsets

the mysqld was started with general log active and this is included the tarball

ok. I've looked through the massif logs. They don't have a sample snapshot/backtrace the same as the memcheck one so the ones so far aren't that much use.

They do however produce a have a time on the snapshot that can be correlated with the general query log.

I may just need to run the massif long enough to get the snapshot (though I'm thinking this is still a low probability of a hit).

More realistic could tigger debug messages (or some gdb events) to obtain timings at one of the following locations to correlate with the general query log:
==7763== by 0x6BFE55: init_read_record(READ_RECORD*, THD*, TABLE*, SQL_SELECT*, int, bool, bool) (records.cc:595)
==7763== by 0x4D236F: join_init_read_record(st_join_table*) (sql_select.cc:17203)
==7763== by 0x4C25D0: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16329)

That's right, I also saw that there was no this specific one in init_read_record; but there was something in evaluate_join_record instead. Sergei Petrunia is going to take a look at the output to see if it's relevant to the initial memory leak problem.

I've been reading up on https://sourceware.org/gdb/onlinedocs/gdb.html#Tracepoints

I've compiled userspace-rcu-0.5.3/ust-0.11 and built gdbserver (gdb/gdbserver with ./configure --with-ust=/usr/local ).

In record.cc added the following after the existing includes (duplicate and seemingly equivalent defs occurred for likely/unlikely)

#undef unlikely
#undef likely
#include <ust/marker.h>

inside the existing if (! init_rr_cache(thd, info) braces added:

/* is there something more I should be capturing? - text representation of query? or execution plan? */
trace_mark(record,init_read_record,"TIME: %lld ALLOC: %p", (long long int) thd->start_time, info->cache);

configure.cmake
MY_SEARCH_LIBS(marker_probe_cb ust LIBUST)
MY_SEARCH_LIBS(rcu_read_lock urcu-bp LIBURCUBP)

and appending ${LIBUST} ${LIBURCUBP} to then end of CMAKE_REQUIRED_LIBRARIES

Unfortunately is currently not linking with the following error:
.../.libmysqld.a/(libmysqld.arecords.cc.o()libmysqld.arecords.cc.o()_markersrecords.cc.o+:0x20)()::_markers (+0x20): __markersundefined+0x20)undefined: undefined reference reference to `marker_probe_cb'to
../libmysqld.a(records.cc.o`)marker_probe_cb'
:libmysqld.a(__markers+0x28(): undefined records.cc.o reference) :reference(to __markers to+` __mark_empty_function0x28'`
)marker_probe_cb:'
undefined. .reference/ libmysqld.ato( records.cc.o`)__mark_empty_function:'(
_markers+0x28): undefined reference to `_mark_empty_function'

tips on debugging this welcome but haven't spent much time on this.

next I plan to:

with gdbserver running mysqld (or the valgrind --tool=memcheck if I can)

attach gdb and try the following:

set default-collect $args,$locals,$_ret,$_sdata
strace -m record/init_read_record if select!=0 && use_record_cache==1 && print_error==1 && disable_rr_cache==FALSE;

1. this restricts the calls to the join_init_read_record and join_read_record_no_init calls (of which join_init_read_record was seen in the memleak backtrace).
2. if possible add to the condition $_ret == (location after the call in join_init_read_record)

tstart

1. wait for collection
   tstop

tfind start
while ($trace_frame != -1)
printf "Frame %d, = %08X, sdataSP = %s, ret = %08X\n", \
$trace_frame, $_sdata, $_ret
tfind
end

and correlated allocated address to memleak output, or time to general query log.

After switching from Precise to Wheezy, I've got the one in ma_checkpoint_init (which is also in your valgrind output). No those about sub_select/join_init_record yet, but it's already remarkable since i wasn't getting any at all on Precise. I'm going to re-run my other tests on Wheezy, possibly it's something system-related (valgrind version is the same on both, 3.7.0)

Me and Elena went through the available information again, including the massif logs, and could not get any clue.

Daniel, could you try exploring with the attached patch, psergey-mdev4974-xpl1.diff ?

The patch makes the server to write a file, /tmp/leaks-track.sql. Into that file it will write, at appropriate times, statements:

insert into track_mem (query_id, ptr, query) values (%lld, %p, '%s');
delete from track_mem where query_id=%lld and ptr=%p;
insert into leaks select * from track_mem where query_id=%lld;

the idea is that one can pipe leaks-track.sql into another mysql server, and as soon as table "leaks" is not empty, it will have the query that has caused the leak.

Sergie, thanks for the patch. I have compiled, run and uploaded the result to ftp://ftp.askmonty.org/private/.

Seems to only include the insert into leaks command and no insert/delete from/into track_mem. I tried moving track_mem_alloc(thd, info->cache); before DBUG_RETURN(1); in sql/record.cc but it didn't seem to have an effect.

Assigned to Sergei Petrunia for the follow-up on the debug output.

I'm happy to try another patch. I'm still getting this memory leak.

I've looked through 4974-leaks-track.sql and have found this:

insert into track_mem (query_id, ptr, query) values (29476, 0x7f277f242000, 'SELECT DISTINCT node.nid AS nid,
node.type AS node_type,
node.title AS node_title,
users.name AS users_name,
users.uid AS users_uid,
node_comment_statistics.comment_count AS node_comment_statistics_comment_count,
node_comment_statistics.');

This gives us something!

Btw, the above is the only such query in all the uploaded files.

• For some reason, the query text has been cut. danblack, is it possible to grep through query logs, or the app you're running and find the full query that starts with

SELECT DISTINCT node.nid AS nid,
node.type AS node_type,
node.title AS node_title,
users.name AS users_name,
users.uid AS users_uid,
node_comment_statistics.comment_count AS node_comment_statistics_comment_count,
node_comment_statistics.

(If there are multiple candidates, we would like all of them).

• When you've found the query, could you provide

• the query text (essential)
• the table DDLs for the involved tables (just as essential)
• what you get if you run EXPLAIN for the query (nice to have)

• (not essential, but nice to have) Could you start the server with the above provided patch, run the query manually, and then provide leaks-track.sql? This is to check whether the query always leaks data. If it doesn't leak data, we would still like to have query text, DDLs, etc - there are still chances I'll spot the leak.

Interesting - that looks like Drupal 6 query. Other deployments where we observed the leak don't use Drupal, but it's good to see that we can catch at least one aspect of the problem in this way. We'll find out what the full query is.

I don't see another patch?

drupal mysqldump --no-data

'''# Time: 131121 10:35:06

1. User@Host: oq-drupal[oq-drupal] @ [74.207.241.159]
2. Thread_id: 1024112 Schema: oq_drupal QC_hit: No
3. Query_time: 3.298782 Lock_time: 0.000182 Rows_sent: 25 Rows_examined: 224767
4. Full_scan: No Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes
5. Filesort: Yes Filesort_on_disk: Yes Merge_passes: 13
   use oq_drupal;
   SET timestamp=1385030106;
   SELECT DISTINCT node.nid AS nid,
   node.type AS node_type,
   node.title AS node_title,
   users.name AS users_name,
   users.uid AS users_uid,
   node_comment_statistics.comment_count AS node_comment_statistics_comment_count,
   node_comment_statistics.last_comment_timestamp AS node_comment_statistics_last_comment_timestamp
   FROM node node
   INNER JOIN users users ON node.uid = users.uid
   INNER JOIN node_comment_statistics node_comment_statistics ON node.nid = node_comment_statistics.nid
   INNER JOIN node_access na ON na.nid = node.nid WHERE (na.grant_view >= 1 AND ((na.gid = 0 AND na.realm = 'all') OR (na.gid = 1 AND na.realm = 'forum_access'))) AND ( node.status = 1
   )ORDER BY node_title ASC
   LIMIT 16075, 25;

MariaDB [oq_drupal]> explain ^MSELECT DISTINCT node.nid AS nid,
-> node.type AS node_type,
-> node.title AS node_title,
-> users.name AS users_name,
-> users.uid AS users_uid,
-> node_comment_statistics.comment_count AS node_comment_statistics_comment_count,
-> node_comment_statistics.last_comment_timestamp AS node_comment_statistics_last_comment_timestamp
-> FROM node node
-> INNER JOIN users users ON node.uid = users.uid
-> INNER JOIN node_comment_statistics node_comment_statistics ON node.nid = node_comment_statistics.nid
-> INNER JOIN node_access na ON na.nid = node.nid WHERE (na.grant_view >= 1 AND ((na.gid = 0 AND na.realm = 'all') OR (na.gid = 1 AND na.realm = 'forum_access'))) AND ( node.status = 1
-> )ORDER BY node_title ASC
-> LIMIT 16075, 25;;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------+

--------------------------------------------------------------------------------------------------------------------------------------------------------------------+

--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
'''
Variants include the last ORDER by being node_comment_statistics_comment_count, user_name and some of these are DESC.

''

Don't worry - patch is still applied.

SELECT DISTINCT node.nid AS nid,
node.type AS node_type,
node.title AS node_title,
users.name AS users_name,
users.uid AS users_uid,
node_comment_statistics.comment_count AS node_comment_statistics_comment_count,
node_comment_statistics.last_comment_timestamp AS node_comment_statistics_last_comment_timestamp
FROM node node
INNER JOIN users users ON node.uid = users.uid
INNER JOIN node_comment_statistics node_comment_statistics ON node.nid = node_comment_statistics.nid
INNER JOIN node_access na ON na.nid = node.nid WHERE (na.grant_view >= 1 AND ((na.gid = 0 AND na.realm = 'all') OR (na.gid = 1 AND na.realm = 'forum_access'))) AND ( node.status = 1
)ORDER BY node_title ASC
LIMIT 16075, 25;

( cat /var/log/mysql/mariadb-slow.log.1 ; zcat /var/log/mysql/mariadb-slow.log*z ) | grep A 12 '^{SELECT DISTINCT node.nid AS nid' | grep -v '}-$' | mysql oq_drupal

see drupal_data_MDEV4974.sql in private ftp for extracted tables relevant to these query. users data is just uid and a name in a form.

Ok, I was able to repeat the leak with the provided data. Thanks a lot!

The query plan uses "Using temporary; Using filesort" and "Distinct". What that means:
1. The join is executed, and the results are put into the temporary table (Using temporary). The table has a unique key, this is how "Distinct" is achieved.
2. The temporary table is then passed to filesort(). filesort() produces an ordered stream (an IO_CACHE of rowids).
3. Then, a join (a degenerate one, a result of make_simple_join()) is run the temptable + IO_CACHE. Reading the table is initialized with
init_read_record(), which calls init_rr_cache().
4. sub_select() finishes with NESTED_LOOP_QUERY_LIMIT
5. JOIN::join_free() is invoked. it calls JOIN::cleanup(full=TRUE)
which calls free_io_cache(table). free_io_cache() frees something, but that's not what init_read_record() has allocated.

I am not sure who didn't free what init_read_record() has allocated. Will need to debug further.

Awesome Sergei, glad to hear you were able to replicate the prob using the data provided by us.
Hopefully we'll see the final find soon!

One question that will of course be interesting to know the answer to is when and where the problem was introduced.

Hey Sergei. Any luck isolating which function should free the init_read_record allocation?

Hi Daniel,

I'll need to debug to find out which part of the code fails to free the memory. This bug is at the top of my todo, but I am currently on vacation and is unable to work on it. I hope to start working on it at around Dec, 16th.

I'm glad you weren't working on it on your vacation and I hope you enjoyed yourself. If there's anything I can help with now that you're back please let me know.

Thanks for the ping. I have investigated the problem:

The EXPLAIN is:
-------------------------..-----------------------------------------------------------------------+

-------------------------..-----------------------------------------------------------------------+

-------------------------..-----------------------------------------------------------------------+

It means, execution proceeds as follows:
1. Execute the join of 4 tables.
2. Put join output into a temporary table with TABLE::distinct=1. This resolves the SELECT .. DISTINCT part of the query.
3. Pass the temporary table to filesort.
4. filesort() will produce "record pointers" (that is, an array of rowids in the temporary table)
5. then, use that array to read first LIMIT rows in disk order.

(the above is valid for both MariaDB and MySQL)

MySQL code traditionally had a weak spot. On step #1, JOIN::join_tab referred to the original base tables (node, users, node_comment_statistics, na). Before step #5, the executor would set JOIN::join_tab to point to an array of one element, with temptable created on step#2. The original array was gone, though, which caused problems for EXPLAIN and didn't allow to implement SHOW EXPLAIN (or EXPLAIN FOR CONNECTION).

MariaDB has solved this problem in the following cset:

timour@askmonty.org-20120821122443-ut07mf3z3194ch5d
5.5-md447
Fix bug mdev-447: Wrong output from the EXPLAIN command of the test case for lp bug #714999

The fix backports from MWL#182: Explain running statements the logic that
saves the original JOIN_TAB array of a query plan after optimization. This
array is later used during EXPLAIN to iterate over the original JOIN plan
nodes in the cases when this plan could be changed by early subquery
execution during the optimization phase of the outer query.

The idea of the fix was to have two JOIN_TAB arrays (grep for WALK_OPTIMIZATION_TABS and WALK_EXECUTION_TABS). However, we have introduced this bug - we don't call JOIN_TAB::cleanup() for the JOIN_TAB that reads from the temp. table. This works typically (most query plans will just read the temp. table). However, when reading from temporary table employs init_rr_cache() and rr_from_cache() functions, one must free the cache afterwards. MariaDB didn't do this, which caused this bug.

I suspect, MySQL 5.6 also faced this problem. They have made this change:

evgeny.potemkin@oracle.com-20120531162137-iq3wjp1dppk17syi
WL#6071: Inline tmp tables into the nested loops algorithm.

they don't create two JOIN_TAB arrays. Instead, they extend the original join_tab array, and put the join_tab element that reads from temp.table at the very end of that array. I'm not sure which approach is better (WL#6071 is not public, so I need to guess about Evgen's reasoning) but one consequence is that MySQL 5.6 does not have an analog of this bug.

This fix makes the leak go away. I think, it needs a round of testing, the join cleanup area doesnt have a clear architecture so I could be missing something.

Elena, could you please test 5.5 tree, patched with psergey-fix-mdev4954.diff ?

The first round of tests went well, running the next one, now with valgrind.
Although, I am still to try to figure out why the bug itself was not triggered by regular tests. We have DISTINCTs, and joins, and ORDER BYs, and LIMIT, and various combinations of those.

Asked Sanja to review the patch

The patch is OK to push.

I 100% sure that I already faced problem of non-cleanup table_access_tabs (you even found flaw in my firat patch), but I can't find results (probably it was not pushed or passed to you).

Fix pushed into 5.5 tree (I have made a smaller testcase that could be included into mysql-test).

Daniel, thanks a lot for reporting this, and for all the input that allowed us to pin down and fix the problem!

Thanks Sergei, Sanja, and Elena for the fix. Looking forward to a DB that doesn't leak memory and all the other cool stuff and fixes you've done over last months (and years really).

http://bazaar.launchpad.net/~maria-captains/maria/5.5/revision/4013

Here is the smallest test case based on Daniel's data that I could come up so far. Sergei, could you please take a look and if it is good enough, use it as a bugfix test case (I checked that the valgrind error it causes disappeared after the bugfix, but I'm not sure it's the main target problem and not just a side-effect).

CREATE TABLE t1 (a VARCHAR(32000) DEFAULT '', b VARCHAR(32000), id INT);
INSERT INTO t1 (b, id) VALUES
('t1',2), ('t2',1),('t3',2),('t4',1),('t5',2),('t6',1),('t7',2),('t8',1),
('t9',2),('t10',1),('t11',2),('t12',1),('t13',2),('t14',1),('t15',2),('t16',1),
('t17',2),('t18',1),('t19',2),('t20',1),('t21',2),('t22',1),('t23',2),('t24',1),
('t25',2),('t26',1),('t27',2),('t28',1),('t29',2),('t30',1),('t31',2),('t32',1),
('t33',2),('t34',1),('t35',2),('t36',1),('t37',2),('t38',1),('t39',2),('t40',1),
('t41',2),('t42',1),('t43',2),('t44',1),('t45',2),('t46',1),('t47',2),('t48',1),
('t49',2),('t50',1),('t51',2),('t52',1),('t53',2),('t54',1),('t55',2),('t56',1),
('t57',2),('t58',1),('t59',2),('t60',1),('t61',2),('t62',1),('t63',2),('t64',1),
('t65',2),('t66',1),('t67',2),('t68',1),('t69',2),('t70',1),('t71',2),('t72',1),
('t73',2),('t74',1),('t75',2),('t76',1),('t77',2),('t78',1),('t79',2),('t80',1),
('t81',2),('t82',1),('t83',2),('t84',1),('t85',2),('t86',1),('t87',2),('t88',1),
('t89',2),('t90',1),('t91',2),('t92',1),('t93',2),('t94',1),('t95',2),('t96',1),
('t97',2),('t98',1),('t99',2),('t100',1),('t101',2),('t102',1),('t103',2),
('t104',1),('t105',2),('t106',1),('t107',2),('t108',1),('t109',2),('t110',1),
('t111',2),('t112',1),('t113',2),('t114',1),('t115',2),('t116',1),('t117',2),
('t118',1),('t119',2),('t120',1),('t121',2),('t122',1),('t123',2),('t124',1),
('t125',2),('t126',1),('t127',2),('t128',1),('t129',2),('t130',1),('t131',2),
('t132',1),('t133',2),('t134',1),('t135',2),('t136',1),('t137',2),('t138',1),
('t139',2),('t140',1),('t141',2),('t142',1),('t143',2),('t144',1),('t145',2),
('t146',1),('t147',2),('t148',1),('t149',2),('t150',1),('t151',2),('t152',1),
('t153',2),('t154',1),('t155',2),('t156',1),('t157',2),('t158',1),('t159',2),
('t160',1),('t161',2),('t162',1),('t163',2),('t164',1),('t165',2),('t166',1),
('t167',2),('t168',1),('t169',2),('t170',1),('t171',2),('t172',1),('t173',2),
('t174',1),('t175',2),('t176',1),('t177',2),('t178',1),('t179',2),('t180',1),
('t181',2),('t182',1),('t183',2),('t184',1),('t185',2),('t186',1),('t187',2),
('t188',1),('t189',2),('t190',1),('t191',2),('t192',1),('t193',2),('t194',1),
('t195',2),('t196',1),('t197',2),('t198',1),('t199',2),('t200',1),('t201',2),
('t202',1),('t203',2),('t204',1),('t205',2),('t206',1),('t207',2),('t208',1),
('t209',2),('t210',1),('t211',2),('t212',1),('t213',2),('t214',1),('t215',2),
('t216',1),('t217',2),('t218',1),('t219',2),('t220',1),('t221',2),('t222',1),
('t223',2),('t224',1),('t225',2),('t226',1),('t227',2),('t228',1),('t229',2),
('t230',1),('t231',2),('t232',1),('t233',2),('t234',1),('t235',2),('t236',1),
('t237',2),('t238',1),('t239',2),('t240',1),('t241',2),('t242',1),('t243',2),
('t244',1),('t245',2),('t246',1),('t247',2),('t248',1),('t249',2),('t250',1),
('t251',2),('t252',1),('t253',2);

CREATE TABLE t2 (c VARCHAR(60) NOT NULL, id INT(10));
INSERT INTO t2 VALUES ('user1',1),('user2',2);

SELECT DISTINCT * FROM t1 INNER JOIN t2 ON t1.id = t2.id ORDER BY t1.b;

http://bazaar.launchpad.net/~maria-captains/maria/5.5/revision/4013

been running 5.5.35 for 7 hrs now and no memory leaks. It would of leaked about 100M previously.

Definitely fixed and thanks gain.

So, it is really fixed. Thanks for the confirmation.