fedora-20

Hi,

Yes, please attach the function. Can you (also) attach the source code?

binary for UDF huffman_dist

I'm too ashamed to this code will be here forever so I'll send it by email, OK?

Sure, please do.
Alternatively, you can upload it to our FTP, the private section (ftp.askmonty.org/private). This way nobody except for MariaDB team will have access to it.

Please also attach your cnf file(s) and/or output of SHOW VARIABLES.
So far I tried the query with your binary version of the function on a 32-bit machine and with one built from the source on a 64-bit machine, on MariaDB 5.5.35, 5.5.36 and 5.5.37, all works fine:

SELECT count(h.id) FROM wp_bp_album_hash h where  h.id!=8164 AND h.hashv !='' AND huffman_dist(h.hashv,'00BB0000 00000000 00BB9C80 00000000 00C09BF9 00000000 CCC00C04 000000CC DC4C0000 00000CCD FFDDCC0C 00CCCFFB DDFFDC40 00CFFFFD FFF3FFCC DDFDCCDD FFFFDD0C 0CCCCDDF CCCCC4CC 0C0DDDDD CCCC40C0 0000CCCC DCFFF400 0000C04C DCDDC000 00000CDD DD040000 0000CCCD C0000000 00000000 00000000 00000000')< 400;

+-------------+

| count(h.id) |

+-------------+

|         108 |

+-------------+

1 row in set (0.03 sec)

I reproduced this problem on debug server so I can try to help localize it on my side.

PS. MariaDB built from sources by rpmbuild with attached SPEC

Can you get a stack trace from the running process while it is hanging?
Something like gdb --batch --eval-command="thread apply all bt" <binary> <pid> would do (or anything of your choice as long as it produces the stack trace).

Please comment to re-open the report when you have any new information.

Sorry for timeout. Unfortunately I can not debug on the production server, and on debug server the bug occures periodically and is not caught.
Attachment - "mysqladmin proc" just before hanging

Oops, I can't add attachment

Seems that I can't add any attachments until reopening

Re-opened, now you should be able to add the attachments.

err.txt - "mysqladmin proc" just before hang

trace.log - stack trace when some threads hangs
err1 - "mysqladmin proc" just after trace.log

Hi,

Thanks.
Did you take the stack trace snapshot once, or did you try it several times?
In the latter case, were there any changes in the stack traces between the snapshots?

Also, how is the CPU usage when it hangs?

Once, there is no so much time before hangs.

CPU load about 0 - hanging threads don't eats CPU

What about stack traces during hanging (not before)? Do they look any different?

I'm asking about stack traces and CPU because server that is really hanging (forever waiting for something) and server going through an endless loop often look the same – unavailable, irresponsive. The difference is that for a server in a loop consequent stack traces differ and CPU load is [relatively] high. Since your CPU is about 0, I assume it's not the case.

>stack traces during hanging (not before)
I can't do much debug on production server because we lose all subscribers... "killall -9 mysqld" spells automatically when "mysqladmin ping" fails.

Yes its seems that threads are waiting for events forever and eventually use up the entire thread pool so server comes irresponsive.

I tried again to reproduce it, but there is too much mystery in here.

The query quoted in the description hangs in 'Sending data', which can be anything, no way to know without the stack trace and more information (complete process list etc.). It can even be a problem in the UDF function, e.g. something causing an endless loop (only in this case CPU usage should be noticeable).

But the attached processlist and stack trace are from a different occasion, a query which starts with SELECT SQL_SMALL_RESULT h.id, h.hashv, huffman_dist(h.hashv,. It stops not on 'Sending data', but on 'Copying to tmp table'). Unfortunately, there is no complete query anywhere, so we can't even know why it wants to create a temporary table in the first place. Also, the stack trace does not look like something hanging, again it looks more like an endless loop – without a few consequent stack traces, there is no way to tell for sure.

So, if the problem still exists, we need:

• at least a couple (better 3-4) stack traces taken with a short time interval;
• complete processlist;
• complete query which is being run;
• server error log.