This is a difficult one. It is a limitation of vtable. When group_concat_max_len is high then the group_concat gets turned into a TEXT equivalent data type which then no longer works for MariaDB's sorting algorithm (no possible length will be high enough). I don't know exactly what setting value would trigger this but I suspect >= 64KB.

This problem will go away with 1.4 / 10.4. As a workaround a subquery select can be used, something like:

select * from (select variant_id,group_concat(geno) from (select b.variant_id,num_sample_id,geno from qc_parentage_variant a,dairy.dairy_qc_geno b where a.variant_id=b.variant_id and b

.num_sample_id in (90571,113548) ) a group by variant_id order by variant_id limit 100000000) as sq;

It isn't something we would be able to fix before 1.4 / 10.4.

Closed as "Won't fix" because we cannot fix this without heavy modifications to MariaDB Server, there are workarounds and the limitation will go away with 1.4.