Fixed in 5.7.1, documented as:
The server could exit if built to permit a maximum number of indexes per table larger than 64.

In the course of fixing this problem, a -DMAX_INDEXES=N CMake option was added to permit building the server to support a larger maximum number of indexes per table. The default is 64. The maximum is 255. Values smaller than 64 are ignored and the default of 64 is used. (Bug #11761614)

http://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-1.html

When I backported this to 5.6, there are 2 commits needed to make it work. I cannot remember the other commit but its by the same author and the same bug number.

gadnium Yep, added.

psergey Can you please review this?

The patches have been pushed to the following branch :
https://github.com/MariaDB/server/compare/bb-10.1-mdev9021

Monty's response :

> On Fri, Oct 30, 2015 at 2:13 PM, Nirbhay Choubey <nirbhay@mariadb.com> wrote:
>>
>> Hi Monty,
>>
>> I am working on a customer issue MDEV-9021 which is about merging fix for Bug#54127, that I
>> fixed back in Oracle. In that patch, I had added a restriction of MAX_INDEXES <= 255. I do not
>> seem to recollect the rationale behind using 255 as upper limit though.
>
> So, the restriction comes from the fact that in index file header only 8 bits are allocated
> to store the number of keys.
> https://github.com/MariaDB/server/blob/10.1/include/my_compare.h#L27,L37

The other reason to keep a reasonable limit is that there are some
arrays that are allocated in structures and on stack with size
MAX_KEY. Increasing MAX_KEY will
cause more memory usage, even if the user doesn't need that.

The biggest structures depending on MAX_KEY is in tokudb. There is a
risk that if you make MAX_KEY to big, then TokuDB may cause
stack-overflow crashes.

>>
>> In MariaDB code, however, in handler.h, I see the following preprocessor check :
>>
>> #if MAX_KEY > 128
>> #error MAX_KEY is too large. Values up to 128 are supported.
>> #endif
>>
>> Do you know why 128 was chosen as the maximum limit? It is purely to restrict
>> memory usage (practical limitation) or its something else?

Memory and stack usage.

>> Also, in MDEV-9021 should I leave the limit to 128 or move it to 255 (like in mysql-5.7)?

In practice, having more than 128 keys will make any tables REALLY slow.
I would assume that in most cases when this is needed,it's because the
application is wrongly designed.

It would be nice to know the reason/customer that got MySQL to change
the limit to 255. So far I have never met a user that needs that many
keys.
(There are a few users that have needed 1000+ columns to support some
odd designed applications, however adding keys on several of these
would make the thing impossible to use).

I think that 128 is better, except if we have some common application
or important customer that needs this.

Regards,
Monty

128 is fine for us.

We actually only need about 68.

But nevertheless, its over the current limit of 64 so doubling it to 128 will work.

gadnium ok

Expanded nirbhay_c 's diff until it passes the tests: https://gist.github.com/spetrunia/53b07648a7bd7bf61800

index_nr= MAX_KEY means "hash join is used". test_if_skip_sort_order used it in a context where it should have used "no key", for which it normally uses "-1".

(gdb) p sizeof(key_map)

  $1 = 48

(gdb) p sizeof(ulonglong)

  $2 = 8

So, 64-bit bitmap takes 8 bytes, 128-bit bitmap would take 16 bytes, 256 bit bitmap would take 32 bytes, and that
1. would still be less than current size of 48 bytes.
2. would not have a risk of not calling Bitmap::~Bitmap somewhere, and leaking memory

On the other hand, solution that uses Bitmap<n> is already coded, while we would still need to code the in-place bitmaps.

Patch : https://github.com/MariaDB/server/commit/0a03bd1f043861be5bafa5b0cdff72995fac61d5

https://github.com/MariaDB/server/commit/7ec655850397a0edfcea8c1fd82650824297e564