Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18012

Supplied jemalloc results in immediate MariaDB Segfault on aarch64

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.0.37
    • 10.2.5
    • Compiling, Server
    • None
    • AWS aarch64, Amazon Linux AMI (EL7)

    Description

      mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      		 
      [Note] /usr/sbin/mysqld (mysqld 10.0.37-MariaDB) starting as process 30510 ...
      		 
      <jemalloc>: Error in munmap(): Invalid argument
      		 
      [ERROR] mysqld got signal 11 ;

      This is likely to do with aarch64 on EL7 defaulting, against all sanity, to 64KB pages. Unfortunately, it is what it is.

      AFAICT, it is directly related to this bug:
      https://bugzilla.redhat.com/show_bug.cgi?id=1545539
      Unfortunately, that bug long postdates EL7, so a better, working jemalloc will likely need to be supplied.

      Also:
      MariaDB supplied jemalloc is the same as EL7 aarch64 EPEL jemalloc (same version, broken in the same way).

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MDEV-11059 don't build the server with jemalloc

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            gbobic Gordan Bobic created issue -
            serg Sergei Golubchik made changes -
            Field Original Value New Value
            Description mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
            [Note] /usr/sbin/mysqld (mysqld 10.0.37-MariaDB) starting as process 30510 ...
            <jemalloc>: Error in munmap(): Invalid argument
            [ERROR] mysqld got signal 11 ;

            This is likely to do with aarch64 on EL7 defaulting, against all sanity, to 64KB pages. Unfortunately, it is what it is. :-(

            AFAICT, it is directly related to this bug:
            https://bugzilla.redhat.com/show_bug.cgi?id=1545539
            Unfortunately, that bug long postdates EL7, so a better, working jemalloc will likely need to be supplied.

            Also:
            MariaDB supplied jemalloc is the same as EL7 aarch64 EPEL jemalloc (same version, broken in the same way).             		{noformat}
            mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
            [Note] /usr/sbin/mysqld (mysqld 10.0.37-MariaDB) starting as process 30510 ...
            <jemalloc>: Error in munmap(): Invalid argument
            [ERROR] mysqld got signal 11 ;
            {noformat}
            This is likely to do with aarch64 on EL7 defaulting, against all sanity, to 64KB pages. Unfortunately, it is what it is. :-(

            AFAICT, it is directly related to this bug:
            https://bugzilla.redhat.com/show_bug.cgi?id=1545539
            Unfortunately, that bug long postdates EL7, so a better, working jemalloc will likely need to be supplied.

            Also:
            MariaDB supplied jemalloc is the same as EL7 aarch64 EPEL jemalloc (same version, broken in the same way).
            serg Sergei Golubchik added a comment -

            Starting from 10.2 we no longer link the server with jemalloc, so upgrading to 10.2 should fix it.

            serg Sergei Golubchik added a comment - Starting from 10.2 we no longer link the server with jemalloc, so upgrading to 10.2 should fix it.
            gbobic Gordan Bobic added a comment -

            That's all well and good but the other 1000 or so servers my client has in production are 10.0, so unless this can be fixed that rather blows aarch64 out of the water for us.
            Besides, 10.0 is still supported, is it not? Surely this means that 10.0 is broken for everyone on aarch64 RHEL/CentOS?

            gbobic Gordan Bobic added a comment - That's all well and good but the other 1000 or so servers my client has in production are 10.0, so unless this can be fixed that rather blows aarch64 out of the water for us. Besides, 10.0 is still supported, is it not? Surely this means that 10.0 is broken for everyone on aarch64 RHEL/CentOS?
            danblack Daniel Black added a comment -

            What so insane about a 64K default? Would seem to me to make better use of TLB on a process that naturally uses lots of memory on hardware that has a lot of memory.

            Does installing the EPEL jemalloc work compatibly?

            10.0 supported to March 2019 https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-100/

            danblack Daniel Black added a comment - What so insane about a 64K default? Would seem to me to make better use of TLB on a process that naturally uses lots of memory on hardware that has a lot of memory. Does installing the EPEL jemalloc work compatibly? 10.0 supported to March 2019 https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-100/
            gbobic Gordan Bobic added a comment -

            EPEL jemalloc is identical to the rpm bundled with MariaDB, in fact I think the MariaDB supplied package is a copy from the EPEL repository, not a MariaDB build.

            As for 64KB default page size, this link contains some of the most notable words from Linus Torvalds on the subject:
            https://yarchive.net/comp/linux/page_sizes.html

            TL;DR: if you need large pages, use huge pages. 64KB default pages just end up increasing memory wasteage in most cases.

            gbobic Gordan Bobic added a comment - EPEL jemalloc is identical to the rpm bundled with MariaDB, in fact I think the MariaDB supplied package is a copy from the EPEL repository, not a MariaDB build. As for 64KB default page size, this link contains some of the most notable words from Linus Torvalds on the subject: https://yarchive.net/comp/linux/page_sizes.html TL;DR: if you need large pages, use huge pages. 64KB default pages just end up increasing memory wasteage in most cases.
            serg Sergei Golubchik added a comment -

            Just as a side note, 10.0 is still suported, but only for about 3 more months: https://mariadb.org/about/maintenance-policy/
            And with quarterly releases there can be only one new 10.0 release before the EOL.
            Unless there will be a decision to extend 10.0 life span.

            serg Sergei Golubchik added a comment - Just as a side note, 10.0 is still suported, but only for about 3 more months: https://mariadb.org/about/maintenance-policy/ And with quarterly releases there can be only one new 10.0 release before the EOL. Unless there will be a decision to extend 10.0 life span.
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            Component/s Compiling [ 11001 ]
            Fix Version/s 10.2.5 [ 22117 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            gbobic Gordan Bobic added a comment - - edited

            Resolution says the fix is in 10.2.5. What about fixes for the still supported 10.0 and 10.1 branches, particularly since this bug was filed against 10.0?

            gbobic Gordan Bobic added a comment - - edited Resolution says the fix is in 10.2.5. What about fixes for the still supported 10.0 and 10.1 branches, particularly since this bug was filed against 10.0?
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 91249 ] MariaDB v4 [ 155365 ]

            People

              Unassigned Unassigned
              gbobic Gordan Bobic
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.