Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28946

STOP SLAVE or slave errors (ex 1062, 1032) constantly crash MariaDB server

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.6.1, 10.5.15
    • N/A
    • Compiling, Replication
    • None
    • Ubuntu 21.10 / Ubuntu 22.04

    Description

      I've set-up replication between a galera cluster connecting to one node, and each time I stop the slave the slave server crashes immediately.

      I upgraded to 22.04 and replication stopped working but the crash still occured. I reverted back to 21.10, still getting the crash but replication is still not working so trying to play with settings slave position, etc is just a pain while the server is constantly crashing on me.

      I've setup replication back in january and the problem was already there, running 10.4 I believe at the time.

      Any help would be greatly appreciated.

      2022-06-25 9:55:09 35 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.012160', position 19131057; GTID position 300-3-522464027,100-1-102978519
      2022-06-25 9:55:09 35 [Note] master was xxxxx:3306
      220625 9:55:09 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 10.5.15-MariaDB-0ubuntu0.21.10.1-log
      key_buffer_size=268435456
      read_buffer_size=131072
      max_used_connections=2
      max_threads=5002
      thread_count=1
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11273904 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      2022-06-25 9:55:09 0 [Note] InnoDB: FTS optimize thread exiting.
      2022-06-25 9:55:09 0 [Note] InnoDB: Buffer pool(s) load aborted due to user instigated abort at 220625 9:55:09
      2022-06-25 9:55:09 0 [Note] InnoDB: Dumping of buffer pool not started as load was incomplete
      2022-06-25 9:55:09 0 [Note] InnoDB: Starting shutdown...
      2022-06-25 9:55:09 0 [Note] InnoDB: Dumping of buffer pool not started as load was incomplete
      ??:0(my_print_stacktrace)[0x55f980682170]
      ??:0(handle_fatal_signal)[0x55f980207268]
      ??:0(__sigaction)[0x7faeb69a6520]
      ??:0(pthread_kill)[0x7faeb69fa828]
      ??:0(raise)[0x7faeb69a6476]
      ??:0(abort)[0x7faeb698c7b7]
      ??:0(wsrep_write_dummy_event_low(THD*, char const*))[0x55f97fe6754a]
      ??:0(wsrep::wsrep_provider_v26::status() const)[0x55f98076065f]
      ??:0(_Unwind_GetTextRelBase)[0x7faeb69595f4]
      ??:0(_Unwind_ForcedUnwind)[0x7faeb6959ce2]
      ??:0(sem_trywait)[0x7faeb6a012a6]
      ??:0(pthread_exit)[0x7faeb69f9afa]
      ??:0(handle_slave_sql)[0x55f97ff0b9b1]
      ??:0(aria_get_capabilities)[0x55f98040cc23]
      ??:0(pthread_condattr_setpshared)[0x7faeb69f8947]
      ??:0(clone)[0x7faeb6a88a44]

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25633 MariaDB crashes when compiled with link time optimizations

          • Major - Major loss of function.
          • Closed
          links to

          Web Link lp 1979695 (10.6.7), same backtrace

          Activity

            Ascending order - Click to sort in descending order
            View 11 older comments
            danblack Daniel Black added a comment - - edited

            Glad to hear you successful upgrade.

            ref: maintenance policy, so yes 10.5 is 5 years.

            21.10 is EOL at the end of this month so 10.5.16 was the last release for it on ubuntu impish by us. We don't do releases less that what's supported on the distro so there won't be a 22.04 10.5 release as 10.6 is packaged. We will be doing 10.5 20.04 focal packages until the 10.5 eol date if you want to stay on that branch longer.

            Please do report the 10.6 upgrade issues on galera/replication. There are meant to be supported, and documentation has fallen behind MDEV-28483.

            Some constraints on Galera SST upgrades are in MDEV-27437, so if it was SST related and not rsync this may have been one problem along with aio being default on.

            danblack Daniel Black added a comment - - edited Glad to hear you successful upgrade. ref: maintenance policy , so yes 10.5 is 5 years. 21.10 is EOL at the end of this month so 10.5.16 was the last release for it on ubuntu impish by us. We don't do releases less that what's supported on the distro so there won't be a 22.04 10.5 release as 10.6 is packaged. We will be doing 10.5 20.04 focal packages until the 10.5 eol date if you want to stay on that branch longer. Please do report the 10.6 upgrade issues on galera/replication. There are meant to be supported, and documentation has fallen behind MDEV-28483 . Some constraints on Galera SST upgrades are in MDEV-27437 , so if it was SST related and not rsync this may have been one problem along with aio being default on.
            ccounotte COUNOTTE CEDRIC added a comment -

            Thanks for the heads-up.

            I'm using mariabackup as SST method and replication boot-strap method (is there any other for replication?), primarily because servers are live and cannot be taken down. And this method is said not possible for an upgrade unless IST is used, which in turn requires enough gcache.

            I already tried migrating the slave server and it would report differences in mysql DB tables and would not start anymore!? I had to reinstall the server entirely with old OS and re-apply replication using a new mariabackup backup, which takes hours for our 180GB DB.

            I just tried to upgrade to Ubuntu 22.04 while keeping MariaDB 10.5.16, but it doesn't seem possible.

            When it tells me some repo has been disabled I renabled the MariaDB repo you gave me, and proceed, but it ends up with this:

            Error during update

            A problem occurred during the update. This is usually some sort of
            network problem, please check your network connection and retry.

            W:Updating from such a repository can't be done securely, and is
            therefore disabled by default., W:See apt-secure(8) manpage for
            repository creation and user configuration details., E:The repository
            'https://mirror.mva-n.net/mariadb/repo/10.5/ubuntu jammy Release'
            does not have a Release file.

            Anyway to keep 10.5 on Ubuntu 22.04 ?

            ccounotte COUNOTTE CEDRIC added a comment - Thanks for the heads-up. I'm using mariabackup as SST method and replication boot-strap method (is there any other for replication?), primarily because servers are live and cannot be taken down. And this method is said not possible for an upgrade unless IST is used, which in turn requires enough gcache. I already tried migrating the slave server and it would report differences in mysql DB tables and would not start anymore!? I had to reinstall the server entirely with old OS and re-apply replication using a new mariabackup backup, which takes hours for our 180GB DB. I just tried to upgrade to Ubuntu 22.04 while keeping MariaDB 10.5.16, but it doesn't seem possible. When it tells me some repo has been disabled I renabled the MariaDB repo you gave me, and proceed, but it ends up with this: Error during update A problem occurred during the update. This is usually some sort of network problem, please check your network connection and retry. W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., E:The repository 'https://mirror.mva-n.net/mariadb/repo/10.5/ubuntu jammy Release' does not have a Release file. Anyway to keep 10.5 on Ubuntu 22.04 ?
            danblack Daniel Black added a comment -

            I typoed 22.04 focal for 20.04 focal sorry.

            > Anyway to keep 10.5 on Ubuntu 22.04 ?

            Options (though not great); building your own packages, or using a tarball from our download page

            danblack Daniel Black added a comment - I typoed 22.04 focal for 20.04 focal sorry. > Anyway to keep 10.5 on Ubuntu 22.04 ? Options (though not great); building your own packages, or using a tarball from our download page
            ccounotte COUNOTTE CEDRIC added a comment -

            Thanks for fixing the typo, I'm still unsure which is the safest option between reverting our servers to 20.04 or to first upgrade MariaDB to 10.6, then upgrade Ubuntu to 22.04. Reverting seems the safeest option avoiding major version upgrade of MariaDB, but reinstalling 6 servers seems a little overwhelming.

            ccounotte COUNOTTE CEDRIC added a comment - Thanks for fixing the typo, I'm still unsure which is the safest option between reverting our servers to 20.04 or to first upgrade MariaDB to 10.6, then upgrade Ubuntu to 22.04. Reverting seems the safeest option avoiding major version upgrade of MariaDB, but reinstalling 6 servers seems a little overwhelming.
            danblack Daniel Black added a comment - - edited

            Alternative:

            Create a systemd service for a mariadb:10.5 container.

            A slave server should still start on 10.6, even with reporting table differences. Table differences should be resolved with mariadb-upgrade. If possible, I'd like to see a log in a new bug report of it not starting.

            danblack Daniel Black added a comment - - edited Alternative: Create a systemd service for a mariadb:10.5 container. A slave server should still start on 10.6, even with reporting table differences. Table differences should be resolved with mariadb-upgrade . If possible, I'd like to see a log in a new bug report of it not starting.

            People

              danblack Daniel Black
              ccounotte COUNOTTE CEDRIC
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.