Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11035

Restore removed disallow-writes for Galera

Details

    Attachments

      1. sGAaCvfk.txt
        27 kB
        Tobias Genberg
      2. db-logs_2018-03-29.txt
        47 kB
        Tobias Genberg
      3. db-logs_2018-03-29_donor.txt
        15 kB
        Tobias Genberg

      Issue Links

        Activity

          fraggeln Tobias Genberg added a comment - - edited

          After provoking a full sst sync, using rsync as transport. (rm -rf /var/lib/mysql/*) Im not able to sync again.

          The sync fails with this in the logs:
          Mar 29 14:04:59 db3.fraggelberget.nu sh[30134]: 2018-03-29 14:04:57 139896963414144 [Note] InnoDB: Starting final batch to recover 404 pages from redo log.
          Mar 29 14:04:59 db3.fraggelberget.nu sh[30134]: 2018-03-29 14:04:57 139896963414144 [ERROR] [FATAL] InnoDB: Trying to read page number 722386 in space 56113, space name dynmap/Tiles, which is outside the tablespace bounds. Byte offset 0, len 16384
          Mar 29 14:04:59 db3.fraggelberget.nu sh[30134]: 180329 14:04:57 [ERROR] mysqld got signal 6 ;

          the Tiles-tablespace was the first time around 25G, and the 2nd time around 15G

          innodb_use_native_aio=false in the config or innodb_use_native_aio=true does change anything.

          check table returns OK on donor-node

          full log hopefully attached

          fraggeln Tobias Genberg added a comment - - edited After provoking a full sst sync, using rsync as transport. (rm -rf /var/lib/mysql/*) Im not able to sync again. The sync fails with this in the logs: Mar 29 14:04:59 db3.fraggelberget.nu sh [30134] : 2018-03-29 14:04:57 139896963414144 [Note] InnoDB: Starting final batch to recover 404 pages from redo log. Mar 29 14:04:59 db3.fraggelberget.nu sh [30134] : 2018-03-29 14:04:57 139896963414144 [ERROR] [FATAL] InnoDB: Trying to read page number 722386 in space 56113, space name dynmap/Tiles, which is outside the tablespace bounds. Byte offset 0, len 16384 Mar 29 14:04:59 db3.fraggelberget.nu sh [30134] : 180329 14:04:57 [ERROR] mysqld got signal 6 ; the Tiles-tablespace was the first time around 25G, and the 2nd time around 15G innodb_use_native_aio=false in the config or innodb_use_native_aio=true does change anything. check table returns OK on donor-node full log hopefully attached

          unable to reproduce when switching transport to mariabackup.

          fraggeln Tobias Genberg added a comment - unable to reproduce when switching transport to mariabackup.
          seppo Seppo Jaakola added a comment -

          Tobias, if you still have error logs from rsync SST failure, please attach longer history before the failure. I need to see how rsync joiner activities were carried out. So anything starting from SST request sending up to to the node failure would be needed.

          Please attach also error logs from donor node as well, from matching time window.

          seppo Seppo Jaakola added a comment - Tobias, if you still have error logs from rsync SST failure, please attach longer history before the failure. I need to see how rsync joiner activities were carried out. So anything starting from SST request sending up to to the node failure would be needed. Please attach also error logs from donor node as well, from matching time window.

          After Looking very closly to the logs again.
          I Think this might be related to a timeout-event happening in Systemd during startup.

          I have not been able to recreate this, since my db's are now in sync again, when I used mariabackup as transport.
          But I have attached full logs from the node that fails and the donor from one point.

          I hope it helps

          // T

          fraggeln Tobias Genberg added a comment - After Looking very closly to the logs again. I Think this might be related to a timeout-event happening in Systemd during startup. I have not been able to recreate this, since my db's are now in sync again, when I used mariabackup as transport. But I have attached full logs from the node that fails and the donor from one point. I hope it helps // T

          MDEV-14705 extends systemd timeouts on startup and shutdown. Could it solve this? Or would some other similar work be needed? I am not familiar with the details of the snapshot transfer procedure.

          marko Marko Mäkelä added a comment - MDEV-14705 extends systemd timeouts on startup and shutdown. Could it solve this? Or would some other similar work be needed? I am not familiar with the details of the snapshot transfer procedure.

          People

            jplindst Jan Lindström (Inactive)
            jplindst Jan Lindström (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.