Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31332

Galera SST rsync wants to replicate snapshot directory

Details

    Description

      A new Galera Cluster was installed with some hosts and the MariaDB-Datadir is on an NetApp storage with NFS access.
      The bootstrap of the first node worked fine but then at the second node its impossible to start the mariadb.service because of endless restart of the service and permissions denies of the .snapshot directory of the mounted share.

      rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log.00000001" (in rsync_sst) failed: Permission denied (13)
      May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log_control" (in rsync_sst) failed: Permission denied (13)
      May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/debian-10.5.flag" (in rsync_sst) failed: Permission denied (13)
      May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/galera.cache" (in rsync_sst) failed: Read-only file system (30)
      May 23 10:12:22 rsyncd[24965]: sent 48 bytes  received 374 bytes  total size 61
      May 23 10:12:22 rsyncd[24964]: sent 1815 bytes  received 3493527 bytes  total size 3486785
      May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/grastate.dat" (in rsync_sst) failed: Read-only file system (30)
      May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/gvwstate.dat" (in rsync_sst) failed: Read-only file system (30)
      May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_buffer_pool" (in rsync_sst) failed: Permission denied (13)
      May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_logfile0" (in rsync_sst) failed: Permission denied (13)

      Attachments

        Issue Links

          Activity

            dcz01 Daniel Czadek created issue -
            dcz01 Daniel Czadek made changes -
            Field Original Value New Value
            dcz01 Daniel Czadek made changes -
            Component/s Galera SST [ 10121 ]
            Component/s Replication [ 10100 ]
            Component/s wsrep [ 11500 ]
            dcz01 Daniel Czadek made changes -
            Labels galera
            dcz01 Daniel Czadek made changes -
            Description A new Galera Cluster was installed with some hosts and the MariaDB-Datadir is on an NetApp storage with NFS access.
            The bootstrap of the first node worked fine but then at the second node its impossible to start the mariadb.service because of endless restart of the service and permissions denies of the .snapshot directory of the mounted share.

             pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log.00000001" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log_control" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/debian-10.5.flag" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/galera.cache" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:22 pr-be-ic-db-02 rsyncd[24965]: sent 48 bytes  received 374 bytes  total size 61
            May 23 10:12:22 pr-be-ic-db-02 rsyncd[24964]: sent 1815 bytes  received 3493527 bytes  total size 3486785
            May 23 10:12:23 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/grastate.dat" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:23 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/gvwstate.dat" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:23 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_buffer_pool" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:23 pr-be-ic-db-02 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_logfile0" (in rsync_sst) failed: Permission denied (13)
            A new Galera Cluster was installed with some hosts and the MariaDB-Datadir is on an NetApp storage with NFS access.
            The bootstrap of the first node worked fine but then at the second node its impossible to start the mariadb.service because of endless restart of the service and permissions denies of the .snapshot directory of the mounted share.

            rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log.00000001" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/aria_log_control" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/debian-10.5.flag" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:22 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/galera.cache" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:22 rsyncd[24965]: sent 48 bytes  received 374 bytes  total size 61
            May 23 10:12:22 rsyncd[24964]: sent 1815 bytes  received 3493527 bytes  total size 3486785
            May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/grastate.dat" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/gvwstate.dat" (in rsync_sst) failed: Read-only file system (30)
            May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_buffer_pool" (in rsync_sst) failed: Permission denied (13)
            May 23 10:12:23 rsyncd[24966]: rsync: [receiver] open "/.snapshot/hourly.2023-05-23_1005/ib_logfile0" (in rsync_sst) failed: Permission denied (13)
            danblack Daniel Black added a comment - - edited

            Quick solution is to append `.snapshot` to the ignored filelist in the wsrep_sst_rsync script here.

            danblack Daniel Black added a comment - - edited Quick solution is to append `.snapshot` to the ignored filelist in the wsrep_sst_rsync script here .
            dcz01 Daniel Czadek made changes -
            Priority Blocker [ 1 ] Major [ 3 ]
            danblack Daniel Black made changes -
            Fix Version/s 10.5 [ 23123 ]
            danblack Daniel Black made changes -
            Assignee Julius Goryavsky [ sysprg ]
            danblack Daniel Black added a comment -

            sysprg can a .* be filtered out like: -f '- /.*'?

            danblack Daniel Black added a comment - sysprg can a .* be filtered out like: -f '- /.*' ?
            danblack Daniel Black added a comment -

            Can we just ignore the server ignore_db_dirs maybe?

            danblack Daniel Black added a comment - Can we just ignore the server ignore_db_dirs maybe?
            danblack Daniel Black made changes -
            Assignee Julius Goryavsky [ sysprg ] Daniel Black [ danblack ]
            danblack Daniel Black made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            danblack Daniel Black made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            danblack Daniel Black added a comment -

            Hi,

            PR for review with test case:

            https://github.com/MariaDB/server/pull/2702

            danblack Daniel Black added a comment - Hi, PR for review with test case: https://github.com/MariaDB/server/pull/2702
            danblack Daniel Black made changes -
            Assignee Daniel Black [ danblack ] Julius Goryavsky [ sysprg ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            sysprg Julius Goryavsky made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky added a comment - Fix merged as https://github.com/MariaDB/server/commit/9b5275b8f5343bf8f00c1a75fff3b2343afa89c2
            sysprg Julius Goryavsky made changes -
            Fix Version/s 10.4.32 [ 29300 ]
            Fix Version/s 10.5.23 [ 29012 ]
            Fix Version/s 10.6.16 [ 29014 ]
            Fix Version/s 10.10.7 [ 29018 ]
            Fix Version/s 10.11.6 [ 29020 ]
            Fix Version/s 11.0.4 [ 29021 ]
            Fix Version/s 11.1.3 [ 29023 ]
            Fix Version/s 11.2.2 [ 29035 ]
            Fix Version/s 11.3.0 [ 29302 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 11.2.2 [ 29035 ]
            Fix Version/s 11.3.0 [ 29302 ]
            danblack Daniel Black made changes -

            People

              sysprg Julius Goryavsky
              dcz01 Daniel Czadek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.