Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25090

Install libpmem-dev in the CI environments

Details

    Description

      Since MariaDB Server 10.5.1, a build configuration is supported but not enabled in our packages:

      cmake -DWITH_PMEM=ON
      

      This will depend on libpmem-dev and introduce a small library dependency to the generated server package.

      I think that we should enable Persistent Memory support in MariaDB 10.6 on all applicable operating systems. Having the InnoDB redo log in a mount -o dax device should vastly reduce the transaction commit latency. This task would be a prerequisite for that.

      Attachments

        Issue Links

          Activity

            marko Marko Mäkelä created issue -

            marko Which version is required? There is libpmem1-dev and libpmem2-dev available. See "binaries" at https://tracker.debian.org/pkg/pmdk

            otto Otto Kekäläinen added a comment - marko Which version is required? There is libpmem1-dev and libpmem2-dev available. See "binaries" at https://tracker.debian.org/pkg/pmdk
            otto Otto Kekäläinen added a comment - - edited

            Work-in-progress PR available at https://github.com/MariaDB/server/pull/1773/commits (see second commit).

            Old buildbot: http://buildbot.askmonty.org/buildbot/grid?branch=ok-10.6-uring-pmem&category=main&category=experimental

            Currently testing the latter commit on old buildbot does not work as it fails on `Unmet build dependencies: libpmem-dev` until dbart has added package libpmem-dev to builders that run the 10.6 branch.

            otto Otto Kekäläinen added a comment - - edited Work-in-progress PR available at https://github.com/MariaDB/server/pull/1773/commits (see second commit). Old buildbot: http://buildbot.askmonty.org/buildbot/grid?branch=ok-10.6-uring-pmem&category=main&category=experimental Currently testing the latter commit on old buildbot does not work as it fails on `Unmet build dependencies: libpmem-dev` until dbart has added package libpmem-dev to builders that run the 10.6 branch.

            otto, thank you , I was not aware that libpmem2-dev exists. A quick look suggests that libpmem2 is an overkill for us; it seems to provide more high-level operations. We actually only use a tiny subset of what libpmem1 provides.

            It might be acceptable to just invoke the IA-32/AMD64 clflush instruction directly, and in MDEV-14425 (hopefully 10.7) I might just do that. (There are also clflushopt and clwb in more recent processors, but for our limited ‘write-only’ use case of redo log blocks, any of these instructions should be fine.) Obviously, if we decide to go down this route later, we should invoke similar instructions on all supported processor architectures.)

            marko Marko Mäkelä added a comment - otto , thank you , I was not aware that libpmem2-dev exists. A quick look suggests that libpmem2 is an overkill for us; it seems to provide more high-level operations. We actually only use a tiny subset of what libpmem1 provides. It might be acceptable to just invoke the IA-32/AMD64 clflush instruction directly, and in MDEV-14425 (hopefully 10.7) I might just do that. (There are also clflushopt and clwb in more recent processors, but for our limited ‘write-only’ use case of redo log blocks, any of these instructions should be fine.) Obviously, if we decide to go down this route later, we should invoke similar instructions on all supported processor architectures.)

            Note that libpmem-dev is only available on amd64, arm64 and ppc64el (not x86): https://packages.debian.org/search?searchon=names&keywords=libpmem-dev

            otto Otto Kekäläinen added a comment - Note that libpmem-dev is only available on amd64, arm64 and ppc64el (not x86): https://packages.debian.org/search?searchon=names&keywords=libpmem-dev
            axel Axel Schwenke made changes -
            Field Original Value New Value

            libpmem-dev is also not available on Ubuntu Xenial, but that is going away next month so I don't think I will even bother making any releases for that

            dbart Daniel Bartholomew added a comment - libpmem-dev is also not available on Ubuntu Xenial, but that is going away next month so I don't think I will even bother making any releases for that
            dbart Daniel Bartholomew made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            libpmem-dev is now installed on:

            • buster-amd64-build
            • focal-amd64-build
            • groovy-amd64-build
            • sid-amd64-build

            libpmem-devel is now installed on:

            • fedora32-amd64-build
            • fedora33-amd64-build
            • opensuse150-amd64-build
            • sles150-amd64-build
            dbart Daniel Bartholomew added a comment - libpmem-dev is now installed on: buster-amd64-build focal-amd64-build groovy-amd64-build sid-amd64-build libpmem-devel is now installed on: fedora32-amd64-build fedora33-amd64-build opensuse150-amd64-build sles150-amd64-build
            otto Otto Kekäläinen added a comment - - edited

            Thanks dbart!

            Also the following builders would need the libpmem-dev:

            Package libpmem-dev is available on platforms amd64, arm64 and ppc64el.

            On Bionic the apt-madison tool when run by autobake-deb.sh sees libpmem-dev in the repositories so it should be installable (from bionic-updates, which is apparently enabled).

            otto Otto Kekäläinen added a comment - - edited Thanks dbart ! Also the following builders would need the libpmem-dev: kvm-deb-bionic-amd64 kvm-deb-focal-ppc64le kvm-deb-groovy-aarch64 kvm-deb-groovy-ppc64le Package libpmem-dev is available on platforms amd64, arm64 and ppc64el. On Bionic the apt-madison tool when run by autobake-deb.sh sees libpmem-dev in the repositories so it should be installable (from bionic-updates, which is apparently enabled).

            We have a build failure on the following platforms due to missing libpmem-dev:

            I confirmed that the following are using libpmem:

            • kvm-deb-sid-amd64
            • kvm-rpm-fedora32-amd64
            • kvm-rpm-fedora32-amd64

            I also checked that kvm-deb-sid-x86 works, but without the library.

            marko Marko Mäkelä added a comment - We have a build failure on the following platforms due to missing libpmem-dev : kvm-deb-bionic-amd64 amd64-ubuntu-1804-deb-autobake (the ARMv8 and POWER platforms that otto mentioned did not start the build yet) I confirmed that the following are using libpmem : kvm-deb-sid-amd64 kvm-rpm-fedora32-amd64 kvm-rpm-fedora32-amd64 I also checked that kvm-deb-sid-x86 works, but without the library.

            Yes, I'm starting with amd64, moving to other architectures today. Bionic-amd64 is done now.

            dbart Daniel Bartholomew added a comment - Yes, I'm starting with amd64, moving to other architectures today. Bionic-amd64 is done now.

            libpmem-dev/devel is now installed on the following:

            • kvm-deb-groovy-aarch64
            • kvm-deb-groovy-ppc64le
            • kvm-deb-focal-aarch64
            • kvm-deb-focal-ppc64le
            • kvm-rpm-centos73-amd64
            • kvm-rpm-centos74-amd64
            • kvm-rpm-centos75-amd64
            • kvm-rpm-centos8-amd64
            • kvm-rpm-rhel8-amd64
            • kvm-zyp-sles123-amd64

            libpmem-dev/devel is not available on:

            • kvm-deb-bionic-ppc64le
            • kvm-deb-bionic-aarch64
            • kvm-rpm-centos73-ppc64le
            • kvm-rpm-centos73-ppc64
            • kvm-rpm-centos74-aarch64
            • kvm-deb-stretch-ppc64le
            • kvm-deb-stretch-aarch64
            • kvm-deb-xenial-aarch64
            • kvm-deb-xenial-amd64
            • kvm-deb-xenial-ppc64le
            • kvm-deb-xenial-x86

            We don't have a valid subscription for rhel8-ppc64le, so this builder does not
            have libpmem-devel:

            • kvm-rpm-rhel8-ppc64le
            dbart Daniel Bartholomew added a comment - libpmem-dev/devel is now installed on the following: kvm-deb-groovy-aarch64 kvm-deb-groovy-ppc64le kvm-deb-focal-aarch64 kvm-deb-focal-ppc64le kvm-rpm-centos73-amd64 kvm-rpm-centos74-amd64 kvm-rpm-centos75-amd64 kvm-rpm-centos8-amd64 kvm-rpm-rhel8-amd64 kvm-zyp-sles123-amd64 libpmem-dev/devel is not available on: kvm-deb-bionic-ppc64le kvm-deb-bionic-aarch64 kvm-rpm-centos73-ppc64le kvm-rpm-centos73-ppc64 kvm-rpm-centos74-aarch64 kvm-deb-stretch-ppc64le kvm-deb-stretch-aarch64 kvm-deb-xenial-aarch64 kvm-deb-xenial-amd64 kvm-deb-xenial-ppc64le kvm-deb-xenial-x86 We don't have a valid subscription for rhel8-ppc64le, so this builder does not have libpmem-devel: kvm-rpm-rhel8-ppc64le

            If I've missed a builder, let me know. Thanks.

            dbart Daniel Bartholomew added a comment - If I've missed a builder, let me know. Thanks.
            marko Marko Mäkelä added a comment - - edited

            The Debian packaging change (setting WITH_PMEM=yes) that was contributed by otto as well as a change to replace WITH_PMEM:BOOL=OFF with WITH_PMEM:STRING=auto have now been pushed to 10.6 after a (mostly) successful bb-10.6-release staging run.

            marko Marko Mäkelä added a comment - - edited The Debian packaging change (setting WITH_PMEM=yes ) that was contributed by otto as well as a change to replace WITH_PMEM:BOOL=OFF with WITH_PMEM:STRING=auto have now been pushed to 10.6 after a (mostly) successful bb-10.6-release staging run.
            marko Marko Mäkelä made changes -
            Component/s Storage Engine - InnoDB [ 10129 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s N/A [ 14700 ]
            Labels performance
            Priority Major [ 3 ] Blocker [ 1 ]

            Installed libpmem onto the rhel8-install VM as rhel8 is our bintar builder for 10.6 and so the install VM needs the lib to be able to run tests.

            dbart Daniel Bartholomew added a comment - Installed libpmem onto the rhel8-install VM as rhel8 is our bintar builder for 10.6 and so the install VM needs the lib to be able to run tests.

            marko probably we need to do the same for ES?

            abychko Alexey Bychko (Inactive) added a comment - marko probably we need to do the same for ES?

            abychko Yes. We'll need libpmem-dev/devel installed on the ES builders.

            dbart Daniel Bartholomew added a comment - abychko Yes. We'll need libpmem-dev/devel installed on the ES builders.
            marko Marko Mäkelä made changes -

            Assigning to abychko to install libpmem-dev/devel into the Jenkins builders. After which this issue can be closed.

            dbart Daniel Bartholomew added a comment - Assigning to abychko to install libpmem-dev/devel into the Jenkins builders. After which this issue can be closed.
            dbart Daniel Bartholomew made changes -
            Assignee Daniel Bartholomew [ dbart ] Alexey Bychko [ abychko ]
            dbart Daniel Bartholomew made changes -
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s 10.6 [ 24028 ]
            abychko Alexey Bychko (Inactive) added a comment - libpmem-dev doesn't exist for debian-9 https://packages.debian.org/search?searchon=names&keywords=+libpmem-dev
            abychko Alexey Bychko (Inactive) added a comment - libpmem-dev is available from bionic and later https://packages.ubuntu.com/search?keywords=+libpmem-dev&searchon=names

            added libpmem dev package to debian-10-buster, rhel-7, rhel-8, sles-12, sles-15, ubuntu-1804-lts, ubuntu-2004-lts.
            testing now

            abychko Alexey Bychko (Inactive) added a comment - added libpmem dev package to debian-10-buster, rhel-7, rhel-8, sles-12, sles-15, ubuntu-1804-lts, ubuntu-2004-lts. testing now
            abychko Alexey Bychko (Inactive) made changes -
            issue.field.resolutiondate 2021-03-22 06:55:36.0 2021-03-22 06:55:36.925
            abychko Alexey Bychko (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 119946 ] MariaDB v4 [ 134406 ]

            People

              abychko Alexey Bychko (Inactive)
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.