Details

    Description

      Scope

      This task covers the changes required to make the amd64-windows and amd64-windows-packages compatible with
      winx64-debug and winx64-packages from https://buildbot.mariadb.net.

      Details

      Debug builder

      The builders configure step is the same between old and new:

      • cmake src -G "Visual Studio 17 2022" -A x64  -DPLUGIN_ROCKSDB=NO -DMYSQL_MAINTAINER_MODE=ERR -DWITH_DBUG_TRACE=OFF -Wno-dev
        

      • The build step matches.

        cmake --build  .  --verbose --config RelWithDebInfo -- -m
        

      • The tests run are invoked from buildbot_suites.bat, found under mysql-test/collections/
      • There is also a separate run for connect suite.

      Packages builder

      • The cmake line is shared between old and new builders:

        cmake . -G "Visual Studio 17 2022" -A x64  -DBUILD_CONFIG=mysql_release -DWITH_THIRD_PARTY=HeidiSQL -DWITH_EMBEDDED_SERVER=0 -DSIGNCODE=ON -DSIGNTOOL_PARAMETERS="/fd;SHA256;/a;/t;http://timestamp.globalsign.com/?signature=sha2" -DWITH_UNIT_TESTS=0 -DMYSQL_MAINTAINER_MODE=ERR
        

      • The package compile step "matches". The new buildbot assumes there is a CPackZIPConfig.cmake, which is true for all MariaDB supported versions.
        Zip packaging

        cmake --build . --config RelWithDebInfo --target win_package
        

        MSI packaging

        cmake --build  .  --config RelWithDebInfo --target MSI
        

      • Upload step: Packages are uploaded to ci.mariadb.org, can later be part of the package signing process.
      • MTR tests: are run the same, buildbot_suites.bat and connect suite.
      • There is currently no install test in the new buildbot. These should be covered.

      Requirements:

      1. Update windows builder to the latest supported Windows Server version.
      2. Update Visual studio and the compiler to the latest version currently available.
      3. Implement install tests for MSI packages.

      dojob ^"^(rename mariadb*.msi mariadb-64.msi ^&^& ^(start /b /wait msiexec /i mariadb-64.msi PORT=3364 INSTALLDIR=\^"C:\Program Files\MariaDB\^" SERVICENAME=MariaDB-64 /qn /l*v msi_install.txt^)^) ^|^| ^(echo ERROR: Installation failed ^&^& type msi_install.txt ^&^& exit 1^)^"
      

      Check the service is running:

      dojob ^"dir C:\\\^"Program Files\^"\MariaDB ^&^& ^(echo Check that the built and installed mysqld.exe are identical ^&^& ^(diff sql\RelWithDebInfo\mysqld.exe \^"C:\Program Files\MariaDB\bin\mysqld.exe\^" ^|^| ^(stat sql\RelWithDebInfo\mysqld.exe ^&^& stat \^"C:\Program Files\MariaDB\bin\mysqld.exe\^" ^&^& sql\RelWithDebInfo\mysqld.exe --version ^&^& C:\\\^"Program Files\^"\MariaDB\bin\mysqld.exe --version ^&^& exit 1^)^)^) ^&^& ^(echo Check that the server is up and running on the right port ^&^& C:\\\^"Program Files\^"\MariaDB\bin\mysql -uroot --port=3364 -e \^"select ^@^@version, ^@^@version_comment; show status like 'Uptime'; select 'Stat' t, variable_name name, variable_value val from information_schema.global_status where variable_name like '%%have%%' union select 'Vars' t, variable_name name, variable_value val from information_schema.global_variables where variable_name like '%%have%%' order by t, name\^"^)^"
      

      Uninstall the msi package:

      dojob ^"start /b /wait msiexec /i mariadb-64.msi REMOVE=ALL /qn /l*v msi_uninstall.txt ^|^| type msi_uninstall.txt ^&^& exit 1^"
      

      Attachments

        Activity

          serg The winx64-debug builder in old buildbot uses a weird way of fetching the source code. It uses

           git init src && cd src && git fetch --depth 1 https://github.com/MariaDB/server <hash> && git checkout FETCH_HEAD
          

          instead of unpacking a tarball like any other builder.

          Is this a requirement? I only see potential issues with this approach, and no clear benefits. For example the build is not reproducible if the commit hash no longer exists on GitHub. Why was it done like this?

          cvicentiu Vicențiu Ciorbaru added a comment - serg The winx64-debug builder in old buildbot uses a weird way of fetching the source code. It uses git init src && cd src && git fetch --depth 1 https: //github .com /MariaDB/server < hash > && git checkout FETCH_HEAD instead of unpacking a tarball like any other builder. Is this a requirement? I only see potential issues with this approach, and no clear benefits. For example the build is not reproducible if the commit hash no longer exists on GitHub. Why was it done like this?
          wlad Vladislav Vaintroub added a comment - - edited

          cvicentiu, it is done like this because people should be able to build from a github, not just from a"tarball", where you bake bison output and other stuff. Why would you want to build if commit hash no longer exists on github? You'd literally build something that no more there. If it is stored in local repo, a push will restore it back, otherwise what's the interest in commit which is lost forever

          wlad Vladislav Vaintroub added a comment - - edited cvicentiu , it is done like this because people should be able to build from a github, not just from a"tarball", where you bake bison output and other stuff. Why would you want to build if commit hash no longer exists on github? You'd literally build something that no more there. If it is stored in local repo, a push will restore it back, otherwise what's the interest in commit which is lost forever

          Also, -DWITH_SIGNCODE=ON should not be there anymore. Corporation did not purchase a certificate that allows to sign non-interactively, or something like this. I think certificate authorities do not do this anymore, or those certificates are limited to 100 usages or something like that, so this won't work if we sign every executable and shared library like we did in the good past, or the certificates that allow would be much more expensive. You'll find out once authenticode certificate for the foundation is there.

          wlad Vladislav Vaintroub added a comment - Also, -DWITH_SIGNCODE=ON should not be there anymore. Corporation did not purchase a certificate that allows to sign non-interactively, or something like this. I think certificate authorities do not do this anymore, or those certificates are limited to 100 usages or something like that, so this won't work if we sign every executable and shared library like we did in the good past, or the certificates that allow would be much more expensive. You'll find out once authenticode certificate for the foundation is there.
          wlad Vladislav Vaintroub added a comment - - edited

          Ok, if you want to exclude me from this conversation, as per agreement, I do not protest

          The intention was to help you , and to answer your question, but I do not really need all that.

          wlad Vladislav Vaintroub added a comment - - edited Ok, if you want to exclude me from this conversation, as per agreement, I do not protest The intention was to help you , and to answer your question, but I do not really need all that.
          cvicentiu Vicențiu Ciorbaru added a comment - - edited

          wlad I believe we used a self-signed certificate. Do we not want to keep this to be able to run install tests on MSIs? We'll get to figuring out the codesigning certificate part too after we've done: MDBF-885.

          cvicentiu Vicențiu Ciorbaru added a comment - - edited wlad I believe we used a self-signed certificate. Do we not want to keep this to be able to run install tests on MSIs? We'll get to figuring out the codesigning certificate part too after we've done: MDBF-885 .

          Hi wlad!
          First of all, thanks a lot for your input.
          Regarding this "I think certificate authorities do not do this anymore, or those certificates are limited to 100 usages"

          Is there any documentation you can point me to regarding this limitation?
          I am doing my homework, investigating how we can get a certificate from DigiCert to store on a HSM FIPS compliant service but I don't see any limitation.
          The only limitation is for their (DigiCert) KeyLocker Cloud Service to store the private key (1000 usages for one 1 year cert, 3000 for 3, but can be increased $)
          But that's not the case for us, as I am looking for a dedicated HSM service (probably AWS).

          rvarzaru Varzaru Razvan-Liviu added a comment - Hi wlad ! First of all, thanks a lot for your input. Regarding this "I think certificate authorities do not do this anymore, or those certificates are limited to 100 usages" Is there any documentation you can point me to regarding this limitation? I am doing my homework, investigating how we can get a certificate from DigiCert to store on a HSM FIPS compliant service but I don't see any limitation. The only limitation is for their (DigiCert) KeyLocker Cloud Service to store the private key (1000 usages for one 1 year cert, 3000 for 3, but can be increased $) But that's not the case for us, as I am looking for a dedicated HSM service (probably AWS).
          wlad Vladislav Vaintroub added a comment - - edited

          rvarzaru, it might be that cloud HSM is something usable, but Corp did not purchase that. We got a new certificate last year, and since there were "new requirements since June 1, 2023", something about safety of the private key of the Authenticode certificate, the old-way signing broke.

          Now, we have corporation employee, who is in Sophia, where our build servers are located. He does "manual" signing of MSIs now, prior to release. A physical access to machine where certificate is installed is required, the corresponding private key is in a dongle, and is password-protected. signtool brings up dialog window, to input the password. Only MSI-to-be-released is signed. But this is mostly hearsay, I have not dealt that new certificate myself.

          I remember there were some other option mentioned, before buying certifcate, those were better suited for automation, but still with restricted number of signs (which we with SIGNCODE=ON would exceed after a couple of builds, since every exe and DLL was signed), or cloud HSM but it was considered expensive, so now we got that cheapest "manual-only with physical access to machine" option.

          wlad Vladislav Vaintroub added a comment - - edited rvarzaru , it might be that cloud HSM is something usable, but Corp did not purchase that. We got a new certificate last year, and since there were "new requirements since June 1, 2023", something about safety of the private key of the Authenticode certificate, the old-way signing broke. Now, we have corporation employee, who is in Sophia, where our build servers are located. He does "manual" signing of MSIs now, prior to release. A physical access to machine where certificate is installed is required, the corresponding private key is in a dongle, and is password-protected. signtool brings up dialog window, to input the password. Only MSI-to-be-released is signed. But this is mostly hearsay, I have not dealt that new certificate myself. I remember there were some other option mentioned, before buying certifcate, those were better suited for automation, but still with restricted number of signs (which we with SIGNCODE=ON would exceed after a couple of builds, since every exe and DLL was signed), or cloud HSM but it was considered expensive, so now we got that cheapest "manual-only with physical access to machine" option.

          cvicentiu, I think there's a value in verifying that the server can be built from git repo. Old buildbot has few builders that do that, not only winx64-debug. I would agree that the old buildbot has too many builders that build from the git repo. Still, I'd suggest to keep this logic for winx64-debug and for at least one non-windows builder.

          serg Sergei Golubchik added a comment - cvicentiu , I think there's a value in verifying that the server can be built from git repo. Old buildbot has few builders that do that, not only winx64-debug. I would agree that the old buildbot has too many builders that build from the git repo. Still, I'd suggest to keep this logic for winx64-debug and for at least one non-windows builder.

          People

            rvarzaru Varzaru Razvan-Liviu
            cvicentiu Vicențiu Ciorbaru
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 6d
                6d
                Remaining:
                Remaining Estimate - 6d
                6d
                Logged:
                Time Spent - Not Specified
                Not Specified