Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • N/A
    • N/A
    • Buildbot
    • None

    Description

      Apply the new changes to the MSAN builder as described in the updated MDEV-20377 and MDEV-26758.

      Attachments

        Issue Links

          Activity

            Either all libraries must be instrumented, or uninstrumented libraries must be unavailable for the MSAN build. An example of a rare dependency is this:

            CURRENT_TEST: plugins.two_password_validations
            mysqltest: At line 21: query 'grant select on *.* to foobar identified by 'q$%^&*R1234ty'' failed: <Unknown> (2013): Lost connection to server during query
            …
            Uninitialized bytes in __interceptor_strchr at offset 0 inside [0x7f54277492e0, 1)
            ==267325==WARNING: MemorySanitizer: use-of-uninitialized-value
                #0 0x7f54247d1f43  (/usr/lib/x86_64-linux-gnu/libcrack.so.2+0x3f43)
                #1 0x7f54247d2356 in FascistCheckUser (/usr/lib/x86_64-linux-gnu/libcrack.so.2+0x4356)
                #2 0x7f5424ff00c5 in crackme /buildbot/amd64-ubuntu-1804-msan/build/plugin/cracklib_password_check/cracklib_password_check.c:39:13
                #3 0x56518621b228 in do_validate(THD*, st_plugin_int**, void*) /buildbot/amd64-ubuntu-1804-msan/build/sql/sql_acl.cc:2178:10
            

            marko Marko Mäkelä added a comment - Either all libraries must be instrumented, or uninstrumented libraries must be unavailable for the MSAN build. An example of a rare dependency is this: CURRENT_TEST: plugins.two_password_validations mysqltest: At line 21: query 'grant select on *.* to foobar identified by 'q$%^&*R1234ty'' failed: <Unknown> (2013): Lost connection to server during query … Uninitialized bytes in __interceptor_strchr at offset 0 inside [0x7f54277492e0, 1) ==267325==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x7f54247d1f43 (/usr/lib/x86_64-linux-gnu/libcrack.so.2+0x3f43) #1 0x7f54247d2356 in FascistCheckUser (/usr/lib/x86_64-linux-gnu/libcrack.so.2+0x4356) #2 0x7f5424ff00c5 in crackme /buildbot/amd64-ubuntu-1804-msan/build/plugin/cracklib_password_check/cracklib_password_check.c:39:13 #3 0x56518621b228 in do_validate(THD*, st_plugin_int**, void*) /buildbot/amd64-ubuntu-1804-msan/build/sql/sql_acl.cc:2178:10

            Thank you, this is much better now. Some work remains to be done:

            • The ENCRYPT() function is broken in some tests.
            • some compression libraries need to be instrumented, or removed from the test run altogether
            • libcrack needs to be instrumented (as noted in my previous comment)
            • some tests are too slow or big for MSAN and should be tweaked somehow, possibly by extending the have_sanitizer variable and skipping tests on MSAN
            marko Marko Mäkelä added a comment - Thank you, this is much better now. Some work remains to be done: The ENCRYPT() function is broken in some tests. some compression libraries need to be instrumented, or removed from the test run altogether libcrack needs to be instrumented (as noted in my previous comment) some tests are too slow or big for MSAN and should be tweaked somehow, possibly by extending the have_sanitizer variable and skipping tests on MSAN

            The problems related to compression libraries has been filed as MDEV-26757. I had posted some commands to build one compression library there.

            Either all compression libraries should be built with MSAN instrumentation, or (my preference, because at least when it comes to InnoDB, I do not expect anything to be caught) the compression libraries should be made unavailable during MSAN builds.

            marko Marko Mäkelä added a comment - The problems related to compression libraries has been filed as MDEV-26757 . I had posted some commands to build one compression library there. Either all compression libraries should be built with MSAN instrumentation, or (my preference, because at least when it comes to InnoDB, I do not expect anything to be caught) the compression libraries should be made unavailable during MSAN builds.
            vladbogo Vlad Bogolin added a comment -

            I have disabled compression tests. Hopefully,

            --skip-test=.*compression.*
            

            is the way to do it and that doesn't miss anything. Now I am trying to see why instrumented libcrack doesn't find the dictionary.

            vladbogo Vlad Bogolin added a comment - I have disabled compression tests. Hopefully, --skip-test=.*compression.* is the way to do it and that doesn't miss anything. Now I am trying to see why instrumented libcrack doesn't find the dictionary.
            vladbogo Vlad Bogolin added a comment -

            All changes should now be added to the new MSAN builder https://buildbot.mariadb.org/#/builders/348.

            I managed to solve the libcrack dictionary problem (see MDBF-229 for details). The updated build-msan.sh script is attached.

            vladbogo Vlad Bogolin added a comment - All changes should now be added to the new MSAN builder https://buildbot.mariadb.org/#/builders/348 . I managed to solve the libcrack dictionary problem (see MDBF-229 for details). The updated build-msan.sh script is attached.
            vladbogo Vlad Bogolin added a comment -

            Updated to clang-12 as discussed on Slack. All changes should be live.

            vladbogo Vlad Bogolin added a comment - Updated to clang-12 as discussed on Slack. All changes should be live.

            After I worked around MDEV-27964 in the tests, the MSAN builds should now be green, for all main branches 10.5 to 10.9.

            marko Marko Mäkelä added a comment - After I worked around MDEV-27964 in the tests, the MSAN builds should now be green, for all main branches 10.5 to 10.9.

            People

              vladbogo Vlad Bogolin
              vladbogo Vlad Bogolin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 1d 1.5h
                  1d 1.5h