Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23648

s3.partition_move 'innodb' test failure - object deleted before copy attempted

    XMLWordPrintable

    Details

      Description

      worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
      s3.partition_move 'innodb'               [ fail ]
              Test ended at 2020-09-02 14:11:20
       
      CURRENT_TEST: s3.partition_move
      mysqltest: At line 61: query 'alter table archive exchange partition p1 with table p1' failed: 29: Expected object 's3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/data/000001' didn't exist
      

      fake s3 server

      04:11:19.772 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?delimiter=%2F&prefix=test%2F  127.0.0.1         271µs       ↑ 73 B ↓ 513 B
      04:11:19.775 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?delimiter=%2F&prefix=performance_schema%2F  127.0.0.1         158µs       ↑ 73 B ↓ 527 B
      04:11:19.777 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?delimiter=%2F&prefix=test%2F  127.0.0.1         209µs       ↑ 73 B ↓ 513 B
      04:11:19.778 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?delimiter=%2F&prefix=mtr%2F  127.0.0.1         165µs       ↑ 73 B ↓ 512 B
      04:11:19.810 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p0/frm 127.0.0.1         315µs       ↑ 73 B ↓ 225 B
      04:11:19.812 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p0/frm 127.0.0.1         213µs       ↑ 73 B ↓ 225 B
      04:11:19.820 [404 Not Found] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/seq_1_to_99/frm 127.0.0.1         374µs       ↑ 73 B ↓ 664 B
      04:11:19.823 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         267µs       ↑ 73 B ↓ 225 B
      04:11:19.827 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/aria 127.0.0.1         325µs       ↑ 73 B ↓ 225 B
      04:11:19.839 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/aria 127.0.0.1         11.79ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.845 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/index/000001 127.0.0.1         4.269ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.850 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/data/000001 127.0.0.1         4.199ms      ↑ 16 KiB ↓ 259 B
      04:11:19.852 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         262µs       ↑ 73 B ↓ 225 B
      04:11:19.857 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         3.79ms       ↑ 1.1 KiB ↓ 259 B
      04:11:19.858 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/par 127.0.0.1         299µs       ↑ 73 B ↓ 225 B
      04:11:19.862 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/par 127.0.0.1         3.565ms      ↑ 114 B ↓ 259 B
      04:11:19.866 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         825µs       ↑ 73 B ↓ 1.4 KiB
      04:11:19.868 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/aria 127.0.0.1         946µs       ↑ 73 B ↓ 8.3 KiB
      04:11:19.871 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/t1/frm 127.0.0.1         369µs       ↑ 73 B ↓ 225 B
      04:11:19.873 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/t1/frm 127.0.0.1         277µs       ↑ 73 B ↓ 225 B
      04:11:19.877 [404 Not Found] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/seq_100_to_399/frm 127.0.0.1         413µs       ↑ 73 B ↓ 670 B
      04:11:19.881 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/frm 127.0.0.1         300µs       ↑ 73 B ↓ 225 B
      04:11:19.883 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/frm 127.0.0.1         297µs       ↑ 73 B ↓ 225 B
      04:11:19.893 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         290µs       ↑ 73 B ↓ 225 B
      04:11:19.898 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/frm 127.0.0.1         4.21ms       ↑ 1.0 KiB ↓ 259 B
      04:11:19.903 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         4.384ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.908 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/index/000001 127.0.0.1         4.031ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.913 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/data/000001 127.0.0.1         3.358ms      ↑ 16 KiB ↓ 259 B
      04:11:19.915 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         748µs       ↑ 73 B ↓ 1.4 KiB
      04:11:19.918 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/aria 127.0.0.1         345µs       ↑ 73 B ↓ 225 B
      04:11:19.922 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/aria 127.0.0.1         3.906ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.927 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/data/000001 127.0.0.1         4.077ms      ↑ 8.1 KiB ↓ 259 B
      04:11:19.931 [200 OK] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         950µs       ↑ 73 B ↓ 355 B
      04:11:19.932 [204 No Content] s3.DeleteObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         603µs       ↑ 73 B ↓ 226 B
      04:11:19.937 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         4.089ms      ↑ 1.1 KiB ↓ 259 B
      04:11:19.939 [200 OK] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/par 127.0.0.1         735µs       ↑ 73 B ↓ 353 B
      04:11:19.940 [204 No Content] s3.DeleteObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/par 127.0.0.1         545µs       ↑ 73 B ↓ 226 B
      04:11:19.944 [200 OK] s3.PutObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/par 127.0.0.1         3.34ms       ↑ 122 B ↓ 259 B
      04:11:19.947 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         912µs       ↑ 73 B ↓ 1.4 KiB
      04:11:19.949 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p0/aria 127.0.0.1         826µs       ↑ 73 B ↓ 8.3 KiB
      04:11:19.951 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive/frm 127.0.0.1         809µs       ↑ 73 B ↓ 1.4 KiB
      04:11:19.953 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/aria 127.0.0.1         783µs       ↑ 73 B ↓ 8.3 KiB
      04:11:19.955 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/frm 127.0.0.1         764µs       ↑ 73 B ↓ 1.3 KiB
      04:11:19.957 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         978µs       ↑ 73 B ↓ 8.3 KiB
      04:11:19.958 [200 OK] s3.GetObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/data/000001 127.0.0.1         746µs       ↑ 73 B ↓ 16 KiB
      04:11:19.961 [200 OK] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         692µs       ↑ 73 B ↓ 355 B
      04:11:19.962 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2Fp1%2Findex  127.0.0.1         663µs       ↑ 73 B ↓ 921 B
      04:11:19.969 [200 OK] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/#sql-exchange-1bbf43-4/index/000001 127.0.0.1         6.156ms      ↑ 104 B ↓ 523 B
      04:11:19.971 [204 No Content] s3.DeleteObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/index/000001 127.0.0.1         665µs       ↑ 73 B ↓ 226 B
      04:11:19.972 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2Fp1%2Fdata  127.0.0.1         626µs       ↑ 73 B ↓ 920 B
      04:11:19.977 [200 OK] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/#sql-exchange-1bbf43-4/data/000001 127.0.0.1         4.723ms      ↑ 104 B ↓ 523 B
      04:11:19.979 [204 No Content] s3.DeleteObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/data/000001 127.0.0.1         566µs       ↑ 73 B ↓ 226 B
      04:11:19.984 [200 OK] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/#sql-exchange-1bbf43-4/aria 127.0.0.1         4.198ms      ↑ 104 B ↓ 523 B
      04:11:19.985 [204 No Content] s3.DeleteObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         438µs       ↑ 73 B ↓ 226 B
      04:11:19.987 [200 OK] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/archive#P#p1/aria 127.0.0.1         692µs       ↑ 73 B ↓ 355 B
      04:11:19.989 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2Farchive%23P%23p1%2Findex  127.0.0.1         440µs       ↑ 73 B ↓ 566 B
      04:11:19.990 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2Farchive%23P%23p1%2Fdata  127.0.0.1         613µs       ↑ 73 B ↓ 939 B
      04:11:19.991 [404 Not Found] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/data/000001 127.0.0.1         495µs       ↑ 104 B ↓ 662 B
      04:11:19.992 [404 Not Found] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         451µs       ↑ 104 B ↓ 648 B
      04:11:19.994 [404 Not Found] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         223µs       ↑ 73 B ↓ 225 B
      04:11:19.997 [200 OK] s3.HeadObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/#sql-exchange-1bbf43-4/aria 127.0.0.1         965µs       ↑ 73 B ↓ 355 B
      04:11:19.999 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2F%23sql-exchange-1bbf43-4%2Findex  127.0.0.1         852µs       ↑ 73 B ↓ 961 B
      04:11:20.000 [404 Not Found] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/index/000001 127.0.0.1         735µs       ↑ 104 B ↓ 664 B
      04:11:20.002 [200 OK] s3.ListObjectsV1 127.0.0.1/justdoit/?prefix=s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c%2F%23sql-exchange-1bbf43-4%2Fdata  127.0.0.1         582µs       ↑ 73 B ↓ 960 B
      04:11:20.003 [404 Not Found] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/data/000001 127.0.0.1         496µs       ↑ 104 B ↓ 662 B
      04:11:20.004 [404 Not Found] s3.CopyObject 127.0.0.1/justdoit/s3_test_5ac0aae8ecd211ea8032cabbbeb0fc5c/p1/aria 127.0.0.1         508µs       ↑ 104 B ↓ 648 B
      

      The server logs show that the missing object was deleted by s3 (04:11:19.979) before it tried to access it later.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              monty Michael Widenius
              Reporter:
              danblack Daniel Black
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration