[MDEV-28419] subsequent runs of debian/autobake-deb.sh are not idempotent Created: 2022-04-26  Updated: 2024-01-22

Status: Stalled
Project: MariaDB Server
Component/s: Packaging
Fix Version/s: 10.5, 10.6

Type: Task Priority: Minor
Reporter: Daniel Bartholomew Assignee: Vicențiu Ciorbaru
Resolution: Unresolved Votes: 0
Labels: None


 Description   

If you run the debian/autobake-deb.sh multiple times on the same source directory it will fail with the error:

dh: error: debian/control has a duplicate entry for mariadb-plugin-columnstore

This happens because the columnstore section of the autobake-deb.sh script has a simple `cat` statement to add the mariadb-plugin-columnstore package config and it doesn't check if the config has already been added to debian/control, it just blindly adds an additional package config block to the end of debian/control every time it is run.

This won't ever show up in buildbot because we're always running debian/autobake-deb.sh on a fresh directory. But it becomes a pain when trying to debug package builds locally because you have to remember to remove the columnstore package config from the end of debian/control prior to re-running the script.

Running the script should be idempotent, meaning back-to-back runs should have the same result assuming no changes have been made to the source tree. The way to fix would be to either:

  1. Add a check to the columnstore section of the script to not run the cat command unless the mariadb-plugin-columnstore package config is missing from debian/control
    ...or...
  2. Add the mariadb-plugin-columnstore package config to debian/control so that it doesn't have to be added at compile time

 Comments   
Comment by Daniel Black [ 2022-04-26 ]

The idempotent is provided by + '[' '!' -f debian/mariadb-plugin-columnstore.install ']'. Is that not being picked up? I guess that git checkout . is still leaving the file around. I'll look at doing something better still.

root@4405d53ce3e1:/source# ls -la debian/mariadb-plugin-columnstore.install
 
ls: cannot access 'debian/mariadb-plugin-columnstore.install': No such file or directory
 
root@4405d53ce3e1:/source#  AUTOBAKE_PREP_CONTROL_RULES_ONLY=1 bash -x debian/autobake-deb.sh 
+ set -e
 
+ export 'DEB_BUILD_OPTIONS=nocheck '
 
+ DEB_BUILD_OPTIONS='nocheck '
 
+ source ./VERSION
 
++ MYSQL_VERSION_MAJOR=10
 
++ MYSQL_VERSION_MINOR=5
 
++ MYSQL_VERSION_PATCH=16
 
++ SERVER_MATURITY=stable
 
+ [[ -n '' ]]
 
+ '[' -d storage/columnstore/columnstore/debian ']'
 
+ sed /-DPLUGIN_COLUMNSTORE=NO/d -i debian/rules
 
+ '[' '!' -f debian/mariadb-plugin-columnstore.install ']'
 
+ cp -v storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.install storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.postinst storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.postrm storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.prerm storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.triggers debian/
 
'storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.install' -> 'debian/mariadb-plugin-columnstore.install'
 
'storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.postinst' -> 'debian/mariadb-plugin-columnstore.postinst'
 
'storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.postrm' -> 'debian/mariadb-plugin-columnstore.postrm'
 
'storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.prerm' -> 'debian/mariadb-plugin-columnstore.prerm'
 
'storage/columnstore/columnstore/debian/mariadb-plugin-columnstore.triggers' -> 'debian/mariadb-plugin-columnstore.triggers'
 
+ echo
 
+ cat storage/columnstore/columnstore/debian/control
 
++ dpkg-architecture -q DEB_BUILD_ARCH
 
+ architecture=amd64
 
++ lsb_release -sc
 
+ CODENAME=sid
 
+ case "${CODENAME}" in
 
+ '[' -n 1 ']'
 
+ exit 0


diff --git a/debian/control b/debian/control
 
index 5352e9902ed..560cd409611 100644
 
--- a/debian/control
 
+++ b/debian/control
 
@@ -861,3 +861,24 @@ Description: MariaDB database regression test suite - data files
 
  ease of use.
 
  .
 
  This package has the architecture independent data files for the test suite.
 
+
 
+Package: mariadb-plugin-columnstore
 
+Architecture: amd64
 
+Depends: binutils,
 
+         libjemalloc1 | libjemalloc2,
 
+         libsnappy1 | libsnappy1v5,
 
+         mariadb-server-10.5 (= ${server:Version}),
 
+         net-tools,
 
+         python3,
 
+         ${misc:Depends},
 
+         ${shlibs:Depends}
 
+Breaks: libthrift-dev,
 
+        mariadb-columnstore-libs,
 
+        mariadb-columnstore-platform
 
+Replaces: libthrift-dev,
 
+          mariadb-columnstore-libs,
 
+          mariadb-columnstore-platform
 
+Description: MariaDB ColumnStore storage engine
 
+ The MariaDB ColumnStore storage engine is a high-performance columnar
 
+ analytical engine, aimed at rapid processing of analytical queries on very
 
+ large amounts of data.
 
diff --git a/debian/rules b/debian/rules
 
index 57636eb028a..713970bb9d7 100755
 
--- a/debian/rules
 
+++ b/debian/rules
 
@@ -45,7 +45,6 @@ endif
 
 
 # ColumnStore can build only on amd64 and arm64
 
 ifneq (,$(filter $(DEB_HOST_ARCH_CPU),amd64 arm64))
 
-    CMAKEFLAGS += -DPLUGIN_COLUMNSTORE=NO
 
 endif
 
 
 ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
 
@@ -100,7 +99,6 @@ endif
 
            -DPLUGIN_TOKUDB=NO \
 
            -DPLUGIN_CASSANDRA=NO \
 
            -DPLUGIN_AWS_KEY_MANAGEMENT=NO \
 
-                       -DPLUGIN_COLUMNSTORE=NO \
 
            -DDEB=$(DEB_VENDOR)
 
 
 # This is needed, otherwise 'make test' will run before binaries have been built


root@4405d53ce3e1:/source#  AUTOBAKE_PREP_CONTROL_RULES_ONLY=1 bash -x debian/autobake-deb.sh 
+ set -e
 
+ export 'DEB_BUILD_OPTIONS=nocheck '
 
+ DEB_BUILD_OPTIONS='nocheck '
 
+ source ./VERSION
 
++ MYSQL_VERSION_MAJOR=10
 
++ MYSQL_VERSION_MINOR=5
 
++ MYSQL_VERSION_PATCH=16
 
++ SERVER_MATURITY=stable
 
+ [[ -n '' ]]
 
+ '[' -d storage/columnstore/columnstore/debian ']'
 
+ sed /-DPLUGIN_COLUMNSTORE=NO/d -i debian/rules
 
+ '[' '!' -f debian/mariadb-plugin-columnstore.install ']'
 
++ dpkg-architecture -q DEB_BUILD_ARCH
 
+ architecture=amd64
 
++ lsb_release -sc
 
+ CODENAME=sid
 
+ case "${CODENAME}" in
 
+ '[' -n 1 ']'
 
+ exit 0


~/repos/mariadb-server-10.5 10.5
 
$ git diff
 
diff --git a/debian/control b/debian/control
 
index 5352e9902ed..560cd409611 100644
 
--- a/debian/control
 
+++ b/debian/control
 
@@ -861,3 +861,24 @@ Description: MariaDB database regression test suite - data files
 
  ease of use.
 
  .
 
  This package has the architecture independent data files for the test suite.
 
+
 
+Package: mariadb-plugin-columnstore
 
+Architecture: amd64
 
+Depends: binutils,
 
+         libjemalloc1 | libjemalloc2,
 
+         libsnappy1 | libsnappy1v5,
 
+         mariadb-server-10.5 (= ${server:Version}),
 
+         net-tools,
 
+         python3,
 
+         ${misc:Depends},
 
+         ${shlibs:Depends}
 
+Breaks: libthrift-dev,
 
+        mariadb-columnstore-libs,
 
+        mariadb-columnstore-platform
 
+Replaces: libthrift-dev,
 
+          mariadb-columnstore-libs,
 
+          mariadb-columnstore-platform
 
+Description: MariaDB ColumnStore storage engine
 
+ The MariaDB ColumnStore storage engine is a high-performance columnar
 
+ analytical engine, aimed at rapid processing of analytical queries on very
 
+ large amounts of data.
 
diff --git a/debian/rules b/debian/rules
 
index 57636eb028a..713970bb9d7 100755
 
--- a/debian/rules
 
+++ b/debian/rules
 
@@ -45,7 +45,6 @@ endif
 
 
 # ColumnStore can build only on amd64 and arm64
 
 ifneq (,$(filter $(DEB_HOST_ARCH_CPU),amd64 arm64))
 
-    CMAKEFLAGS += -DPLUGIN_COLUMNSTORE=NO
 
 endif
 
 
 ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
 
@@ -100,7 +99,6 @@ endif
 
            -DPLUGIN_TOKUDB=NO \
 
            -DPLUGIN_CASSANDRA=NO \
 
            -DPLUGIN_AWS_KEY_MANAGEMENT=NO \
 
-                       -DPLUGIN_COLUMNSTORE=NO \
 
            -DDEB=$(DEB_VENDOR)
 
 
 # This is needed, otherwise 'make test' will run before binaries have been built

Comment by Daniel Black [ 2022-04-26 ]

Acceptable? https://github.com/MariaDB/server/pull/2104 there for review

Comment by Sergei Golubchik [ 2022-04-27 ]

I don't understand how you use "git checkout . is still leaving the file around" to explain that "subsequent runs of debian/autobake-deb.sh are not idempotent".

It seems that your PR#2104 has nothing to do with autobake-deb.sh not being idempotent

Comment by Daniel Black [ 2022-04-27 ]

Wrong git command - git clean -dfx will remove debian/mariadb-plugin-columnstore.* files as they aren't checked but leave debian/control with the added package. I'm assuming this was dbart's patten. So if that was used between consecutive debian/autobake-deb.sh then multiple Package: mariadb-plugin-columnstore in debian/control instances will result.

So the PR purges an existing Package: mariadb-plugin-columnstore in debian/control before pulling the definition from the submodule and also unconditionally copies the other columnstore files. I saw this useful in case the submodule was updated between subsequent runs.

Generated at Thu Feb 08 10:00:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.