[MDEV-14938] make buildbot to include galera into bintars Created: 2018-01-12  Updated: 2022-01-10  Resolved: 2022-01-10

Status: Closed
Project: MariaDB Server
Component/s: Packaging
Fix Version/s: 10.2.42, 10.3.33, 10.4.23, 10.5.14, 10.6.6

Type: Task Priority: Major
Reporter: Sergei Golubchik Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: beginner-friendly


 Description   

Currently bintars that buildbot produces don't contain libgalera_smm.so.

Publishing scripts repackage bintars to add it.

We want to produce correct packages in buildbot and release exactly what buildbot builds.

Either cpack should pick up libgalera_smm.so and add it to bintar or buildbot needs to repackage bintars itself.



 Comments   
Comment by Vladislav Vaintroub [ 2018-06-22 ]

I do not think cmake should be involved, iin packaging something we do not produce during the make. Since it is something very special, I'd prefer the workarounds to be done on buildbot side, and since dbart already has the repackaging scripts, reassigning to him.

Comment by Sergei Golubchik [ 2018-06-22 ]

On the other hand, cmake does repackage rpms in cpack_rpm.cmake. It'd be fairly consistent to do the same for galera. And it'd allow anyone to build a proper tarball without additional post-processing.

Comment by Vladislav Vaintroub [ 2018-06-22 ]

will take another look.

Comment by Sergei Golubchik [ 2018-06-22 ]

sorry, I didn't reload the page and didn't see you took it

Comment by Sergei Golubchik [ 2020-12-18 ]

No, see my comment above

Comment by Sergei Golubchik [ 2021-01-06 ]

dbart, I'm looking at the prep script. It seems there are two reasons to repack bintars:

  • move galera files to usr/bin/ and usr/lib/
  • rename the directory where tarball unpacks into to match the tarball name (like, to have glibc214 etc)

Is that all? I can try to make cpack to do both, with, say, -DBINTAR_NAME=... (or, perhaps, -DBINTAR_SUFFIX=... to leave the variable part out) and, say, -DGALERA_PATH=.... Would that work?

Comment by Daniel Bartholomew [ 2021-01-14 ]

serg Yes, that's all my prep script is doing. If you could do that in cpack that would be perfect.

Comment by Alexey Bychko (Inactive) [ 2021-03-18 ]

do we need to do the same for ES?

Comment by Sergei Golubchik [ 2021-03-18 ]

Let's do it in CS first, then it'll reach ES on the next merge/rebase

Comment by Alexey Bychko (Inactive) [ 2021-10-20 ]

agreed with dbart to do that at cmake level similar to --compat package logic

Comment by Sergei Golubchik [ 2021-10-20 ]

No, please, don't do thus compat auto-magic. It should be explicit, like cmake -DPACK_GALERA=/path/to/galera

Comment by Alexey Bychko (Inactive) [ 2021-10-21 ]

added cmake module to include built galera.
added EXT_GALERA_PATH cmake variable to use the path to external galera.

to use it:
1 - run cmake and make in galera directory. do not run make install
2 - run cmake -DEXT_GALERA_PATH=<path_to_compiled_galera> ... from server
3 - run make -jNCPU package to create server package with galera inside

Comment by Alexey Bychko (Inactive) [ 2021-10-22 ]

the branch is https://github.com/MariaDB/server/tree/bb-10.2-MDEV-14938

Comment by Alexey Bychko (Inactive) [ 2021-11-15 ]

hello Daniel!

could you please take a look at my branch?

to test:

  • clone galera-3, build it (just run cmake and make), do not install
  • clone server, checkout bb-10.2-MDEV-14938
  • run cmake and point it to built galera with -DEXT_GALERA_PATH=<path_to_built_galera>
  • run make package to create binary tarball with galera included
Comment by Sergei Golubchik [ 2021-11-15 ]

Note this email

Comment by Alexey Bychko (Inactive) [ 2021-11-16 ]

replied, answered questions

Comment by Alexey Bychko (Inactive) [ 2021-11-16 ]

stuck a bit with reworking the patch according to email.
stuck point is symlink galera/<galera_library> -> ../<libgalera_smm.so>
serg could you please help? do I need to use already implemented INSTALL_SYMLINK?

Comment by Alexey Bychko (Inactive) [ 2021-11-17 ]

found the root cause.
cmake/cpack is ignoring files which are created in-place and not specified with INSTALL(... DESTINATION ...).
so, the only way is to create libgalera_smm.so -> galera/<galera_library> somewhere inside source directory and then INSTALL() it.
a bit tricky, but it works

Comment by Alexey Bychko (Inactive) [ 2021-11-17 ]

please review 57c68068a771e338ccb456da55e8974fd38fa07d

changes:

  • only garbd and galera library are packed
  • created symlink for galera under lib
  • changed the logic to use either galera files, galera tree, unpacked galera tarball
  • changed bintar name to include glibc_<version>
Comment by Sergei Golubchik [ 2022-01-07 ]

What a prep script does (beyond many other things):
1. rename bintar, e.g.

mv -vi *-linux-x86_64.tar.gz ${version}-linux-glibc_214-x86_64.tar.gz

2. unpack, rename dir

tar -zxf ${version}-linux-glibc_214-${galera_arch}.tar.gz
mv -vi *-linux-${galera_arch} ${version}-linux-glibc_214-${galera_arch}

3. put galera in, repack, e.g.

rsync -av ${dir_galera}/galera-${ver_galera_preferred_real}/bintar/galera-${ver_galera_preferred_real}-glibc_214-${galera_arch}/usr/bin/ ${version}-linux-glibc_214-${galera_arch}/bin/
rsync -av ${dir_galera}/galera-${ver_galera_preferred_real}/bintar/galera-${ver_galera_preferred_real}-glibc_214-${galera_arch}/usr/lib/ ${version}-linux-glibc_214-${galera_arch}/lib/
tar -czf ${version}-linux-glibc_214-${galera_arch}.tar.gz ${version}-linux-glibc_214-${galera_arch}

Generated at Thu Feb 08 08:17:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.