[MCOL-1242] Remote CpImport Created: 2018-03-05  Updated: 2023-10-26  Resolved: 2018-10-16

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: None
Fix Version/s: 1.2.0

Type: New Feature Priority: Major
Reporter: Jens Röwekamp (Inactive) Assignee: Zdravelina Sokolovska (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Attachments: File MariaDB ColumnStore Remote CpImport-1.2.0-1-x64.msi    
Issue Links:
Blocks
blocks MCOL-1740 mcsimport - depend on mcsapi only Closed
is blocked by MCOL-1281 mcsapi Windows support Closed
is blocked by MCOL-1720 ColumnStoreDateTime(dateTime, format)... Closed
Duplicate
duplicates MCOL-806 Create mcsapi CSV import tool Closed
PartOf
includes MCOL-1773 Add mcsimport to PATH Closed
Relates
relates to MCOL-1736 mcsimport - support for different wri... Closed
relates to MCOL-1735 mcsimport Mac OS support Closed
relates to MCOL-1741 Debian 9 - data-adapter package depen... Closed
Sprint: 2018-17, 2018-18, 2018-19

 Description   

More input needed

 Comments   
Comment by Sasha V [ 2018-03-06 ]

Could you please not extend exiting cpimport with this new functionality, but provide a separate new tool for that, say, mcsimport?

Comment by Jens Röwekamp (Inactive) [ 2018-03-06 ]

Yes, it is planned to be a separate tool

Comment by Jens Röwekamp (Inactive) [ 2018-09-21 ]

As discussed with project management Mac OS support and the support for different Columnstore (write) modes is scoped in additional tickets and not part of this ticket any more.

Comment by Jens Röwekamp (Inactive) [ 2018-09-25 ]

mcsimport for Columnstore 1.2 on Linux and Windows.

Test suite executed successfully on Windows 10, Debian 9, and CentOS 7 using mcsapi 1.2.0 [1] for the build, and ColumnStore 1.2.0 running on CentOS 7.

For QA:

  • review if included tests are sufficient or if there are any test cases missing
  • execute test suite (through ctest or the python script directly)

mcsimport needs to be build with mcsapi 1.2.0 and tested against ColumnStore 1.2.0.

[1] 95c70691eaf9b8ec7571ae3df3e30c97c5bc5573

Comment by Andrew Hutchings (Inactive) [ 2018-10-03 ]

jens.rowekamp can you please add something that will stop cpimport from being built on CentOS 6?

Comment by Jens Röwekamp (Inactive) [ 2018-10-03 ]

LinuxJedi Sure. I just documented the cmake flags in README.md.

Use:

cmake -DREMOTE_CPIMPORT=OFF .

to avoid building remote cpimport on CentOS 6.

Comment by Andrew Hutchings (Inactive) [ 2018-10-06 ]

Still breaks CentOS 6 due to requiring a C++11 compiler which CentOS doesn't have. Also still appears to break all the other builders.

Comment by Jens Röwekamp (Inactive) [ 2018-10-08 ]

Buildbot errors were mostly caused due to removed packages.sh. I restored a modified version of packages.sh that doesn't include the folders mcsimport, build, resources and cmake to the binary package.
Currently we distributed one binary package with all platform independent shell scripts. Therefore, I don't know if we want to include mcsimport into it or not.

The buildbot errors for CentOS 6 and Suse are caused by not using the -DREMOTE_CPIMPORT=OFF flag while invoking cmake.

On CentOS 6 we can't build remote cpimport as it requires mcsimport which depends on C++ 11. On Suse buildbot currently fails as buildbot doesn't build and install mcsapi prior building the tools.

I tested the package build on CentOS 6 with above mentioned -DREMOTE_CPIMPORT=OFF flag on a local VM and the rpm package was built successfully.

[jens@centos6 ~]$ cat /etc/centos-release
 
CentOS release 6.10 (Final)
 
[jens@centos6 ~]$ git clone git@github.com:mariadb-corporation/mariadb-columnstore-tools.git
 
Initialized empty Git repository in /home/jens/mariadb-columnstore-tools/.git/
 
remote: Enumerating objects: 251, done.
 
remote: Counting objects: 100% (251/251), done.
 
remote: Compressing objects: 100% (133/133), done.
 
remote: Total 476 (delta 113), reused 228 (delta 100), pack-reused 225
 
Receiving objects: 100% (476/476), 179.76 KiB, done.
 
Resolving deltas: 100% (229/229), done.
 
[jens@centos6 ~]$ cd mariadb-columnstore-tools/
 
[jens@centos6 mariadb-columnstore-tools]$ git checkout MCOL-1242
 
Branch MCOL-1242 set up to track remote branch MCOL-1242 from origin.
 
Switched to a new branch 'MCOL-1242'
 
[jens@centos6 mariadb-columnstore-tools]$ ls
 
backuprestore  cmake  CMakeLists.txt  COPYRIGHT.txt  LICENSE.txt  mcsimport  monitoring  README.md  resources  VERSION
 
[jens@centos6 mariadb-columnstore-tools]$ mkdir build && cd build
 
[jens@centos6 build]$ cmake .. -DREMOTE_CPIMPORT=OFF
 
-- The C compiler identification is GNU 4.4.7
 
-- The CXX compiler identification is GNU 4.4.7
 
-- Check for working C compiler: /usr/bin/cc
 
-- Check for working C compiler: /usr/bin/cc -- works
 
-- Detecting C compiler ABI info
 
-- Detecting C compiler ABI info - done
 
-- Check for working CXX compiler: /usr/bin/c++
 
-- Check for working CXX compiler: /usr/bin/c++ -- works
 
-- Detecting CXX compiler ABI info
 
-- Detecting CXX compiler ABI info - done
 
-- Running cmake version 2.8.12.2
 
-- MariaDB-Columnstore 1.2.0
 
-- -----------------------------------------------
 
-- CMAKE_INSTALL_PREFIX = /usr/local/mariadb/columnstore
 
-- CMAKE_BUILD_TYPE =
 
-- TEST_RUNNER = OFF
 
-- BACKUPRESTORE = ON
 
-- MONITORING = ON
 
-- REMOTE_CPIMPORT = OFF
 
-- RPM = OFF
 
-- DEB = OFF
 
-- Change a values with: cmake -D<Variable>=<Value>
 
-- ------------------------------------------------
 
--
 
-- Configuring done
 
-- Generating done
 
-- Build files have been written to: /home/jens/mariadb-columnstore-tools/build
 
[jens@centos6 build]$ make
 
[jens@centos6 build]$ cmake .. -DRPM=centos6
 
-- Running cmake version 2.8.12.2
 
-- MariaDB-Columnstore 1.2.0
 
-- -----------------------------------------------
 
-- CMAKE_INSTALL_PREFIX = /usr/local/mariadb/columnstore
 
-- CMAKE_BUILD_TYPE =
 
-- TEST_RUNNER = OFF
 
-- BACKUPRESTORE = ON
 
-- MONITORING = ON
 
-- REMOTE_CPIMPORT = OFF
 
-- RPM = centos6
 
-- DEB = OFF
 
-- Change a values with: cmake -D<Variable>=<Value>
 
-- ------------------------------------------------
 
--
 
-- Configuring done
 
-- Generating done
 
-- Build files have been written to: /home/jens/mariadb-columnstore-tools/build
 
[jens@centos6 build]$ sudo make package
 
[sudo] password for jens:
 
Run CPack packaging tool...
 
CPack: Create package using RPM
 
CPack: Install projects
 
CPack: - Run preinstall target for: Project
 
CPack: - Install project: Project
 
CPack: Create package
 
CPackRPM:Debug: rpmbuild version is <4.8.0>
 
CPackRPM:Debug: LSB_RELEASE  = lsb_release not installed/found!
 
CPackRPM:Debug: processing URL
 
CPackRPM:Debug: using CPACK_RPM_PACKAGE_URL
 
CPackRPM:Debug: User defined Url:
 
 http://www.mariadb.com
 
CPackRPM:Debug: processing REQUIRES
 
CPackRPM:Debug: using CPACK_RPM_PACKAGE_REQUIRES
 
CPackRPM:Debug: User defined Requires:
 
  libxml2 rsync
 
CPackRPM:Debug: processing SUGGESTS
 
CPackRPM:Debug: processing PROVIDES
 
CPackRPM:Debug: using CPACK_RPM_PACKAGE_PROVIDES
 
CPackRPM:Debug: User defined Provides:
 
  mariadb-columnstore-tools
 
CPackRPM:Debug: processing OBSOLETES
 
CPackRPM:Debug: processing PREFIX
 
CPackRPM:Debug: processing CONFLICTS
 
CPackRPM:Debug: processing AUTOPROV
 
CPackRPM:Debug: processing AUTOREQ
 
CPackRPM:Debug: processing AUTOREQPROV
 
CPackRPM:Debug: User defined more define spec line specified:
 
 
 
        %define ignore #
 
 
 
CPackRPM:Debug: Initial list of path to OMIT in RPM:
 
CPackRPM:Debug: CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST= /etc;/etc/init.d;/usr;/usr/share;/usr/share/doc;/usr/bin;/usr/lib;/usr/lib64;/usr/include
 
CPackRPM:Debug: Final list of path to OMIT in RPM: -o;-path;./etc;-o;-path;./etc/init.d;-o;-path;./usr;-o;-path;./usr/share;-o;-path;./usr/share/doc;-o;-path;./usr/bin;-o;-path;./usr/lib;-o;-path;./usr/lib64;-o;-path;./usr/include
 
CPackRPM:Debug: Handling User Filelist: </usr/local/mariadb/columnstore/tools/COPYRIGHT.txt;/usr/local/mariadb/columnstore/tools/LICENSE.txt;/usr/local/mariadb/columnstore/tools/README.md;/usr/local/mariadb/columnstore/tools/VERSION;/usr/local/mariadb/columnstore/tools/backuprestore/columnstoreBackup;/usr/local/mariadb/columnstore/tools/backuprestore/columnstoreRestore;/usr/local/mariadb/columnstore/tools/backuprestore/README.md;/usr/local/mariadb/columnstore/tools/monitoring/check_mariadbcs;/usr/local/mariadb/columnstore/tools/monitoring/README.md;%ignore /usr;%ignore /usr/local>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/COPYRIGHT.txt>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/LICENSE.txt>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/README.md>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/VERSION>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/backuprestore/columnstoreBackup>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/backuprestore/columnstoreRestore>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/backuprestore/README.md>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/monitoring/check_mariadbcs>
 
CPackRPM:Debug: F_PREFIX=<>, F_PATH=</usr/local/mariadb/columnstore/tools/monitoring/README.md>
 
CPackRPM:Debug: F_PREFIX=<%ignore>, F_PATH=</usr>
 
CPackRPM:Debug: F_PREFIX=<%ignore>, F_PATH=</usr/local>
 
CPackRPM:Debug: CPACK_TOPLEVEL_DIRECTORY          = /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM
 
CPackRPM:Debug: CPACK_TOPLEVEL_TAG                = Linux
 
CPackRPM:Debug: CPACK_TEMPORARY_DIRECTORY         = /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
CPackRPM:Debug: CPACK_OUTPUT_FILE_NAME            = mariadb-columnstore-tools-1.2.0-1-amd64-centos6.rpm
 
CPackRPM:Debug: CPACK_OUTPUT_FILE_PATH            = /home/jens/mariadb-columnstore-tools/build/mariadb-columnstore-tools-1.2.0-1-amd64-centos6.rpm
 
CPackRPM:Debug: CPACK_PACKAGE_FILE_NAME           = mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
CPackRPM:Debug: CPACK_RPM_BINARY_SPECFILE         = /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/SPECS/mariadb-columnstore-tools.spec
 
CPackRPM:Debug: CPACK_PACKAGE_INSTALL_DIRECTORY   = mariadb-columnstore-tools 1.2.0
 
CPackRPM:Debug: CPACK_TEMPORARY_PACKAGE_FILE_NAME = /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6.rpm
 
CPackRPM: Will use GENERATED spec file: /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/SPECS/mariadb-columnstore-tools.spec
 
CPackRPM:Debug: You may consult rpmbuild logs in:
 
CPackRPM:Debug:    - /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/rpmbuild.err
 
CPackRPM:Debug: *** + umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ LANG=C
 
+ export LANG
 
+ unset DISPLAY
 
+ mv /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/tmpBBroot
 
+ exit 0
 
+ umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ '[' /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 '!=' / ']'
 
+ rm -rf /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
++ dirname /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ mkdir -p /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM
 
+ mkdir /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ LANG=C
 
+ export LANG
 
+ unset DISPLAY
 
+ '[' -e /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 ']'
 
+ rm -rf /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ mv /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/tmpBBroot /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ /usr/lib/rpm/check-buildroot
 
+ /usr/lib/rpm/redhat/brp-compress
 
+ /usr/lib/rpm/redhat/brp-strip /usr/bin/strip
 
+ /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
 
+ /usr/lib/rpm/redhat/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump
 
+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python
 
+ /usr/lib/rpm/redhat/brp-python-hardlink
 
+ /usr/lib/rpm/redhat/brp-java-repack-jars
 
warning: Could not canonicalize hostname: centos6
 
+ umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ exit 0
 
 ***
 
CPackRPM:Debug:    - /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/rpmbuild.out
 
CPackRPM:Debug: *** + umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ LANG=C
 
+ export LANG
 
+ unset DISPLAY
 
+ mv /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/tmpBBroot
 
+ exit 0
 
+ umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ '[' /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 '!=' / ']'
 
+ rm -rf /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
++ dirname /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ mkdir -p /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM
 
+ mkdir /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ LANG=C
 
+ export LANG
 
+ unset DISPLAY
 
+ '[' -e /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6 ']'
 
+ rm -rf /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ mv /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/tmpBBroot /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/mariadb-columnstore-tools-1.2.0-1-amd64-centos6
 
+ /usr/lib/rpm/check-buildroot
 
+ /usr/lib/rpm/redhat/brp-compress
 
+ /usr/lib/rpm/redhat/brp-strip /usr/bin/strip
 
+ /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
 
+ /usr/lib/rpm/redhat/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump
 
+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python
 
+ /usr/lib/rpm/redhat/brp-python-hardlink
 
+ /usr/lib/rpm/redhat/brp-java-repack-jars
 
warning: Could not canonicalize hostname: centos6
 
+ umask 022
 
+ cd /home/jens/mariadb-columnstore-tools/build/_CPack_Packages/Linux/RPM/BUILD
 
+ exit 0
 
 ***
 
CPack: - package: /home/jens/mariadb-columnstore-tools/build/mariadb-columnstore-tools-1.2.0-1-amd64-centos6.rpm generated.
 
[jens@centos6 build]$ ls
 
backuprestore   CMakeFiles           CPackConfig.cmake  CPackSourceConfig.cmake  Makefile                                             monitoring
 
CMakeCache.txt  cmake_install.cmake  _CPack_Packages    install_manifest.txt     mariadb-columnstore-tools-1.2.0-1-amd64-centos6.rpm  VERSION.dep
 
[jens@centos6 build]$

Comment by Elena Kotsinova (Inactive) [ 2018-10-12 ]

Win 10 Pro msi installation and functionality works fine.
Minor issue for UI of installer is reported separately.

Generated at Thu Feb 08 02:27:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.