[MCOL-1976] Amazon AMI with EBS - movepmdbrootconfig didnt update the configuration Created: 2018-11-28  Updated: 2023-10-26  Resolved: 2019-07-10

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.2.2
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: Unassigned
Resolution: Won't Fix Votes: 1
Labels: None
Environment:

Amazon AMI EC2 with EBS 1um 2pm

Issue Links:
Relates
relates to MCOL-1732 movePmDbrootConfig on external disk f... Closed

 Description   

Ran the movepmdbrootconfig, but the dbroot didnt get moved to the correct pm. It stayed on the original pm. EBS did get attached to the new pm. Now errors was logged

mcsadmin> getstor
getstorageconfig Wed Nov 28 15:55:01 2018

System Storage Configuration

Performance Module (DBRoot) Storage Type = external
User Module Storage Type = internal
System Assigned DBRoot Count = 2
DBRoot IDs assigned to 'pm1' = 1
DBRoot IDs assigned to 'pm2' = 2

Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot1: vol-0f52ae0becfa0a398, /dev/sdg, /dev/xvdg
Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot2: vol-075b3f607ad1d5a29, /dev/sdh, /dev/xvdh

mcsadmin> move pm2 2 pm1
movepmdbrootconfig Wed Nov 28 15:55:10 2018

DBRoot IDs currently assigned to 'pm2' = 2
DBRoot IDs currently assigned to 'pm1' = 1

DBroot IDs being moved, please wait...

DBRoot IDs newly assigned to 'pm2' = 2
DBRoot IDs newly assigned to 'pm1' = 1
mcsadmin> getstor
getstorageconfig Wed Nov 28 15:56:03 2018

System Storage Configuration

Performance Module (DBRoot) Storage Type = external
User Module Storage Type = internal
System Assigned DBRoot Count = 2
DBRoot IDs assigned to 'pm1' = 1
DBRoot IDs assigned to 'pm2' = 2

Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot1: vol-0f52ae0becfa0a398, /dev/sdg, /dev/xvdg
Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot2: vol-075b3f607ad1d5a29, /dev/sdh, /dev/xvdh

mcsadmin>

pm1 – shows that is attached to pm1

Disk /dev/xvdg: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/xvdh: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

NO ERRORS LOGGED ON PM1 OR PM2 - this is PM1 where the command was run from

Nov 28 15:55:10 ip-172-31-41-24 oamcpp[13951]: 10.735757 |0|0|0| D 08 CAL0000: manualMovePmDbroot: 2 from pm2 to pm1
Nov 28 15:55:10 ip-172-31-41-24 oamcpp[13951]: 10.750452 |0|0|0| D 08 CAL0000: mountDBRoot api, umount dbroot2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.758712 |0|0|0| I 17 CAL0000: MSG RECEIVED: Unmount dbroot : 2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.763382 |0|0|0| D 17 CAL0000: send unmountDBRoot to pm: 2/pm2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.768300 |0|0|0| D 17 CAL0000: sendMsgProcMon: Process module pm2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.787889 |0|0|0| I 17 CAL0000: UnMount Completed status: 0
Nov 28 15:55:10 ip-172-31-41-24 oamcpp[13951]: 10.788939 |0|0|0| D 08 CAL0000: amazonReattach function started
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.799583 |0|0|0| I 17 CAL0000: MSG RECEIVED: Unmount dbroot : 2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.804992 |0|0|0| D 17 CAL0000: send unmountDBRoot to pm: 2/pm2
Nov 28 15:55:10 ip-172-31-41-24 ProcessManager[15882]: 10.809524 |0|0|0| D 17 CAL0000: sendMsgProcMon: Process module pm2
Nov 28 15:55:11 ip-172-31-41-24 ProcessManager[15882]: 11.802857 |0|0|0| I 17 CAL0000: UnMount Completed status: 0
Nov 28 15:55:20 ip-172-31-41-24 oamcpp[13951]: 20.737236 |0|0|0| D 08 CAL0000: amazonReattach, detachEC2Volume passed on vol-075b3f607ad1d5a29
Nov 28 15:55:25 ip-172-31-41-24 oamcpp[13951]: 25.944734 |0|0|0| D 08 CAL0000: amazonReattach, attachEC2Volume passed on vol-075b3f607ad1d5a29:pm1
Nov 28 15:55:25 ip-172-31-41-24 oamcpp[13951]: 25.950930 |0|0|0| D 08 CAL0000: mountDBRoot api, mount dbroot2
Nov 28 15:55:25 ip-172-31-41-24 ProcessManager[15882]: 25.961220 |0|0|0| I 17 CAL0000: MSG RECEIVED: mount dbroot : 2
Nov 28 15:55:25 ip-172-31-41-24 ProcessManager[15882]: 25.965681 |0|0|0| D 17 CAL0000: send mountDBRoot to pm: 2/pm2
Nov 28 15:55:25 ip-172-31-41-24 ProcessManager[15882]: 25.970668 |0|0|0| D 17 CAL0000: sendMsgProcMon: Process module pm2
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 25.999908 |0|0|0| I 17 CAL0000: Mount Completed status: 0
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 26.019163 |0|0|0| I 17 CAL0000: MSG RECEIVED: Distribute Config File system/Columnstore.xml
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 26.019275 |0|0|0| D 17 CAL0000: distributeConfigFile called for system file = Columnstore.xml
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 26.030897 |0|0|0| D 17 CAL0000: sendMsgProcMon: Process module um1
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 26.031923 |0|0|0| D 17 CAL0000: um1 distributeConfigFile success.
Nov 28 15:55:26 ip-172-31-41-24 ProcessManager[15882]: 26.038554 |0|0|0| D 17 CAL0000: sendMsgProcMon: Process module pm2
Nov 28 15:55:27 ip-172-31-41-24 ProcessManager[15882]: 27.000319 |0|0|0| D 17 CAL0000: pm2 distributeConfigFile success.
Nov 28 15:55:27 ip-172-31-41-24 ProcessManager[15882]: 27.000409 |0|0|0| I 17 CAL0000: Distribute Config File Completed system/Columnstore.xml

 Comments   
Comment by David Hill (Inactive) [ 2018-11-28 ]

work around fix:

Use EC2/EBS to detach/attach back to original PM

Comment by John Dutchover [ 2018-12-04 ]

I have noticed that the AWS console will report "(detaching)" at the end of the "Attachment information" status for EBS volume after movePmDbrootConfig (error below) is issued.

{{DBroot IDs being moved, please wait...
ERROR: amazonReattach, detachEC2Volume failed on vol-0xxxxxxxxxxx
ERROR: amazonReattach api failure}}

In my case a forced detach from instance1 and attachment to desired instance was needed, even though dbroot was no longer appeared mounted from Linux OS perspective.

Comment by David Hill (Inactive) [ 2019-01-11 ]

This issue also reported by a customer using non amazon setup.

Generated at Thu Feb 08 02:32:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.