Centos 7

you pass plugin-load=aws_key_managment on the command line? This is not necessary, mariabackup already has corresponding option in my-backup.ini

Could you provide the full output as text file, redirecting stdout of mariabackup to a file?

I'm in a secure environment, copy paste isn't accessible nor can I move data files in and out. (sorry its a pain)

When removing the plugin-load=aws_key_management, the following error occurs:

1. The aws keys are there from the initial mariabackup.

Can you prepare backup on the same machine, as the same user where you took the backup first?
It can be anything, ranging from aws_key_management.so being missing, to missing aws credentials.

if the .so is there, you can edit backup-my.cnf file to switch on the AWS KMS plugin trace logging to see what happens there.

Using mariabackup v10.2.9

1. Executed the backup and the restore on the same server
2. Outcome is the same
3. Set aws log to trace, but I do not see any trace logs, it almost looks like the plugin isn't being loaded.
   

well, then I guess you need to examine the contents of backup-my.cnf.
the backup does store some information about encryption plugin (actually, all plugin parameters there). And, prepare is written to read the contents of backup-my.cnf .

Please see the attached crash report, I've tried several attempts.

mariabackup --backup --parallel=8 --rsync --user=root --password=x

mariabackup --prepare --parallel=8 --user=root --password=x --innodb-encrypt-log=1 --plugin-load=aws_key_management --plugin-dir=/usr/lib64/mysql/plugin/

mariabackup-crash-report.txt

I am unable to get any sort of reasonable output with just a simple --prepare command like such:

mariabackup --prepare --parallel=8 --user=root --password=x

--prepare does not need user, or password, it does not contact any server.. , it does not understand --innodb-encrypt-log either, it is not a server.

Kudos on getting text output. I hope now you can send also the backup-my.cnf in the backup directory, or maybe even a picture of it would also suffice

can you try all of this without ---rsync. We did not test it here, not (I suspect) percona had.
I'd recommend to provide --target-dir parameters as well (unless you use streaming)

Great news, removing --rsync and adding --target-dir= produced a successful backup and prepare!

I assume it failed because mariabackup had not found rsync utility. I'll now check it earlier before starting the backup.
If you happen to have the log of "mariabackup --backup", you can attach it here.