As part of this task, we must test if MariaDB can be upgraded from 10.1 to 10.2 when log encryption is in use. Here is my test on Debian GNU/Linux Sid:

sudo apt install mariadb-server-10.1

mkdir /dev/shm/f; cd /dev/shm/f; cat > logkey.txt << EOF

1;36D6CB74CA7D4586CCC7261E174079CC5639E5F681D500ADFA887C165AD49301

2;F51F5108CF6048B4C9C88BA6CE1C13F9F1CBEE82080F7FA0F979DEF5D4B94509

EOF

/usr/sbin/mysqld --file-key-management-filename=logkey.txt --plugin-load=file_key_management.so  --basedir=/usr --datadir=/dev/shm/f --innodb --innodb-encrypt-log

After shutting down the 10.1 server (for me, it would should shut down because another mysqld instance was already started on port 3306), start 10.2 in gdb on the same files:

break recv_log_format_0_recover

run --gdb --datadir=/dev/shm/f --file-key-management-filename=logkey.txt --plugin-dir=/dev/shm/t/plugin/file_key_management --plugin-load=file_key_management.so --innodb --innodb-encrypt-log --lc-messages-dir=/dev/shm/t/sql/share

continue

Observe that the following message is displayed:

2017-01-24 10:18:21 140737353952960 [ERROR] InnoDB: Upgrade after a crash is not supported. This redo log was created before MySQL 5.7.9, and it appears corrupted. Please follow the instructions at http://dev.mysql.com/doc/refman/5.7/en/upgrading.html

I think that the message should be revised to refer to MariaDB 10.2.2, and that we should attempt to decrypt the data. Upgrade with a clean encrypted redo log from 10.1 should definitely be allowed.

I think that as part of this effort, in 10.2 we must rebuild the redo log files on startup if the requested innodb_encrypt_log setting differs from the redo log header, just like we would rebuild the redo log files when upgrading from 10.1 or MySQL 5.6 or earlier.
We should also remove the log checkpoint call after fil_crypt_threads_init(), which according to jplindst was added to handle the case that the innodb_encrypt_log setting was changed.

In the new encrypted redo log format for MySQL 10.2, the function log_block_checksum_is_ok() should be used so that we always know beforehand (based on the redo log header block) whether subsequent blocks are supposed to be encrypted. I would forcibly enable the redo log checksum when redo log encryption is used.

To support an upgrade from an encrypted 10.1 redo log, the function recv_log_format_0_recover() needs to call log_decrypt_after_read() after reading the redo log block.

The code so far is available in 10.2. The test encryption.innodb_encrypt_log_corruption demonstrates that we can upgrade from an encrypted 10.1 redo log file.
But we still need to introduce a redo log format tag that distinguishes the MariaDB encrypted redo log files from the MySQL 5.7.9 format.

I committed two preparatory patches to bb-10.2-marko for review:
Remove recv_sys->last_block
Add separate functions for reading 10.1 encrypted log

ok to push above preparatory patches to 10.2.

bb-10.2-mdev-11782

Revised patch with less memcpy() and consistently using a 4-byte cleartext log block header: bb-10.2-mdev-11782

ok to push after addressing review comments that are open.

Revised patch, addressing the review comments and test failures, and also with some changes to the startup logic: We must not generate redo log before rebuilding the redo logs.
bb-10.2-marko (includes also MDEV-12061)

commit 2af28a363c0ac55c9b91aa9eb26949fc9ecf043a
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date: Fri Feb 10 12:11:42 2017 +0200

MDEV-11782: Redefine the innodb_encrypt_log format

Write only one encryption key to the checkpoint page.
Use 4 bytes of nonce. Encrypt more of each redo log block,
only skipping the 4-byte field LOG_BLOCK_HDR_NO which the
initialization vector is derived from.

Issue notes, not warning messages for rewriting the redo log files.

recv_recovery_from_checkpoint_finish(): Do not generate any redo log,
because we must avoid that before rewriting the redo log files, or
otherwise a crash during a redo log rewrite (removing or adding
encryption) may end up making the database unrecoverable.
Instead, do these tasks in innobase_start_or_create_for_mysql().

Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some
unreachable code and duplicated error messages for log corruption.

LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo
log format.

log_group_t::is_encrypted(), log_t::is_encrypted(): Determine
if the redo log is in encrypted format.

recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED.

srv_prepare_to_delete_redo_log_files(): Display NOTE messages about
adding or removing encryption. Do not issue warnings for redo log
resizing any more.

innobase_start_or_create_for_mysql(): Rebuild the redo logs also when
the encryption changes.

innodb_log_checksums_func_update(): Always use the CRC-32C checksum
if innodb_encrypt_log. If needed, issue a warning
that innodb_encrypt_log implies innodb_log_checksums.

log_group_write_buf(): Compute the checksum on the encrypted
block contents, so that transmission errors or incomplete blocks can be
detected without decrypting.

Rewrite most of the redo log encryption code. Only remember one
encryption key at a time (but remember up to 5 when upgrading from the
MariaDB 10.1 format.)