# This is a (partial) log of Apache Airflow orchestrating our APT mirrors using a custom wrapper called mirrorhandler that abstracts aptly (https://github.com/aptly-dev/aptly) commands.
# To anyone reading this, please don't steal the wrapper name, I am still hoping to release this under a free license and would like to avoid confusion ;)
# Step update - This fetches new packages in maxscale 2.5 bionic (everything goes fine here)
*** Reading local file: /logs/mirrorhandler.refresh.channel.maxscale-2.5-bionic/mirror.update.maxscale-2.5-bionic/2021-09-14T11:00:00+00:00/6.log
[2021-09-14 12:35:52,462] {taskinstance.py:896} INFO - Dependencies all met for
[2021-09-14 12:35:52,495] {taskinstance.py:896} INFO - Dependencies all met for
[2021-09-14 12:35:52,495] {taskinstance.py:1087} INFO -
--------------------------------------------------------------------------------
[2021-09-14 12:35:52,495] {taskinstance.py:1088} INFO - Starting attempt 6 of 6
[2021-09-14 12:35:52,495] {taskinstance.py:1089} INFO -
--------------------------------------------------------------------------------
[2021-09-14 12:35:52,507] {taskinstance.py:1107} INFO - Executing on 2021-09-14T11:00:00+00:00
[2021-09-14 12:35:52,517] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'mirrorhandler.refresh.channel.maxscale-2.5-bionic', 'mirror.update.maxscale-2.5-bionic', '2021-09-14T11:00:00+00:00', '--job-id', '92136', '--pool', 'mirrorhandler_pool', '--raw', '--subdir', 'DAGS_FOLDER/mirrorhandler.refresh.channel.maxscale-2.5-bionic.py', '--cfg-path', '/tmp/tmpltwb8zsh', '--error-file', '/tmp/tmpp1ngy2_p']
[2021-09-14 12:35:52,517] {standard_task_runner.py:77} INFO - Job 92136: Subtask mirror.update.maxscale-2.5-bionic
[2021-09-14 12:35:52,513] {standard_task_runner.py:52} INFO - Started process 159467 to run task
[2021-09-14 12:35:52,586] {logging_mixin.py:104} INFO - Running on host cb5014894f2c
[2021-09-14 12:35:52,655] {taskinstance.py:1302} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=
AIRFLOW_CTX_DAG_ID=mirrorhandler.refresh.channel.maxscale-2.5-bionic
AIRFLOW_CTX_TASK_ID=mirror.update.maxscale-2.5-bionic
AIRFLOW_CTX_EXECUTION_DATE=2021-09-14T11:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-09-14T11:00:00+00:00
[2021-09-14 12:35:52,655]
[2021-09-14 12:35:52,788] {ssh_skip_operator.py:130} INFO - Running command: sudo -u mirrorhandler --execute -v -v mirror update "maxscale-2.5-bionic"
[2021-09-14 12:35:52,857] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:35:52+0200 [INFO] mirrorhandler 0.0.7
2021-09-14 14:35:52+0200 [INFO] Parsed command line arguments, set-up logging and read configuratiton.
2021-09-14 14:35:52+0200 [INFO] Aggregating mirror config files from '/mirror-enabled'.
2021-09-14 14:35:52+0200 [INFO] Beginning work on 1 mirrors.
[2021-09-14 12:35:52,858] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:35:52+0200 [INFO] Working on mirror 'maxscale-2.5-bionic'.
[2021-09-14 12:35:52,957] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:35:52+0200 [INFO] mh out: aptly mirror update -config=/etc/aptly.conf -keyring=/mirror-keys/maxscale-2.5-bionic.keyring maxscale-2.5-bionic
aptly mirror update -config=/etc/aptly.conf -keyring=/mirror-keys/maxscale-2.5-bionic.keyring maxscale-2.5-bionic
[2021-09-14 12:35:54,762] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/dists/bionic/InRelease...
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading & parsing package files...
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/dists/bionic/main/binary-amd64/Packages.gz...
2021-09-14 14:35:54+0200 [INFO] aptly out: Building download queue...
2021-09-14 14:35:54+0200 [INFO] aptly out: Download queue: 4 items (54.04 MiB)
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/pool/main/m/maxscale/maxscale_2.5.15~bionic-1_amd64.deb...
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/pool/main/m/maxscale-devel/maxscale-devel_2.5.15~bionic-1_amd64.deb...
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/pool/main/m/maxscale-experimental/maxscale-experimental_2.5.15~bionic-1_amd64.deb...
2021-09-14 14:35:54+0200 [INFO] aptly out: Downloading https://downloads.mariadb.com/MaxScale/2.5/ubuntu/pool/main/m/maxscale-cdc-connector/maxscale-cdc-connector_2.5.15~bionic-1_amd64.deb...
2021-09-14 14:35:54+0200 [INFO] aptly out: Mirror `maxscale-2.5-bionic` has been successfully updated.
2021-09-14 14:35:54+0200 [WARNING] aptly err: gpgv: Signature made Wed 18 Aug 2021 15:24:23 CEST
2021-09-14 14:35:54+0200 [WARNING] aptly err: gpgv: using RSA key 4C470FFFEFC4D3DC59778655CE1A3DD5E3C94F49
[2021-09-14 12:35:54,766] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:35:54+0200 [WARNING] aptly err: gpgv: Good signature from "MariaDB Enterprise Signing Key "
2021-09-14 14:35:54+0200 [INFO] mh out: {'changed': true, 'changed_items': ['maxscale-2.5-bionic']}
{'changed': true, 'changed_items': ['maxscale-2.5-bionic']}
[2021-09-14 12:35:54,809] {taskinstance.py:1211} INFO - Marking task as SUCCESS. dag_id=mirrorhandler.refresh.channel.maxscale-2.5-bionic, task_id=mirror.update.maxscale-2.5-bionic, execution_date=20210914T110000, start_date=20210914T123552, end_date=20210914T123554
# Some snapshot dance excluded here, successfully resulting in snapshot publication_maxscale-2.5-bionic_2021-09-14-123559
# Step publish - This actually changes what is served by our mirror
[2021-09-14 12:36:02,127] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:36:02+0200 [INFO] Working on publication 'maxscale-2.5-bionic'.
[2021-09-14 12:36:02,195] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:36:02+0200 [WARNING] Ignoring changed configuration for publication as it is bogus. Changes are: ['sources']
[2021-09-14 12:36:03,062] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:36:03+0200 [INFO] mh out: aptly publish switch -config=/etc/aptly.conf -gpg-key= -batch -passphrase-file= bionic upstream/mariadb/maxscale/2.5 publication_maxscale-2.5-bionic_2021-09-14-123559
aptly publish switch -config=/etc/aptly.conf -gpg-key= -batch -passphrase-file= bionic upstream/mariadb/maxscale/2.5 publication_maxscale-2.5-bionic_2021-09-14-123559
[2021-09-14 12:36:03,295] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:36:03+0200 [INFO] aptly out: Loading packages...
2021-09-14 14:36:03+0200 [INFO] aptly out: Generating metadata files and linking package files...
[2021-09-14 12:36:03,295] {ssh_skip_operator.py:162} INFO - 2021-09-14 14:36:03+0200 [WARNING] aptly err: ERROR: unable to publish: unable to process packages: error linking file to /public/upstream/mariadb/maxscale/2.5/pool/main/m/maxscale/maxscale_2.5.15~bionic-1_amd64.deb: file already exists and is different
subprocess.CalledProcessError: Command '['aptly', 'publish', 'switch', '-config=/etc/aptly.conf', '-gpg-key=', '-batch', '-passphrase-file=', 'bionic', 'upstream/mariadb/maxscale/2.5', 'publication_maxscale-2.5-bionic_2021-09-14-123559']' returned non-zero exit status 1.
[2021-09-14 12:36:03,300] {ssh_skip_operator.py:162} INFO -
airflow.exceptions.AirflowException: SSH operator error: error running cmd: sudo -u mirrorhandler --execute -v -v publication publish "maxscale-2.5-bionic", error:
[2021-09-14 12:36:03,318] {taskinstance.py:1551} INFO - Marking task as UP_FOR_RETRY. dag_id=mirrorhandler.refresh.channel.maxscale-2.5-bionic, task_id=publication.publish.maxscale-2.5-bionic, execution_date=20210914T110000, start_date=20210914T123601, end_date=20210914T123603
[2021-09-14 12:36:03,375] {local_task_job.py:149} INFO - Task exited with return code 1