From your scripts, it looks like you set wsrep_sst_receive_address to 127.0.0.1. To my understanding, it should be anything but 127.0.0.1 – with that, your node is basically supposed to receive SST from itself. According to your configuration from MDEV-4163, your wsrep_sst_receive_address should apparently be 192.168.122.138 ?
I also think it's better to set wsrep_node_address explicitly too, since WSREP complains about it as well.
(it will probably make wsrep_sst_receive_address obsolete, but it won't hurt anyway).
Of course, I set wsrep_node_address to the good IP. The value in the script is the default one but I modify it on the server before the bash action. I will try with wsrep_node_address set too.
I have the same error. my mariadb.cnf is:
[mysqld]
wsrep_node_address='192.168.122.137'
wsrep_sst_receive_address='192.168.122.137'
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_sst_method=rsync
Here is the syslog:
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] /usr/sbin/mysqld: Normal shutdown
Feb 11 14:59:47 mariadb2 mysqld:
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: Stop replication
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] Event Scheduler: Purging the queue. 0 events
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: dtor state: CLOSED
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: apply mon: entered 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: apply mon: entered 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: apply mon: entered 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: cert index usage at exit 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: cert trx map usage at exit 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: deps set usage at exit 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: avg deps dist 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 [Note] WSREP: Flushing memory map to disk...
Feb 11 14:59:47 mariadb2 mysqld: 130211 14:59:47 InnoDB: Starting shutdown...
Feb 11 14:59:48 mariadb2 mysqld: 130211 14:59:48 InnoDB: Shutdown completed; log sequence number 1598303
Feb 11 14:59:48 mariadb2 mysqld: 130211 14:59:48 [Note] /usr/sbin/mysqld: Shutdown complete
Feb 11 14:59:48 mariadb2 mysqld:
Feb 11 14:59:48 mariadb2 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Feb 11 14:59:50 mariadb2 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Feb 11 14:59:50 mariadb2 mysqld_safe: WSREP: Running position recovery with --log_error=/tmp/tmp.J8JLjR0crR
Feb 11 14:59:53 mariadb2 mysqld_safe: WSREP: Recovered position 64dc5b89-742e-11e2-0800-9315c248faac:0
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Warning] option 'table_cache': unsigned value 2097152 adjusted to 524288
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: wsrep_start_position var submitted: '64dc5b89-742e-11e2-0800-9315c248faac:0'
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: Read nil XID from storage engines, skipping position init
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: Found saved state: 64dc5b89-742e-11e2-0800-9315c248faac:-1
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: Passing config to GCS: base_host = 192.168.122.137; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: The InnoDB memory heap is disabled
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: Compressed tables use zlib 1.2.3.4
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: Using Linux native AIO
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: Initializing buffer pool, size = 256.0M
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: Completed initialization of buffer pool
Feb 11 14:59:53 mariadb2 mysqld: 130211 14:59:53 InnoDB: highest supported file format is Barracuda.
Feb 11 14:59:54 mariadb2 mysqld: 130211 14:59:54 InnoDB: Waiting for the background threads to start
Feb 11 14:59:55 mariadb2 mysqld: 130211 14:59:55 Percona XtraDB (http://www.percona.com) 1.1.8-29.1 started; log sequence number 1598303
Feb 11 14:59:55 mariadb2 mysqld: 130211 14:59:55 [Note] Plugin 'FEEDBACK' is disabled.
Feb 11 14:59:55 mariadb2 mysqld: 130211 14:59:55 [Note] Event Scheduler: Loaded 0 events
Feb 11 14:59:55 mariadb2 mysqld: 130211 14:59:55 [Note] /usr/sbin/mysqld: ready for connections.
Feb 11 14:59:55 mariadb2 mysqld: Version: '5.5.28a-MariaDB-a1~squeeze' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution, wsrep_23.7rc1.rXXXX
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5082]: Upgrading MySQL tables if necessary.
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5085]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5085]: Looking for 'mysql' as: /usr/bin/mysql
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5085]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5085]: This installation of MySQL is already upgraded to 5.5.28a-MariaDB, use --force if you still need to run mysql_upgrade
Feb 11 14:59:55 mariadb2 /etc/mysql/debian-start[5096]: Checking for insecure root accounts.
In your config above, I don't see wsrep_cluster_address – what is it supposed to be?
What is this (problematic) node connecting to, is the other one up and running? If so, does its syslog show any attempt of the other one to connect?
I thought when we set wsrep_sst_receive_address manually we didn't have to set wsrep_cluster_address.
the wsrep_cluster_address is supposed to be gcomm://192.168.122.137 the ip of the first server which is up and running.
As you can see:
root@mariadb1:~# mysql --defaults-file=/etc/mysql/debian.cnf -e "SHOW STATUS LIKE 'wsrep_%';"
----------------------------------------------------------------+
| Variable_name | Value |
----------------------------------------------------------------+
| wsrep_local_state_uuid | c71c6b4f-742e-11e2-0800-c1770497f8e0 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 160 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_received | 2 |
| wsrep_received_bytes | 139 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.500000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 0 |
| wsrep_causal_reads | 0 |
| wsrep_incoming_addresses | 192.168.122.138:3306 |
| wsrep_cluster_conf_id | 1 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | c71c6b4f-742e-11e2-0800-c1770497f8e0 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 23.2.2(r137) |
| wsrep_ready | ON |
----------------------------------------------------------------+
In the syslog from the first server there is no appearance of the second:
Feb 11 13:41:14 mariadb1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Feb 11 13:41:14 mariadb1 mysqld_safe: WSREP: Running position recovery with --log_error=/tmp/tmp.GDBZirqZv8
Feb 11 13:41:17 mariadb1 mysqld_safe: WSREP: Recovered position c71c6b4f-742e-11e2-0800-c1770497f8e0:0
Feb 11 13:41:17 mariadb1 mysqld: 130211 13:41:17 [Warning] option 'table_cache': unsigned value 2097152 adjusted to 524288
Feb 11 13:41:17 mariadb1 mysqld: 130211 13:41:17 [Note] WSREP: wsrep_start_position var submitted: 'c71c6b4f-742e-11e2-0800-c1770497f8e0:0'
Feb 11 13:41:17 mariadb1 mysqld: 130211 13:41:17 [Note] WSREP: Read nil XID from storage engines, skipping position init
Feb 11 13:41:17 mariadb1 mysqld: 130211 13:41:17 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Found saved state: c71c6b4f-742e-11e2-0800-c1770497f8e0:160
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Passing config to GCS: base_host = 192.168.122.138; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Assign initial position for certification: 160, protocol version: -1
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: wsrep_sst_grab()
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Start replication
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Setting initial position to c71c6b4f-742e-11e2-0800-c1770497f8e0:160
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: protonet asio version 0
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: backend: asio
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: GMCast version 0
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: (54a4800c-7448-11e2-0800-03692350c6db, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: (54a4800c-7448-11e2-0800-03692350c6db, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: EVS version 0
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: PC version 0
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer ''
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: view(view_id(PRIM,54a4800c-7448-11e2-0800-03692350c6db,1) memb
joined
{ Feb 11 13:41:18 mariadb1 mysqld: }left
{ Feb 11 13:41:18 mariadb1 mysqld: }partitioned
{ Feb 11 13:41:18 mariadb1 mysqld: })
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: gcomm: connected
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Opened channel 'my_wsrep_cluster'
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Waiting for SST to complete.
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 54a596b6-7448-11e2-0800-fd4ae6056076
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: STATE EXCHANGE: sent state msg: 54a596b6-7448-11e2-0800-fd4ae6056076
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: STATE EXCHANGE: got state msg: 54a596b6-7448-11e2-0800-fd4ae6056076 from 0 (mariadb1)
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Quorum results:
Feb 11 13:41:18 mariadb1 mysqld: #011version = 2,
Feb 11 13:41:18 mariadb1 mysqld: #011component = PRIMARY,
Feb 11 13:41:18 mariadb1 mysqld: #011conf_id = 0,
Feb 11 13:41:18 mariadb1 mysqld: #011members = 1/1 (joined/total),
Feb 11 13:41:18 mariadb1 mysqld: #011act_id = 160,
Feb 11 13:41:18 mariadb1 mysqld: #011last_appl. = -1,
Feb 11 13:41:18 mariadb1 mysqld: #011protocols = 0/4/2 (gcs/repl/appl),
Feb 11 13:41:18 mariadb1 mysqld: #011group UUID = c71c6b4f-742e-11e2-0800-c1770497f8e0
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Flow-control interval: [16, 16]
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Restored state OPEN -> JOINED (160)
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Member 0 (mariadb1) synced with group.
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 160)
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: New cluster view: global state: c71c6b4f-742e-11e2-0800-c1770497f8e0:160, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 [Note] WSREP: SST complete, seqno: 160
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: The InnoDB memory heap is disabled
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Compressed tables use zlib 1.2.3.4
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Using Linux native AIO
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Initializing buffer pool, size = 256.0M
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Completed initialization of buffer pool
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: highest supported file format is Barracuda.
Feb 11 13:41:18 mariadb1 mysqld: 130211 13:41:18 InnoDB: Waiting for the background threads to start
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 Percona XtraDB (http://www.percona.com) 1.1.8-29.1 started; log sequence number 1598303
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] Plugin 'FEEDBACK' is disabled.
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] Event Scheduler: Loaded 0 events
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] WSREP: Assign initial position for certification: 160, protocol version: 2
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] WSREP: Synchronized with group, ready for connections
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb 11 13:41:19 mariadb1 mysqld: 130211 13:41:19 [Note] /usr/sbin/mysqld: ready for connections.
Feb 11 13:41:19 mariadb1 mysqld: Version: '5.5.28a-MariaDB-a1~squeeze' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution, wsrep_23.7rc1.rXXXX
Feb 11 13:41:20 mariadb1 /etc/mysql/debian-start[1771]: Upgrading MySQL tables if necessary.
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1777]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1777]: Looking for 'mysql' as: /usr/bin/mysql
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1777]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1777]: This installation of MySQL is already upgraded to 5.5.28a-MariaDB, use --force if you still need to run mysql_upgrade
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1865]: Checking for insecure root accounts.
Feb 11 13:41:21 mariadb1 /etc/mysql/debian-start[1889]: Triggering myisam-recover for all MyISAM tables
Feb 11 14:08:49 mariadb1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67
Feb 11 14:08:49 mariadb1 dhclient: DHCPACK from 192.168.122.1
Feb 11 14:08:49 mariadb1 dhclient: bound to 192.168.122.138 – renewal in 1340 seconds.
Feb 11 14:17:01 mariadb1 /USR/SBIN/CRON[2156]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 11 14:31:09 mariadb1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67
Feb 11 14:31:09 mariadb1 dhclient: DHCPACK from 192.168.122.1
Feb 11 14:31:09 mariadb1 dhclient: bound to 192.168.122.138 – renewal in 1533 seconds.
Feb 11 14:56:42 mariadb1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67
Feb 11 14:56:42 mariadb1 dhclient: DHCPACK from 192.168.122.1
Feb 11 14:56:42 mariadb1 dhclient: bound to 192.168.122.138 – renewal in 1415 seconds.
Feb 11 15:17:01 mariadb1 /USR/SBIN/CRON[2183]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 11 15:20:17 mariadb1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67
Feb 11 15:20:17 mariadb1 dhclient: DHCPACK from 192.168.122.1
Feb 11 15:20:17 mariadb1 dhclient: bound to 192.168.122.138 – renewal in 1552 seconds.
Feb 11 15:46:09 mariadb1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67
Feb 11 15:46:09 mariadb1 dhclient: DHCPACK from 192.168.122.1
Feb 11 15:46:09 mariadb1 dhclient: bound to 192.168.122.138 – renewal in 1494 seconds.
the first server has the ip 192.168.122.138 so I changed the config in the mariadb.cnf to match. It doesn't change anything.
No, you still need wsrep_cluster_address.
wsrep_sst_receive_address is the address of the current node, while wsrep_cluster_address is where it is supposed to connect to and receive the state from. Without it, it doesn't know anything about an existing cluster, that's why WSREP stops at
WSREP: Assign initial position for certification: 0, protocol version: -1
and doesn't proceed with replication.
Please set the cluster address back as usual.
It proceeded much further now, so we are making progress 
Now it says
Feb 11 16:13:47 mariadb2 mysqld: #011Read: 'rsync daemon already running.'
Feb 11 16:13:47 mariadb2 mysqld: 130211 16:13:47 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.122.241' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '11358': 114 (Operation already in progress)
While I'm not sure that's what happens in your case, I've seen the error before when rsync would hang running from previous attempts.
Please check if you have rsync running on each of the machines (sender and receiver), kill it if it's hung there and try again.
If it's hanging it will probably require a brutal SIGKILL, at least that's how it used to be for me.
Okay, we had one just like that not long ago. Please check this comment and the next ones in the same report:
You might have either the common problem with Debian passwords described at stackoverflow.com (the link provided in the comment), or the Galera-specific trick with password replication. In the latter case, the reporter described how he solved the problem, please try to follow his advice.
Good luck with your Galera cluster, hope you'll enjoy it.