I set up MariaDB replication for my phpBB forum database, computerbb, from a socket-only MariaDB instance on the master server to a MariaDB replica running inside a hardened FreeBSD jail on the slave server.
The goal was to keep a live replicated copy of the phpBB database inside a FreeBSD jail without exposing MariaDB over a public TCP port.
Final design:
Code: Select all
master MariaDB socket
-> SSH tunnel
-> slave jail 127.0.0.1:3307
-> MariaDB replica inside jaildb
Environment
Master
- MariaDB 10.6.x
- Socket-only MariaDB
- No public TCP listener
- phpBB database: computerbb
- Binary logs enabled
- FreeBSD 15.x
- VNET jail named jaildb
- MariaDB 10.11.x
- Replica database server
- Persistent SSH tunnel using autossh
1. FreeBSD jail networking
The jail was created as a thick jail:
Code: Select all
bsdinstall jail /usr/local/jails/jaildb
The slave host uses:
Code: Select all
bridge0
lagg0
epair interface for jaildb
Initially, I had lagg0 using load balancing, but DHCP replies for the jail were not coming back reliably. Switching lagg0 to failover fixed it.
Working rc.conf network section on the slave:
Code: Select all
ifconfig_em0="up"
ifconfig_igb0="up"
cloned_interfaces="lagg0 bridge0"
ifconfig_lagg0="laggproto failover laggport em0 laggport igb0 up"
ifconfig_bridge0="addm lagg0 SYNCDHCP"
kld_list="if_bridge if_epair"
jail_enable="YES"
jail_list="jaildb"
Useful checks:
Code: Select all
ifconfig bridge0
ifconfig lagg0
netstat -rn -f inet
Code: Select all
service jail start jaildb
jls
2. jaildb configuration
The jail configuration file on the slave:
Code: Select all
/etc/jail.conf
Code: Select all
jaildb {
host.hostname = "jaildb";
path = "/usr/local/jails/jaildb";
vnet;
vnet.interface = "jaildb0";
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
exec.prestart = "ifconfig epair10 create";
exec.prestart += "ifconfig epair10a descr jaildb-host";
exec.prestart += "ifconfig epair10a up";
exec.prestart += "ifconfig bridge0 addm epair10a";
exec.prestart += "ifconfig epair10b name jaildb0";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 deletem epair10a";
exec.poststop += "ifconfig epair10a destroy";
mount.devfs;
devfs_ruleset = 10;
persist;
allow.raw_sockets = 0;
allow.set_hostname = 0;
allow.sysvipc = 0;
allow.mount = 0;
allow.mount.devfs = 0;
allow.mount.nullfs = 0;
allow.mount.procfs = 0;
allow.mount.tmpfs = 0;
allow.chflags = 0;
enforce_statfs = 2;
children.max = 0;
}
On the slave host:
Code: Select all
ee /etc/devfs.rules
Code: Select all
[devfsrules_jaildb=10]
add include $devfsrules_jail
add path 'bpf*' unhide
Code: Select all
service devfs restart
Code: Select all
/usr/local/jails/jaildb/etc/rc.conf
Code: Select all
hostname="jaildb"
ifconfig_jaildb0="SYNCDHCP"
sshd_enable="NO"
sendmail_enable="NONE"
syslogd_flags="-ss"
clear_tmp_enable="YES"
dumpdev="NO"
moused_nondefault_enable="NO"
Code: Select all
service jail start jaildb
jexec jaildb ifconfig jaildb0
3. Installing MariaDB inside the jail
Enter the jail:
Code: Select all
jexec jaildb /bin/sh
Code: Select all
pkg update
Code: Select all
pkg install mariadb1011-server mariadb1011-client
Code: Select all
sysrc mysql_enable=YES
Code: Select all
mkdir -p /usr/local/etc/mysql/conf.d
ee /usr/local/etc/mysql/conf.d/replica.cnf
Code: Select all
[mysqld]
server_id = 226
relay_log = jaildb-relay-bin
read_only = ON
skip_name_resolve = ON
# Only replicate the phpBB database.
replicate-wild-do-table=computerbb.%
Code: Select all
service mysql-server start
service mysql-server status
Code: Select all
mariadb -e "SHOW VARIABLES WHERE Variable_name IN ('server_id','read_only','relay_log');"
Code: Select all
server_id = 226
read_only = ON
relay_log = jaildb-relay-bin
4. Enabling binary logs on the master
The master MariaDB instance was originally socket-only and started with binary logs disabled.
Original startup script included:
Code: Select all
--skip-log-bin
The minimal change was to remove/comment out skip-log-bin and add binary logging options.
Master startup script:
Code: Select all
#!/bin/bash
# start MariaDB
/usr/bin/mysqld_safe --skip-networking \
--socket=/home/$USER/.config/mysql/mysqld.sock \
--datadir=/home/$USER/.config/mysql/data \
--pid-file=/home/$USER/.config/mysql/mariadb.pid \
--log-error=/home/$USER/.config/mysql/mysqld.err \
--server-id=1 \
--log-bin=/home/$USER/.config/mysql/mariadb-bin \
--binlog-expire-logs-seconds=604800 \
--event-scheduler=ON \
--nowatch > /dev/null
Code: Select all
--skip-networking
Code: Select all
--log-bin=/home/$USER/.config/mysql/mariadb-bin
Code: Select all
--binlog-expire-logs-seconds=604800
Code: Select all
mariadb-admin --socket="$HOME/.config/mysql/mysqld.sock" shutdown
~/.config/mysql/start
Code: Select all
mariadb --socket="$HOME/.config/mysql/mysqld.sock" -e "
SHOW VARIABLES WHERE Variable_name IN ('log_bin','server_id','skip_networking','socket','binlog_expire_logs_seconds');
SHOW MASTER STATUS;
"
Code: Select all
log_bin ON
server_id 1
skip_networking ON
binlog_expire_logs_seconds 604800
Code: Select all
mariadb-bin.000001 4062
5. Creating a replication user on the master
Because the master is socket-only, the replication connection arrives locally through the SSH tunnel. I created the replication user as localhost.
On the master:
Code: Select all
mariadb --socket="$HOME/.config/mysql/mysqld.sock"
Code: Select all
CREATE USER 'repl'@'localhost' IDENTIFIED BY 'REDACTED_PASSWORD';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'localhost';
FLUSH PRIVILEGES;
Code: Select all
mariadb --socket="$HOME/.config/mysql/mysqld.sock" -e "SELECT User, Host, plugin FROM mysql.user;"
6. Creating an SSH key inside the jail
On the slave host, enter the jail:
Code: Select all
jexec jaildb /bin/sh
Code: Select all
ssh-keygen -t ed25519
Code: Select all
/root/.ssh/id_ed25519
Code: Select all
cat /root/.ssh/id_ed25519.pub
Code: Select all
~/.ssh/authorized_keys
Code: Select all
mkdir -p ~/.ssh
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Code: Select all
ssh [email protected]
7. Testing a manual SSH socket tunnel
Inside jaildb, run:
Code: Select all
ssh -N \
-L 127.0.0.1:3307:/home/USERNAME/.config/mysql/mysqld.sock \
[email protected]
The -N option means SSH opens the tunnel but does not run a shell.
Leave that command running.
Open another shell on the slave and enter the jail again:
Code: Select all
jexec jaildb /bin/sh
Code: Select all
mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
Code: Select all
10.6.x-MariaDB-log
Code: Select all
jaildb 127.0.0.1:3307
-> SSH tunnel
-> master MariaDB Unix socket
8. Running the SSH tunnel in the background
Once the manual test worked, I tested the same tunnel in the background:
Code: Select all
ssh -f -N \
-L 127.0.0.1:3307:/home/USERNAME/.config/mysql/mysqld.sock \
[email protected]
Code: Select all
sockstat -4 -l | grep 3307
Code: Select all
mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
Code: Select all
ps aux | grep '[s]sh.*3307'
pkill -f 'ssh.*3307'
9. Installing autossh for a persistent tunnel
A plain background SSH tunnel works, but if it dies, replication stops.
So I installed autossh inside jaildb:
Code: Select all
pkg install autossh
Create an rc.d service inside the jail:
Code: Select all
ee /usr/local/etc/rc.d/whatbox_db_tunnel
Code: Select all
#!/bin/sh
# PROVIDE: whatbox_db_tunnel
# REQUIRE: NETWORKING
# KEYWORD: shutdown
. /etc/rc.subr
name="whatbox_db_tunnel"
rcvar="whatbox_db_tunnel_enable"
command="/usr/local/bin/autossh"
load_rc_config $name
: ${whatbox_db_tunnel_enable:="NO"}
command_args="-M 0 -f -N \
-o ExitOnForwardFailure=yes \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
-L 127.0.0.1:3307:/home/USERNAME/.config/mysql/mysqld.sock \
[email protected]"
run_rc_command "$1"
Code: Select all
chmod +x /usr/local/etc/rc.d/whatbox_db_tunnel
Code: Select all
sysrc whatbox_db_tunnel_enable=YES
Code: Select all
service whatbox_db_tunnel start
Code: Select all
sockstat -4 -l | grep 3307
Code: Select all
mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
Code: Select all
service whatbox_db_tunnel restart
mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
Code: Select all
service jail restart jaildb
jexec jaildb sockstat -4 -l | grep 3307
jexec jaildb mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
10. Dumping the phpBB database from the master
The phpBB database is:
Code: Select all
computerbb
Code: Select all
mariadb-dump \
--socket="$HOME/.config/mysql/mysqld.sock" \
--single-transaction \
--routines \
--triggers \
--events \
--master-data=2 \
--databases computerbb \
> computerbb-replication-seed.sql
Code: Select all
grep -m1 "CHANGE MASTER" computerbb-replication-seed.sql
Code: Select all
-- CHANGE MASTER TO MASTER_LOG_FILE='mariadb-bin.000001', MASTER_LOG_POS=21956084;
11. Copying and importing the dump on the slave
On the slave host:
Code: Select all
scp [email protected]:~/computerbb-replication-seed.sql /tmp/
Code: Select all
cp /tmp/computerbb-replication-seed.sql /usr/local/jails/jaildb/tmp/
Code: Select all
jexec jaildb mysql < /tmp/computerbb-replication-seed.sql
Code: Select all
jexec jaildb mariadb -e "SHOW DATABASES;"
Code: Select all
jexec jaildb mariadb computerbb -e "SHOW TABLES;"
12. Starting MariaDB replication on the slave
Enter MariaDB inside the jail:
Code: Select all
jexec jaildb mariadb
Code: Select all
STOP SLAVE;
CHANGE MASTER TO
MASTER_HOST='127.0.0.1',
MASTER_PORT=3307,
MASTER_USER='repl',
MASTER_PASSWORD='REDACTED_PASSWORD',
MASTER_LOG_FILE='mariadb-bin.000001',
MASTER_LOG_POS=21956084,
MASTER_CONNECT_RETRY=10;
START SLAVE;
SHOW SLAVE STATUS\G
Code: Select all
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:
Master_Server_Id: 1
13. Useful replication health checks
From the slave host:
Code: Select all
jexec jaildb mariadb -e "SHOW SLAVE STATUS\G"
Code: Select all
jexec jaildb mariadb -e "SHOW SLAVE STATUS\G" | egrep 'Slave_IO_Running|Slave_SQL_Running|Seconds_Behind_Master|Last_IO_Error|Last_SQL_Error'
Code: Select all
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:
Code: Select all
jexec jaildb sockstat -4 -l | grep 3307
Code: Select all
jexec jaildb mysql -h 127.0.0.1 -P 3307 -u repl -p -e "SELECT VERSION();"
14. Optional replication test table
On the master:
Code: Select all
mariadb --socket="$HOME/.config/mysql/mysqld.sock" computerbb -e "
CREATE TABLE replication_test (
id INT PRIMARY KEY AUTO_INCREMENT,
msg VARCHAR(100),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO replication_test (msg) VALUES ('replication works');
"
Code: Select all
jexec jaildb mariadb computerbb -e "SELECT * FROM replication_test;"
Clean it up from the master:
Code: Select all
mariadb --socket="$HOME/.config/mysql/mysqld.sock" computerbb -e "DROP TABLE replication_test;"
15. Current final state
The final replication status showed:
Code: Select all
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:
Code: Select all
computerbb on master MariaDB
-> master binary logs
-> SSH socket tunnel
-> slave jail port 127.0.0.1:3307
-> MariaDB replica inside jaildb
The slave has a live replicated copy of the phpBB database inside a FreeBSD jail.