Install HAProxy on Percona XtraDB custers

I have posted how to install Percona XtraDB cluster previously. And I want to show how to use the DB clusters from web server.

To use the multiple nodes from one or multiple web servers, it need load balancer. There’s several LB but HAProxy is one of most popular and easy to install.
Here’s documentation from Percona how to set up HAProxy on Percona XtraDB clusters.

Let’s say there are Percona DB nodes synced each other and one web server.

10.7.13.81 web
10.7.13.91 node1
10.7.13.92 node2

And I would like to make web server connected to 2 db nodes.

Config and install HAProxy on web server

Download and Install HAProxy on web server.

sudo add-apt-repository ppa:vbernat/haproxy-1.8
sudo apt-get update
sudo apt-get install haproxy

Then config HAProxy.

sudo vi /etc/haproxy/haproxy.cfg

Then add this below. We will use port 3307 and localhost to connect Mysql nodes. And HAProxy will routes the traffic to db nodes using port 3306.

frontend pxc-front
  bind *:3307
  mode tcp
  default_backend pxc-back

frontend stats-front
  bind *:80
  mode http
  default_backend stats-back

frontend pxc-onenode-front
  bind *:3306
  mode tcp
  default_backend pxc-onenode-back

backend pxc-back
  mode tcp
  balance leastconn
  option httpchk
  server node1 10.7.13.91:3306 check port 9200 inter 12000 rise 3 fall 3
  server node2 10.7.13.92:3306 check port 9200 inter 12000 rise 3 fall 3

backend stats-back
  mode http
  balance roundrobin
  stats uri /haproxy/stats
  stats auth pxcstats:secret

backend pxc-onenode-back
  mode tcp
  balance leastconn
  option httpchk
  server node1 10.7.13.91:3306 check port 9200 inter 12000 rise 3 fall 3
  server node2 10.7.13.92:3306 check port 9200 inter 12000 rise 3 fall 3 backup

Install clustercheck on nodes

Install Clustercheck on each db nodes. Clustercheck is checking mysql health and display the status on web port 80. So that HAProxy knows which node is live and available.

First create clustercheckuser on mysql.

GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!'
FLUSH PRIVILEGES;

Then download clustercheck from git repository and place into /usr/bin/clustercheck on node server.

git clone git@github.com:olafz/percona-clustercheck.git
mv  /root/clustercheck /usr/bin/clustercheck

The edit the downloaded file.

vi /usr/bin/clustercheck

Here’s important part, there’s typo in the programming where the mysql username and password recorded. Fix it like below, you can change the user name and password but it should be matched with the mysql user information created above. For me, this took an hour to find out this bug. No body reported this bug on the git repository although this is at least 3 years old.

MYSQL_USERNAME="${MYSQL_USERNAME:-clustercheckuser}"
MYSQL_PASSWORD="${MYSQL_PASSWORD:-clustercheckpassword!}"

Add clustercheck in mysqlchk

Configure mysqlchk file and designate where is clustercheck file located. (/usr/bin/clustercheck)

vi /etc/xinetd.d/mysqlchk
# default: on
# description: mysqlchk
service mysqlchk
{
        disable = no
        flags = REUSE
        socket_type = stream
        port = 9200
        wait = no
        user = nobody
        server = /usr/bin/clustercheck
        log_on_failure += USERID
        only_from = 0.0.0.0/0
        per_source = UNLIMITED
        
}

Install xinetd

xinetd is service where we can monitor using port 80. Add mysqlchk service in the xinetd.

vi /etc/services

Then searching for xinetd and add after below.

mysqlchk    9200/tcp    # MySQL check

We need to install xinetd if it is not installed.

sudo apt-get update -y
sudo apt-get install -y xinetd

Start xinetd using below command.

sudo service xinetd start

You can check health status of nodes from web browser, port 9200.
http://10.7.13.91:9200/
http://10.7.13.92:9200/

Make sure the message saying: Percona XtraDB Cluster Node is synced.
If it says Percona XtraDB Cluster Node is not synced. then check if the clusteruser login information matched with mysql user and credential on the file (/usr/bin/clustercheck)

You can also check through terminal.

curl http://10.7.13.91:9200/

Connectivity from web server

From web server, check if the connection to db node is working through port 3306.

mysql -uyourmysqluser -p -P 3306 -h 10.7.13.91 -e "show variables like 'wsrep_node_name';"

If there’s no problem, also check connection using port 3307, through HAProxy.

mysql -uyourmysqluser -p -P 3307 -h 127.0.0.1 -e "show variables like 'wsrep_node_name';"

If everything works fine, you will see below and now mysql is connected using host 127.0.0.1 and port 3307.

Installing Percona XtraDB Cluster on Ubuntu 18.04

Mysql DB is great, it’s free and has good performance. Also has good combination with Apache and PHP.

But we need to use multiple DB servers for High availability. And the each db clusters need to be synced and keep the same data for each server while it delivers or updates records.

Percona XtraDB could be great solution for it. I would like to present how to install percona XtraDB on 3 servers.

We need to have 3 DB servers with Ubuntu 18.04. The original documentation can be found on Percona website.

Open Ports

I set it up with 3 servers, with these IPs.

Percona1 - 10.0.21.1
Percona2 - 10.0.21.2
Percona3 - 10.0.21.3

Before starting, some of server ports need to be opened for communicate each other.

Open 3306, 4444, 4567, 4568 ports for each servers. And open the ports using below commands.

iptables -A INPUT -i eth0 -p tcp -m iprange --src-range 10.0.21.1-10.0.21.3 --dport 3306 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m iprange --src-range 10.0.21.1-10.0.21.3 --dport 4444 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m iprange --src-range 10.0.21.1-10.0.21.3 --dport 4567 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m iprange --src-range 10.0.21.1-10.0.21.3 --dport 4568 -j ACCEPT

If you are using AWS, you will need to open the ports as well on security groups. Just like this.

Remove apparmor or Mysql

I strongly recommend do not install mysql before installing Percona DB. It will conflict and may not working properly.
Also remove apparmor before installing.

sudo apt-get remove apparmor

Install Percona XtraDB package

Using below command, install package. During installation, you will need to set up root password.

wget https://repo.percona.com/apt/percona-release_0.1-6.$(lsb_release -sc)_all.deb 
sudo dpkg -i percona-release_0.1-6.$(lsb_release -sc)_all.deb
sudo apt-get update
sudo apt-get install percona-xtradb-cluster-full-57

Prepare same for all 3 servers. If you are using cloud service, like AWS, you may want to take snapshot and duplicate the Percona servers.

Login into all servers and stop mysql service.

sudo service mysql stop

Config mysql on master

I have prepared these 3 nodes.
Percona1 – 10.0.21.1
Percona2 – 10.0.21.2
Percona3 – 10.0.21.3

And config mysql on master server(10.0.21.1). All servers will be set up as master, but at the beginning, we need to designate one server for master and others could be synced into it.

vi /etc/mysql/my.cnf
[mysqld]

wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://10.0.21.1,10.0.21.2,10.0.21.3

wsrep_node_name=pxc1
wsrep_node_address=10.0.21.1

wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sstuser:sstuser_password

pxc_strict_mode=ENFORCING

binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/percona-xtradb-cluster.conf.d/

Set all node with same configuration but you will need to change wsrep_cluster_address for all your nodes and configure below line.

wsrep_node_name=pxc1
wsrep_node_address=10.0.21.1

Also set your sstuser and password for this line:

wsrep_sst_auth=sstuser:sstuser_password

Create SST user

Login to each nodes and login to mysql. Then create SST users.

CREATE USER 'sstuser'@'%' IDENTIFIED BY 'sstuser_password';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'sstuser'@'%';

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'sstuser_password';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

Open port from terminal

Double check if port is opened and it communicate each other.

iptables --append INPUT --in-interface eth0 --protocol tcp --match tcp --dport 3306 --source 10.0.21.0/24 --jump ACCEPT
iptables --append INPUT --in-interface eth0 --protocol tcp --match tcp --dport 4567 --source 10.0.21.0/24 --jump ACCEPT
iptables --append INPUT --in-interface eth0 --protocol tcp --match tcp --dport 4568 --source 10.0.21.0/24 --jump ACCEPT
iptables --append INPUT --in-interface eth0 --protocol tcp --match tcp --dport 4444 --source 10.0.21.0/24 --jump ACCEPT

Run first node

For the first node, this need to be executed.

/etc/init.d/mysql bootstrap-pxc

After that check status with mysql command below.

show status like 'wsrep%';
wsrep_local_state_comment should show as synced and wsrep_cluster_size will be 1

Run all other nodes

Run 2nd nodes using mysql command and check the status

/etc/init.d/mysql start
show status like 'wsrep%';
wsrep_cluster_size should be 2 and wsrep_local_state_comment should show as Synced

Run all other nodes with same way. wsrep_cluster_size should be same node number as all node servers at the end.

Delete ._ files using linux command

I found out my files are corrupted files starting with ._
I don’t know why / how these files created but I don’t like all duplicated with trash files.

I guess this is created when I copy files from my mac computer to the server. Any how, I wanted to delete these files.

Solution is simple, run this command then it will delete all the sub directories recursively.

find . -iname '._*' -exec rm -rf {} \;

Vagrant error: failed to open /dev/vboxnetctl: No such file or directory

I haven’t used vagrant + virtualbox for a while and an error shows up when I tried to reuse.

I have googling about it and I found solution.

sudo /Library/StartupItems/VirtualBox/VirtualBox restart

But I got different error when I tried this.

sudo: /Library/StartupItems/VirtualBox/VirtualBox: command not found

And I have checked vagrant version.

vagrant version
Installed Version: 1.9.1
Latest Version: 2.2.3

I found my vagrant version is quite old. So I have downloaded new vagrant file and reinstalled. You can download updated version from Vagrantup.com

Also I have upgraded Virtualbox as well.

Then Vagrant start working as before.

$ vagrant up

It’s always good to have things are updated.

Switch to HTTPS, Install free SSL Certificates using Let’s Encrypt

SSH, also known as Secure Socket Shell, is a network protocol that provides administrators with a secure way to access a remote computer. SSH also refers to the suite of utilities that implement the protocol. Secure Shell provides strong authentication and secure encrypted data communications between two computers connecting over an insecure network such as the Internet.

Without SSH, browser marked as “Not Secure”  on the address line in Chrome.  This is essential to have secure browsing.

If you have SSH access, use command line.

1. get cerbot

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install python-certbot-apache

2. Then install cerbot

$ sudo certbot --apache

3. Automating renewal for testing

sudo certbot renew --dry-run

4. Add below to crontab for auto renewal

43 6 * * * certbot renew --post-hook "systemctl reload nginx"

If you don’t have access to SSH then check control panel if there’s Let’s Encrypt option.

In my case,
1. After login plesk control panel, go to Let’s Encrypt.

Let's Encrypt

2. Then enable it.

Let's Encrypt

3. If your website is not redirecting to https automatically, go to host settings and setup redirect from HTTP to HTTPS.

Let's Encrypt

Change server timezone on Linux CentOS

Here’s way to change timezone on CentOS.

  1. Access to SSH through root account.
  2. Check timezone setting using date command.
  3. Check the timezone list using timedatectl list-timezonescommand.
  4. Change timezone, for example.
    timedatectl set-timezone America/New_York
  5. Check if that applied using date command again.

mysql allow remote access from any host

Granting mysql access from remote is not secure, so it’s not recommendable but sometimes we need to do it. Here’s how to do.

  1. Login to mysql with root and create user with remote connect.
    GRANT ALL PRIVILEGES ON remote_db.* TO 'remote_user'@'%' IDENTIFIED BY 'passwordhere';
    
    FLUSH PRIVILEGES;
  2. We need to allow remote connection from mysql configuration. mysql.cnf or mysqld.cnf files locations are differ but in my case, it is located in /etc/mysql/mysql.conf.d/ folder.
    vi /etc/mysql/mysql.conf.d/mysqld.cnf And comment out  below line.
    #bind-address = 127.0.0.1
    After that mysql restart desired.
    service mysql restart
  3.  Now if you have any firewall or network security, allow the port. In my case, I needed to allow 3306 port in AWS inbound rule.
  4. Try to connect from remote.

    mysql -u {username} -p -h {ip address}

    mysql remote connection
    mysql remote connection

Mysql root password reset on Ubuntu

Stop mysql service

sudo /etc/init.d/mysql stop

Start without password

sudo mysqld_safe --skip-grant-tables &

Connect to Mysql

mysql -uroot

Reset password

use mysql;

update user set password=PASSWORD("passwordgoeeshere") where User='root';

flush privileges;

quit

Restart mysql

sudo /etc/init.d/mysql stop
sudo /etc/init.d/mysql start

Amazon S3 folder open to public with bucket policy

When upload file to S3, files are not allowed to open to public.
We need to make policy to open public.

There’s Bucket Policy on permission tab.
Add policy with Json format like following:

{
    "Version": "2012-10-17",
    "Id": "AWS-some-name-of-id",
    "Statement": [
        {
            "Sid": "AWS-some-name-of-id",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::arn-url-get-from-above/*"
        }
    ]
}

We need to keep Version date, it’s not today’s date, but it is Amazon policy version date.
Also Resource ARN code could get from above the policy editor.

Varnish command

Varnish is web application to increase web contents delivery time by cache. It can be installed in front of any servers with HTTP. Varnish is very fast, so it can speed up 300 to 1000 times faster.

Using Ubuntu, here’s varnish config file located:

vi /etc/varnish/default.vcl

After config varnish vcl file, you can test it using this command:

varnishd -C -f /etc/varnish/default.vcl

To varnish start / stop / restart:

service varnish start
service varnish stop 
service varnish restart

To view varnish status:

service varnish status

In my case, varnish stopped working with no reason.
I found out that varnish disk was full, so I had to find out which file has the most big size.

du -h / | grep '[0-9\.]\+G'

And I narrow down and found varnish log file was too big.
After delete the file and start, it worked as normally.

Happy caching!
Bravo Varnish!