In Patroni We Trust

It’s a funny incident, that occurred in a test environment. We were upgrading the Patroni version from 1.x.x to 2.x.x on all self-hosted nodes in staging. The procedure is pretty simple as per the following instructions.

Previous installations were done by some other dba and dba who upgraded current version didn’t notice that the upgraded executable is in /var/lib/postgresql/.local/bin/ but the systemd unit file we have configured via ansible was expecting it in /usr/local/bin/ so the running patroni instance was still using the old 1.6.5 executable.

But when Patroni version was checked it showed 2.1.7.

After that another DBA and analytics platform team spent hours trying to solve the mystery of why Patroni 2.X.X is not copying logical slot information on switchover as promised. The mystery of Patroni 2.X behaving the same as Patroni 1.X was solved by identifying which version was actually running on the node. That can be done in any of the following ways:

postgres@XXXX-db-01:~$ curl -s XXXX-db-01.tw.ee:8008 | jq
{
...
  "patroni": {
    "version": "1.6.5",
    "scope": "XXXXdb"
  }
...

Or When you use “patronictl version clustername” rather than just “patronictl version”, all nodes in a patroni cluster will show the current version.

Here is a safe way to upgrade cluster, just make sure you know where your binary from previous installation is located.

-- update pip and setuptools before upgrade
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade setuptools

step1 :
patronictl pause
step2 -- on any replica nodes
sudo systemctl stop patroni
step3 -- upgrade
python3 -m pip install --upgrade  --user patroni[etcd]==2.1.7
step4  -- rename old patroni file and create symlink for new
mv /usr/local/bin/patroni /usr/local/bin/patroni.old
ln -s /var/lib/postgresql/.local/bin/patroni /usr/local/bin/patroni
or -- depends on your previous setup
mv /usr/bin/patroni /usr/bin/patroni.old
ln -s /var/lib/postgresql/.local/bin/patroni /usr/bin/patroni
step5 -- start patroni on that node
sudo systemctl start patroni
step6 -- repeat 2,3,4,5 for all nodes and then
patronictl resume

It turns out Bitcoiners were right after all.