That month, I had to build a cluster of nodes with automatic scaling on Azure. As I’m not an expert on K8S, and by lack of time, I decided to build a master Virtual Machine and use it to create the future nodes.
This master VM includes an open source product that needed a smtp relay email notifications through an external Email Provider. I chose Postfix for that.
The app ran perfectly on this VM. That would not be the case later in the cluster.
Next, in order to create a Virtual Machine Scale Set (VMSS) on Azure, I needed to create an Image and a Version from this VM.
I decided to use automated scaling. That means that each time a criteria of my choice, say CPU load percentage, has been reached, Azure will create a new node from the version of the master VM. You need to set the default number of instances (nodes) at the initialization of the VMSS. I chose three.
Everything ran fine. Then I ran a performance test tool like Gatling to stress the cluster and force it to scale up to 40 instances. My tests consisted in displaying the subscription page and edit an email as a username. This action should result in an email notification to confirm the email address.
I observed that many emails were not sent.
I began a boring job to check each postfix log (/var/log/mail/mail.log) of these 40 instances and saw that the smtp relay didn’t work on some nodes.
In Postfix logs, I saw that the smtpd daemon had a DNS resolving error on some nodes, not all, but some.
After a heavy googling, I found this 10 year-old fantastic archive :
So, by default, resolv.conf file is copied by /etc/init.d/postfix at launch.
To disable it :
I disabled smtpd chroot in master.cf in the master VM I used to create.
nodes.sudo nano /etc/postfix/master.cf
--- set smtpd to N
sudo systemctl restart postfixRecreate an Image, Version and VMSS with it.
And all worked fine !
Conclusion :
Postfix documentation needs a new UI for the sake of Developer Experience.
Postfix is huge, take the time to understand how it works : RTFM.
