One of the most painful outcomes of using PaaS services is the fact that they expose their endpoints via FQDN that resolves to a dynamic IP List that the cloud provider might change.
If you need to connect from on-prem to your PaaS service and whitelist that service using an IP address, you are in trouble!
At least at Azure, you can get the IP list of Service Tags using REST, Powershell, or Azure CLI. But, let’s say that you provide a solution for thousands of customers – do really want the headache of following up and updating them with IP list changes? And then waiting for them to implement the changes on their network appliances?
I had to develop a workaround for a customer that provides an IoT solution to thousands of clients. Each client had my customer’s IoT devices on-prem and needed to send data through their own Network Appliances (like Firewall, Proxies, etc) to Azure IoT Hub, which as we all know exposes FQDN that resolves to dynamic IP address.
My workaround involved creating the following Azure components:
- Azure Load-Balancer Standard SKU
- VMSS with HAProxy VMs (Spot)
- Shared Image Gallery
- Private link
- IoT Hub
- Google Cloud GKE - for POC and load testing the solution that has been created in Azure.
I used Hashicorp Packer to create the HAProxy images and push them into the Azure shared image gallery.
As VMSS VMs are stateless and used only for HAProxy, to reduce costs I used Spot VMs.
Data flow explained
IoT devices at the clients’ sites send data to an Azure load-balancer that has a static PIP (Public-IP).
Each client can whitelist the IoT hub connection, by configuring its network appliance to allow outbound traffic to that IP.
From the load-balancer, traffic flows into VMSS with HAProxy installed, and HAProxy reroutes traffic to the Azure IoT Hub private link IP address (Private IP).
We configure the Azure NSG, to allow traffic from our client’s IPs and on port 443 and MQTT (8883) and/or AMQP (5671).
We then configure the IoT hub to admit traffic only from private links, thereby creating a much more secure environment.
Automation Explained
Packer
For the VMSS HAProxy, I used Hashicorp Packer ARM builder (HCL2) which creates the images and pushes them to the Azure shared image gallery.
Be aware that the shared image gallery and image definition need to be manually created beforehand, using IaC on manually !
Packer code
HAProxy.cfg
We can configure the Azure VMSS to do a rolling update whenever we push a new HAProxy image to the Azure shared image gallery.
Load testing our configuration
For load tests, I used Microsoft’s IoT Telemetry Simulator. The repo contains a helm chart that we can deploy and simulate thousands of devices that send data to our IoT Hub.
For the purpose of resolving my IoT Hub FQDN to the load-balancer Public IP, I added it to the helm chart deployment.yaml (lines 30–33) and the corresponding values in values.yaml (lines 27–28).
deployment.yaml
values.yaml
I deployed the helm chart into GKE. All this was performed with the following pulumi code
__main__.py
Pulumi.dev.yaml
Conclusion
Although our solution is not fully supported by Microsoft, and there are a number of ‘moving’ parts involved, we did manage to solve our IoT Hub Dynamic Public IP address issue, and achieved a more secure IoT Hub flow.
Just remember to POC the solution properly and load test it to resemble your production environment.
