Thanks for answering!
No firewall (host-based/network Firewall) blocking connection to RD Gateway URL. Using Microsoft fabric as DNS.
WVDAgentUrlTool.exe shows all addresses are accessible.
Agents are all registered and show healthy in Pool settings.
Session host performance is normal and shows a standard workload (based on Azure performance metrics)
The issue being sporadic is not helping at all. It randomly stops communicating on random servers with rd gw.
We are using Azure Load Balancer for outgoing communications. So that's the IP address that Azure RD gW fabric sees. I thought it might help.