Monitoring Server Port Availability with Powershell

I recently ran into an interesting issue with some web front ends in various Windows Network Load Balance clusters. The servers were configured with 2 NICs each: one for general networking and one to be a member of the NLB. Occasionally, the NLB NIC would become unresponsive. The adapter state still showed as connected. The NLB itself was fully converged, and NLB hostname was pingable. However when that NIC was pinged, it would time out, and the service offered by the NLB would fail if your session went to that member.

nlb-01

The ultimate root cause was a problem with the virtual hardware configuration within a farm, but until that issue could be isolated, I needed to know when this was occurring. The difficultly with this was that the server is up, all services and components on that server are running, and the NLB is converged. Transaction monitoring only had a 50/50 shot at best of catching this because the transaction may go to the working NLB members. Likewise, I wasn’t convinced that simple up/down monitoring on the NLB NIC IP would catch this. Yes, pings appeared to timeout, but I didn’t want to risk a ping being returned, but the application port itself being inaccessible.

My solution was to script the creation of a TCP client connection on the application port. If that connection failed, then an alert would be sent out. Since these were web front ends, the port would be 443. I didn’t work on HTML formatting of the email since this is to be a temporary fix, but getting it too look pretty with colors and bolding would be handy for a longer term solution.  This is what the code looks like:

Client Connectivity Monitor
# This is a simple script to monitor the load balancer NICs
$serverlist = “server1-lb.domain.com”, “server2-lb.domain.com”
# Nulls out variables from previous run in case they are present$errorserver = $null
$errorpresent = $null
$testconnection = $null# Processes the front end servers
ForEach ($server in $serverlist)
{
$friendlyname = $null
$testconnection = (New-Object System.Net.Sockets.TCPClient -ArgumentList $server,443)
If ($testconnection -eq $null)
{
If ($errorpresent -eq $null)
{
$ErrorPresent = “Yes”
$friendlyname = $server -replace “-lb.*”
$errorserver = $friendlyname
}
Else
{
$friendlyname = $server -replace “-lb.*”
$errorserver += ” and “+$friendlyname
}
}
}

# Sends a message if errors are found
If ($errorpresent -ne $null)
{
$messagesubject = “CRITICAL – $($errorserver) connection error!!”
$messagebody = “There is a connectivity issue with $($errorserver) to port 443 of the LB NIC interface. This requires immediate attention and may be causing a client connectivity issue.”
$recipient = “<{recipient@domain.com>”
$sender = “Client Connectivity Tester <sender@domain.com>”
Send-MailMessage -smtpserver relayserver.domain.com -To $recipient -Subject $messagesubject -From $sender -Body $messagebody
}

As you may have guessed by looking at the code, the NLB NICs do have a DNS name of {hostname}-lb. I strip out the “-lb” in the notification for clarification’s sake. I have a scheduled task running this script every 10 minutes. It allowed us to head off client connectivity issues at the pass, and it ensured that when an issue did start, it didn’t last long.

This turned out to be a simple but effective piece of scripting that I’ll probably reuse a lot during a lot of deployments or for testing connectivity when needed.

 

 

 

tales01

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s