SFB CloudPBX Load/Stress Testing Made Easy

A Must Do task before (even after) we deploy CloudPBX, What we as well as customers should be fully aware of, is that how a customer’s network is performing under a duress and contentious concurrent Audio traffic load.

Also, whether a customer’s network meets network performance requirement/benchmark recommended by Microsoft. So, two main critical objectives here:

1. To have full understanding and visibility on how a customer’s network is performing under a stress and contentious concurrent Audio traffic load.

2. To determine and pinpoint whether a customer’s network meets network performance requirement/benchmark set by Microsoft

The most critical aspect of SFB CloudPBX is practically Audio traffic while all other traffics such as Video, and Application sharing are just unifying features rather than fundamental or survival ones.

So, now we need to generate Audio traffic load from users’ network(s). The tool that I am going to use to do just that is Microsoft Network Assessment ToolRef: https://www.microsoft.com/en-us/download/details.aspx?id=53885

And I am largely following Skype for Business Operations Framework guidelines with my own automation and customization approach to fit in a common customer’s network environment.

SOF Ref: https://www.skypeoperationsframework.com/Downloads?SelectedIDs=4_2_0_1

Under “Determine Network Readiness”

1 -Assess-Network Readiness Assessment-Delivery Guide-V4 6 - Assess-Network Readiness-Tool Guidance V2.1 7- Assess-Network Readiness-Tool Result Guidance V2.1

Approach in running MNAT

Download MNAT (Microsoft Network Assessment Tool) and extract it on users’ PCs – may be under C: drive as c:\mnat. You could use USB stick to copy and paste it on all users’ PCs or use a network share to copy and paste the extracted MNAT folder.

MNAT use a command line to generate an audio traffic lasting 17 seconds (you can’t change 17 seconds call duration)

MNAT should run on, if possible, all users’ PCs or at least on 10-to-20 PCs based on possible numbers of concurrent users – the point is that we need to generate 10-20 or your desired concurrent audio calls traffic using users’ PCs.

o Don’t run 2 MNAT command shells on the same PC – it doesn’t make sense – a user does one call at a time – not two calls at a time. That means if you want to generate 2 concurrent audio calls – user two PCs running MNAT command each.

Since MNAT must be run by a user, I have created a script to automatically run it.o My approach is that a PC will run it for an hour 2 times a day of you choosing,

for example, peak hours like daily 9:30 am and 2:00pm for 2-3 days or a week through Microsoft Windows Task Scheduler on users’ PCs while they are doing their daily activities as usual.

o The reason we need to run it for daily at a certain time for 2-3 days or a week is to ensure that we have collected a reliable traffic usage trend by users in the network(s)

o Try to run and collect the usage from users’ PC that they station in the office. Don’t run it on users who are frequently outside.

o If you want it to run more frequently, for example, run an hour 3-4 times a day, just add your desired time in the Microsoft Windows Task Schedulers accordingly.

o By default, the MNAT command shell windows pops up when it runs but I customized a script to run it silently in the background – it means users won’t notice when it’s running and their foreground windows are not disturbed.

Implementation

Download and extract MNAT and extract it under c:\mnat as follow.

Copy and paste mnat_generate_audio_traffic.bat and mnat_silence_run.vbs into c:\mnat folder as follow

mnat_generate_audio_traffic.bat uses the following commands; essentially, it’s running Audio traffic generation tool, NetworkAssessmentTool.exe, once it’s done running, copy result.tsv file as “results_date_time” format.

Since result.tsv file will be overwritten every time “NtetworkAssessmentTool.exe” runs. And we need to run it for 2 times a day for 2-3 days or a week, it’s important that we keep all result file for later compilation. I’ll detail more about it later.

mnat_generate_audio_traffic.bat@echo offcd c:\mnatNetworkAssessmentTool.exexcopy c:\mnat\results.tsv "c:\mnat\results_%date:~10,4%-%date:~4,2%-%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,5%.tsv*"clsexit

mnat_generate_audio_traffic.bat pops up and run in foreground, we need it to run silently and in the background without even users noticing it.

So, I use “mnat_silence_run.vbs” script and nested mnat_generate_audio_traffic.bat in it as follow. And then use “mnat_silence_run.vbs” in Windows Task Scheduler to generate the audio traffic workload at a specified date and time.

mnat_silence_run.vbsSet WshShell = CreateObject("WScript.Shell") WshShell.Run chr(34) & "C:\mnat\mnat_generate_audio_traffic.bat" & Chr(34), 0Set WshShell = Nothing

You also need to customize “NetworkAssessmentTool.exe.config” file to suit your requirements as shown below.

For example, if you need to generate 1 hour amount of audio traffic, you must set “<add key="NumIterations" value="157"/> in “NetworkAssessmentTool.exe.config file and save it.

There is nothing much to change, leave all value as default as possible, apart from number of iterations since we want it to generate 1 hour amount of audio traffic workload.

The next task it to put “mnat_silence_run.vbs” in the Windows Task Scheduler to run it at a specified time.

I am running it at 9:30 am and 2:00 pm – 2 times every day. It’s important that you set the same timing and steps for all users’ PCs to simulate concurrent calls at the same time.

Choose c:\mnat\mnat_silence_run.vbs

For a test run, modified your networkassessmenttool.exe.config file to 5 iterative calls “<add key="NumIterations" value="5"/>” and right-click the task and run it manually to see how it goes. You should see something like that below.

Please note that the audio call duration is 17 seconds + 5 sec waiting before to initiate 2nd call = 22 seconds per call = for 5 calls it will take 22 seconds * 5 = 110 sec / 60 = 1.833 minutes to complete this task.

I manually ran this task at 9:09 AM today, 28-05-2017.

Go to c:\mnat\ you will see the following new file, results_results_2017-28-05_ 9_11_31.70 – in fact it’s a copy of results.tsv file and copied as results_results_2017-28-05_ 9_11_31.70 automatically by the script, mnat_silence_run.vbs by task scheduler.

Results.tsv file contain network performance metric/data that was generated by NetworkAssessmentTool.exe.

We’ll use these values to determine how our network(s) is performing under 10/20/50/100 etc concurrent audio calls load and whether if our network meets the network performance benchmark required by CloudPBX.

Open results*.tsv files using Microsoft Excel.

Remember, every time we run audio load test, result.tsv file will be overwritten, so the script will automatically copy and save as results_date_time so that we could use them later for traffic performance data compilation and performance result assessment.

Once it’s confirmed running smooth, modify your networkassessmenttool.exe.config file to 157 iterative calls “<add key="NumIterations" value="157"/>” to get it run (generating audio traffic load) for an hour.

Gathering and Accumulating Results

Assumed you run (generate) “audio traffic load” script 2 times a day for 3 days, you’ll get 6 results_date_time.tsv files per PC/user. As mentioned before every time we run audio load test, results.tsv file will be overwritten.

So, open earliest results_date_time.tsv in Excel, open second earliest results_date_time.tsv file in Excel and copy all metric values but leaving header into earliest results_date_time.tsv file in Excel – well, essentially you are combining performance metric values from all other 5 results_date_time.tsv into earliest results_date_time.tsv in Excel to become a single consolidated results_earliest_date_time.tsv file as follow.

For me, I just ran 5 audio load calls for 3 times manually to demonstrate the purpose.

Red is the earliest metrics follows by second (blue) and third (green) earliest network performance metrics and so on. That’s how you compile all performance metrics into a single consolidated results_earliest_date_time.tsv file.

Getting average(consolidate) network performance results.

ResultsAnalyzer.exe has an ability to analyse and output an average network performance metrics from a consolidated results.tsv file or results_date_time.tsv file

Now you have a consolidated results_earliest_date_time.tsv file, the final step is to run ResultsAnalyzer.exe against results_earliest_date_time.tsv

As shown above, average consolidated performance metric under “Client to Microsoft Edge” category:Packet Loss Rate: 0.41814348% is less than 1% = PASSEDRTT Latency is less than 12.9 is less than 100ms = PASSED Jitter 17.728386 is less than 30ms = PASSEDPacket Recorder Ratio 0.34610632 is MORE than 0.05% = FAILED.

Now you know which aspect of network, in our case, “Packet Recorder Ratio” doesn’t meet the target/benchmark number recommended by Microsoft.

Without this audio load assessment tool, we wouldn’t know how a network is performing and if it meets CloudPBX Network Target/Benchmark recommended by Microsoft.

Now we know what is going on, and where we are failing.

You would need to consult with your network engineer on how to improve “Packet Recorder Ratio” from your network going to Internet (Microsoft Edge).

This in fact is a network performance by a PC/user, and remember that we are running this audio load tool on 10-20 PCs. How are we going to do about it?

You could do the same steps and approaches mentioned above on each PC one by one, if you have 10 PCs, you will have 10 consolidated network performance results.tsv files and 10 varied network performance metric results after checking them against ResultsAnalyzer.exe.

Or in fact you can also consolidate all 10 (assumed u ran it on 10 PCs) varied consolidated results.tsv files into one big consolidated results.tsv file and check it against ResultsAnalyzer.exe to get 1 single consolidated (average) network performance metric results.

For example, you run it 2 times a day for 3 day = 6 files then they are consolidated into 1 consolidated file per PC.

If you have 10 PCs = 10 consolidated files then they are consolidated again into 1 big consolidated file for 10 PCs.

It’s important to take note that the audio load we did so far is from client network(s) to Microsoft Edge (anycast IP13.107.8.2) but there is one more endpoint we need to test as well which is from your Edge Router to Microsoft Edge (Internet - 13.107.8.2) .

You could use a PC connected directly to your network edge router to perform audio load like the way we did from client network(s) to Microsoft Edge (anycast 13.107.8.2 defined in NetworkAssessmentTool.exe file) above.

Once you have a consolidated results.tsv file or results_date_time.tsv file, run it against ResultsAnalyzer.exe to get consolidated (average) network performance metric results then compare it with recommended CloudPBX performance requirement by Microsoft to realize if your current network meets the target/benchmark under “Customer Edge to Microsoft Edge” category as follow.

In my point of view, since CloudPBX users are directly register and use Microsoft Edge directly for most of the workload such as audio, video, conferencing and IM, it’s more than good enough that if you only measure from Client network(s) to Microsoft Edge. Therefore, audio load assessment from customer edge to Microsoft Edge is an optional to me.