By Sumanth MS, Sreekanth G
Have you heard of something like Parallel test execution ever? People who usually write unit/integration or Ui tests (Browser automation using selenium) might have heard this or may be even doing parallel execution in there development/test environment. With today’s mode of software execution, when using a software execution model like agile with continuous delivery and continuous integration your tests should be compatible for parallel execution and they should be executed within a very short span of time. Now a days industry is going with incremental and sequential changes in apps and releasing them in sprints and with Continuous Integration & Continuous Delivery, releases are even more frequent. Thus, for testing in an agile environment, you must be ready to make changes in test requirements and scenarios as and when your application undergoes changes. By using parallel execution, we can reduce the ‘execution time’ as tests are executed simultaneously in different threads, it is always important to test application in different browsers. We can perform automation on multiple browsers using selenium and testing. If there are more number of tests that need to be executed parallel on different browsers also, we can do this using parallel execution testing. And there is another challenging task such executing tests in different browser versions and different Operating systems. This can be achieved with parallel test execution.
This is a fully supported, open source and cross-platform implementation of the MSTest test framework which can be used to write tests targeting .NET Framework, .NET Core and ASP.NET Core on Windows, Linux, and Mac.
MStest V2 offers test case execution at assembly, where it will force us to split the dll’s for parallel execution, making tedious job. So now maintainers are working on this to correct the approach of parallelism. Since it is in development mode we don’t know yet when it will ready and how it’s going to be (usefull or not).
To avoid waiting time and to achieve the parallel test execution using MStest v2 we have come up with a unique idea of executing the test cases using the command line version of VSTest. We have developed a console application which takes the advantage of the cli feature which MStest offers and executed our test cases which is now numbered around 1000 and 1500.
In the console application, that we have developed, we are expecting to give a simple command for the execution to take place. You can find the vstest console.exe usually in your visual studio installation path. For me following is the path:
Below is the example command to list the tests in the assembly or dll which we often use:
>vstest.console.exe “<Your Project path\bin\Debug\project.dll” /Settings:”You can mention any settings” /Tests:TestName /logger:trx
However, the above does execute only a single test case, and if you give the option as /TestCaseFilter:”TestCategory=” then all the test cases under that category would be executed but sequentially. With our console application we have use the above test category or the assembly or test cases format as given in the command prompt and execute them as each test case if invoked individually.
Following is the simple command format for our console app which expects:
-TestCategories: -ThreadCount:1 -Browsers:chrome/logger:trx
In the console app we determine the pass argument is a test case or a test category. If there are some test cases given, then we split them up based on the comma and add them to a list of test cases to be executed. Then each test case is given as command to vtest .console.exe. Our console app forms the command for each test case like as shown below:
>vstest.console.exe “test.dll” /Tests: /Settings:”.runsettings” /logger:trx. Each such command is formed for all the test cases defined. If its a test category we firstly get all the test cases from the category and for each test case we form the same command as shown above.
All such formed commands are given to another class where C# language Parallel class is used to iterate over the list of commands formed.
Below is the snapshot of how we iterated over the commands and execute each of test case as invoked individually.
The other two arguments what our console application expects is the thread count and type of browser.
With the thread count as 5, our console application would invoke 5 parallel browsers and execute the test cases on them. Other argument is the browser, on which type of browser you want the test cases to get executed.
Our console application can also be integrated with Jenkins. Jenkins offers the execution of batch command using its build section. You can give the batch execution command in the following way. Below is the example of how we integrated our console application to Jenkins.
Below is the snapshot in jenkins:
In the above snapshot -ThreadCount: is the number of browsers to be invoked parallel and executed upon. Browsers:, you can have browsers separated by comma to get executed.
It is just a work around what we have come up in the absence of parallel execution in MSTest V2 at the test class or method level.
One can go through the RFC proposed to be implemented in MSTest V2 by going through the below link: https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md
Also can give some suggestions or requirements as below example:
Some test case shouldn’t be executed in parallel as they may cause other tests to get failed. To explain that context, a test case where you’re trying to change password for an user and the same user is being used by some other test cases for other functional test cases then other test cases will get failed because of password change. You can give your feedback to MSTest at the github to introduce attributes such as [No Parallel] where would allow you to exclude this test case from parallel execution.
Such kind of [No Parallel] attributes can be given either at class level or assembly level to exclude test cases from parallel execution. Compared to other test frameworks report generation such as TestNG and Extent reports MSTest trx report are not very much informative. Extent reports generate pie chart and bar diagrams for the execution whereas MSTest doesn’t have any such.
Like above similar such suggestions where testers feel it should be included into MSTest can be made at the GitHub.