We created this whitepaper to provide an essential, independent guide to the testing apps in the Atlassian world. The Marketplace is a vast and rapidly changing, enabling organizations a multitude of choices for many solution areas – of which, testing is well represented. With the gradual demise of the old HP ALM platform, many organizations have realized that they already have and could utilize Jira, an excellent platform to support all of these testing functions.
The term “apps” is used throughout the article; Atlassian’s terms have included also “plugins” or “add-ons”. This paper provides a context for assessing testing apps, identifying the relevant ones for discussion, analyzing their strengths and weaknesses. We are not going to focus on feature matrices – these are easily manufactured and dependent on each app vendor’s own development efforts. Instead, we provide a logical framework to help you select the right tool for you.
Our analysis takes a deeper dive inside the selected apps, to understand how they address the “secret sauce” of testing – how together teams can identify problems and fix them quickly.
Scope
Although described in more detail below, it is important to distinguish Atlassian tools’ functionality from what additional apps have to offer. In this paper, we are not evaluating applications developed by Atlassian, but turning our attention to the add-ons that enhance testing capabilities in Jira.
The vast variety of Jira testing apps can, based on how they’re built, be divided into two categories:
- Jira-based apps – built for Atlassian specifically, entangled and fully integrated with Jira (Cloud / Server);
- Standalone apps – designed to integrate/communicate with Jira (e.g. via REST API calls).
Another means of distinguishing Jira apps, is to assess available hosting options: Cloud / Server / Data Center.
In this analysis, we have focused on the most common scenario, the Jira Server application. We will provide a Cloud vs Server comparison, and a more thorough analysis on standalone apps in our next review. Although constantly improving, today’s Cloud apps tend to be less feature rich than those on Server.
The standalone testing apps in the market have a rich set of capabilities, and are designed to seamlessly integrate with Jira, unlike some available testing platforms (e.g. the old HP ALM). We decided that most organizations will do their own very focused review on standalone testing apps so we remained focused on the Jira Server versions.
Applications Reviewed
In this paper, we focused on three current testing apps, market leaders for Atlassian Jira (Core / Software):
- Test Management for Jira (TM4J)– formerly developed by Kanoah, re-designed and -branded by Adaptavist from early 2017;
- Xray Test Management for Jira (Xray)– first released in 2013 by Xpand IT, and enhanced by the same vendor till this day;
- Zephyr for Jira (Z4J)– launched in 2012 by the company of the same name (Zephyr), this app has just very recently been acquired by Smartbear.
All the above mentioned are excellent apps – but each comes at the testing space from different vectors. Contegix has experience implementing all three. We have practical real world observations of their strengths and weaknesses. It’s this we share.
The Atlassian Stack and Context for Testing
In the diagram below the very basic model of how Jira works is explained. Jira calls all its record items “issues”. Essentially types of issues are created that really can be anything – not only epics and stories but also risks and other project-control records, and potentially tests.
The key is that once issues are created they can be managed in the various facilities that Jira provides such as Filters, Projects and Boards. When considering third party apps an important question is: “how does the application work with other Jira facilities, rather than as a standalone product?”
Conversely what does the new application provide that is not natively available in Jira. A further element is that reporting apps such as eazyBI can generate excellent reports if the items or issues in question conform to the basic Jira architecture displayed below.
The next aspect to understand is how the various core Atlassian applications work together to support the Software Development Lifecycle (SDLC). Testing is best thought of in the context of the SDLC main functions. As an example, when code is built using a build engine such as Bamboo, it will automatically detect and expose any breaks; this in itself is a form of initial basic testing. Moreover, build engines can trigger further automated testing which will create defects that need triage in the development backlog.
Best practices for using Atlassian tools for SDLC is generally as follows:
- Document requirements more completely in Confluence with appropriate flow charts and context of how Epics and stories are mapped against the application. (Later this will form an essential part of the application documentation).
- Create and link the relevant planning issues in Jira (rather than carry all the requirement detail around in Jira). As stories are worked on in Jira, code (branches) are created in Bitbucket that can be linked all the way back to Confluence.
- As each branch is finished it is checked in; Bamboo notices and will automatically build. Testing can then commence.
- As each branch is finished it is committed, Bamboo notices and will automatically rebuild.
Thus traceability is ensured throughout the entire lifecycle, documentation gets centralized in a documentation platform and relationships are clear.
Notionally, Testing could be supported by creating an issue type called “test case” with more functional information carried in Confluence. These test cases could be excluded from the relevant scrum boards. The problem with this is that a test case can be run over and over and in different environments. Thus test in the traditional sense needs more attention.
Testing Solution Architecture Options and Background
So if we add testing into this mix – we essentially need some additional entities or objects. Pure agile calls for a fully integrated testing approach but in reality testing applications will typically support the following functional model described below. Fundamentally, defects certainly need to end up back in the Jira backlog for teams to work on.
They need to be a part of the release management process as well. When reviewing testing apps, how they deal with this matter is a major consideration. Testing specialists will look at the elements below and consider how well supported they are and how manageable the various aspects of the overall process.
Given the needs of Test Planning, Test Cases and Test Execution, the question is how various apps tie these elements together. To answer, we have to incorporate the above with the Atlassian overall model. This helps us truly appreciate how a testing app will work with the Atlassian stack. In the model below we have three stakeholder perspectives:
- The Developer/Scrum team functions
- The specialist Testing view
- DevOps which lives in the middle (serves both of these previous functions)
As can be seen below manual testing is going to generate defects but so obviously will DevOps initiated automated testing.
We appreciate that with agile best practices there should not really be a divide between the core Agile development process and testing but also it’s important to accept that for platforms and other major systems there are often many pre-existing test cases and automated testing scripts, especially for regression testing.
Any solution that enables visibility and reporting across this spectrum will be very advantageous because it will enable teams to determine where there are bottlenecks, and provide full traceability, deployment speed and other key performance metrics. It’s important to point out that the end result of any test is either a pass, or the creation of a Jira issue such as a defect or bug. This Jira issue creation forms the work items in the backlog of the development team.
Thus testing apps ultimately trigger the creation of defect issues that involve the larger whole of the project and team. Because of the highly “intermingled” nature of development and testing we believe that any world class app needs to be “intermeshed” as much as possible with core Jira functionality.
This is in addition to the straight race to “feature richness”, as well as integration with build engines and other related tools.
Downloads & Review
There is one more additional dimension to our analysis and that’s perceived market share. At the time of this review we are showing the relative number of downloads each product has below. In our experience this is indicative of the popularity of the app. We always encourage clients to inspect the marketplace download statistics – just to ensure that they appreciate the feedback and popularity of apps. It’s our belief that Zephyr has been traditionally the market leader and so has dominated the space.
This doesn’t mean it’s the richest testing app in the market. Rather it’s the first mover and current market leader by volume. There is so much feedback on these apps that it’s difficult to intuit anything from this data set. The results though are clear below – Zephyr is larger by 4* than Xray and Xray is 50% larger than TM4J. We know that downloads are not sales, but in this case we’re assuming that Xray and TM4J convert similarly, while Zephyr has a relatively poorer close rate (nevertheless leaving them clearly the market leader).
Overall Results
The testing space is a complicated arena because there are three, very well qualified solutions, competing for your business. Yet as will be seen they come at the challenge from varying directions which means that it’s possible to distill the true differences between them. We feel that it’s important to reflect a number of key considerations in how you view these offerings:
- Mesh with Jira – why? because we believe that testing is not an isolated activity and that teams should test together. Jira has been designed to enable this and is richer for it. Isolated solutions may work in some contexts (complex reporting for instance) but not testing.
- Feature richness – this is hard to argue with and is always a major consideration in this kind of report.
- Market share. Of course any first mover will achieve such an advantage but it’s important not to overlook what features or even lack of complexity might be driving this. Success cannot be overlooked.
As an FYI we reviewed pricing and did not consider it a material consideration.
Below is our Jira Testing app excellence matrix:
What does this mean to you the buyer?
We think that it’s important to consider your own context.
- A) If you are a smaller organization with say less than 100-200 IT staff and want a simpler application for testing – Zephyr could be a great choice.
- B) If you are a larger organization and testing is fairly compartmentalized and all the features of the app, being more “meshed” with Jira are not important, then look at TM4J.
- C) If you are a larger enterprise and looking for a full strength, highly configurable solution for testing – go for Xray.
It doesn’t surprise us that all three products are finding success, likely because of some of the factors identified above.
Want more? We have more.
This is a short version of a longer research paper. To get the research version please use below opt-in to download.
Editor’s Note: This whitepaper is branded as Blended Perspectives as it was created prior Blended Perspectives being acquired by Contegix.