How to handle failed test in pipeline

Types of e2e test failure in Jenkins pipeline (https://build.tis.nhs.uk/jenkins/job/e2e-tests/) :

• Test failed due to an application bug
• Test failed due to regression failed introduced by some changes
•  Test failed due to poor application performance forcing timeouts
• Test being flaky

1. Test failed due to an application bug: If Test starts failing because of a bug exist in application than a ticket should be created after review the failing test and PO should be informed about the bug so that based on the severity of the bug, that can be fixed before deploying code to production and prevent any severe bug occurrence in production.

2. Test failed due to regression failed introduced by some changes: Test failed due to regression failed for any/multiple components introduced by some changes, a ticket should be created after the review of failing test and PO should be informed about the bugs so that based on the severity of the bugs, they can be fixed before deploying code to production and application is usable by end user.

3. Test failed due to poor application performance forcing timeouts: Test might fail due to poor application performance since TIS components performance is not consistent and sometimes some changes introduce performance degrade in the application which causes the test to fail. If the test is failing due to this reason a ticket should be created for failing the test due to performance and preventing them to execute to test any functionality. If application performance is consistently poor, a ticket to fix that performance issue should also be created so that based on the priority of the issue, it can be fixed and e2e tests can test the application, any changes going in production without any timeout error and end users don't face performance issue while using the application.

4. Test being flaky: Test failed due to test being flaky sometimes, due to data issues etc. A ticket should be created so that the flaky test can be fixed and can be executed again so that we remove the risk of deploying untested functionality in production. Flaky test can be avoided till they are not fixed and we should make sure we test the avoided test functionality manually If any changes need to get deployed in production to make sure there is no bug exist in the application.

Note: When we comment any test due to being flaky/ will fix later, we should always create a ticket with stack trace of failing test and add the ticket number in test comment in code so that team can fix those tests based on the tracked ticket.

5. E2E Test Report: E2E test report gets created after each build and an alert notification appears in e2e-results channel in slack so that anyone can view the status of test pack, including the screenshot of the failing test.

Screenshot of the commented e2e test in the pipeline which needs to be fixed and uncommented.