E2E Tests Notes Part 1 & 2

• perfect code that never fails - don't need tests?

(AN) - @Philip Wilsdon (Unlicensed) I don’t understand what this means, can you (or someone else) clarify, please?

Came from your comment, see 36 seconds in video 1

• Make it harder to break production

(AN) - Absolutely

• Nightly tests - smoke - already have some stories in place for this

(AN) - Smoke tests in Prod

• Testing user journey - start to end

(AN) - Absolutely

• Testing of a feature based on acceptance criteria in our stories

(AN) - Absolutely

• Also touch on performance, security... integrated environment - system as a whole rather than a single piece of code

(AN) - Absolutely

No - we should test one of every type of field (e.g. so we are covering whether the mandatory/optional flag on fields is working, and whether validation is working, rather than all fields that are mandatory and all fields that have validation).

Unit testing

• We should use unit tests for individual fields (including whether it is mandatory, is there validation, etc)

• Form validation - write unit tests for each and every field in a form

• If a field changes from non-mandatory to mandatory - more work is needed with testing

(AN) - and the work needed is an alteration to the unit test for that field

• We generally have good unit tests coverage

E2E testing

• E2E tests - test one scenario of every type

(AN) - trust the unit tests that the scenarios have been applied correctly to the correct fields

• E2E tests - should be testing the Acceptance Criteria (AC) (e.g. check the UI a bit more, look for error UI elements to check when form is not valid, getting data from BE)

• Are the E2E tests we have doing too much - such as testing the libraries we’re using?

• E2E tests should be to generally test the user journey through the application

(AN) - there should not be large-scale replication of what the unit tests are testing

• E2E tests should not usually become entirely redundant, but will require refactoring as code develops

(AN) - the ACs for a small initial vertical slice, should reflect only that small initial vertical slice. So unit tests and E2E tests should be relatively simple as a result. As the feature develops, more unit tests will be required and the E2E tests will likely need refactoring. NOTE#1

• If 4 fields need to be completed to submit the form - why only test 1 field?

(AN) - If all 4 fields are optional and have no validation, the user should be able to submit the form with nothing entered, so the E2E test should test just that.
If any of those fields are mandatory the E2E test would need to test that the form throws an error if one of the mandatory fields is not filled in and the user tried to submit the form.
If any of those fields have validation, the E2E test should highlight that validation kicks in when invalid code is entered into one of the fields with validation.

This way, if there’s already one field which is mandatory and one field with validation, and another field is added, the E2E test would not need modifying.

• We should not be testing data with E2E tests - mock data imported into live system - can use this with unit tests

(AN) - most common reason for E2E test failure is absence of the specific data expected. If this dependency can be removed, E2E tests become much more of a useful tool for the team

• Objective - avoid testing the same thing in both unit and E2E Tests

• We need to consider the maturity of a feature in determining when to write and run E2E tests

(AndyD) - So for TIS Self-service, is there value in creating, refactoring and running E2E tests in advance of users starting to use the app. And even when we have users testing the app, where they will be anticipating issues, how much value is there in putting loads of effort into having written, refactored and run E2E test many multiple times? For TIS, however, this is a ‘live’ system, where we need to protect its integrity as much as possible, so the effort going into E2E tests is much more valuable.

Other

• Need to consider how maintainable the code is - quite bloated.

(AN) - @Philip Wilsdon (Unlicensed) did we mean “code” here or “E2E” tests?

Pretty sure it was code - think this came from shivani - something around making sure codebase in the future is easier to maintain or tests easier to maintain

• Test harness - the capability to create the data - before running of E2E Tests - import that data into the database then will avoid test data

(AN) - so is this something we need to build then? @Andy Dingley / @Joseph (Pepe) Kelly

• AC can in some cases be better written / refined

• We need to anonymise / obfuscate data in non-Prod environments - unclear why we moved away from this?

(AN) - if our environments are secured with logins, and anyone with access to those pre-Prod environments have signed up (contractual small print) to being professional in their handling of data, why do we need to anonymise / obfuscate data? Is that a stupid question?

• Staging not just staging - need somewhere for users to test

(AN) - there’s a bit of a conflict to manage here, I think. Obfuscating data in non-Prod environments, but having an environment for users to run data-reliant testing (e.g. Bulk Upload).

• Need a UAT for users to practice?

(AN) - perhaps we need to spin up and destroy an environment for users to test/play with, on demand?

• PRs reviewed - all looks good, merged and then E2E test issue - in next sprint - what happens?

(AN) - @Philip Wilsdon (Unlicensed) Could you clarify the “in newsprint - what happens” - are you asking for how this sort of thing would be handled by a team working with a physical product, or do you mean “in the next Sprint - what happens”?

Typo - should be next sprint. I think the question was; if in sprint A, the coding and E2E tests all done and working well. However, in Sprint B, the E2E Test then fails, how do we handle this

• Improving the quality of code reviews

(AN) - would be good to have some How’s here

• The person doing the review - doesn't need to check if PR matches AC - they don't know the ticket, could be multiple PRs - if code is solid all good

(AN) - we can’t expect a reviewer to match the PR to the AC - it won’t always be obvious which AC a PR is designed to meet. It should be clear which AC an E2E test is designed to meet though.
(Femi) - Use GitFlow to manage code commits via feature branches that multiple devs are working on, which are then collated into a single release branch at which point the E2E tests are run
(AndyD) - Not keen on this approach as it moves us away from the CI/CD approach we’re adopting

• Fundamentally, should E2E test block code going to Prod?

(AndyD) - No. This goes against the CI/CD approach we’re adopting.
(AN) - But we should be in a position to run E2E tests often enough to catch code that interferes with other code before it breaks Prod (an aside: is this the issue where Feature flags could help?). And in parallel we should be carrying out tech improvements to reduce dependencies between services such that if Prod breaks, only a small part of it does, and user can continue to use large parts of the app while we fix it. NOTE#2

• Small PRs are good

• E2E tests - could be in one PR - many PRs as part of the ticket

• AC can't pass on every single PR

• Roles and responsibilities - mainly on devs for fixing E2E Tests. A PR that breaks an E2E test should not be approved until that E2E test passes

• Multiple PRs - comment out the test, then put it back?

• Feature branches - still rely on local instances of BE services for checking things work

• It is crucial that we don’t inadvertently create a ‘stage gate’ of testing at the end of the development process

(AN) - testing is the responsibility of all developers, not just Shivani

• Additions to the Dev handbook in Git mean that we are able to run E2E tests before Production, run locally or on Stage

• DoD - E2E tests for AC

• We don’t have a ‘formal’ process for ensuring E2E tests meet the DoD for a Story

• TDD / Dev writing the tests - E2E Tests difficult at the start

• There is a distinction between TIS as released product in live use vs TIS Self Service / Reval under development - users understand if something breaks

• Always remember “Testing is your friend” (whether it is hard, or painful!). And when something is hard, the best way to handle it is to ‘bring it forward’ - front up to it early, rather than leave it to bite you in the bum later on!

Current

Dev works on code -->

PR process -->

Code merged to master -->

masterauto-deploys to Pre-Prod -->

E2E tests run -->

Manual push from masterto Prod

Ideal?

Dev works on code -->

PR process -->

Code merged to master -->

masterauto-deploys to Pre-Prod -->

E2E tests run -->

Pre-Prod automatically becomes Prodon successful completion of E2E tests,
Prodbecomes Pre-Prod. NOTE#3