Improving QA: Short-term changes applied to the production pipeline

The Giveth Platform
1

Hey Givetheres,

We recently had nearly 100 issues in UAT, going back and forth and usually keeping the environments was hard, so we decided to manage it more to ensure the QA is getting better before release. Also, the waterfall responsibility over QA can be more attainable due to changes we made recently.

So what were the short-term changes?

image
image1366Ă—383 24.6 KB

  1. Unit Testing environment added with Feature Branches.

Feature branches are defined based on Epic or hot-fixes, they are going to be on vercell domain.

Examples of feature branches:

  • Givpower
  • Givfilter
  • Verification

Note: Backend currently doesn’t have feature branches. It means the related backend issues need to be merged in staging in order for the tester can start testing.

  1. Two different steps were added to Zenhub: Unit Testing / Unit Testing Passed.

Unit Testing means testing features on feature-braches was successful.

  1. We’re going to have 2 different UAT environments (**) from now on: one for Giveconomy and one for Dapp.

Note: Backend UAT env is also on staging / Subgraph also have only staging for tests.

The flow guide:

Unit Test Column:

1- Input:
Who? Code- Reviewer
When? After Review was ok.

The reviewer should merge the changes to Feature Branches, put them in the Unit Testing Column, and delete the previous branch.

2- Output:
Who? Tester
When? After Unit Test was ok.

The Tester should put the issue in the Unit Tests Passed column, preferably with a comment on the issue that the test was OK.
If the test was not ok, the Tester should comment the scenario and evidence and let the issue be there, preferably tag the coder or reviewer about the problem.

Unit Test Passed Column:

1- Input:
Who? Tester (who wears the tester hat)
When? After the unit test passed and the issues were ok.

The Tester should put the issue in the Unit Tests Passed column, preferably with a comment on the issue that the test was OK.

2- Output:
Who? Devs (*)
When? After PM confirmed that the whole feature is delivered.

The PM should be responsible to follow up on the issues in the Unit Test passed and letting them go to UAT test env for final tests. PM asks for release and creates the required release on ZenHub and tries to connect all related issues to the Release, then accompanies the dev team to see them merge in UAT.
Now test parties can happen because we’re pretty sure that the value of the feature is delivered.

Note:
1- DApps Features merge into staging branch
2- Giveconomy Features merge into Giveconomy branch
3- Backend related issues should

UAT Column:

1- Inputs:
Who? Devs ()
When? After PM confirmed that the whole feature is delivered.*

2- Outputs:
Who? Testers (who wear the tester hat)
When? After all test scenarios of the DApp / Giveconomy are working file.

The Tester with the PM should be responsible to do the “System Integration Test”, which means the whole application should work fine with the new feature merged into the current branch. Tester should comment that the feature is working fine and move it to done.

If tests are not passed:
The tester should comment on the issue, and mention the coder and let the issue be there. Devs can bring them back into In-Progress and fix it. In this case, another issue can be created based on this comment, and PM should put it in Sprint backlog and attach the epic (or feature) so devs can pick it up.

Note:

  1. All issues should be connected to a release (preferably with the number)
  2. All pages, all previous test scenarios, and also the Admin side, should be checked and approved by the tester.

Done Column:

1- Inputs:
Who? Tester
When? After UAT tests passed.

2- Outputs:
Devs? Devs (*)
When? After PM confirmed that the UAT version is final and can go live

PM asks for release and creates (or recheck) the required release on ZenHub and tries to connect all related issues to the Release, then accompanies the dev team to see them merge into Production (main branch)

Note: Conflicts should resolve by Devs () while merging into the main branch, and PM should now the risks of removing these conflicts. (**)

Side Notes:
*_ In these cases, preferably DevOps Teams should accept the responsibility and do the Ops, not the Dev team, but for now, Dev team does the operations, usually Cherick and Mateo on Front End and Carlos and Mohammad on the Backed. Amin usually does the ops on subgraph repos.

**_ Why we have 2 UATs:

  • It’s not standard we have to change it after DevOps team established the “isolated UAT environment” so we can split out the SIT and UAT.
  • Giveconomy and Dapp have 2 different release plan and 2 different roadmaps, sometimes Giveconomy don’t want to get blocked on staging to be able to simply add farms.
  • Giveconomy and Dapp has the least overlaps. Only subgraph issues

***_ Once we have “isolated UAT environment” ran by DevOps we can let the current staging as SIT env and have a stable UAT to do ab testing or have beta version. :slight_smile:

4 Likes
2

@MoeShehab do you have any comments?

4

Hey @MoeNick

Just a quick recap, I see that here we want the UAT to be acting just like the production environment (main branch on production) with a little distinction (It is not used for production). And a lot of tests will be made before anything reaches the UAT Environment.

To know what is exactly required for this new Environment, I have some questions:

  • Who should Use the UAT Environment?
  • Should these be real users? Should this be real data?
  • How are we going to approach this AB testing on UAT?? Feature Toggles, Maybe?
  • New resources must be deployed, is increasing the infra bill an issue??

Git suggestions:
1- UAT should be an exact copy of the main branch → New Features are merged as Feature branches to UAT → Approved By PM & Create a release → (After Successful Testing) UAT is merged to production (Main Repo) → Deployed to Production Infrastructure

2- After the merge, a new UAT branch will be created which will be an exact copy of the Main repo, and the old UAT branch is deleted

@geleeroyale do you have any suggestion or feedback?

2 Likes
5

Thanks @MoeShehab foe contributing

  • Who should Use the UAT Environment? Testers / closed circle of users.
  • Should these be real users? Should this be real data? Real users but they can be fake data. with test networks and test tokens.
  • How are we going to approach this AB testing on UAT?? Feature Toggles, Maybe? No at this step, but silimar configurations with production are needed. (like single configs on rate limits/ real emails / separate DB which controlled by DevOps not devs)
  • New resources must be deployed, is increasing the infra bill an issue?? we have to do it if we want to scale. I don’t think so but I can’t confirm. maybe @geleeroyale can talk with Griff

I agree with the Git suggestions also.

2 Likes
6
  • As far as the infrastructure bill goes - until further notice I am advised to enable all the power we need (of course we still try to optimize and think about cost, but it should not be a blocker)
  • I would be in favor of reducing the environments as much as possible and instead use only the production infrastructure.
  • Testing needs to be easy. We should have a path for general testing and possibly even for features (i.e. general release goes to testing.giveth.io - we could even token gate or password gate the testing environment). The testing environment uses the production database, but with testing data (i.e. stored in the same database but with a “testing” flag set which makes the data invisible on the main application). Additionally for these testing environments Ethereum test networks can be enabled. The backend used should be production as well.
  • If we imagine above in a Kubernetes system, we can see that these things must be part of the same system even. That would make even less sense - a whole Kubernetes cluster for staging.

To advance the whole issue, my recommendation is to set a path for realizing our current infrastructure on Kubernetes (i.e. lets really start with staging and replicate it in Kubernetes, but switch it off once we are comfortably running the production version on Kubernetes)

1 Like
7

Thanks for your input @geleeroyale

I agree with your points but have some comments:

  • For testing flags, and setting a testing environment on our production database, I think this needs some development work to be implemented (Not to take from it’s importance, as I see it is very important especially when describing it as a UAT environment)

  • As for moving things to Kubernetes, I think we are a half way to it. I’ve had a meeting with @renjer today and setup a plan what we want to move the Kubernetes (I shared it on the Devops channel on discord here , and I am currently working on these as well.

1 Like