My adventures with improving test automation

I want to share some thoughts and experiences that I worked on for over a year now - "Improving test automation in our product". It has be a great experience, learning and very challenging one for me.

Before I move on to my next adventure, I would like to pause, look back, retrospect and document my experiences and learning. 

Background

     Two years ago (2017) as part of an organizational goal, we had to improve our automation. Our product is a huge enterprise product with approximately 2.5 million lines of Java code (not counting Java script, angularJs, JSP, HTML, XML lines). The product has a code coverage on higher side of 30% with about approximately 90K tests. It had a good mix of unit tests, integration tests and selenium tests

Though we started writing tests for the new code where ever possible, It is the majority of critical old code that is not covered by any automation.

Goal

   - Our goal was to increase code coverage in critical areas by 8% in FY19.

Challenges:

While driving this effort, we faced several challenges.

    - Managers/team members were not convinced that unit tests for legacy code is possible or of any value.

    - Some generic questions/comments from my peer managers have been

• What is the ROI of this effort?
• Are we investing in right area?
• How will it add value to customer?
• Is the code coverage right metrics for judging effectiveness of automation?
• With the effort spent on automation, I could have fixed more customer reported issues.
• I have already spent improving code coverage in my area, but there are still large number of customer reported issues
• My area has lot of automation tests and the maintenance cost of these automation is  huge. - we doubt if increasing tests an effective strategy
• For more effective test automation, it should be end to end testing or UI based testing involving customer scenarios and it should be QA Engineers problem

    - Many developers are problem solvers and are not truely extreme programmers - equipped with test driven development - Automation tests/testable code is always an after thought

    - Some code areas are no go areas. No one dares to touch it and hence automation has always been on back burner.

Challenges can be broadly categorized into two

   1. Mindset problem

   2. Gap in skill set

Execution

Overcoming the mindset

   To be frank, though I have tried to reason with many managers about importance of focused effort of increasing automation and code coverage, It was very difficult to convince managers. The conclusion of all our private talks was, our product is different. It is legacy code and the effort is not worth spending.

Here are some of my views about this effort.

It is not worth spending the effort on automation for legacy product. It can be effective for new product

I don’t agree with this view. If the product is almost dead and/or is living on life support ( is on maintenance revenue,  the customers are not upgrading to new products or version) then it could be a case where we need to evaluate if we want to spend on such effort.

For a product that is  growing, such an effort can give the product a new lease of life. The effort can reduce the cost of testing by many folds. With greater automation, developer can get faster feedback of newly added feature which in turn will reduce the development cycles. A unit test help developers refactor and clean the code.

Is code coverage a right metrics for effective automation?

   Yes code coverage could be deceiving for a product like ours. Let's take an example, lets' say, I have 100 lines of code in my module and the module has automation tests that covers 90 lines of the code.

If the customer majorly uses features that are part of uncovered 10 lines, my 90% code coverage has no meaning. (In other words these 90 lines should be relooked at if they are really required. It could be that they are not needed in the first place)

In short mindless code coverage improvement is worthless. You would not achieve value.

 Before we start this kind of effort, we need to analyse the areas within the product that are critical and needs coverage.

   Some strategies could be where

• There are more customer reported issues
• The code is frequently changing
• Critical "No Go" areas
• Difficult to understand code.  

    Code coverage is one of output variable - "metric" which indicates that we are in right direction and at right pace.

The other outputs of effective automation are reduced customer reported issues, refactorability of code, increase developers productivity. These variables can only be measured in long run. We cannot measure these in a short term. For short term measure to guide us through this effort, code coverage seems to be the right metric.

It is an involved effort. What is the ROI of this effort? How do we measure the effectiveness of this effort at the end of the year?

    It definitely an involved effort. But once it gets rolling it becomes easier.

    The ROI of automation effort would be to reduced feature regressions and reopen or rework in long run. In short run it is improved code coverage.

To see the benefits we need to reach certain point of code coverage. – A tipping point. In my opinion it is at 60% of your code.

There are several input variable to achieve this

• Focused and strategical automation efforts. No point in automating those areas where the footfall is less or rare.
• Writing Unit test which will enable refactoring of code.
• Covering customer or internal QA filed issues with automation (These are missed use cases)
• Removal of dead piece of code. This can be easily identified after writing some unit tests.

It is legacy code. What is the point of automating to improve code coverage?

  More so, it needs automation. Tests also work as documentation which help us in future refactoring.

Those modules in functional areas which are currently stable, critical but don’t have automation test are good candidates to be automated first.

We lose out on new product development capacity?

Initially those teams who have less code coverage and spent less time in automation will have a pinch.

Once the teams mature and start a habit of coding with TTD, the new product development effort will increase, build breaks and rework will reduce.

Such focused effort may not go down well with developers?

The increase automation effort should not be projected as dirty work. It is managers duty to ensure it is projected in right way. Few of my thoughts here.

• If you recall the progression of developers over years, they began with procedural programming, then to OOPS then to extreme programming. When we are taking this effort we are asking our developers to move on to become an extreme programmer. - Simple, effective, refactorable, testable coding.

A way to start is change the mind set of developers from fixing problems to writing testable code.

TDD works only for new code and new products?

May not be true. The first step in TDD is to write tests and keep refactoring. This works for old legacy code too.

• It is managers responsibility to make this effort interesting. One interesting way is to spend whole team 1 day to automate. May be first day of the sprint.

    

What is the point of writing tests which have no meaning but only improves code coverage?

   Yes agreed. This is where teams should have strategy to go thru the customer filed issues and other available data to automate the critical modules and critical use cases. Automate those areas so that you would secure them from future regressions.

After all these explanation and reasoning , there are managers who still have problems with this effort.

Yes the benefits of this effort seems very less and invisible in short term. But the benefits of these if implemented correctly in long run will be huge.  

Gap in skill set

Writing unit tests for legacy code is a big challenge. Sometimes it is difficult for developers to write unit tests for legacy code without refactoring and refactoring cannot be done with confidence without the tests backing it. A vicious circle which needs to be broken by writing tests…

Developers need to be skilled in writing unit tests with mocks. There is a learning curve for some developers and managers need to understand this. The code reviewers have to be cautious in this phase as the unit tests sometimes can be  too shallow.

A pair programming with a senior developer will help learning writing tests quickly.

Coding forums will help propagate best practices of writing tests across the teams.

Our Plan for increasing automation

Increase in code coverage and improving effective automation can be achieved by following

1. Adding more tests
2. Refactoring tests and code
3. Removing dead code

Adding more tests

Adding more tests should not be arbitrary. You need to identify which tests are apt.

Automation Strategy: 

Test automation pyramid pattern of test automation are ideal pattern for teams to reach. This pattern should be ultimate goal to reach for developing team.

Why automation pyramid?

If we look at the cost of test automation we have cost of creation, cost of execution and cost of maintenance.

The cost of automation depends on type of testing, availability of test frameworks and tools. There are good tools available for automation like junit, jsunit, jasmine, selenium. Assuming that every team has good tools and framework the cost of test automation mainly varies on type of tests.

 

  

Every time the code needs to be changed the automation tests need to be run

As the code keeps changing with more feature additions, there is an amount of maintenance cost associated with automation tests.  Test automation pyramid achieve more code coverage with optimal maintenance cost. 

To get the team to the best patterns of test automation, you need to understand where the team currently is.

Identify the teams with no automation, heavy on unit tests, heavy on integration tests or heavy on UI tests.

The goal of the module is to get a right automation pattern either by writing unit tests, integration tests and selenium tests  

Refactoring code and Remove dead code

As developers we sometimes believe that adding more code, more features are important and forget that clean-up is as important as adding new features. This effort gave us opportunity to clean up or refactor some code.

Conclusion

Though we have completed this exercise and met our yearly goal of improving code coverage, some of the "Nay Sayers" are still not convinced that the effort was wisely spent. The short term code coverage shows an improvement, but we don’t have reported metrics of increased developer and QA members productivity nor we have metrics to show there is reduction of customer issues.

The immediate change that I could see is a cultural change. The developers now believe that code written/changed has to be testable.

Some nice reads that helped me in the process

https://martinfowler.com/articles/practical-test-pyramid.html
https://abstracta.us/blog/test-automation/best-testing-practices-agile-teams-automation-pyramid/
https://www.guru99.com/code-coverage.html