Posted on

Our Selenium to Cypress Journey

Here at ServiceTitan, we are always working to improve our automation test framework. Back in May 2020, we realized that our team had out-grown our existing Selenium framework; so, a few of our key members got together and discussed what was next for us. Then came Cypress — an all-in-one JavaScript testing framework, assertion library, with mocking and stubbing without Selenium. As a team, we instantly fell in love with it and pushed forward on a plan for transition. In this post, I am going to talk about all the fun and challenges during our path as we converted our technology from Selenium to Cypress.

Our “Selenium” Issues

Changing tools or frameworks is never easy, but a good plan helps. Before we dive into the details on how we transition from Selenium to Cypress, let’s take a look at our issues with our existing Selenium framework.

Existing Framework: C# with Selenium WebDriver, bundled within the main ServiceTitan application codebase. Since our tests/framework reside within our main application, our test data generation objects were basically extended from some of the core application’s existing models and controllers. Our framework was designed to run our tests against a locally hosted/built application. The main approach of this architecture allowed each developer to run tests locally against their work branch before merging back to the main release/master branch. With that method, we reduced long regression test cycles.

Developing with Selenium

Our company is growing very quickly with developers being hiring at lightning speed, and with our current approach, scaling our tests had becoming a problem due to the following reasons:

Expensive

  • Slow startup, setup, and teardown
  • High test maintenance costs

Unstable

  • Out-of-process communication
  • Dependence on waits, builder specs, and amount of data
  • The dependency on the main application creates a lot of flaky tests

Rigid

  • Not portable enough to run on a live environment
  • Dependency on the main application.

How Does Cypress Help?

We started looking at other frameworks based on our problems. We POCed a few others like nightwatch.js, puppeteer, etc. But when we started to explore Cypress, we felt like we might have found a match! Given all the features provided by Cypress, we are now circling back to the problems I mentioned earlier:

Inexpensive

  • Cypress is fast to deploy and execute tests
  • Cypress debugger runs live in the browser; allowing for fast test updates and maintenance and requires no external driver

Stable

  • Automatic waits and retries

Flexible

  • Can run on any environment by pointing tests at a new base URL
  • With options like ‘decoupling’ from the main app, tests are able to run against any environment

Cypress Architecture

Cypress uses a different architecture compared to Selenium. The Cypress engine directly operates inside the browser. In other words, it is the browser that is executing your test code. It also means it has native access to your Document Object Model (DOM) and all the web elements on your page; giving you absolute control.

from https://www.edgewordstraining.co.uk/cypress-vs-selenium/

At ServiceTitan, we established 4 basic principles for our QA automation engineers to implement UI tests into our Cypress framework:

  1. Isolation — Keep tests flow against specific pages and isolates them from the rest of the application. Tests will not need to navigate outside of the target flow.
  2. Separation of Concerns — Stubbing the backend service calls through mockup data as much as possible.
  3. Independent — Tests should not depend on each other. One test should not know of the existence of the other ones and, therefore, should not conflict should they be run in parallel.
  4. Stateless — Tests should be able to restart anytime or shouldn’t depend on the state of test data.

All QA Automation Engineers follow the above principles when writing integration UI tests. This way, the support boundary for these tests is well defined with this clean implementation.

Scaling

One of the biggest drawbacks to building tests on our old Selenium framework is scalability and performance. We don’t use Selenium grid; we set up 20 core CPU window servers as our test agents so that we can customize our framework to run tests in parallel. Also, we realize that once we run more than 10 threads (10 parallel tests) at the same time, the test results were flakier than when running tests with a lower number of parallelism. Basically, we have to pick between test quality and test performance.

On the other hand, by using Cypress, we do not need to sacrifice either test performance or test quality. As cypress tests, we are building through Javascript/Typescript on top of node.js, we can package all the tests and the framework and run them through our low-cost virtual Linux-base agents instead of our expensive window servers. Also, test results are much more stable.

Timeline and Results

As a result, we started our POC about 2 years ago, and today, we finally go live and have 100% replaced our old Selenium framework. And, the results of the switch are stunning. Let’s review the following improvements:

Within 18 months, we have built more than 3000+ UI tests, compared to our old framework that ran about 800 tests in 4 years.

The test performance is significantly faster, meaning we can run more regression cycles within a week.

And the most important part, we got happier and more motivated QA engineers.

Conclusion

The Cypress framework is far from perfect and can still be challenging with the implementation and adoption. There is no current support for multiple tabs/browsers. Also, they have a smaller support community compared to Selenium; but these problems are small compared to the numerous benefits that have been gained through the transition and we are excited about the future.

I would like to give a shoutout for all the hard work from my team. Especially Carlos S, Parin P, Michael R, and the entire ST Cypress-Council group. We will not be here without your hard work!

Posted on

Year 1 at ServiceTitan

Another year is done and dusted! Time flies! I worked for ServiceTitan over one year now. Looking back at the QA organization at the time I joined and comparing it to now, we made a lot of changes. So, it’s time to review all of them and grade the results.

Here is the observation from the first two months since I started:

All the great stuff: (Highlights)

  1. We have some of the most incredibly talented and hard working Engineers, QAs, and product managers I have ever worked with.
  2. The overall testing process is good, and the organization saw great value from our QAs.
  3. Developers, QAs, and PMs collaborated very well along with each other

Stuff, that might need some help: (Opportunities)

  1. The application produced a lot of bugs, (mostly regressions), after each release.
  2. There is substantial Automation tests backlog.

The solution seems simple; we need to run regression tests consistently, and invest more in test automation! Well, what if you don’t have the time and resources to do both? From here, I will discuss some of the changes I made in attempting to address these two issues as listed. I will also walk through the outcome of these changes.

Organizational Decision

The team I inherited when I joined Service Titan had the following organizational structure:

This simple QA organization structure worked very well when we were small, and there was only a single product owner/development team. When the team started to grow, the QA team was running into a lot of scaling issues with this organization structure.

Why you might ask? Here are the main concerns:

  1. Manual testers don’t do regression testing, and the assumption is that automation QAs will automate all the test cases handed off by them as part of the CI/CD automation process. Due to this assumption, our regression tests coverage were low and often time release code with a lot of bugs.
  2. With the ever increasing of the complexity of our software design, automation QA engineers were depending on manual testers to “explain” how the feature(s) worked (it won’t be enough to create good automation if by just reading test cases alone without understanding the use cases). Often time, manual testers were too busy to accommodate the needs of the automation QAs, this caused a vast backlog and delay of the automation tests.

So after I studied all these problems carefully, I decided to roll out the following organizational changes.

  1. Break down the silo between Automation QA teams vs manual testers, create QA teams which based on the functionality of the application(squad) which align with Dev/Product team. By doing this, we were able to identify QA leads for each group and provide growth opportunities.
  2. Since the cost differences are minimal between a QA engineer vs a Manual Tester, I decided that we will only hire QA engineers who have experience in automation on going forward. (know how to code)
  3. Create a new team called: QA framework, and I hired senior developers in this team whose primary function is to manage the test frameworks and train all the existing manual tester how to do automation.

Now the new structure looks like this:

Technical Decision

ST core application is written in C#, .NET framework. Our Automation framework is also written in C# nunit with selenium webdriver. By choosing nunit using parallelizable attribute allow us to run tests in parallel. This enabled test suite to scale when adding more tests.

Performance first approach: The depth of the tests are important, but as a team, we decide to make our technical decision based on “test performance” as a priority. Adherence to this decision led the team to make easy technical trade-offs such as: if tests should part of this framework, refactoring is needed, or approving test code PR. We also refactor test steps within cases to use Headless Chrome (no visual verification needed) instead of using Webdriver. This saves us 5% of execution times for all tests.

CI/CD ready: We always build our tests/framework based on the fact that we need to CI/CD ready. Tests can be run against each branch, PR and should be able to start by anyone within the team.

Process Decision

Team Communication

Something simple, set-up a weekly team meeting once a week for 15–30 minutes to do a quick check-in. It also served as meeting to provide any leadership level information pass downs amongst the team. This meeting became a primary communication channel for the overall teams.

Introducing functional/Regression test plan

There were no test planning conducted as manual testers were treated more like a support team. Stories were coming to their queue as first come first serve base. There wasn’t any document that captured what and how as a team we did our testing.

The first step on what a QA should do when testing code, is to plan how to test. I picked a couple of manual testers and automation QAs with more experiences in the team, and instruct them to create a test plan template. As a group, we also develop a process of reviewing these test plans. After we tried with a couple of teams, we then roll it out to every QA.

Create QA onboarding training plan for new hire

Training material for new hire is critical. I spent the first six months to come up with a 25 points onboarding checklist which I felt like if I knew these things during the time I was hired, I would be in so much better shape. The checklist includes things like: which group mailing alias to added or required slack channels to be included, to something like walking through the deployment process; to running your automation tests locally. I established eight goals that each new hire must complete within the first three months. Also, by providing a go-to contact within the team for each point of the checklist so that the new hire can find that help when needed.Summary

Summary

As a result

Post Release Production Issues

This graph shows our production bugs post-release. In other places, we also call these “site issues”.

Automation Test Coverage:


Automation QA Engineer by Quarter:

Automation QAs finally outnumbered Manual Testers.

Nothing is perfect, there are things still not working as expected…

Communication within the team is still not perfect, but it’s improving.

We still can’t catch all the bugs! As our test coverage improves, so do all the edge cases out there. The ideal scenario is that if we can create a bug finder which keeps crawling our application code and chew on bugs like PacMan would be awesome!

Our process is still not perfect, just like every other place I worked.

Finally! We can improve “eat club” menu. J/K. I am pretty satisfied with the stuff that they provided, but don’t mind if they throw in a couple of lobsters as the appetizer.

Posted on

Why the f**k we still write test cases?

Why the f**k that we still write test cases manually? I haven’t heard of the single person at work that loves read them or review them. I know it’s essential to test software, but how useful to write test cases?

So, let’s look at all the key stakeholders when developing software:

Product Manager – How often they review the test cases in details?
They don’t care as long as they know their requirements have tests coverage.

Project Manager – They don’t care, as long as you tell them when testing completes.

Developers – They don’t care, genuinely don’t care.

QA – Only yourself care about your own cases, but you don’t read them yourself anyway because you already know how to run those cases right off your head.

Developers/Managers – They don’t care, they probably don’t know if there is a test management system existed anyway

QA Leads/Managers – They might review them (sometimes). But often, they don’t care as long as they know there are test coverage

The only time people will ask for test cases in details is when something break in production, people (besides QA, which is yourself) will ask about where the F**K are the test cases and why the problems were not caught initially?

Now, let’s plug in some numbers, assuming QA spends ~15% write new test cases, and modify existing test cases, (6 hours per week, 312 hours per year) spends ~20% automating test cases, (8 hours per week, 416 hours per year) and production major issues happen ~5% of time (assuming 20 releases per year).

And assuming the cost:

  • Writing test case per hour cost: $15
  • Writing automation test case per hour cost: $45

The saving of not writing test cases:

  • $4680 saved from writing test cases, OR
  • We can add 104 hours of automation time, AND
  • by adding 104 hours to automation (25% increase), which will now lower the percentage that major production issues happen to 3.75%

I will take this saving anytime. So, why we still f**king write test cases?

(To be continue…)

Posted on

Love and hate against the usage of SLACK!!

I love the invention of slack; but at the same time, I hate it so much I want to throw my phone away everytime I heard that “ding” sound.

Slack is a very powerful tool, especially at work. The tool allows me to talk to people and get an instant response without leaving my desk, or get my hand off my computer. The conversation between me and my team members was recorded and easily searchable. Communication and work collaboration become easy with the call and screen share features.

Because of it’s so easy to use, it makes me never stop working! My co-workers start to add me to all different private and public channels and it is quite overwhelming. Same discussions happen in multiple channels with a different group of people. At the end of the day, people ended up not responding to any critical slack messages and commented on useful facts.

This is the time that I miss the good old days when meetings run on paper and pen.

Posted on

Software Quality Management 101

It’s been a while that I check my blog email, a piece of content was submitted from an unknown writer.  I am going to share the content here.  I think SQM has been a forgotten term in 2018, as technology companies are going into matrix organization model and automation within their engineering organization.

So enjoy… 


Software Quality Management (SQM) is a process that manages the quality of software in such a way that the product meets the required level of software and ensures that the quality is achieved when it reaches the users and ultimately they are satisfied by its performance. 
Quality software refers to the software which meets the requirements and is maintainable with no defect, exactly according to the expectations of the users. 
There are three basic components of Software quality management activity;
Software Quality Assurance − Software Quality Assurance (SQA) involves the activities that are focused for the development of a framework of organizational procedures and standards which are intended to ensure that it meets desired quality measures that ultimately result in quality software products. It involves process-focused action. 
Software Quality Control − Software Quality Control (SQC) involves the activities that ensure the quality of software products. These activities focus on determining the defects in the actual products produced. It involves product-focused action.

Quality Planning − The selection of appropriate procedures and standards from this framework and adapt for a specific software project.


In the software engineering context, software quality reflects both functional qualities as well as structural quality.
Software Functional Quality reflects how well it satisfies a given design, based on the functional requirements or specifications.
Software Structural Quality deals with the handling of non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability, and the degree to which the software was produced correctly.
Quality management provides an independent check on the software and software development process. It ensures that project deliverables are consistent with organizational standards and goals.
Software measurement provides a numeric value for some quality attribute of a software product or a software process. Comparison of these numerical values to each other or to standards draws conclusions about the quality of software or software processes. Software product measurements can be used to make general predictions about a software system and identify anomalous software components.
Software metric is a measurement that relates to any quality attributes of the software system or process. It is often impossible to measure external software quality attributes, such as maintainability, understandability, etc., directly. In such cases, the external attribute is related to some internal attribute assuming a relationship between them and the internal attribute is measured to predict the external software characteristic. 
Software metrics can be classified into three categories −
Product metrics − Describes the characteristics of the product such as size, complexity, design features, performance, and quality level.
Process metrics − These characteristics can be used to improve the development and maintenance activities of the software.
Project metrics − These metrics describe the project characteristics and execution. Examples include the number of software developers, the staffing pattern over the life cycle of the software, cost, schedule, and productivity.
Some metrics belong to multiple categories. For example, the in-process quality metrics of a project are both process metrics and project metrics.
Software quality metrics are a subset of software metrics that focus on the quality aspects of the product, process, and project. These are more closely associated with the process and product metrics than with project metrics.
The various factors, which influence the software, are termed as software factors. They can be broadly divided into two categories. The first category of the factors is of those that can be measured directly such as the number of logical errors, and the second category clubs those factors which can be measured only indirectly. For example, maintainability but each of the factors is to be measured to check for the content and the quality control.
Software Quality Metrics simply means the measurement of attributes, pertaining to software quality along with its process of development. The term “software quality metrics” illustrate the picture of measuring the software qualities by recording the number of defects or security loopholes present in the software
Software testing metrics – Improves the efficiency and effectiveness of a software testing process.” Software testing metrics or software test measurement is the quantitative indication of extent, capacity, dimension, amount or size of some attribute of a process or product.
Software quality managers require software to be tested before it is released to the market, and they do this using a cyclical process-based quality assessment in order to reveal and fix bugs before release. This testing implies;
The software specification should reflect the characteristics of the product that the customer wants. However, the development organization may also have requirements such as maintainability that is not included in the specification.
Certain software quality attributes such as maintainability, usability, reliability cannot be exactly specified and measured.
At the early stages of the software process, it is very difficult to define a complete software specification. Therefore, although software may conform to its specification, users don’t meet their quality expectations.
So, their job is not only to ensure their software is in good shape for the consumer but also to encourage a culture of quality throughout the enterprise.

Posted on

Interview Questions for QA Automation Engineer

Throughout my career, I believe I’ve interviewed at least 500 candidates for QA positions. I’ve been “lucky” enough to hire some very bright engineers, who ended up having very successful careers. Now that I come to think about it… how did I find these bright engineers? I guess most people today would think “LinkedIn.” But I can tell you that my most successful hires have always come from referrals.

I believe that the screening and interview process is very important. I also believe that each hiring manager needs to create an interview plan and identify the “features” of the type of engineers they are looking for.

What do I mean by a “feature”? A feature is a set of traits that you believe will improve an engineer’s chance of being successful in the role. For me, I usually look for interviewees who love to listen to others, are patient, have good communication skills, and are good at the process of elimination.

Once I’ve identified the features I’m looking for, I need a hiring/interview plan.

Here you go:

  1. Create a clear job description and define roles/responsibilities for the hiring role.
  2. Identify a few key interviewers for the hiring panel.
  3. Set up a good set of technical tests for candidates. I like to use hackerrank to set up my tests. One of the features of hackerrank that I really like is that you can create test cases and test each candidate’s solution.
  4. Make sure you come up with a list of problems to pose to your candidates that can help you judge whether they have all the features you’re looking for.
  5. Don’t forget to sell your team and vision to the candidates! Remember, the evaluation process goes both ways! You want to make sure the candidate likes what you’re offering.

Good luck on your next hiring!

PS, I have my list of favor questions, DM me and I’ll share with you!

Posted on

My Experience on using JIRA Cloud API to customize your release and quality data

I am sure a lot of us used JIRA for bug tracking, sprint planning, story telling, features logging, and etc…  Most of the QA I know somewhat touches JIRA one way or the other.  I will walk you thought how to connect to JIRA api cloud, also give out some of the pain points I had from my experiences.

Before I listed out the steps, I expected you know some basic about JIRA on how to setup projects as admin, create issues, and use jql.  If not, please view this video

or (https://confluence.atlassian.com/jira/jira-documentation-1556.html) to get help.

Assuming you already have JIRA cloud setup, like  for an example, http://yourproject.atlassian.net, and have an atlassian cloud user account, here is the step by step, on how to connect to your JIRA cloud API.

1. First you need to encoded your username (JIRA cloud email login) and password in base64.  Let say your JIRA cloud login address is abc@hello.com and password is 1234,  you will encode this as: #echo  – n “abc@hello.com:1234” | base64 to get the encoded string for your basic JIRA api authentication.  Make sure you use “-n” because echo will attach a trailing newline char at the end.

2.Now you can connect using this simple ruby script:

require 'httparty'

## Create your JQL query here
jql = 'text~ "' + "find X".to_s + '"' + " and project = YOURPROJECT"
yourencoded = "" ## Put your encoded string on step #1 above

## Header
@jurl= "https://yourcompany.atlassian.net/rest/api/2/search?jql=" + URI.encode(uri_text)

### Now Loop through each issue from the search
 response=HTTParty.get(
 @surl,
 headers: {
 "Authorization"=> 'Basic '+ yourencoded.to_s,
 "Content-Type"=> 'application/json'
 }
 )
 result=JSON.parse(response.body)
 result["issues"].each do | issue |
    ### Now refer to the JIRA issue doc, you can do your logic here
 end

3. you are done, I will put this in some kind of chart tool so you can create your own dashboard

Posted on

Should I “dress up” for a software QA engineer interview?

One of my previous report asked me recently, “should I wear a tie for my interview?” (Sorry, my friend is a guy, generally, he meant should we dress up before we head to an QA interview?

My simple answer is, yes.  “Dressing up” doesn’t necessary mean you have to wear a tie, or suit, or nice evening gown.   “Dressing up” to me meaning business causal.  Something that it’s proper for the company business environment that you are interviewing for.

A typical QA interview includes 4 major parts

  1. Personality test (whether you fit the team, able to work with others)
  2. Technical test (mostly coding skill)
  3. Testing process/development process test  (test you knowledge on testing and development, also, if you can prioritize tasks)
  4. Problem solving skill

Dressing up before you for go an interview doesn’t guarantee you the job.  But it provide an impression to your interviewer that you respect the company and you treat the interview seriously.  At the point that if you and another candidate did very well on the first 4 parts, and you dressed up and look more professional than the other candidate, you will have a slice advantage to get the job.

Rule of thumb, dress to impress, but don’t over-dress.
Posted on

I need help to figure how to test massive credit card transactions

The testing problem I faced today is that I have a feature which go through thousands of customers, each of them with a credit card information, we need to batch the payment transaction at once.  The key is to test the following:

1. It’s able to handle over 5000 transactions at once

2. If the third party gateway can handle the same load, if not, what’s the best scenario they can handle.

3 . Test random fail/success transaction with the large batch, make sure the failed transaction will not stop the full load

4. Build a baseline on performance

Now we know our test cases, but creating test data become very challenging.  What we are given is that, our vendor doesn’t have a sandbox api for testing.  Also they only provided a hand full of credit cards(test) for us to do functional tests.

Can someone provide me some possible solution that I can obtain 5000 unique credit cards? (Once you submit your ideas, I will provide my solution later in the comment section)