Our Selenium to Cypress Journey

Here at ServiceTitan, we are always working to improve our automation test framework. Back in May 2020, we realized that our team had out-grown our existing Selenium framework; so, a few of our key members got together and discussed what was next for us. Then came Cypress — an all-in-one JavaScript testing framework, assertion library, with mocking and stubbing without Selenium. As a team, we instantly fell in love with it and pushed forward on a plan for transition. In this post, I am going to talk about all the fun and challenges during our path as we converted our technology from Selenium to Cypress.

Our “Selenium” Issues

Changing tools or frameworks is never easy, but a good plan helps. Before we dive into the details on how we transition from Selenium to Cypress, let’s take a look at our issues with our existing Selenium framework.

Existing Framework: C# with Selenium WebDriver, bundled within the main ServiceTitan application codebase. Since our tests/framework reside within our main application, our test data generation objects were basically extended from some of the core application’s existing models and controllers. Our framework was designed to run our tests against a locally hosted/built application. The main approach of this architecture allowed each developer to run tests locally against their work branch before merging back to the main release/master branch. With that method, we reduced long regression test cycles.

Developing with Selenium

Our company is growing very quickly with developers being hiring at lightning speed, and with our current approach, scaling our tests had becoming a problem due to the following reasons:

Expensive

Slow startup, setup, and teardown
High test maintenance costs

Unstable

Out-of-process communication
Dependence on waits, builder specs, and amount of data
The dependency on the main application creates a lot of flaky tests

Rigid

Not portable enough to run on a live environment
Dependency on the main application.

How Does Cypress Help?

We started looking at other frameworks based on our problems. We POCed a few others like nightwatch.js, puppeteer, etc. But when we started to explore Cypress, we felt like we might have found a match! Given all the features provided by Cypress, we are now circling back to the problems I mentioned earlier:

Inexpensive

Cypress is fast to deploy and execute tests
Cypress debugger runs live in the browser; allowing for fast test updates and maintenance and requires no external driver

Stable

Automatic waits and retries

Flexible

Can run on any environment by pointing tests at a new base URL
With options like ‘decoupling’ from the main app, tests are able to run against any environment

Cypress Architecture

Cypress uses a different architecture compared to Selenium. The Cypress engine directly operates inside the browser. In other words, it is the browser that is executing your test code. It also means it has native access to your Document Object Model (DOM) and all the web elements on your page; giving you absolute control.

from https://www.edgewordstraining.co.uk/cypress-vs-selenium/

At ServiceTitan, we established 4 basic principles for our QA automation engineers to implement UI tests into our Cypress framework:

Isolation — Keep tests flow against specific pages and isolates them from the rest of the application. Tests will not need to navigate outside of the target flow.
Separation of Concerns — Stubbing the backend service calls through mockup data as much as possible.
Independent — Tests should not depend on each other. One test should not know of the existence of the other ones and, therefore, should not conflict should they be run in parallel.
Stateless — Tests should be able to restart anytime or shouldn’t depend on the state of test data.

All QA Automation Engineers follow the above principles when writing integration UI tests. This way, the support boundary for these tests is well defined with this clean implementation.

Scaling

One of the biggest drawbacks to building tests on our old Selenium framework is scalability and performance. We don’t use Selenium grid; we set up 20 core CPU window servers as our test agents so that we can customize our framework to run tests in parallel. Also, we realize that once we run more than 10 threads (10 parallel tests) at the same time, the test results were flakier than when running tests with a lower number of parallelism. Basically, we have to pick between test quality and test performance.

On the other hand, by using Cypress, we do not need to sacrifice either test performance or test quality. As cypress tests, we are building through Javascript/Typescript on top of node.js, we can package all the tests and the framework and run them through our low-cost virtual Linux-base agents instead of our expensive window servers. Also, test results are much more stable.

Timeline and Results

As a result, we started our POC about 2 years ago, and today, we finally go live and have 100% replaced our old Selenium framework. And, the results of the switch are stunning. Let’s review the following improvements:

Within 18 months, we have built more than 3000+ UI tests, compared to our old framework that ran about 800 tests in 4 years.

The test performance is significantly faster, meaning we can run more regression cycles within a week.

And the most important part, we got happier and more motivated QA engineers.

Conclusion

The Cypress framework is far from perfect and can still be challenging with the implementation and adoption. There is no current support for multiple tabs/browsers. Also, they have a smaller support community compared to Selenium; but these problems are small compared to the numerous benefits that have been gained through the transition and we are excited about the future.

I would like to give a shoutout for all the hard work from my team. Especially Carlos S, Parin P, Michael R, and the entire ST Cypress-Council group. We will not be here without your hard work!

September 3, 2018December 10, 2018

Interview Questions for QA Automation Engineer

Throughout my career, I believe I’ve interviewed at least 500 candidates for QA positions. I’ve been “lucky” enough to hire some very bright engineers, who ended up having very successful careers. Now that I come to think about it… how did I find these bright engineers? I guess most people today would think “LinkedIn.” But I can tell you that my most successful hires have always come from referrals.

I believe that the screening and interview process is very important. I also believe that each hiring manager needs to create an interview plan and identify the “features” of the type of engineers they are looking for.

What do I mean by a “feature”? A feature is a set of traits that you believe will improve an engineer’s chance of being successful in the role. For me, I usually look for interviewees who love to listen to others, are patient, have good communication skills, and are good at the process of elimination.

Once I’ve identified the features I’m looking for, I need a hiring/interview plan.

Here you go:

Create a clear job description and define roles/responsibilities for the hiring role.
Identify a few key interviewers for the hiring panel.
Set up a good set of technical tests for candidates. I like to use hackerrank to set up my tests. One of the features of hackerrank that I really like is that you can create test cases and test each candidate’s solution.
Make sure you come up with a list of problems to pose to your candidates that can help you judge whether they have all the features you’re looking for.
Don’t forget to sell your team and vision to the candidates! Remember, the evaluation process goes both ways! You want to make sure the candidate likes what you’re offering.

Good luck on your next hiring!

PS, I have my list of favor questions, DM me and I’ll share with you!

February 26, 2018February 26, 2018

My Experience on using JIRA Cloud API to customize your release and quality data

I am sure a lot of us used JIRA for bug tracking, sprint planning, story telling, features logging, and etc… Most of the QA I know somewhat touches JIRA one way or the other. I will walk you thought how to connect to JIRA api cloud, also give out some of the pain points I had from my experiences.

Before I listed out the steps, I expected you know some basic about JIRA on how to setup projects as admin, create issues, and use jql. If not, please view this video

or (https://confluence.atlassian.com/jira/jira-documentation-1556.html) to get help.

Assuming you already have JIRA cloud setup, like for an example, http://yourproject.atlassian.net, and have an atlassian cloud user account, here is the step by step, on how to connect to your JIRA cloud API.

1. First you need to encoded your username (JIRA cloud email login) and password in base64. Let say your JIRA cloud login address is abc@hello.com and password is 1234, you will encode this as: #echo – n “abc@hello.com:1234” | base64 to get the encoded string for your basic JIRA api authentication. Make sure you use “-n” because echo will attach a trailing newline char at the end.

2.Now you can connect using this simple ruby script:

require 'httparty'

## Create your JQL query here
jql = 'text~ "' + "find X".to_s + '"' + " and project = YOURPROJECT"
yourencoded = "" ## Put your encoded string on step #1 above

## Header
@jurl= "https://yourcompany.atlassian.net/rest/api/2/search?jql=" + URI.encode(uri_text)

### Now Loop through each issue from the search
 response=HTTParty.get(
 @surl,
 headers: {
 "Authorization"=> 'Basic '+ yourencoded.to_s,
 "Content-Type"=> 'application/json'
 }
 )
 result=JSON.parse(response.body)
 result["issues"].each do | issue |
    ### Now refer to the JIRA issue doc, you can do your logic here
 end

3. you are done, I will put this in some kind of chart tool so you can create your own dashboard

January 23, 2018January 25, 2018

Building test tool – Setup Ruby on Rails in windows through ubnutu

You always need an easy way to develop simple web tools to assist on your testing. Ruby on Rails provide that simple framework. Not too much overhead of installation and deployment, as well as simple setup to get your local webpages running. Today, I will walk through how you can set this up in window 10.

First, you need to setup Ubuntu in your PC. (Read here to see how you can do it)
Open up a Ubuntu shell
Check if your ruby is install. If not, do #apt install ruby
I usually install the rvm on top so I can manage my ruby version and gemsets: #
```
curl -sSL https://get.rvm.io | bash -s stable --rails
```
Get the nodejs, sqlite3 # apt install sqlite3
#sudo apt-get install -y nodejs
incase if step 4 didn’t install your rails, do the #gem install rails
Create your new rails app #rails new app
cd app
rails server

Now open browser, and go to localhost:3000

you will see:

Notes:

#1 You need to shutdown your IIS, if localhost is taken and you will not able to start your rails server.

#2 Git your changes!

January 21, 2018January 24, 2018

How to test application that uses machine learning?

When working with application that uses machine learning to drive decision. Testing becomes very challenging. For examples, systems like review spam detection, images categorization applications and etc. So as a testers, how do you write your tests for these applications? What kind of automation framework we should use?

I recall that I tested one application which using machine learning algo to detect spam reviews from user submissions. The way I constructed my test plan, I broke down into 3 parts:

Technical functionalities: all the test cases/automation which check the most basic technical functionalities. (if all these test passed, means the application runs and have some output, but it won’t tell you if the review spam is catching the right stuff.)
Established baseline reviews for both spam/not spam: Manually create 50 reviews that are “spammy” and 50 legit reviews. Run through the application to make sure they did it correctly. We automated using a simple PHP program to run through this step
Established an on going process (what I call AI for testing) to automatically feed more spam/not spam reviews from the output of the application and feed it back to step #2.

This framework did help us to find a few regression bugs. Not perfect, but yet it works.

December 20, 2017January 20, 2018

Step by Step automation guide – Selenium using Ruby Mini-Test framework

Setup Automation using Selenium/Ruby Mini-Test in 10 minutes (step by step) The assumption is that you are using a MAC…. (sorry PC users)

1. 1. Download a webdriver (in this case, I am getting a chrome driver: https://sites.google.com/a/chromium.org/chromedriver/downloads . Make sure you set the PATH which you where your webdriver was install to.
  2. get rvm (which is to set your ruby version, and get different gems (ruby libraries) https://rvm.io/rvm/install
  3. install the following gem libraries:
    1. gem install minitest (Mini Test framework for all the assert)
    2. gem install selenium-webdriver (this is pretty obvious)
  4. Now create a test_login_file.rb like the following code, then run: ruby test_login_file.rb
```
require "minitest/autorun"
require "selenium-webdriver"

class TestLogin &lt; Minitest::Test
 def setup
   @driver=Selenium::WebDriver.for :chrome
   @driver.navigate.to "http://yoursite.com/login"      
 end

 def test_login_correct
       @driver.find_element(:id,"login").send_keys "username" 
       @driver.find_element(:id,"password").send_keys "password"
       @driver.find_element(:id,"submit").click
       sleep (1)
       if @driver.find_elements(:id,"logins").size <= 0 
	   assert false, "Bad Login"
       else
       	   assert_match "success",ps, "Page Failed"
       end
 end
end
```
Now there you go with your first automation test! easy?
Okay, I guess I still own you some explaination

The setup function basically tells your program to start chrome browser and go to url “yourlogin.com”. find_element is part of the function within the webdriver class, which do different thing within your browser window (you can find all the reference here: https://gist.github.com/huangzhichong/3284966)

Each Minitest class represents 1 test case.