Software Test Engineering at Axial

At Axial, all of our Software Test Engineers are embedded in the development teams. We come in early on in the development life cycle to do the testing. We firmly believe that quality can be built-in from scratch, therefore, we spend a lot of time educating other members of the team how to think of testing. We encourage developers to test, write unit tests, and pair program. Everybody on the team should be able to test and write automation to support the testing activities. Our developers allocate a certain amount of time to write unit tests and are responsible for them. Both testers and developers have shared ownership over the integration and API tests. Here is where we pair up to build out test suites based on test ideas. We use mind mapping tools, stickies, and brainstorm together to determine what needs to be automated. A good rule of thumb is to focus on automating the repetitive regression tests as well as the scenarios based on recurring bug fixes. Automation offers invaluable support to to us testers, and it allows us to focus on the exploratory and smarter type of testing that lets us catch more corner cases.

Pairing is something that is very important when it comes to testing. Having a fresh set of eyes look at a piece of software helps us broaden our perspective. To not only pair with other testers, but also, with developers, product managers, customer service, sales representatives, and even end-users helps us get better perspective and momentum.

We spend time creating documentation both in a written format, but also, in visual and video format. This allows for the ability to get new people on board quickly and share this knowledge across departments. As testers, we are serving many stakeholders. By keeping documentation up-to-date, we are enlightening the organization about testing, and that generates more interest in testing. One thing we value a lot at Axial is “dogfooding”; to have people from all departments test is valuable to all. It provides us with feedback so that we can develop better software.

Some might say that testing is dead and that everything can be automated, but that could not be further from the truth. Testing is a craft. It is an intelligent activity performed by humans. Computers help us save time and money, by automating what is already known, but having smart testers embedded in development teams brings real value. To communicate well and to ask questions about development, risks, requirements, etc. are two of the most important skills a tester can have, especially in an Agile team. Asking questions to gain knowledge and communicating well will help you to quickly identify risk areas and prevent development issues ahead of time potentially saving the team both time and money. These abilities are something that differentiate an excellent tester from a regular tester whose mindset is set entirely on just finding bugs.

There are many branches of testing to think of. Functional is one, security, performance, UX, location and accessibility testing are some others. Tools can help us do our job better. But it is how we think and act as testers that will make a difference. Keeping a positive, solution-minded attitude and looking at issues in an objective manner helps to eliminate personal constraints in a team. We share the quality interest, and work, together as one to ship a quality product.

Staying connected to the software testing community, going to conferences, meetups such as CAST, TestBash, NYC Testers and Let’s Test, and interacting with great people in the field really helps motivate us, gives us new ideas and brings our craft to a whole new level.

We read a lot, listen, follow (and question) some of the thought leaders in the field. Here are three testing quotes from some of my favorites:

“The job of tests, and the people that develop and run tests, is to prevent defects, not find them” – Mary Poppendieck

“Documentation is the castor oil of programming.” – Gerald M. Weinberg

“Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous.” – James Bach

 

If you are interested in chatting about testing, automation tools and different methodologies, please feel free to reach out to us @axialcorps

In the upcoming blog posts we will do on Software testing, we are going to post hands-on tutorials, videos on how to set up test environments with Protractor and, Gatling, as well as break down how we do pair testing. Stay tuned!

Test Data Generation

Testing software is reducible to the Halting Problem. No software program, no matter how small, can be proven (or tested) to show that it will “always” work as expected. Even the simplest program:

#!/usr/bin/env python
print (“hello world”)

can break if the python executable is moved to a different directory on your system, or due to operating system failures or physical hardware failures.

Realistically, today’s programs are increasingly complex and built on top of multiple frameworks, each with its own bugs. And all of this before you get to testing different browsers and browser versions for web applications. What test engineer hasn’t heard: “This does not work on IE9”? There are an infinite number of factors that can cause a program to fail.

In addition to the challenge of testing complex programs, there is increasing pressure to test and ship faster. To keep pace, you have to automate a sufficient amount of testing. When testing web interfaces, even with popular functional testing tools like Selenium-Web Driver or QuickTest Pro (QTP), your automated tests can quickly become brittle and obsolete against a constantly changing feature set.

Data-Driven Automation

To keep tests from becoming overly brittle, test engineers try to separate test data from testing logic, choosing to store test data in a database or config file rather than hard-coding test data with test code. Tools like pytest’s mark.parametrize allow you to make the most of this test data by treating each test entry as a separate test-case.

Remember that the goal is still not full coverage, you’re not trying to test every possible permutation of bytes for every text field. The goal is good coverage with minimal test data, taking a tip from The Art of Software Testing we can quickly get to “better coverage” with a minimum of test data by using equivalence partitioning, and boundary testing, but even with consistent application of these techniques we’re left with a few questions:

  1. Who is going to write out all the test data for each field?
  2. With multiple interdependent form fields, how are we going to maintain the data?

The Answer? Automation!

Think of each field in a form as a node of a directed graph. If each field is a node in a graph, all you need to automate test data generation is a data-format for representing each node, and a graph generation utility. Given inputs that conform to our equivalence partitioning and boundary testing, we can then start to generate a minimum amount of test data with a trivial amount of code.

A first pass at graphing a simple form looks like this:

fig1

  1. Start -> ObjSell, ObjRaise, ObjExplore
  2. ObjSell, OppExplore -> NextButton
  3. ObjRaise -> CapSenior, CapMezz, CapEquity, CapAll
  4. CapSenior, CapMezz, CapEquity, CapAll ->  RangeTrue, RangeFalse
  5. RangeTrue -> CapMin, CapMax
  6. RangeFalse -> CapMin
  7. CapMin -> CapMax, NextButton
  8. CapMax -> NextButton

Which would generate test cases like this:

  1. START->ObjSell->NextButton
  2. START->ObjRaise->CapSenior->RangeTrue->CapMin->CapMax->NextButton
  3. START->ObjRaise->CapSenior->RangeFalse->CapMin->NextButton
  4. START->ObjRaise->CapMezz->RangeTrue->CapMin->CapMax->NextButton
  5. START->ObjRaise->CapMezz->RangeFalse->CapMin->NextButton
  6. START->ObjRaise->CapEquity->RangeTrue->CapMin->CapMax->NextButton
  7. START->ObjRaise->CapEquity->RangeFalse->CapMin->NextButton
  8. START->ObjRaise->CapAll->RangeTrue->CapMin->CapMax->NextButton
  9. START->ObjRaise->CapAll->RangeFalse->CapMin->NextButton
  10. START->ObjExplore->NextButton

With this one simple example, you can start to see the power of using graphs to generate maintainable test data. If fields change, you need only update the node description to reflect any changes and regenerate your test data.

The next step is automatically generating test cases and hooking it up to a test runner. Stay tuned!

 

 

Pytest – Testing Axial-Style Part 3

Squashing Bugs

Selenium and the Page Object Model serve as building blocks for our testing suite at Axial. But we’re missing the glue that connects it all together: pytest. Pytest is a popular alternative to python’s builtin unittest framework offering many highly useful features unavailable in unittest. Our favorite features include:

  • Test collection and execution
  • Fixtures and hooks (setup/teardown on steroids)
  • Reporting

To demonstrate pytest we’ll take a break from selenium and focus on sharing resources among test cases. In this demo we’ll write a pytest plugin for a hypothetical testing suite where the majority of test cases connect to our AMS (Axial Messaging Service) API. Wouldn’t it be nice if all our tests made use of the same session key and API connection? Not always, but for this demo the answer is yes.

Pytest Hooks

Our first step is to get an Axial session key that can be shared among all tests. Our plugin can do this by specifying a pytest_sessionstart hook, which is a function that will be called at the beginning of the testing session (i.e. when py.test is invoked). A full list of pytest hooks can be found here. Our sessionstart hook will generate a UUID and add it as a property to pytest’s session object.

from uuid import uuid4
def pytest_sessionstart(session):
    session.key = uuid4()

From here on any tests that make use of pytest’s request fixture can access our session key using request.session.key. More on fixtures in the next section.

Pytest Fixtures

At this point all tests can access the same session key but we still want a convenient way for each test to get connection to the AMS API. To solve this problem we will define a custom fixture. As the pytest documentation explains, the purpose of a fixture is to “provide a fixed baseline upon which tests can reliably and repeatedly execute”. Lets dive right in and define a fixture that provides an API connection to be shared among all test cases:

@pytest.fixture(scope="session")
def ams_api(request):
    ''' Return a connection to the AMS API '''
    return AxlClientApiManager(
        AMS_HOST,
        AMS_PORT,
        AMS_API_PATH,
        session_key=request.session.key
    ).api

The decorator tells pytest this is a fixture and the scope=”session” kwargs means the returned connection will be shared among all tests. The request argument is an optional instance of the FixtureRequest object, which tells us everything there is to know about the requesting test case, including the session key we set in our sessionstart hook. In fact, the request instance here is itself a fixture, made available using function arguments. The process of exposing fixtures using function arguments is a prime example of dependency injection. Our new fixture can be used much like the request fixture: as a function argument. For example, to test the echo function of the AMS API all we need to do is:

def test_echo(ams_api):
    ams_api.echo('Hello world!')

Pytest Plugins

In order to make pytest aware of these fixtures and hooks we need to define a plugin. There are many ways to define a plugin but for our little test suite the most convenient option is to create a conftest.py file that contains our fixtures/hooks and place it in the directory that contains our tests. As a result, any tests run within this directory or any subdirectories will be able to oh-so-magically access the same session key and API connection as described above.

Pytest Collection and Execution

The last step is actually running our test suite. The most common way to execute tests is with the py.test command. Tests are collected by recursively searching the specified directory for modules that begin with test_*, classes that begin with Test* and functions that begin with test_*. Use the –collectonly argument to see which tests will be collected without actually running them.

A rich set of arguments gives control over test execution and configuration. My py.test invocations usually contain the following options. Note that I can define a pytest.ini file with these options in the same directory as conftest.py and they will be used as default options.

-v
Show more when reporting, looks much nicer
--tb-short
Show shorter tracebacks
--capture=no
Don’t capture test stdout. My tests rarely write to stdout and when they do I want to see it right away
--pdb
Open an interactive pdb session on when an unhandled

In part 4 we’ll see how Selenium, the Page Object Model and pytest can be combined to yield a stupid simple web testing framework.

[image credit: shutterstock]

The Power of Socializing Performance

F

Back in the day, when I worked as the Python guru at a big investment bank, my basic mission was to make everyone’s code better.  With hundreds of thousands of scripts laying around the system written by programmers and quants of spectacular to dubious talent, this was a rough task!

When I first started, I would go around and sit at a quant’s desk and look for inefficiencies.  My first day, in fact, an MD pointed me at some random code on his screen and said, “I heard you make Python code faster.  Fix that.”  I told him to make a couple of quick changes and boldly stated it would be 15% faster.  He timed it.  It was 14.7% faster.  I apologized for being off so much.  He laughed and immediately sent off a message telling his minions to look for the inefficiency I pointed out and eliminate it in their code.

This socialization of information is important and useful.  It is, in fact, how I keep gainful employment.  As a guru, I seed the firm with better ideas and help spread them.

But information is slow to propagate. And there is only one of me…

At the bank, there were many hundreds of programmers spread all over the world that used the system. There were hundreds of thousands of scripts written and copied with little understanding of how or why they worked.  This phenomena of “cargo culting” propagates and perpetuates some really bad code.

How to get rid of it?

We had a peer review and testing process in place that was supposed to help us keep bad code out of the system.  Any code that was “in production” had to have a programmer from the group that “owned” the code check it and OK it before it hit the servers.  Similarly, some code had tests associated with them and the tests had to pass.

In principle, code was supposed to get better as well informed code blessers carefully oversaw the curation of the code.  As each script was touched, it would get better and better.  Nirvana.

In practice, as you might imagine, this provided little real benefit.  The blessing step was often a rubber stamp.  People had favored blessers who they would call on to do their checks.  After a quick glance and a perfunctory test it was blessed and pushed out.  When tests broke, they were sometimes just commented out.  And this did nothing for the large umbra of scripts that were not in production, but were important for many tasks.

We had an in-house lint tool that I worked on with an associate in my group.  It started as a simple analyzer and became more sophisticated over time.  We had some standard rules that applied to everyone though each group could add specialized rules.  It produced a letter grade (A-F) depending on the number of warnings and fatals it detected.

Sample rules were like: Don’t import string (except to get things like string.uppercase).  Use a generator expression rather than a list comprehension in sum (it’s much faster).

The tool was integrated into our IDE and when you violated a rule, it could pop up a discussion of why the rule was there and what a better form was.  Some of the rules even had an auto-correct feature that would edit the code for you (e.g. remove the extraneous [] in the sum with list comprehension).

We hooked it into the configuration manager to run the check.  If you got an F, you were supposed to abort the check in.  You could, in an emergency, check in anyway.  Also, if you suspected that the lint tool was wrong, you could mark a box and also proceed.  We hoped that this would catch problems early and keep bad code out of the system.

It didn’t.

Most people blindly checked the “Lint is broken” box and checked in code with F’s.  Blessers continued to rubber stamp bad code.  Crappy old scripts were still in circulation being blindly copied into new code.

We needed something better

We assigned “ownership” to every script.  If you checked it in or if you blessed it, then you owned it.  Then we built a auto-updated database that had the grade for every script.  It meant that we could give a grade, not just to every piece of code, but rather to every programmer!  We graded on a very generous curve.  You could get an A with something like 70% of the best grade in your group. We emailed everyone a weekly report showing them their grades.  Most people filtered it away.

How, we wondered, could we make people care? We tossed around some ideas, and in the end, we did something evil…

Like many banks, we used an internal chat program to keep tabs on problems and to communicate between groups.  One nice feature it had was a tool-tip that displayed a person’s full name, picture, phone number, and location when you hovered the cursor over their handle.  We decided to leverage that.

My minion updated the tool-tip display to show everyone’s grade in the text line at the bottom of the picture.  So, if you hovered over my handle, you would see my smiling mug and an A(95%) that denoted my overall script score.  But suddenly, there were a lot of people whose C’s, D’s, and F’s were suddenly visible to the whole world!

Now remember, a big bank is filled to the brim with people that never got an F in their life.  They went to the right schools.  They were top of their class.  They checked all the boxes in life … now they had an “C” or “D” or “F” next to their name.  A public “C” or “D” or “F”.

This. Would. Not. Stand.

Work ground to a halt as people started looking at and fixing all code that they owned.  Realizing that the easiest way to kill an “F” was to delete the script, thousands of crufty old scripts were deleted (a huge, unanticipated benefit!).  Blessers, realizing that they got tagged with ownership when they approved a crappy script started demanding passing lint grades on a piece of code before they would even consider looking at something.

As we found new bad code patterns, we added new rules.  When things broke in production, we added new rules.  This kept grades a bit of a moving target, but people quickly adjusted to the new regime.  A lot of the inefficient code that had been buried in scripts that were run every day, but rarely looked at was dramatically improved.  The overall safety of the code was improved.  We unleashed the power of socializing grades to fix the problem.

Then we added a second grade that measured the code coverage of tests…

[image via shutterstock]

Selenium for Automation – Testing Axial-Style Part 1

Testing Axial-Style

The challenge with testing is no different than most: allocating resources. Its hard to sacrifice resources for testing at the cost of new features and still feel a sense of accomplishment – especially when such a sacrifice needs to be made every time you release. At Axial we release every day and have come to realize that such sacrifices need to be made consistently in order to maintain a top quality product. We recognized the need to make our testing efforts more reusable. We accomplished this by doing what the corpus does best: build. From this we built Axium, a test automation suite that is easily executed, understood, maintained and configured. This is the first of a series of posts that shows how we built Axium and contains the following independent parts:

  • Selenium – For test automation

  • The Page Object Model – For test maintainability

  • Pytest – For test configuration

  • Axium – Bring it all together

Those testing web apps using Python will benefit the most, although topics covered are applicable to most languages and testing needs. 

Selenium

Selenium is a test automation tool that comes in two flavors:

  1. Selenium IDE – a Firefox extension that simulates browser interactions and optionally records them

  2. Selenium WebDriver – a to run interactions in the browser programmatically. Selenium WebDriver is the successor of Selenium Remote Control.

Part 1 covers how Selenium works using the IDE, WebDriver is covered in part 2.

Installing Selenium IDE

  1. From Firefox, visit this download page and click on the latest version of Selenium IDE

  2. Firefox will detect this is an add-on and ask for permission, click “Allow”

  3. Open with Tools -> Selenium IDE

  4. There are also a bunch of very useful plugins on Selenium’s download page 

To install Selenium WebDriver

pip install selenium

Example

Following is an example of creating a test case to get the weather at weather.com

  1. Open the Selenium IDE from Firefox using Tools > Selenium IDE

  2. Click Record (red circle on top right-hand corner)

  3. Switch back to your Firefox browser and …

  4. Switch back to Selenium IDE and …

    • Click Record again to stop recording

    • Run your test case by clicking “Run Test Case”

 

You should see something similar to the screenshot below. Selenium IDE shows the different commands that makeup the test case. Each command does the following:

  1. Open the root directory of the Base Url (www.weather.com)

  2. Click on the element with an ID of typeaheadBox (the search box)

  3. Enter “New York, New York” into the search box

  4. Trigger the click() event for each div with a class of “wx-searchButton”. Wait for the page to respond before continuing (although we don’t continue in this example).

  5. To verify correct output:

    • Highlight some expected output

    • Right click and select “Show All Available Commands” -> “AssertTextPresent was added successfully”


Selenium Constructs

Commands

Commands have three parts: the name of the command, the target and the value.

  • Actions – commands that perform operations, such as navigating a page

  • Accessors – are commands that allow you to get and set variables

  • Assertions – are commands that allow you to verify expected behavior

Locators

Element Locators are used to specify which HTML elements a command should operate on. There are many different and convenient ways to locate elements. Those who prefer jQuery selectors should check out Sizzle.

Sizzle

Sizzle is a library that allows you to select elements with jQuery and CSS 3 selectors in Selenium.

Installing Sizzle

  1. Download Sizzle on GitHub

  2. Extract to a permanent location

  3. Create user-extensions.js with the following contents:

 

PageBot.prototype.locateElementBySizzle = function(locator, inDocument) {
  var results = [];
  window.Sizzle(locator, inDocument, results);
  return results.length > 0 ? results[0] : null;
}
  1. In Selenium IDE, select Options -> Options and add sizzle.js and user-extensions.js to “Selenium Core Extensions”

  2. Restart Selenium IDE

 

Test installation with by executing the verifyElementPresent command with a target of sizzle=body and a value of true. When you execute the command it should highlight green, not red.

Patterns

Patterns can be used to locate text with regular expressions or wildcards. The majority of commands that accept patterns are assert and verify. There are some good examples here.

Variables

Variables are set with any of the store commands and can be accessed with ${var_name} anywhere text is accepted.

Real-World Examples

Testing via Django messages.info

Recording a test case that creates a Company through our admin interface is straightforward – creating a User for that Company is not. How do we know the ID of the Company that was just created? In this case we are lucky because our admin displays a message in the form of “The company ‘8528’ was added successfully.” We can grab the Company ID from this message using the storeEval command with the following JavaScript as the target:

window.document.getElementsByClassName('info')[0].innerHTML.match('\"([0-9]*)\"').pop()

Example of a Weak Selector

These Selenium commands fill out a form capturing historical financial information. Do you see anything that could be improved in order to make this more flexible over time?

A better test case would not explicitly target years by matching inputs with an ID of id_*-revenue and id_*-ebitda. This prevents the test case from failing when the year changes.

iframes

Selenium allows you to select iframes with the selectFrame command.

Waiting for ajax requests

If you want your test case to wait for an ajax request then call waitForElementPresent with the identifier of whatever it is you are ultimately looking for. The following test script sends a message to another user by (1) waiting for an ajax response with a send button (i.e. a button whose id starts with “send-”) to complete and then (2) clicking the button to send.


Further Reading