Introduction

There are a few words, like unit testing, integration testing, mocking, dependency injection, object oriented proramming. These are all friends, and areall related to each other. In this post I’m going to talk how, and then it will start to make sense to you.

This post is divided into these subsections:

• Preamble
• The Testing part
• The OOPs part
• The dependency injection part
• A real life problem

I’ll start with how I started with unit testing.

While you read this post, take a moment to connect with me on LinkedIn.

Preamble: My journey of testing

In my journey of TDD, at first there was unit test…. I was not aware of test driven development from start. It took me some time to gain traction on unit testing. And even if I because aware to it, practicing it in Animation and VFX industry was really tough.

• Production always wants things done as quickly as possible.
• It was hard to test stuff inside DCC packages like Maya, Nuke or Houdini etc. It is a direct jump to integration test or UI testing.
• I was not trained in testing like software engineers are.

I don’t say that studios don’t write tested code. The company I currently work in tests. But for whatever business reasons, some development teams don’t invest time in testing. And I know, this could be true for any company which is not a tech company.

When I was working with Go, I read a book called Learn Go with Tests. This book encouraged me to embrace Red-Green-Refactor mantra. That is how I came to know about Uncle Bob. Testing is also crucial from the perspective of deployment and CI/CD. The build/deployment should fail if any of the test is failing.

After realizing all these, it was time for me to embrace testing. But I still find some people who say testing does not add any value to their workflow. That is just bullshit.

…then there there came databases. Testing an add function is easy. You test it with different values, and even with different type of data types. But have your tried testing an HTTP endpoint which writes row/document to a database? If you have, then you might know that if you test a route, whose handler writes rows to database would fail for second time endpoint is hit. See the link I metioned above to know what I’m talking about.

There are possibily two things that can be done in this case:

1. Use a test database inside current dbms and delete the test table/collection every time test is run.

This is more of an integration test. Because we are dealing with real database.

Although integration tests have their own charm in software testing, I don’t like this method of testing specifically in this case. Why? Think of running your integration test in CI environment. You need a real database instance there, and I don’t want that. Also, a core rule for unit tests is that they should be autonomous.

2. Second option is to mock the call to database and stay in the realm of unit tests.

This mocking is achieved by a technique called dependency injection. I have written a whole section dedicated to dependency injection below.

For now, let’s talk about both integration test and mocking a little bit before we dive deep into DI. Then we’ll circle back to this database example.

The Testing part: Interation Test and Mocking

In last section we had a situation, we have to test our application, and also do not have to forget that our application depends on a database.

Integration test in real life

You can continue unit testing with databases. But we’ll call database an external dependency here. And when a test is dependent on external resources, it is an integration test. And one of the core things about unit test is that it has to be self-contained i.e. not depend on external resources.

When you depend on external resources, such as a database, it no longer remains a unit test, it becomes an integration test. Integration tests are different than unit tests. This is one stage above the unit test as seen in below diagram.

        /\
       /  \        UI / End-to-End
      /----\
     /      \     Integration/System
    /--------\
   /          \          Unit
  --------------

Example of a laptop manufacturer

Let us understand this scenario with an example of laptop production company. Like any other production company, its very unlikely that a single company will produce the entire laptop. Nope, that’s not what usually happens. Each component can come from a different company. The processor can come from company X whereas the hard drive come from company Y.

When company X produces a processor, they will test it before it sells to the laptop manufacturer. We can consider the testing they do as unit test; that is because they test each single unit of processor. Same thing happens when hard drive manufacturer has to sell a unit to laptop manufacturer. They unit test each unit.

How how does this relates to our programming scenario? We’ll see next…

When the laptop company receives the parts from company X and Y, the laptop company also runs the same unit test from the specific companies. Thereafter the laptop company plugs the parts into rest of the laptop and runs a so called integration test to check how different units behave in combination to other parts.

In a way you can say that integration tests are the big brother to unit tests; and is more like a next step to unit tests. They are used in a combination in software production as well as laptops, cars and whatnot.

We’ll not be talking about integration test here in this post, because I just wanted you to get acquainted with the word integration testing and don’t be confused between unit test and integration test.

Mocking in real life

Mocking is an art of replacing the part of the application you are testing with a dummy version of that part called a mock.

Mock objects are simulated objects that mimic the behavior of real objects in controlled ways, most often as part of a software testing initiative. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.

How do we do this in code? Let’s see an example in Python progamming language. Suppose the function below is part of a rather very large application.

Let’s assume the function below is part of a very large application and depends on a third party API.

1
2
3
4
5
6

def get_repo_by_username(username):
    """Get a repos for a specific username."""

    request_url = "https://api.github.com/users/{user}/repos?per_page=1&page=1".format(user=username)
    response = requests.get(request_url)
    return response.json()

In above code, the connection to api.github.com is an external dependency. Of course you can test this code, but you should keep these things in mind.

• You will need access to api.github.com which might or might not be problem for you or your organisation.
• If test for this function is running on a CI/CD server like Jenkins, that machine also need to have access to api.github.com.
• For every test that connect to external server like in above case github.com, each test will have some connection overhead. This time will be added to total time which needs to run the test suite.

Basically in Python, you can use @patch decorator from unittest.mock module. Then you can also make use of return_value or side_effect argument with the patched method. I’m not going to replicate all of what is already written in Getting Started with Mocking in Python

All in all, you can run your unit tests without use of mocking, but it will slow down the test execution time and will be dependent on external resources.

Note: If you don’t mock above function, don’t forget that it is by theory not a unit test, but an integration test as we discussed in last section. This integration test tests integration between the application and GitHub.

The OOPs part: A principle from the OOPs world

Until now, we know what is integration test, what is unit test, and what is difference between them. We also know how we can use a technique called mocking to mock a call to external dependency.

Heard of SOLID principles? Yes? You know what D stands for? D stands for Dependency Inversion Principle. If you have not yet heard about SOLID principle, please consider reading this wonderful post by Samuel Oloruntoba. The code example in that post uses Java, but you can easily map it to your favourite language OOPs contruct.

Dependency Inversion Principle states that:

Entities must depend on abstractions, not on concretions. It states that the high-level module must not depend on the low-level module, but they should depend on abstractions.

You know how does it looks like in a picture?

What I mean to say by above diagram is, DIP states that our code should not directly depend on the EmailGateway class. Rather it should depend on some other class which is based on abstract class IEmailGateway.

This helps us easily replace the EmailGateway code without affect the actual EmailGateway.

The dependency injection part: So where does DI fits in?

According to Wikipedia, dependency injection is a technique in which an object receives other objects that it depends on, called dependencies.

Remember what I said in mocking section? GitHub in that case was a denpendency to our application. Did we do dependency injection when we mocked the call to GitHub? Yes, we did. While that example was not a very great one to explain this concept, I’ll go back to the application example which interacted with database.

But first, visualize a Composite Pattern in your mind. Suppose your application depends on a database to function, you’d not want go ahead and write the database interaction code tightly coupled to the application. Instead the way to do is to write an interface first. This helps us create our components in the application as orthogonal as possible.

This interface will have methods like, create record, read record, edit record, delete record. Then you can go ahead and fulfill the contract of the interface.

This might seem a long process now, but here are the benifits:

1. You can fulfill the contract for multiple database systems if you change them in future.

Suppose you are using Postgres right now. You only need to implement the interfaces for postgres. You write methods to do create, read, edit, delete in postgres style.

Later on if you decided to use MongoDB, you can re-implement those interfaces separately, without editing the postgres one.

2. This also complies with Open-Close principle of SOLID.

Remember that the intent behind dependency injection is to achieve separation of concerns of construction and use of objects. This can increase readability and code reuse.

https://en.wikipedia.org/wiki/Dependency_injection

A real life problem

This exampleis mostly derived from one of my previous series which is called TDD Auth with FastAPI.

The problem

My fictious product has a signup system where user can come up and sign up. And as far as possible, I wanted to test this authentication system with same passion as other part of my systems.

The problem is when a test is run, it registers a new user. But when the tests run for second time, it fails. After some investigation, I found out that that entry is not removed when that test is finished.

1. I don’t want to the test to execute something which writes to production database.

I can definately rollback changes done to production database with something called fixtures (or tearDown mechanism), but that’s not an option.

2. Can’t afford to start a database server when testing.

I wanted to unit test my API server which connected to mongo instance. I just can’t test it because I can’t afford to start a separate database server just for sake of testing.

The solution

The solution is of course, our Dependency Injection!

The only way left to deal with this problem was to use a database like SQLite for testing purpose.

Separate database session object for testing and production

The database management system I was using was PostgreSQL. While both Postgres and SQLite are SQL databases, the ORM I was using played an important role to solve this problem as most of the database abstraction by done by SQLAlchemy.

The only thing which I had to do was to create a mechanism which returned a connection to SQLite when test was running and connection to Postgres when postgres was running. I created this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

class BaseDBInit:
    def __init__(self, db_uri) -> None:
        self.db_uri = db_uri
        self.create_engine()
        models.Base.metadata.create_all(bind=self.engine)

    @abstractmethod
    def create_engine(self):
        pass

    def get_session(self):
        session = sessionmaker(autocommit=False, autoflush=False, bind=self.engine)
        return session


class DBInitTest(BaseDBInit):
    def create_engine(self):
        self.engine = create_engine(
            self.db_uri, connect_args={"check_same_thread": False}
        )


class DBInit(BaseDBInit):
    def create_engine(self):
        self.engine = create_engine(self.db_uri)

I know it’s mostly inheritence, I had to do this because Python does not have a interface keyword, but you got the gist.

The client of these classes can pass in a db_uri and call the method get_session. Here is an example from test code.

1
2
3
4
5
6
7
8

def get_test_db():
    session = None
    try:
        session = DBInitTest("sqlite:///./test.db").get_session()
        session = session()
        yield session
    finally:
        session.close()

Inject database dependency on runtime

Whenever my test runs, I have used a dependency injection mechanism which comes pre-baked with FastAPI which makes this work really easy. So here is what I had to do in addition to the code I wrote above.

1

app.dependency_overrides[get_db] = get_test_db

What above code does is it overrides get_db. This get_db is actual production version of database connector. This function looks something like this:

1
2
3
4
5
6
7
8

def get_db():
    session = None
    try:
        session = DBInit("postgresql+psycopg2://postgres:postgres@localhost/fastauth").get_session()
        session = session()
        yield session
    finally:
        session.close()

You see how can same BaseDBInit be used to create connectors for both production and testing? This is the power of object oriented programming. app.dependency_overrides[get_db] = get_test_db is how I inject database dependency into the application.

This along side with pytest fixture I can remove SQLite database after every test run. My fixture simply looks like this:

1
2
3

@pytest.fixture
def delete_database():
    os.unlink("./test.db")

Which indeed removes test.db file as we defined in the get_test_db. I call this fixture when I have to run test which writes to database.

For an example in Go, I’d direct you to https://appliedgo.net/di/.

And this is how we come to end of this post. If you are a first time reader and you liked this post, please consider subscribing to this blog via email on the main page. Also feel free to leave feedback in comments. You can reach me @sntshk. See you later.

If you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.