Improving Data Integration Test Performance with Mock

Have you ever wondered how Data Engineers test code components that involves data endpoints? Testing in data engineering typically starts with unit tests — checking if a single encapsulated function returns the expected results given input. Besides unit tests, integration tests are also used to evaluate the behaviours of multiple functions or modules combined together. In data engineering, integration testing usually involved components of data endpoints, such as SFTP, databases, Blob storage, and so on. As best practices, integration tests should never make requests to actual servers, because that would make the test flow dependent on the uptime of the dependencies. For example, if the service is momentarily unavailable, the integration test will fail even if the logic of your code is correct. In addition, exchanging data with actual servers can significantly increase the total runtime of integration tests as the codebase gets bigger. Therefore, data endpoints are often Mocked in integration tests to improve test reliability and performance. In this blog, I will talk about what Mock is and the benefits of Mock, and walk through integration test examples using Mock library to test functions that exchange data with SFTP servers. <a href="https://blog.det.life/improving-data-integration-test-performance-with-mock-ef6b357456da">Click Here</a>