Situation: While working on working multiple countries/regions file (with one file dedicated to each country), we encountered the requirement to execute a specific set of data cleaning, validation, transformation and enrichment task for each of these countries. To streamline this process and enable the concurrent execution, I harnessed the capabilities of AWS Lambda, S3, and ECR to effectively achieve a highly efficient workflow.
Task: The task at hand is to set up a serverless system that can concurrently and seamlessly trigger multiple AWS Lambda(s), each of which is based on one of the rows in the configuration file. To accomplish this, it’s essential to create a well-coordinated system where one Lambda function acts as an orchestrator and another Lambda acts as the worker for these tasks.