SDK + AWS LAMBDA

SDK has recently started to leverage AWS LAMBDA more to help with data processing and cleaning in a few of our client's projects. We have also started to use it to help import data for COVID Mapping Project, like the recently added Apple Mobility data. It's become a great tool to quickly spin up an isolated script or collection of scripts that manipulate some of our data without having to deploy a website or push something to an EC2 instance.

Most of the time that we have utilized LAMBDA in SDK, we are not only importing but also transforming the data. Over the course of SDK's life we have constantly run into the need to cleanup or transform data after its been collected or imported. This is usually done to allow the data be more consumer friendly in the end. It also allows for some data-points to be created, like averages or the summing of values. Since data cleaning is done on a case by case basis, using LAMBDA allows us to separate out the dependency of the SDK codebase. Another benefit of this is that as we build in more features and tools for internal use, our clients will also be able to access these new features, tools (in most cases) and even create custom integration utilizing changes we've made for ourselves.

SDK started to leverage LAMBDA because we started to have to deploy the SDK codebase too much when we updated some of our scripts. To remove that step, we now instead house the code in LAMBDA. That code can then be triggered by an endpoint or setup with a cron job. Also, this allows us to house the code in a central location that can be shared between developers more easily. As our collection of scripts has grown, so to has our need to keep track of all these scripts. To do this, we have now created another LAMBDA script that helps us monitor our LAMBDA scripts. This allows us to funnel it into SDK, which can then be displayed on an internal dashboard.

AWS LAMBDA has become a key tool in SDK's tool belt. From importing data to cleaning data, LAMBDA is the go to for one off scripts or isolated jobs. Going forward, we even potentially plan to even split some core logic of SDK off into LAMBDA, allowing us to isolate and scale some key areas more easily.

‍