Simplifying Cloud Workflows: Migrating from AWS SWF to Lambda

Cloud computing is constantly moving toward solutions that are faster, simpler, and more cost-effective. Though AWS Simple Workflow Service (SWF) was once a key tool for managing complex workflows, but over time it changed. The need for more flexibility and lower operational overhead led many organisations to adopt AWS Lambda-a serverless compute service. One organisation adopted this shift which wasn't just about switching tools-it was about rethinking old decisions based on how cloud services have changed.

This transition was guided by the insights and direction of senior engineers on the team. Their deep understanding of how the systems behaved helped spark a new approach. Sai Krishna Chirumamilla was one of the engineers who contributed to putting this change into action. Working closely with senior team members, he helped adapt the new architecture and turn it into a working solution. "I'm proud to have been part of a successful implementation that has significantly benefited our team and our users," he added.

While both SWF and Lambda are serverless services, they serve different purposes. SWF is well-suited for long-running workflows with complex coordination and state tracking. However, it requires considerable setup and ongoing maintenance. Developers must manage task workers, deciders, and failure handling logic manually, which can lead to increased operational complexity over time. In contrast, AWS Lambda is designed for short-lived, event-driven tasks. It scales automatically, integrates natively with many AWS services, and simplifies development by handling the infrastructure behind the scenes. This makes it a better fit for many modern, streamlined use cases.

Chirumamilla explained that his team had used SWF and other workflow engines for years, and at the time, it made sense. But things had changed. AWS Lambda functions could now run for up to 15 minutes (up from the previous 5), and most failures in SWF were occurring during activity invocations-due to timeouts, retries, or communication issues-not midway through execution. With this in mind, the team decided to explore a simpler approach. Instead of splitting everything into separate steps, they created a single Lambda function using Python's asyncio module. This allowed them to run multiple asynchronous tasks concurrently within a single function.

The results were great-the new setup completed tasks in just 2.4 seconds, the code was easier to read and fix, and it ran much faster. Execution time dropped by 70%, and the system became more reliable, with success rates going up from 92% to 99.9%. Problems during busy times also dropped by 68%, which saved the team time and stress.

Sharing the transition's impact, Chirumamilla mentioned, "The shift to Lambda streamlined our workflows and reduced the operational complexity associated with managing them." He also noted, "The pay-per-use model of Lambda also optimized our costs, as we only pay for the compute time actually used by our functions. This allowed us to allocate resources more efficiently and focus on delivering new features and improvements." Furthermore, testing improved with everything in one Python function. The team used Pytest, boosting test coverage from 60% to 95% and building features 40% faster.

The migration, however, wasn't instantaneous. Refactoring legacy workflows required time, effort, and careful planning. The team had to consider how to handle failures, manage state, and enforce timeouts within a single long-running function. They started with a small pilot, validated the approach, and gradually expanded it to additional workflows-minimizing risk while building confidence. The results justified the effort. "Through collaboration with my senior engineer and leveraging AWS's robust tooling, I successfully navigated these challenges and contributed to a seamless migration," Chirumamilla shared.

Importantly, workflow engines like SWF weren't completely dropped off-they are still used for complex, long-running tasks. But now simpler tools like Lambda are preferred when possible. This shift led to faster, smoother workflows and better customer experiences, inspiring other teams to follow. It shows how rethinking old choices can lead to big improvements.

Now, looking forward, as the cloud would keep evolving, significant improvements will come not just from using latest tools, but from stepping back and asking, "Is there a simpler way to do this now?" Often, the answer can lead to better performance, smoother operations, and more time to focus on building great products.