Until now, building versatile automation by simply writing code – and without the drudgery of dealing with repetitive, boilerplate infrastructure tasks – was a fantasy.
Most developers have encountered the challenge of automating tasks that connect various applications or services. Whether these automations were seemingly simple or complex, there was always one constant frustration: the considerable time investment required for automation. The truth is that even the most straightforward automation task involves a slew of technical challenges, such as handling APIs setup and authentication, ensuring reliability, recovery from failures, deployment, managing automations, and more.
To tackle this problem, we decided to build a platform that would simplify automation. This platform is built around an exciting new concept – we call it “automation-as-code” – that’s at the heart of what we’re doing at autokitteh. This novel approach aims to empower developers to code and manage reliable, long-running, versatile automations with minimal effort and a basic level of technical expertise.
But before we elaborate on the benefits of automation-as-code, let’s quickly review the challenges it is intended to solve.
What is automation/orchestration (in this context)?
Automation, in software development, refers to utilizing tools, scripts, or software to execute tasks, processes, or operations autonomously, devoid of manual intervention. In our context, automation entails crafting orchestration that interconnects APIs from diverse systems or services. This orchestration, often termed workflow automation, is pivotal for today’s distributed systems. Automation is everywhere, touching various use cases in software development, DevOps, business operations, and more.
What makes automation so challenging?
The development of automation, and mission-critical automation in particular, is anything but simple. Beyond implementing the business logic, developers must address numerous other challenges, including the following:
- API authentication and configuration: Managing authentication, handling APIs across multiple interfaces, and more.
- Synchronization and data sharing of multiple automations: Navigating the complexities of data synchronization across multiple automations that depend on one another.
- Robustness in connectivity: Addressing network glitches, temporary service unavailability, etc.
- Automation reliability: Handling server crashes that necessitate recovery, often requiring stateful application development.
- Visibility and automation management: Ensuring effective administration and automation tracking and version management.
- Long-running automation support: Some automation requires asynchronous workflows, especially when a human is involved in the process. Such workflows, which can take a long time to complete, are implemented as asynchronous tasks based on states.
- Deployment: Execution of automation, whether on-prem or in the cloud, requires resources and knowledge in DevOps.
- Monitoring the server and the automation execution – Once the automation is deployed, monitoring and error handling are required.
Tasks for building mission-critical automation
As can be seen, constructing robust automation for mission-critical tasks surpasses mere scripting; it’s a multifaceted endeavor requiring effort and vast technological knowledge in many disciplines.
Understanding the limitations of existing automation tools
No-Code and Low-Code Tools
Recognizing the complexity of automation and connecting APIs in general, no-code and low-code platforms have stepped in to simplify the process. Tools such as Zapier, Make, n8n, Pipedream, and others excel at streamlining automation, particularly when integrating public applications such as Slack, Google, Github, etc. However, these same platforms are limited in supporting custom workflows and use cases and do not allow writing scripts freely as done in code. The limitations of no-code and low-code platforms include, among others:
- API Subset: Some applications offer only partial access to APIs of the integrated service, potentially hindering diverse integration needs.
- Long-running constraints: In some tools, it is impossible to build long-running or asynchronous automation since executions are limited in run time.
- DAG Dependency: Certain tools mandate Directed Acyclic Graph (DAG) workflow configurations, constraining flexibility.
- CI/CD: No-code and some low-code tools store the automation in the platform. Thus, version management, bug tracking, and many other features in the development life cycle are not part of the development CI/CD.
- Reliability: Though no-code and low-code tools are reliable, the developer still needs to take care of temporal connectivity failures and recovery due to failures in the applications they connect to.
- Cloud/on-prem – Some platforms are cloud-native, which could pose security restrictions for automation of the organization’s mission-critical tasks.
Do It Yourself Automation
At the other end of the spectrum, building automation in code from scratch enables highly versatile and limitless automation. However, this is not as easy as it sounds. To make it all work, the developer needs to be proficient in authentication, connecting to APIs, deployment, reliability, monitoring, and many other issues. In fact, coding the program logic of automation is just a small (and relatively simple) part of the whole. For this reason, a seemingly straightforward automation might take days to build and require technical knowledge across many technologies.
Mission Critical Automation Platforms for Developers
In the middle ground, there are various frameworks for developing mission-critical complex workflows, such as Temporal, Apache Airflow, AWS Step Functions, and more. Those frameworks provide high flexibility and are great for developing reliable workflows. However, they require deep technical expertise and leave high-level features, such as connecting to APIs, for the developer to deal with.
Workflow automation solutions landscape
Automation-as-Code to the Rescue
To address the gap between less flexible tools that require basic technical skills and highly flexible frameworks designed for professionals, we have introduced the “automation-as-code” concept.
Automation-as-code lets the platform handle the various fundamental automation challenges mentioned earlier. By providing an abstraction layer, developers can focus on coding the business logic in a familiar programming language, while the platform takes care of the rest, reducing the need for complex boilerplate code.
To implement automation-as-code, a platform is required to enable the execution of “code” offering the full spectrum of programming language capabilities – loops, conditions, variables, and library imports. Under the hood, the platform seamlessly handles essential aspects related to building robust distributed systems, such as reliability and temporal error network handling. At the same time, it provides the tools for easily connecting to various APIs as well as visibility for monitoring and debugging. Once the framework is deployed, the developer does not have to worry about infrastructure (i.e., serverless).
Automation-as-code can be deployed on-prem, in the cloud, in a hybrid environment, or even on a PC to support various use cases while addressing security and organizational restrictions.
What does automation-as-code look like for a developer?
Using such a platform, the developer can easily perform the following tasks:
- Set up APIs: Configure and authenticate the APIs
- Develop automation business logic: Code the workflow in StarLark / Python / Typescript or other languages.
- Deploy: Validate and deploy automation in the platform
- Manage and monitor automations executing in the system
The platform transparently handles the rest:
- Provides an environment to deploy and execute the automations
- Assists with connecting easily to APIs
- Provides facilities for data sharing and synchronizing between automations
- Seamlessly handles temporal failures in the network or connectivity to services.
- Automatically recovers in case of failure – no need to manage states in automation
- Provides monitoring and visibility for running and completed automations
- Manages users, versions and resources
Automation-as-code marks a significant stride in the evolution of automation and workflow orchestration. By blending the prowess of programming languages with comprehensive platform capabilities, this approach empowers developers to construct resilient, adaptable, and reliable automated workflows.
If you’re interested in hearing more or sharing your thoughts and use cases, drop us a line: firstname.lastname@example.org