Serverless Orchestration Using Cloudify

Serverless is a buzzword in cloud computing world where developers do not need to worry about servers management at all as they only need to focus on writing code and deploy it. Amazon introduced its Serverless service AWS Lambda in 2014 and then other cloud providers started introducing their Serverless services such as Azure Function, Google Functions, IBM Cloud Functions and many others.

Every cloud provider has its own way for configuring, deploying and managing Serverless. For example AWS Lambda used AWS CloudFormation or SAM, Azure Functions use ARM and Google Functions use Cloud Deployment Manager. However, managing Serverless functions from different providers is not an easy job especially when it comes to use CI/CD on Multi-Cloud environment. There are multiple tools/frameworks used to abstract and hide the complexity of the underlying Serverless providers using one interface and unified yaml syntax. Serverless Framework is one of popular tools which can be used to deploy functions across different cloud providers.

Serverless Framework

Serverless Framework

It’s an open source framework which allows to deploy and configure Serverless functions across multiple cloud providers. I’m going to create a simple function and deploy it to AWS Lambda using framework itself before integrating anything with Cloudify.

Note: Serverless Framework is written in NodeJS which requires NodeJS to be installed on your machine. More information can be found here

1. Make sure that Serverless is installed on your machine.

Serverless version

2. Create Python3 function on AWS called demo.

Serverless boilerplate

3. The above command will generate two files: serverless.yml & handler.py

4. We can use the generated boilerplate and deploy it to AWS.

5. Now it’s time to deploy the function to AWS but make sure that AWS credentials are set.

export AWS_ACCESS_KEY_ID=<your-key-here>
export AWS_SECRET_ACCESS_KEY=<your-secret-key-here>

6. Run serverless deploy

Serverless Deployment

7. Once the function get deployed to AWS we can invoke it using serverless invole -f hello

Serverless Function Invocation

8. Serverless Framework also allows us to remove all of the created resources associated with the deployed function by running serverless remove

Severless Function Removal

So all the previous steps are required in order to manage the full lifecycle of Serverless function using AWS lambada. But wait !!! someone will ask , the title of this blog is Serverless Orchestration Using Cloudify, what is cloudify ? and where is Cloudify from this ? be patient will come to this in a moment ;)

Cloudify

Cloudify is an open source cloud orchestration framework which is not limited to infrastructure provisioning but allows to model applications and services and automate their entire life cycle, including deployment on any cloud or data center environment, monitoring all aspects of a deployed application, detecting issues and failure and its based on TOSCA DSL.

Cloudify

Pluggability is one of the cool features provided by Cloudify where it can be integrated with any configuration management tool, cloud providers, containers, Kubernetes and the list goes on. Cloudify is written in Python and has already a lot of official plugins that can run on top of it. I decided to build a Cloudify plugin for Serverless Framework which enables to write standard TOSCA blueprint (DSL) that describe the functions I need to deploy.

Cloudify Setup

1. Install Cloudify from here.
2. Run Hello World App on AWS using Cloudify from here

Steps to create Plugin

1. Prepare the DSL file that describe the new node types.
2. Add the required Python code which mapped to the DSL node types.

Note: In order to create a custom plugin follow this doc

Cloudify Serverless Plugin

This plugin is a POC that shows how Cloudify can easily be integrated with any third party tool such as Serverless Framework.

The plugin mainly does the following:

1. Service management.
2. Functions Management.
3. Function Invocation.

This blueprint defines the following

1. Service Configuration.
2. Serverless provider (AWS Lambda).
3. Functions associated with the service to deploy on AWS.

Note: Before deploy the above blueprint and start actual execution we need to make sure to do the following:

1. Clone cloudify-serverless-plugin plugin (Contains example + code)
2. Cloudify manager has the NodeJS runtimes and Serverless Framework itself .
3. Upload plugin to Cloudify manager
4. Create Cloudify Secrets aws_access_key_id & aws_secret_access_key

Steps to deploy Serverless functions in Cloudify

1. Upload blueprint cfy blueprint upload -b aws.yaml aws-serverless

Cloudify Blueprint Uploading

Blueprint in Cloudify Console

2. Create Deployment from the uploaded blueprint.

Cloudify Deployment Creation

Deployment in Cloudify Console

3. Run Install workflow to start creating resources on AWS using Serverless

Cloudify Install Workflow

4. The installation result created a boilerplate serverless configuration ready for deployment and then deploy all functions as specified on the blueprint

5. The plugin also has the ability to invoke functions using execute_operation workflow

cfy executions start execute_operation -d aws-serverless -p inputs.yaml
allow_kwargs_override: true
operation_kwargs:
  functions:
   - hello_1
   - hello_2
operation: 'serverless.interface.invoke'
node_instance_ids:
  - example-aws-serverless_uionc3

Cloudify Functions Invocations

6. Tear down all resources using cfy uninstall -b aws-serverless

Cloudify Uninstall Workflow

To Recap, the cloudify-serverless-plugin is just a POC showing how Cloudify can be easily used to orchestrate frameworks/tools such as Serverless using standard DSL format. The code is available on Github https://github.com/cloudify-incubator/cloudify-serverless-plugin