Building Global Hybrid-Cloud Infrastructure Automation with Pipes and Python3

Building Global Hybrid-Cloud Infrastructure Automation with Pipes and Python3

Building automation for foundational hybrid-cloud infrastructure deployments in hybrid cloud environments is hard. At Cohesive Networks we’ve helped our users build out networks that span datacenters, public clouds, IOT networks, even cell networks for bridging SMS communications. So we can find ourselves stitching together cloud infrastructure using different APIs in different clouds assuming different credentials etc. It can seem a little daunting. Now, If you’re in single public cloud, life’s a little bit easier: each cloud has their own deployment manager and Infrastructure as Code (IaC) templating system that more or less suffices. The complexity arises when you need to, say, build some infrastructure in AWS, then switch context and build in Azure, and switch back to AWS with state from the Azure infrastructure to continue building in AWS. The goal here is to step through an example architecture and simplify our deployment using pipes and python3.

The Hybrid-Cloud Architecture

So let’s run through a common hybrid-cloud architecture, break it down into discrete steps and see if we can’t simplify our lives.

VNS3 federation Plane

Here’s a simplified architecture we see quite a bit: Account B in Cloud Y is providing connectivity (down and across) to Account A in Cloud X. Account A would like its applications to have secure access to any applications running on premise. At Cohesive we call the routing plane in Account B the “federation plane” and the whole network the “federation network”. It provides highly available connectivity to any new virtual private clouds (VPC) spun up in Cloud XYZ.

Automated VPC Cloud Construction

Ok, so let’s run through automating the construction of a new VPC in Cloud X that automatically has connectivity to on-premise applications. The steps for us look something like:

  1. Fetch network configuration for new VPC subnet (e.g. CIDR with your favorite subnets)
  2. Create new VPC and virtual router/firewall (VNS3) for Account A in Cloud X
  3. Create fednet routing – routes for traffic “up” to
  4. Create new VPC routing – routes to on-prem, traffic to controller interface for routing traffic “down” to fednet
  5. Temporarily open API access for configuring VNS3 network routes and firewalls
  6. Connect new VPC with federation network by IPSEC peering controller in account A with federation network controllers
  7. Remove API access
  8. Teardown any configuration resources

Each step requires state from the previous step. This is really just a simple pipe: “echo parameters | fetch-network-config | create-new-vpc | …”. Ah, the beauty in simple ideas decades old. Each step accepts the previous step’s state, runs the step’s task, enriches the data with any new state and returns it. Now, I’ve always been partial to python for its ease of use and readability for things like this. Unfortunately, it doesn’t have native pipelining functionality. So let’s build it. Python3 also has some nice async features that we can take advantage for optimizing our automation.

The Python 3 Pipeline

A first pass at a simple pipelining function could be as simple as this:

Python Pipe Code

I like to use python3’s new typing functionality because I think it makes for more maintainable code. Here it tells maintainers that a step is a tuple of length 2, where the first element is a string and the second is a callable function. We also copy the initial data so as not to mutate our caller’s data and then simply loop through our functions. So a pipeline function for our architecture might look like this:

Python Pipe Code v2

So what’s going on here?

  1. We initialize some cloud clients that have permissions for provisioning resources in our cloud environments
  2. We define each step with a name (for readability and logging purposes) and a function. The function partial binds the parameters passed (template) to the function provided as the first argument. You’ll see each step can target whatever cloud we want using whatever IaC templates we like and the state will be passed through. In fact, each step can do anything so long as it respects the function signature expected (ie. it accepts and returns a dictionary).

A Little Optimization

This works quite nicely and it’s very simple. You might notice that some steps could perhaps be combined or run concurrently. Python3’s asyncio makes that quite easy:

Python Async Pipe Code

Our new pipe function has a couple changes:

  1. It now accepts different kinds of steps as indicated by the Union typing parameter. Our pipe now accepts a step that provides a list of functions rather than just one. Using asyncio.gather we can run each sub-step concurrently (each sub-step must implement the async/await paradigm )
  2. We also updated our step functions to return a dictionary that has an optional “outputs” key for passing along in the pipe.

Add thats it. Pretty minor changes to take advantage of steps that can be run concurrently. Here’s what our deployment pipeline looks like now:

Python run function awsync pipe

We’ve found this approach simple and effective. I’ll end with putting out a feeler: is there any interest in an open source python3 library for hybrid-cloud deployment automation? It would be purposefully simple and pluggable, adopting only a few powerful idioms like a pipeline. Let us know!

Announcing AWS Quick Start Reference Deployment for VNS3

Announcing AWS Quick Start Reference Deployment for VNS3

Want a HIPAA/HITECH compliant application deployed to AWS in minutes? Read on!

We’re proud to announce the release of our first AWS Quick Start reference deployment for configuring and launching our VNS3 overlay network for your cloud application. Working closely with Amazon we’ve leveraged the proven power of AWS CloudFormation to take our secure and scalable solution and make it even more accessible. With our Quick Start deployment, VNS3 can easily secure your cloud application to HIPAA and HITECH standards in as few as fifteen minutes, supported by best practice tools and strategies for automating your infrastructure deployments.

Check out our Quick Start Guide here! Keep reading for more information about this release.

VNS3 AWS Quickstart Architecture

Save Time

Our Quick Start was built by AWS and Cohesive Networks solutions architects to help you automatically deploy a VNS3 topology quickly and easily. Don’t worry about high availability and security, we’ve included it for no extra charge! Build your production deployment fast and start using it now.

Reduce Complexity

Simple (not to be confused with simplistic) is secure. VNS3 provides a generalized approach to encryption across your cloud deployment. This enables you to field a clean VPC Route Table and Security Group configuration to reduce attack surface and minimize misconfigurations.

Control Encryption

AWS provided and controlled, symmetric encryption with common shared keys isn’t enough for regulated industries. Customer controlled encryption with VNS3 is essential to securing PII/PHI in order to pass HIPAA audits. VNS3 as demonstrated in this Quick Start Guide provides a simple and programmatic way for achieving HIPAA compliance.

Added Bonus

Do you use blocked protocols like UDP multicast? The VNS3 encrypted overlay network deployed by this guide allows you to redistribute UDP multicast within your AWS VPC deployment. Now you can apply the same design principles to your cloud applications, whether designing cloud native or lifting and shifting.

Moving Forward

Following the successful launch of our first AWS Quick Start Guide, we’re excited to move forward and create new reference deployments for all the various use cases VNS3 supports. We’re cooking up AWS Quick Start Guides that deal with more complex peered VNS3 topologies, demonstrating different High Availability and Network Federation capabilities. We are also working on an Azure QuickStart template for deploying the encrypted Overlay Network for Microsoft Windows VMs later this summer.