Stand Apart – but be Cohesive (part 2)

Stand Apart – but be Cohesive (part 2)

Some more thoughts about standing apart, but being Cohesive.

As Cohesive Networks grows, we do struggle to keep the elements of our company that are fundamental, without being too much of a constraint on evolution and responsiveness to changing markets.

Separate from very specific policies are the stories, mantras, aphorisms that guide you. Not too long ago we had a partner company ask us to tell them why they should work with us. In response we distilled some of the essence we try to hold forth in our words and actions. Here they are with a bit of elaboration.

Our favorite code is code we never write, but we solve a customer problem.
In general people do what you pay them to do, not what you tell them to do. If your reward and identity systems are built around writing and delivering code, you get a lot of code. You get change for change’s sake. We do not do that. The expression of our brutalist approach to software development is:
– Our favorite code is code we never write, but we solve a customer problem.
– Our second favorite code is the code we removed from the most recent release.
– Our third favorite code, is that code, which sadly, exists in order to have a product.

“Run on the cloud not in the cloud.” (Avoid the economically defined architecture)
We go more in depth in this post. To boil it down, the cloud vendors have built massive feature sets that both provide value and extract payments. As a consumer of cloud one needs to ask continuously, am I getting the value?

Unfortunately, certifications in cloud expertises can become so dominant in reward and identity systems that cloud services are incorporated not to serve the ultimate customer, but to serve a certification sense of satisfaction. Customers not acutely aware of this are at risk of implementing an ‘economically defined architecture’, which is a working system architecture, but while correct, it also maximizes cloud vendor revenue. Sometimes “over the top” visible services are a better path to multi-cloud and cost control.

Networks are addresses, routes, and rules; everything else is implementation detail.
This simple statement has gotten me rapidly exited from meetings at one large scale networking vendor interested in acquiring Cohesive Networks. The product management team I was talking to took insult that I could be so reductive. That said, we as a team stand by it. There are names and numbers for things, paths to these things, and the ability to say ‘yes you can’ or ‘no you can’t’ get to those things. The question then becomes how to manifest that in a way that gives enough power to someone with in-depth network expertise, but also empowers a person “just trying to connect these two things”.


Do your best to not surface implementation detail in customer experience.

This is the corollary to above. We definitely have not succeeded at this in all parts of the VNS3 Network Platform, but we try.  Obviously we provide a networking system, so there are elements of the implementation that have to surface technical detail.

For example, there are such things as “IPsec connections” and IPsec has elaborate configuration constraints which need to be managed. That said, it can probably still be simplified. As an example, our single page UI for defining an ipsec endpoint tries to achieve this objective. I had one user tell me, “The first time I saw your IPsec UI, I wondered what college intern wrote it. After I used it for a while I decided it was genius.”

Support is the critical sales function.
Some companies will claim this to be true, and then they have the support staff hound you with upsell offers, surveys, and other actions that have NOTHING to do with solving your problems. What we mean at Cohesive is, when a customer has a question or concern, your ability to quickly and knowledgeably assist them is a critical part of your trust relationship.

There can be no ulterior motive other than you want to help them in that distinct moment. In our case where the majority of our customers are SaaS/BPaaS providers, initial connectivity issues are a key obstacle to time-to-value for both the SaaS provider and their customer. Eliminating any issues in this dimension helps our customer and their customer alike.

FNA – full network accountability is a company culture.
Multi-party networking and security by definition is a collaborative endeavor. When there is a problem, there is a risk that collaboration devolves to finger pointing. When you have a solution like the VNS3 Network Platform that is SO reliable it can be difficult to look at your self as the culprit when a customer has issues. Well over 90% of all inbound “P1” support issues that we receive come as a result of our customer’s customer has <changed misconfigured broken> something in the network path.

It is tempting to enter the interaction with a bias towards a “it’s not us, it’s you” mindset. To counter this we practice what we call “Full Network Accountability” which means to the best of our ability we will work with our customer, their customer, their customer’s outsourced networking people, whomever, to solve the problem. In doing so, our team has to start with the premise, that no matter how unlikely, our customer has run into a never-before experienced, one time only, horrible bug in Cohesive VNS3, until proven otherwise. Then we move forward through the entire connection path with customers (as desired) to find the ultimate issue.

This list is not exhaustive, but is indicative of how we have approached the long term health of the VNS3 platform and our company Cohesive Networks. Software, security and network skills are the table stakes for serving customers in our space, but how to have those skills and deliver them to customers through time and across hype cycles is the additional critical capability that we strive for.

Owning our mistakes, “mea culpa” from Cohesive’s CEO

Owning our mistakes, “mea culpa” from Cohesive’s CEO

We owe our customers an apology as a several issues were recently found by the Trend Micro Zero Day Initiative.

Two of them had a high score because of the potential for an unauthenticated user to trigger remote code execution.

(Note: Like with other recent industry exploits, if your control plane access is limited as we advise, then the risks are significantly lower.)

If you follow the CVEs that are released every day, we are not alone, but that is still not an excuse.

While our engineering management and company management are intimately involved in all code changes and releases, as our solution footprint has grown and our responsibilities to our customers have grown, we did not expand some of our processes comensurately.

We have worked with customers since the disclosure getting them patches and new cloud images.

We have added additional code review tools and processes to our releases.

Thank you to Trend Micro ZDI and Mehmet INCE @prodaft for the discovery and working us through the disclosure process. We are grateful for the help from the community to make our products and customers more secure.

Any users of the VNS3 Network Platform who need assistance can always reach us via support@cohesive.net and we will assist with patching and upgrades.
https://cohesive.net/support/security-responses/

– Pat Kerpan
CEO, Cohesive

To agent or not agent, the CrowdStrike question

To agent or not agent, the CrowdStrike question

Some thoughts on the CrowdStrike events at a broad level. We don’t provide the type of capabilities they do, so I don’t feel comfortable inveighing on what happened and how it happened.  But, we do have some thoughts on when to take the risk of deploying agents into your systems.

At Cohesive we have a standard response to customers wanting to install <fill in the blank agent> into our VNS3 Network and Security platform virtual appliances. It is always a difficult situation as it is being pushed by the security team and CISO, at what tends to be our larger, more revenue significant customers. It is difficult because our answer is a polite, yet vehement “no”.

The VNS3 control plane should only be open to an allow-listed operations ingress address. The data plane should be open to the ports and services you use our platform for. (We are often a cloud edge firewall, people vpn, data center vpn, waf, and more.) The biggest liability to the security and operational integrity of our devices would be to add an agent that can be exploited or can itself cause an outage. This has happened in the industry (but not to us) numerous times over the recent years via things like SolarWinds, Microsoft OMS agent, and now CrowdStrike to name a few.

One of the oft-used phrases at Cohesive is “horses for courses” and we use that British idiom as part of our approach to making architectural and deployment security recommendations. With respect to the panoply of agents that vendors and security teams alike argue to be installed in all your systems, we think it is quite apropos. We think you need to look at what the purpose and placement of the computing devices is in order to make this decision.

Seinfeld - His father was a mudder, his mother was a mudder

If it is a computer used by a person, especially hybrid use or remote use, how are are you going to protect it? For the most part you can’t protect the network it is on, since it is running on networks you don’t control.

There are some options outside of agenting, for example if using Windows operating system you could use group policy to force the machine onto a virtual network with no ability to access the Internet other than through your VPN’s edge, but it is bad user experience and expensive to have acceptable points of presence.

And even if forcing users onto a VPN, what about email phishing links and attachments, what about Microsoft Office macro exploits that leak into your shared drives? Super tough, and I am glad we aren’t in that business.

People make mistakes. Even well-trained, diligent people can make mistakes in where they browse to or what they click on, and you need to be able to address possible exploits there at the point of attack. So I can’t really blame someone for wanting to believe these types of products are the answer.

At Cohesive which admittedly is not a huge company, our answer has evolved from ‘Microsoft Office is disallowed’, to now ‘Windows is not allowed’. Right there the “people attack surface” is minimized. However, that is a long migration or a “never migration” for many companies. In these cases, I think “agent away” and be judicious in the vendor you choose.

What about business servers, the computers that are not the personal device of an end-user, but rather are there to perform a function like database or API serving or message queuing? Again, for Cohesive, easier in that Windows is not allowed, business services are built on Linux. That said, whether Windows or Linux, in these scenarios why are you ‘protecting’ the computer instead of the network? Those business servers are not going to click on links, browse bad websites, or download malware. They are automatons performing specific, repetitive operations.

This is where we are skeptical of the agent approach and will not allow it for our network appliances, and as applicable would not allow for business services. (CrowdStrike apparently created a similar issue previously on some Linux distributions: https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools)

In these cases the VNS3 Network Platform (or similar vendor offerings) is there to protect and monitor ingress and egress to the network, along with other elements of what should be thought of as a security lattice. Using cloud or data center virtualization, you have your cloud network security groups and network ACLs, along with your OS firewall, both of which should be working in tandem with network appliances like VNS3. You can then run WAF or IPS at these ingress/egress points as well – run them inside VNS3 or run them via an appropriate cloud service offering.

We think these are the rules to follow:

  • Computer primarily used by human and it has to be Windows, agent it and anti-virus it.
  • Computer primarily used by human and is Mac or Linux, anti-virus it.
  • Computer used by computers, both micro-segment and protect the network, use a security lattice. The risk from any agents are too high.
  • (OPTIONAL) Computers used by computers are ideally Linux servers using the distributions’ minimal install with at least some elements of CIS hardening.

But then you loudly proclaim, “we need visibility into the VNS3 (or other) appliances for our <DataDog, Syslog Server, SumoLogic, AlertLogic, etc..>!”

OK – you got us. We DO allow visibility to logs, network configuration and statistics, netflow, and some API calls for system status to plugins that run inside VNS3 – but don’t have full host access.

When you are running VNS3 controllers, whether a single one or a larger cluster, you are running a mini “docker cloud” inside your virtual network. We have a set of pre-built plugins as well as a published set of elements needed to make any open source, proprietary source, or commercial source that can be containerized, into a compliant plugin. This gives you the ability to put analysis and monitoring INTO the network, yet not be able to disrupt the network security device; observability with security.

(And now the sound of Cohesive patting itself on the back for this balanced approach to agenting.)

Cisco owes me a sandwich!

Cisco owes me a sandwich!

In 2010 we were so excited to meet with a team at Cisco to discuss our network function virtualization offering that had been live in the clouds since 2008.

IPv6 VNET/VPC - Grains of Sand

The folks at Futuriom (@rayno, aka R. Scott Raynovich) have put up a nice piece about Cisco’s struggles to integrate its many acquisitions and fundamentally retain relevance.  Now of course Cisco is a behemoth compared to Cohesive Networks, we are a lean and cute marsupial next to it the hulking mammal, but we do compete with them all the time and they do represent the other side of about 50% of all our customer’s connections.  (My point being we are quite familiar.)

So, way back when cloud was young – our parent company CohesiveFT developed a very strong champion at Cisco.

He loved what we did with the VNS3 platform (then the VPN-cubed appliance) and was both fascinated and skeptical that via an API you could create any over-the-top network and route traffic through it, indifferent to the actual underlying networks. You could have a flat network anywhere with dynamic routing protocols in place, a form of server motion which separated network location from network identity, thus enabling very flexible and secure application systems.  In many ways, unbeknownst even to ourselves, in 2008 we had created the first commercial cloud native NFV (network function virtualization) solution with a restful API.

We ended up invited to Cisco campus in California for what would be a pretty significant (for us) meet up with some of their key technical and business development decision makers about our virtual appliances. We were thrilled and hoped for a partnership to accelerate our business.

The calendar snippet above shows that it was a morning meeting – and maybe I have conflated events through time, as in my memory it was thereafter rescheduled into a longer, later lunch meeting (catering provided) with time for a demo and hopefully some whiteboarding with the team.

The meeting was well attended by our champion, key technical people and decision makers.

A network bridge to the compute clouds.

We launched into our pitch about how you could build networks anywhere you had access to compute and bulk network ingress/egress.  Some questions came and went – and we were thrilled to be talking about the flexibility and security that comes with network virtualization.

At about 20 minutes in, product managers were scribbling away in their notebooks, but then the “biggest” of the technical folks looked up and said “Wait, wait, this is a hardware appliance, right?”

“No, no,” we say, “it’s virtual; all software, really flexible.”

To which he said, “No one will ever put network traffic through software” and he closed his notebook, stood up, and began exiting the room.

With the exception of our champion, everyone else followed.

As we were leaving the room, there was the catering cart outside that had not yet had a chance to come in to our truncated meeting. My memory is sandwiches – but maybe it was breakfast items, regardless it was now NOT on offer, and we were exited from the building.

So, two takeaways, network virtualization was real, driven by software and took Cisco a long time to get their heads around, and I think Cisco owes me a sandwich (or perhaps a croissant).

Migrating to IPv6 – Managing the Madness (Pt. 1)

Migrating to IPv6 – Managing the Madness (Pt. 1)

Names, Names, Names.

Its the year enterprises get focused on moving to Internet 6. We think it is mostly because of the AWS IPv4 address price increase, but as good a reason as any.

IPv6 VNET/VPC - Grains of Sand

While we don’t expect you to start using ALL of the eighteen quintillion, four hundred forty-six quadrillion, seven hundred forty-four trillion, seventy-three billion, seven hundred nine million, five hundred thousand addresses in your cloud VPCs and VNETs, do remember nature abhors a vaccuum.

If you combine the enormity of the address spaces, and the somewhat impossible-to-verbalize representation of addresses, using addresses as a point of reference at the surface of network administration, monitoring, or security is not going to work much longer.

It is time to let loose your “wordsmiths”!

Chief Wordsmith

All of those devices running around your organization need a name, and in fact once we start naming things it won’t stop, we will have multiple names per device, because if one name is good, more are even better.

Of course organizations have been using DNS for years, and the Internet runs on it, but for B2B, enterprise, SaaS we have seen widely varying depths of use.

Some of our more advanced customers have adopted one or more taxonomies for naming devices in their production SaaS environments, albeit perhaps not throughout their internal organization. It can be somewhat of the cobbler’s children having no shoes, we do pay attention to our customers, but often not to ourselves. Here is an anonymized example of such a SaaS taxonomy:

<saas service name abbreviation>.<cloud>.<cloud account abbreviation>.<cloud region>.<production status flag>.<customer id>.<customer abbreviation>.<cost center id>.<machine id>.<cloud instance id>.domain

It is certainly a mouthful, and in fact more so than “fc55:636f:6865:7369:7665::6440:1fe3”, but since most humans DON’T do hex/decimal calculations in their heads, it works by providing understanding via the delineated labels in the full name.

Once you see this type of naming at work in administration and monitoring, you can then see why devices end up with multiple DNS names based on differing organizational taxonomies. We have see technical, organizational, asset management, project-based, and functional naming schemes for devices. At Cohesive we aren’t quite that complex, but if you ask Chatgippety or the LLM of your choice for examples they will provide some overly simplistic examples which give you the gist.

  • dell.laptop.8gb.intel.corei5.lenovo.windows.10.ssd.departmentC.corporate.com
  • marketing.sales.promotions.abrown.laptop.hr.central.branch.corp.com
  • asset.laptop.s345678.la.abrown.decommissioned.finance.southern.us.corp.com
  • project.gamma.phase3.teamC.abrown.task3.testing.chicago.corp.com
  • function.sales.system.crm.module3.midwest.corp.com

Further Reading

If you have all of naming nailed, good on you. If you don’t here is a great overview by the folks at Cloudflare who create some great content for all of us: https://www.cloudflare.com/learning/dns/what-is-dns/.

For those who are more visual, here is excellent work from a redditor who seemed to have become inactive (sadly) three years back:
https://www.reddit.com/r/programming/comments/klaffg/dns_explained_visually_in_10_minutes/.

For more on the emerging challenges of the IPv6 world you can look back at a few Cohesive posts:

And if IPv6 addressing still has you perplexed the team at Oracle did a super job here:
https://docs.oracle.com/cd/E18752_01/html/816-4554/ipv6-overview-10.html.

AND – never forget the Rosetta Stone of IPv4 to IPv6 understanding here:
https://docs.google.com/spreadsheets/d/1pIth3KJH1RbQFJvZmZmpBGq6rMMBhqEmfK-ouSNszNY.

Next up we will show you how VNS3 version 6.6 for Internet6 will help you adopt IPv6 for cloud edge and cloud interior, provding network functions that are better, faster, cheaper than the competing offerings (to NAME just a few reasons to get a Cohesive network).

Moving to IPv6 – herein lies madness (Part 2)

Moving to IPv6 – herein lies madness (Part 2)

This post is short and essential.

The are two Internets now: the IPv4 Internet and the IPv6 Internet.

The IPv6 Internet is not related to the IPv4 Internet as you have configured it inside, outside, or at the edge of your Enterprise.
IPv6 is not the next version of the Internet protocol; it is the next Internet.

This is the best way to think about it in order to have a more intuitive feel of what is happening underneath it all.

AND – the two Internets never meet. Meaning they are side by side, and applications can move data between them, but in their essence they are different Internets.

Both Internets move across the same layer 2 and physical links, and the protocols above them largely work the same on both, but they are distinct at layer 3, where addressing and naming and routing and firewalling and flow control and many other Internet-y things happen.

A computer may exist on either or both Internets concurrently; an application may bind a network socket to either or both; but they are entirely separate and unrelated.

You will hear about NAT64 (and some NAT46) and the implicit description that addresses are being translated from 6-to-4. I am sure that is the name that will “win” in the industry, but this is a place where we will stand apart. We are calling it Proxy64 and Proxy46, as that is a more realistic and familiar description of what is actually happening. It is not address “translation” in the long standing sense, it is packet TRANSMOGRIFICATION* (according to rules which are in some ways “NAT-like.”) It pulls the payload out of an IPv6 packet and creates an entirely new IPv4 packet. (Some concepts don’t even translate well between the two; these cases are handled by different sofware anywhere from “poorly” to “confusingly.”)

Cohesive’s VNS3 6.6.x is moving out into customer’s hands and with it, our Proxy64 and Proxy46 plugins for communicating between the two Internets.

Let us know how we can help your teams get connections up and running at cloud edge using IPv6.

(ps. If you are just coming up to speed on IPv6 here is a good primer from Microsoft. The “producer-centric” nature of IPv6 comes through, but is a good summary of the industry description and belief systems (don’t let the ‘dotnet’ in the URL scare you).

*Yes that is a C&H reference