Immutable Infrastructure and admin access – how do they fit together?

TL;DR. Cloudification introduces a new level of abstraction to IT environments. One potential game changer is the concept of immutable infrastructure where resources, such as servers, services, or generic infrastructure deployments are not changed after they deployed but the entire setup is discarded and replaced with a new one when needed. What are the benefits and what is the impact of such environment to privileged access, or does the immutable paradigm eliminate the need for anyone to access the environment altogether?

Moving to cloud infrastructure means, for many companies, breaking old habits and changing the way they manage their IT environment. A concrete example is NoOps – everything is deployable and operable without human interaction, and without a dedicated “operations team”.

Also, the phrase “immutable infrastructure” (more about the concept a few paragraphs below) is being used a lot more these days by operations teams, and that has a direct impact on the emergence of the cloud.

If creating an immutable infrastructure is important to you and your team, I wanted to point out one important consideration that some businesses overlook: privileged access management (PAM). The specific PAM software that you use to manage your superuser, administrator, developer, 3rd party or IT consultant access to your environment can make or break your goal of designing an immutable infrastructure.

So how does immutable infrastructure change role of PAM, where is PAM needed, where is not relevant?

The short answer: an immutable infrastructure describes resources, such as servers, services, or generic infrastructure deployments that are not changed after they deployed.

In practice, this is how it works: you configure your servers and virtual infrastructure with automated provisioning tools (such as Chef, Puppet, Ansible) or using the Infrastructure as Code (IaC) paradigm where you “program your infrastructure”. One upside of IaC is unlimited scalability, and ability to fail over just by spawning new resources.

A great example of the IaC paradigm is AWS Cloud Development Kit that allows you to deploy you infrastructure with a TypeScript or Python code. Normally you deploy your infrastructure to a new fresh instance on your cloud service provider (CSP) and direct a portion of the live production environment traffic into it. At that point, the entire infrastructure is in “read-only” mode – no other changes are made after that.

You can take Immutability a step further with FaaS (Function as a Service) where your function does not provide access. Instead, the importance of telemetry as a way of monitoring increases, and your PAM becomes a critical part of both code change management and the ‘development environment to staging environment to the production environment’ cycle.

The benefits include:

Furthermore, your operations teams fully understand the current state of their infrastructure and its dependencies. There are no surprises.

In an ideal case there is no need to have access to the environment at all, meaning you don’t need to set up any privileged accounts or manage their access or passwords, and in fact, don’t even need PAM. As a result, the operations team can be more productive and is able to address issues more quickly.

This matches very well the normal software development cycle where you do not “binary patch” the resulting binaries. Instead you make changes to your source code and recompile your application—or infrastructure in our case. That process retains all the predictability and confidence immutable infrastructure is supposed to offer.

To understand why the cloud has made immutable infrastructure more of a priority, let’s start by thinking about legacy infrastructures – the ones that rely on physical, on-premises servers.

Configuring physical servers took time and effort. You had to buy the physical hardware, get it set up and tested, then deploy it. The idea of creating “cloned images” to make infrastructure changes simply didn’t exist – it was too expensive and too much work! So physical environments are inherently ‘mutable.’

The cloud changed that. Now, when you can spin up a new server in seconds via virtualization, immutable infrastructure is much more viable.

One challenge is that cloud-based infrastructures are inherently more intangible. Legacy environments are very concrete, since in at least in some case you can literally touch the problem. In contrast, IaC instead has a very high level of abstraction and might be hard to comprehend at first.

There’s also more complexity. You normally end up having a greater number of servers, containers, micro-services than on the on-prem installation. Cloud service providers have many hosted services (databases, message queues, notification frameworks for external communication, etc.) that make building scalable applications much easier.

But this comes with a price tag. All these components require their own configuration, from resource limits to access permissions. These configurations, components, and other tools add to the complexity, creating more potential dependencies and impacts on your infrastructure in more ways than you might ever be able to diagnose and understand.

That simultaneously makes immutable infrastructure more important in the cloud, but also potentially more difficult to achieve.

A few years ago, we completed an audit for a client that had 14 “golden” master server images for their infrastructure. Using our solution, Universal SSH Key Manager, we found that they had more than 1,000 unique security configurations in their production infrastructure.

They had 14 server images that were supposed to be cloned into production server instances but had in fact diverged into over 1,000 unique configurations.

How could they confidently say what kind of overall security configuration they had in their production infrastructure? What changes were made in the security policies and configurations, and what kind of overlaps or gaps these configuration changes introduce?

The immutable approach can certainly help to avoid this.

Normally, in an immutable environment, you wouldn’t rely on manual work to configure your infrastructure. But depending on what kind of applications you run, there may be some need for manual tasks, checking logs, diagnostics or troubleshooting. In those cases, you need access to the infrastructure. That’s where providing privileged access comes into play.

If you’re following the principles of immutable infrastructure, you don’t make run-time modifications on live production environments.

Instead, you update your automation scripts and programs, make a fresh deployment to a new virtual environment, and once the new instance is up and running, you terminate the old instance.

In this paradigm, where we can't modify anything on production or have alerting in place, we don't want to do things like install software agents. Why? Because 1) there is an alternative, and 2) running extra software components is a cost (you’ll get more resources for your business application without them) 3) they increase the overall system complexity.

However, that’s exactly what many PAM solutions do. By design, they rely on agents to provision new accounts on the actual production server via a run-time modification.

There’s more. In fact, any deviation creates a potential security incident you need to investigate. This includes many traditional PAM features such as:

As a result, agent and password vaulting driven PAM solutions create lots of false flags your operations team needs to waste time investigating.

Granting access that respects the immutable paradigm requires a PAM solution that comes with a few key characteristics. These are:

Agentless. Agents equal overhead. They are another thing to manage, and another risk point to keep in mind. Therefore, immutable-proof PAM is agentless. There is nothing to install on the client or the server but the PAM solution acts as the trusted authority between the users and the target hosts by building the required trust relationships for the environment without touching the setup.

Vaultless and credentialess: Vaults work on the premise that permanent credentials and passwords need to be used in the first place. This means onboarding and rotating passwords which in turn means running extra configurations in an environment that is not supposed to change.

Automated and easy to use. Since in an immutable model the entire environment is discarded if any changes are made to it, and an entirely new instance is spun up when needed, the PAM solution used to access resources in such a setup needs to find those targets automatically. Automatic discovery of cloud hosts is a must feature.

The same goes for assigning the right level of privilege for the right user on establishing the session. Interfacing with IAM/IDaaS solutions – where user identities and their entitlements are hosted – and mapping them automatically to roles (using role based access controls, RBAC) that are hosted in the PAM solution are a big help.

These roles need to be configured only once to the host environment, after which the synchronization between identities/entitlements and privileged roles is automated. The software engineer just single signs-on (SSO) to our solution and can select the target only from the resources available to him or her and within the limits of the level of privilege defined in the role.

Moreover, managing privileged accounts usually translates into making changes to target servers: even if the privileged account (or associated password) is created just-in-time and deleted immediately, it still means configuring the environment.

Instead, we propose a lean PAM solution that is based on Zero Trust access and implemented with ephemeral certificates to provision access. In this model, the secrets needed to establish a privileged connection are baked into an X.509 certificate – or a similar access token - that is automatically created on-demand when the connection is made. Once the connection established, the certificate expires automatically, by default within five minutes.

The user never handles or sees any secrets, never uses any credentials (permanent or one-time) nor is there a need to make any kind of modifications to the environment. There simply are no risky secrets left behind. The whole process is ‘read-only’ and your environment stays immutable.

Moreover, this approach follows Gartner’s principle of zero standing privileges where they recommend not to use permanent credentials in IT environments at all.

At the end of the day, immutable-proof PAM means fewer false alerts, fewer bad server images, and fewer unclear dependencies. All of that means less garbage for your operations team to dig through, and less time spent on unproductive work.

Immutable infrastructure is all about giving you confidence in a clean cloud-based environment. When access is needed to such environment, only immutable-proof PAM can support that type of server paradigm.

Our PrivX is a lean and automation-driven PAM solution that was purpose built for multi-cloud and hybrid IT and that supports the immutable infrastructure paradigm. Our solution was recognized by KuppingerCole as an innovation leader in all their categories in their 2020 Privileged Access Management Leadership Compass. Here’s a quote from the report:

“Instead, PrivX dispenses with the need for a vault full of credentials and issues short-lived, or ephemeral, certificates for on-demand access. It’s an innovative approach but one that does bring functional and security advantages – access is faster, onboarding and offboarding of privileged users is quick and there are no passwords to issue or lose, since there are no permanent leave-behind credentials.”
- Paul Fisher, Senior Analyst, KuppingerCole

Download the full KuppingerCole 2020 PAM Leadership Compass, courtesy of SSH!

PS. Our PrivX project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 881221.