vRO API Explorer

It’s been 5 months since we released the first version of the vCenter Orchestrator API Explorer. Because people seem happy with it (we see a steady stream of feedback and returning users), we continue to improve the tool.

Over the last couple of weeks we’ve worked on:

  • performance improvements – it’s even faster now
  • move to .NET Core 1.0.0 (it was still running RC1)
  • documentation links in the vCenter API objects now link directly to the relevant VMware documentation

Please let us know if you are missing plugins, or would like certain features.

Container confusion

These days I’m working at a client creating workflows for their state of the art private cloud platform. It’s really quite nice: internal clients can use a webportal to request machines, which are then customized with Puppet runs and workflows to install additional software and perform custom tasks like registering the machine in a CMDB. All this is ideal for running legacy workloads like SQL databases.

Other offerings include ‘PaaS’ workloads for running business applications, e.g.: developers can make requests for ‘Scale Out’ application servers, meaning 2 linux VMs with Tomcat installed behind a loadbalancer.

The most popular offering by far is the large VM with a preinstalled Docker engine. In fact, they are so popular you might wonder why.

Is it because Developers have an intrinsic desire to create and run Docker containers? Naively, the current hype around containerization in general and Docker as a specific technology could indeed be explained as such.

However, if you know Developers a bit you know what they really want is to push their code into production every day.

To get to this ideal state, modern development teams adopt Agile, Scrum, and Continuous Delivery. Sadly, especially the latter usually fails to deliver to production in enterprise IT, giving rise to the waterscrumfall phenomenon: Continuous Delivery fails to break the massive ITIL wall constructed by IT Ops to make sure no changes come through and uptime is guaranteed.

So guess what’s happening when your Dev/Business teams request the largest possible deployment of a Docker blueprint?

Yep, you just created a massive hole in your precious wall. You’ll have your CMDB filled with ‘Docker machine’ entries, and have just lost all visibility of what really runs where on your infrastructure.

Docker in production is a problem masquerading as a solution.

Does this mean containers are evil? Not at all. Containers are the ideal shipping vehicles for code. You just don’t want anyone to schedule them manually and directly. In fact, you don’t even want to create or expose the raw containers, but rather keep them internal to your platform.

So how do you use the benefits of containers, stay in control of your infrastructure, and satisfy the needs of your Developers and Business all at the same time? With a real DevOps enablement platform: a platform which makes it clear who is responsible for what – Ops: platform availability, Dev: application availability – and which enables Developers to just push their code.

DevOps stage at VMworld

VMworld 2015: beyond virtualization

What do you base your selection on when buying some piece of technology? Is it the core functionality, or the added features?

As Kit Colbert aptly stated in his VMworld DevOps program Keynote, customers at this point implicitly assume the core functionality of almost any given product will be alright, and base their choices on the extras:

  • when selecting a new home audio set, you select it based on for instance connectivity, wireless options and easy of use. Actual audio quality is perhaps the #10 item on the list
  • a lot of companies make decent tractors, but some (e.g. John Deere) set themselves apart and do great by adding integrated options such as support for GPS (driving in straight lines)
  • the hypervisor used for virtualization was once the unique selling point, where people now buy virtualization suites based on supporting functionalities, e.g.: High Availability, virtualized layer 2 networking (NSX), Dynamic Resource Scheduling

Smart existing companies have recognized this trend of commoditization of the core functionality, which results in a huge drive from the business to add more extra value fast while staying safe and reliable, all in order to stay competitive with the army of disruptive startups coming for (a piece of) the cake.

Developers have been used to working with the short iteration cycles intrinsic to Agile development for years now, since apart from adding value to the business quickly it has the additional benefits of risk reduction and adaptability:

Agile value proposition

Agile value proposition

However, this mode of operation is asking a lot from traditional IT departments as historically IT operations is focused on reliability of infrastructure: a characteristic seemingly best served by no changes ever – diametrically opposed to adding new features on a daily basis.

This has given rise to the waterscrumfall phenomenon: new features developed with short iteration cycles (scrum/agile) will still have to wait for the biannual release weekend to hit production (waterfall), thereby eliminating most of the advantages gained by adopting agile methods in development.

It goes without saying waterscrumfall is not a desirable situation to be in, and therefore people have been experimenting with the logical extension to Agile development to the whole pipeline: the DevOps movement.


DevOps has perhaps over 9000 alternative definitions. The most important thing to note though, is that DevOps is a mix of culture, process and supporting technology. You can’t buy DevOps, and you can’t implement it.

Adopting DevOps requires a permanent push towards a different mindset which enables you to bring changes to production fast, at scale and in a reliable way. There are however some technologies that can help you enforce and enable DevOps principles. It’s here were the most exciting developments took place at VMworld 2015.

Overview of the VMware Cloud Native stack

Overview of the VMware Cloud Native stack

Unified platform: vSphere integrated containers

Interaction between Operations and Development runs most smoothly if Developers don’t have to file tickets for virtual machines, but instead use an API to request some compute resources to run their code. This is where containers come in: originally devised as an Operating System (OS) level virtualization technology, their main popularity these days is not the result of OS overprovisioning capabilities but rather of their ability to serve as shipping vehicles for code enabling reproducible deployment.

The output of a Continuous Integration (CI) process is known as a build artifact. Where usually this is a .war/binary/.zip file, the more modern approaches use containers. Ideally, the next stage of the process – Continuous Deployment (CD) – would subsequently push the container to a container engine (e.g.: Docker) which can schedule it. vSphere integrated containers allow this exact mechanism which nicely seperates Operations and Development concerns:

  • Ops can define special resource pools – Virtual container hosts (VCH) – to keep tabs on the resources available to containerized workloads
  • vSphere exposes a Docker Engine API, which Devs can use to schedule container workloads to a VCH. When a container is scheduled, a Virtual Machine (VM) is forked (instant cloned) to run this workload
vSphere integrated containers

vSphere integrated containers

Note that since the container is running on a VM in a 1:1 relation, the VM is not important here. It just provides the isolation and scheduling features to the container: the first class citizen of the data center – from the perspective of the developer – is the container itself. At the same time, because of the 1:1 mapping, Ops can monitor and manage the just enough VM (jeVM) in the same ways they would legacy workloads.

Continuous Delivery: vRealize Code Stream

Most development teams have some kind of Continuous Integration set up by now, which generates automated builds on a clean system, tests the build and stores the build artifact. The next phase which is pushing the artifact to test, user acceptance test, and ultimately production is not usually done in an automated way in the traditional enterprise environment as this phase requires Ops cooperation to set up – and as described above – this is where traditionally 2 worlds collide.

However, reproducible and therefore automated deployment is essential if you want to work with fast as well as safe pushing of new features into production. Therefore, companies today can only survive the onslaught of disruptive newcomers if they set up some sort of Continuous Delivery practice.

This is where vRealize Code Stream comes in: when a build artifact in the form of a container is output from the Continuous Integration phase of the pipeline, vRealize Code Stream pulls it in and takes care of the Continuous Delivery part in an automated way based on (user defined) rules, checks and tests.

vRealize Code Stream Continous Delivery Automation

vRealize Code Stream Continous Delivery Automation

Integration with cloud native platforms: Photon platform

Scheduling a container directly on vSphere using integrated containers is a great start, but it will not be the typical use case for new applications in production environments. Problems such as scaling, scheduling, dynamic routing and load balancing are universal and so unless you want to reinvent the wheel (a very common developer pastime), it’s much more convenient to use a cloud native application platform to deploy applications. Platforms such as Kubernetes, mesos, docker swarm and Pivotal Cloud Foundry take care of the scheduling, scaling and dynamic routing automatically.

Photon Platform architecture

Photon Platform architecture

At VMworld, VMware announced the missing link for landing cloud native platforms on vSphere – Photon platform – a multi tenant control plane for provisioning next generation (cloud native) application platforms.

Integrated containers or Photon platform?

Integrated containers vs. Photon platform

Integrated containers vs. Photon platform

Cloud native architecture is the future, but applications need to be designed to be cloud native (12 factors), and most existing applications are just not ready. So basically it comes down to this:

  • cloud native applications ⇒ cloud native platform using Photon platform
  • ‘legacy’ applications ⇒ vSphere, with packaging as container if possible

Note that for large applications, it doesn’t have to be one or the other: realistic migrations of existing applications will likely keep a core monolithic/legacy part hosted on the traditional platform, with new extensions or refactored bits – for which a business case can be made – as cloud native applications.

Pivotal Cloud Foundry partnership

Pivotal Cloud Foundry (PCF) is just one of the cloud platforms that can be provisioned on vSphere with Photon controller, so why special attention for PCF? From the VMware perspective this seems obvious: VMware owns Pivotal Software, sure they like to see them do well, there’s $$$ in it.

However, from the impartial enterprise perspective there is a very good case to make for PCF as well:

  • it’s the only platform that has support for enterprise concepts like organisations, projects, spaces, user management
  • it’s the only platform which strongly drives home the distinction between platform operations (managing and monitoring the cloud platform itself) and application operations (managing and monitoring the apps)
  • it’s a structured/opinionated platform which enforces DevOps principles – as opposed to unstructured (more freedom a.k.a. more chaos/management hell) platforms such as Kubernetes and mesos
Pivotal: enabling DevOps

Pivotal: enabling DevOps

Ergo: it’s the only platform right now that’s good enough for general purpose enterprise production use, and it’s the only platform that ‘just works’ on vSphere.

VMware and the commoditization of virtualization

Technology aside, VMworld 2015 was interesting because VMware is in somewhat of a bind: the hypervisor – once the sole reason for buying VMware – has become a commodity. The reason for choosing vSphere is nowadays the management, monitoring and automation suite around it. However, disruptive newcomers are using DevOps and cloud native architectures, and coming from a development background myself, I can see why they are the future, and I was sure there are enough intelligent people at VMware to recognize this as well.

So VMware had to move, and after following the DevOps and cloud native tracks and talking to Kit Colbert privately it became very obvious they are in fact moving.

However, VMware has a strong customer base in Ops in organizations which aren’t known for their aptitude to change; perhaps the willingness to change technology is there, but real change is needed especially on the culture and process fronts in order to keep up.

So it’s pretty clear: VMware realizes exactly what needs to happen, the difficulty is in determining the right pace for change: if they go too fast they alienate their current customer base, and if they go to slow they become legacy themselves. A real balancing act, but the proposition is strong.

.NET on Pivotal Cloud Foundry

In my latest post, I tested Lattice.cf, the single VM smaller brother of Pivotal Cloud Foundry (PCF). Considering a full installation of PCF has a footprint of about 25 Virtual Machines (VM) requiring a total of over 33Gb RAM, 500+ Gb storage, and some serious CPU power, it’s not hard to see why Lattice is more developer friendly. However, that wasn’t my real motivation to try it out: more important was Lattice’s incorporation of the new elastic runtime architecture codename ‘Diego‘ which will replace the current Droplet Execution Agent (DEA) based runtime in due time.

For me, the main reasons to get excited about Diego are two-fold:

  • Diego can run Docker containers: I demoed this in my latest post
  • Diego can run Windows (.NET) based workloads

In this post, I’ll demo the Windows based workloads by deploying an ASP.NET MVC application, which uses the full-fledged production ready .NET stack and requires a Windows kernel – as opposed to .NET Core which is cross-platform, runs on a plentitude of OSes, and is very exciting, but not production ready yet.

Diego on PCF

At this point we have to resort to PCF as Lattice can’t run Windows workloads. This is because Lattice’s strong point (all services in 1 VM) is also its weakness: since Lattice incorporates all required services in a single Linux VM it quite obviously loses the ability to schedule Windows workloads.

Let’s take a quick look at the Diego architecture overview:

Diego architecture overview

Diego architecture overview

Diego consists of a ‘brain’ and a number of ‘cells’. These cells run a container backend implementing Garden – a platform-neutral API for containerization (and the Go version of Warden). The default backend – garden-linux – is Linux based and compatible with various container technologies found on Linux (including Docker).

As soon as we run the various services in the overview on seperate VMs (as we do on PCF), it becomes possible to provision a Windows cell. ‘All’ we need is a Windows backend for garden and the various supporting services, and we should be good to go. Preferably in the form of a convenient installer.

One problem remains: we still need the Diego runtime in Pivotal Cloud Foundry. Kaushik Bhattacharya at Pivotal supplied me with an internal beta of ‘Diego for PCF’ (version 0.2) which I could install in my Pivotal on VMware vSphere lab environment. Installed, the Ops Manager tile looks like this:

Diego for PCF tile

Diego for PCF tile

And the default VMs and resources that come with Diego for PCF (notice the single default cell which is a garden-linux cell):

Diego for PCF default resources

Diego for PCF default resources

garden-windows cell

To provision a Windows cell, we have to create a new VM manually and install a Windows server OS, and run a setup powershell script as well as the installer with the correct configuration. I’ll skip the details here, but when all is done, you can hit the receptor service url (in my case https://receptor.system.cf.lab.local/v1/cells), and it should show 2 cells: 1 installed as part of Diego for PCF, as well as the one we just created:

        "cell_id": "DiegoWinCell",
        "zone": "d48224618511a8ac44df",
        "capacity": {
            "memory_mb": 16383,
            "disk_mb": 40607,
            "containers": 256
        "cell_id": "cell-partition-d48224618511a8ac44df-0",
        "zone": "d48224618511a8ac44df",
        "capacity": {
            "memory_mb": 16049,
            "disk_mb": 16325,
            "containers": 256

Intermezzo: Containerization on Windows?

There must be some readers who are intrigued by the garden-windows implementation by now. After all, since when does Windows have support for containers? In fact, Microsoft has announced container support in the next Server OS, and the Windows Server 2016 Technical Preview 3 with container support was just released. However, this is not the ‘containerization’ used in the current version of garden-windows.

So how does it work?

Some of you may know Hostable Web Core: an API by which you can load a copy of IIS inside your process. So what happens when you push an app to Cloud Foundry and select the windows stack, is that app is hosted inside a new process on the cell, in which it gets its own copy of IIS after which it’s started.

I know what you’re thinking by now: “but that’s not containerization at all”. Indeed, strictly speaking it isn’t: it doesn’t use things like cgroups and namespaces used by Linux (and future Windows) container technologies in order to guarantee container isolation. However, from the perspective of containers as ‘shipping vehicles for code’ it’s very much containerization, as long as you understand the security implications.

Deploying to the Windows cell

Deployment to the Windows cell isn’t harder than to default cells, however, there are a couple of things to keep in mind:

  • as the windows stack isn’t the default, you have to specify it explicitly
  • as for now the DEA mode of running tasks is still default, you have to enable Diego support explicitly

Using the Diego Beta CLI the commands to push a full .NET stack demo MVC app are as follows (assuming you cloned it from github):

cf push diegoMVC -m 2g -s windows2012R2 -b https://github.com/ryandotsmith/null-buildpack.git --no-start -p ./
cf enable-diego diegoMVC
cf start diegoMVC

After pushing and scaling the Pivotal Apps Manager:

Pivotal CF Apps Manager with DiegoMVC .NET  app

Pivotal CF Apps Manager with DiegoMVC .NET app

And the app itself:

DiegoMVC .NET application on Windows cell on Pivotal CF

DiegoMVC .NET application on Windows cell on Pivotal CF


Diego for PCF is still in internal beta, but soon Pivotal Cloud Foundry will have support for running applications using the full .NET stack.

Getting started with Lattice

In April, Pivotal released Lattice, a platform for hosting cloud native applications which is aimed at being accessible and convenient for developers. However, it’s not just that: it’s also a testbed for the new elastic runtime codename ‘Diego’ which we will likely see incorporated in the next version of Lattice’s big – enterprise ready – brother Pivotal Cloud Foundry in due time. This new runtime comes with the ability to run Docker workloads which makes it very interesting.

In this post, I’ll describe the minimal steps required to set up a machine in which we create another VM using vagrant and virtualbox which will run Lattice and host its first containerized workloads. Note: in case you already run VMware fusion/workstation and the VMware integration for vagrant, you don’t need the initial hosting VM, so you can skip the first steps and go directly to ‘Get Lattice’.

Create a virtual machine

In fact, it doesn’t have to be virtual, just get a x64 machine, either physical or using your hypervisor of choice. Since this machine will run its own virtualized workload, it’s essential it has virtualization instructions, either hardware based or virtualized. For example in VMware Workstation this option shows as: Virtualize VTx

Install Ubuntu

Install the latest stable Ubuntu (desktop), and make sure it’s updated.

Install vagrant and virtualbox

In order to spin up the lattice machine, we’ll use virtualbox, and to provision it lattice depends on vagrant. For vagrant we need a version (>1.6) which is not by default in the ubuntu repos, so we install it via direct download:

sudo apt-get install virtualbox
wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.2_x86_64.deb
sudo dpkg -i vagrant_1.7.2_x86_64.deb

Get lattice

Install git and clone the lattice repository:

sudo apt-get install git
git clone https://github.com/cloudfoundry-incubator/lattice.git
cd lattice
git checkout <VERSION>

Here <VERSION> is the version you find in the file ‘Version’.

Next provision the virtual machine with
vagrant up

Of course we could ssh to the lattice VM now, but the idea is to access it via API calls. The lattice CLI wraps the API and offers a convenient interface.

Get/build the lattice CLI

You can build the lattice CLI from source, or download a binary. If you take the download option you can skip the following paragraph.

Building the CLI from source

In order to do this we need Go, and again the version in the ubuntu repos is too old (<1.4):

wget --no-check-certificate --no-verbose https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz

And to setup the build environment:

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

If you want this to persist, add the exports to ~/.bashrc

Now build the CLI binary:

go get -d github.com/cloudfoundry-incubator/lattice/ltc

You can check if the CLI has been build successfully by typing: ltc

Connecting to the lattice

Point the CLI to the lattice instance:
ltc target <system_ip_from_Vagrantfile>.xip.io

If you run into errors or timeouts at this stage, try to ping the lattice VM, or (re)start the lattice VM directly from VirtualBox which will usually tell you what’s wrong.

Running a test workload

A Docker container hosting a simple test web application is available in the Docker hub. You can spin up your first instance with:
ltc create lattice-app cloudfoundry/lattice-app

Check it’s status with
ltc status lattice-app

Next, spin up a few more with
ltc scale lattice-app 4

When this is done, point a browser at lattice-app.<system_ip_from_Vagrantfile>.xip.io and refresh a couple of times to see the built in load balancing and router in action.

Cloud native applications: a primer

At times, IT departments can get so large and influential it can become tempting to believe in the fallacy that the IT department has a right to exist in itself. But here’s the thing:

  • unless you work in a real tech company, the only raison d’être for IT is to support the business, and more specifically to support business applications
  • the business doesn’t care how you set up your infrastructure, as long as they get the apps they want, and as long as they work

App evolution

In the old days in order to run and support one application, you needed to add a new wing to your datacenter, insert tons of equipment and staff it with greasemonkeys to keep it running. Clearly, this was less than ideal as applications had strong dependencies on hardware and Operating System (OS). This meant developers couldn’t just focus on adding value for the business but instead had to spend lots (if not most) of their time on worrying about the hardware, infrastructure and OS.

As technology evolved, we have seen a movement to get rid of those dependencies and let development become as app centric as possible:

  • virtualization has allowed us to run applications without physical hardware dependencies
  • standardization and abstraction made sure we could worry about just one type of virtual hardware in the enterprise (x86, LUN/NFS storage)
  • virtual machine languages (Java/.NET) and run anywhere languages (javascript, python) have abstracted the remaining hardware (x86) and OS dependencies away

Platforms & deployment

The same kind of movement has taken place for app deployment: where the first applications were identified by the mainframe they ran on, the later (2nd) generation apps used a client-server scale up platform, where we now see a movement to scale out cloud applications.


Second and third generation platform applications.

These 3rd generation applications or cloud native applications are characterized by being independent of the hardware and OS, built to scale, built for failure, and running as Software-as-a-Service (SaaS) on a highly automated – 3rd generation – platform.


Containers are the common unit of deployment for cloud native applications, and can be seen as Operating System level virtualization: where virtual machines share the hardware of their hosting physical machine, containers share the OS core (kernel) of their hosting (virtual) machine. Therefore, containers provide a very lightweight deployment mechanism as well as OS level isolation enabling scale out of new instances in seconds (compared to minutes for VMs). In practical terms, a container is the application itself, bundled together with any libraries it requires to run and any user mode customization of the OS.


Containers are isolated but share the OS kernel of the host, thus are lightweight. However, each container must have the same OS kernel.

An important thing to keep in mind is that containers do not lift the Operating System dependency:

  • containers can only run on the same machine if they depend on the same OS kernel
  • the app running inside the container should be made to run on that particular OS
  • the OS kernel has to support containers

Operating system level virtualization support has been in OS kernels for a long time, and the same can be said about container implementations. However, initial adoption was slow until publically available guest OS (Linux) and container implementations (Docker) offered access to developers in a friendly way. Ever since adoption has skyrocketed as containers enable very short application release cycles.

3rd generation platforms

While containers are great, they are really just the deployment artifact of a true 3rd generation platform. A complete platform requires the following essentials:

The various platforms above are very much still in development. Platforms differ most by maturity, supported languages and/or runtimes (java/node/.net), the implied OS (Linux/Windows), out of the box stack integration, intended workloads (big data vs. enterprise apps), centralized logging, management UI capabilities, and maybe most important: community. Some illustrations of the evolving landscape:

  • as it stands in almost all cases the constainer OS is some flavor of Linux, but Microsoft announced Docker engine support in the next server OS, and cloud foundry has alpha engine (BOSH) support for Windows in the form of IronFoundry.
  • Docker is (becoming) the dominant container packaging artifact. To illustrate this, originally Cloud Foundry uses a different container construction technology based on detection of the bare application type (java/node) followed by a scripted, on demand construction of a droplet. The next iteration codename Diego will have support for various container back-ends including Docker.

Containers or VMs?

After seeing all the goodness containers can provide, it’s a natural question to wonder whether we still need virtual machines. After all, if we have OS level virtualization, why would we need to stack it on top of machine level virtualization? The answer is two fold:

  • security: hardware level virtualization is much more mature, and hardware assisted. This provides a battle hardened level of isolation on top of software based container isolation. Moreover, specifying security policies such as firewalls at the hardware level can still be convenient.
  • virtualization suites such as VMware vSphere are not just a hypervisor. The extra functionality provided by for example the ability to dynamically distribute resources and deploy new machines from templates in minutes can still be a convincing argument in favor of hardware virtualization, also when using OS level virtualization on top of it.

Moving to the 3rd generation

Creating new apps for the 3rd generation platform is not rocket science. It does however require a different mindset on what a solution architecture should look like. Developers and businesses should both understand the (im)possibilities in adopting this new platform generation.

Developers: the 12 factor app

I already mentioned a limited set of characteristics of cloud native applications above. Adam Wiggins wrote a manifesto about the so called 12 factor app in which he describes the ingredients/factors for a proper cloud native app. Depending on the specific application, not all factors described are equally relevant, so the 12 factors should be considered as a set of best practices, not a law set in stone. However, developers writing new applications or migrating legacy applications to the cloud would do well to consult this document before taking any definite decisions.

All in or hybrid?

It’s always more easy to start with a clean slate. When you for example have a greenfield with some ESXi hosts, it’s relatively easy to install a 3rd generation platform stack and start hosting 12 factor applications. In practice however, most enterprises will have numerous legacy apps running just fine on (virtualized) hardware. Depending on the number and type of legacy apps in your organisation, it could be you want to go all in on the next generation and make a clean switch, or rather build a hybrid environment.

In the case of a hybrid approach you will have to identify the best candidates for early migration. To do this you can look at:

  • application workload: what apps have a substantial workload which is in your current environment scaled up, not out?
  • the work involved to migrate: stateless, modular applications are much more easy to migrate than stateful, monolithic ones

The reason why existing web applications are usually the top candidates is because when designed well they are stateless and modular. Therefore, an example of a hybrid approach is for instance to migrate specific web applications to the new platform, and keep existing databases on predetermined fixed hardware and provision them as a service on the 3rd platform.