A couple of days ago, I got a question from a customer about an issue he ran into when trying to spin up a container on Windows.

The context was as follows:

> docker container run hello-world:nanoserver
Unable to find image 'hello-world:nanoserver' locally
nanoserver: Pulling from library/hello-world
C:\Program Files\Docker\docker.exe: no matching manifest for windows/amd64 10.0.14393 in the manifest list entries.
See 'C:\Program Files\Docker\docker.exe run --help'.

 

I thought that was very interesting because it pointed out some considerations about Docker image architecture design. First, we must bear in mind that containers and the underlying host share a single kernel by design and the container’s base image must match that of the host.

Let’s first begin with containers in a Linux world because it highlights the concept of Kernel sharing between different distros. In this demo, let’s say I’m running a Linux Ubuntu server 16.04 …

$ cat /etc/os-release | grep -i version
VERSION="16.04.6 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial

 

… and let’s say I want to run a container based on Centos 6.6 …

$ docker run --rm -ti centos:6.6 cat /etc/centos-release
Unable to find image 'centos:6.6' locally
6.6: Pulling from library/centos
5dd797628260: Pull complete
Digest: sha256:32b80b90ba17ed16e9fa3430a49f53ff6de0d4c76ad8631717a1373d5921fa26
Status: Downloaded newer image for centos:6.6
CentOS release 6.6 (Final)

 

You may wonder how it is possible to run different distros between the container and the host and what’s the magic behind the scene? In fact, both the container and the host share the same Linux kernel and even if CentOS 6.6 ships with a kernel version 2.6, while Ubuntu 16.04 ships with 4.4 we usually may upgrade the kernel since it’s backward compatible. The commands below demonstrate the centos container is using the same Kernel than the host.

$ uname -r
4.4.0-142-generic
$ docker run --rm -ti centos:6.6 uname -r
4.4.0-142-generic

 

Let’s say now my docker host is running on the x64 architecture. If we look at the Centos image supported architectures on Docker hub, we notice different ones:

From the output above, we may deduce it should exist a combination of different images and tags for each available architecture and the interesting point is how does Docker pull the correct one regarding my underlying architecture? This is where manifest lists come into play and allow multi-architecture images. A manifest list contains platform segregated references to a single-platform manifest entry. We may inspect a manifest list through the docker manifest command (still in experimental mode at the moment of writing this blog post).

For example, if I want to get a list of manifests and their corresponding architectures for the Centos 7, I can run docker manifest command as follows:

$ docker manifest inspect centos:7 --verbose
[
        {
                "Ref": "docker.io/library/centos:7@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66",
                        "size": 529,
                        "platform": {
                                "architecture": "amd64",
                                "os": "linux"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 2182,
                                "digest": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1"
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                        "size": 75403831,
                                        "digest": "sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df"
                                }
                        ]
                }
        },
        {
                "Ref": "docker.io/library/centos:7@sha256:9fd67116449f225c6ef60d769b5219cf3daa831c5a0a6389bbdd7c952b7b352d",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:9fd67116449f225c6ef60d769b5219cf3daa831c5a0a6389bbdd7c952b7b352d",
                        "size": 529,
                        "platform": {
                                "architecture": "arm",
                                "os": "linux",
                                "variant": "v7"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 2181,
                                "digest": "sha256:8c52f2d0416faa8009082cf3ebdea85b3bc1314d97925342be83bc9169178efe"
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                        "size": 70029389,
                                        "digest": "sha256:193bcbf05ff9ae85ac1a58cacd9c07f8f4297dc648808c347cceb3797ae603af"
                                }
                        ]
                }
        },
        {
                "Ref": "docker.io/library/centos:7@sha256:f25f24daae92b5b5fe75bc0d5d9a3d2145906290f25aa434c43bfcefecd10dec",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:f25f24daae92b5b5fe75bc0d5d9a3d2145906290f25aa434c43bfcefecd10dec",
                        "size": 529,
                        "platform": {
                                "architecture": "arm64",
                                "os": "linux",
                                "variant": "v8"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 2183,
                                "digest": "sha256:7a51de8a65d533b6706fbd63beea13610e5486e49141610e553a3e784c133a37"
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                        "size": 74163767,
                                        "digest": "sha256:90c48ff53512085fb5adaf9bff8f1999a39ce5e5b897f5dfe333555eb27547a7"
                                }
                        ]
                }
        },
        {
                "Ref": "docker.io/library/centos:7@sha256:1f832b4e3b9ddf67fd77831cdfb591ce5e968548a01581672e5f6b32ce1212fe",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:1f832b4e3b9ddf67fd77831cdfb591ce5e968548a01581672e5f6b32ce1212fe",
                        "size": 529,
                        "platform": {
                                "architecture": "386",
                                "os": "linux"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 2337,
                                "digest": "sha256:fe70670fcbec5e3b3081c6800cb531002474c36563689b450d678a34a89b62c3"
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                        "size": 75654099,
                                        "digest": "sha256:39016a8400a36ce04799adba71f8678ae257d9d8dba638d81b8c5755f01fe213"
                                }
                        ]
                }
        },
        {
                "Ref": "docker.io/library/centos:7@sha256:2d9b27e9c89d511a58873254d86ecf96df0f599daae3d555d896fee9f49fedf4",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:2d9b27e9c89d511a58873254d86ecf96df0f599daae3d555d896fee9f49fedf4",
                        "size": 529,
                        "platform": {
                                "architecture": "ppc64le",
                                "os": "linux"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "size": 2185,
                                "digest": "sha256:c9744f4afb966c58d227eb6ba03ab9885925f9e3314edd01d0e75481bf1c937d"
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                        "size": 76787221,
                                        "digest": "sha256:deab1c539926c1ca990d5d025c6b37c649bbba025883d4b209e3b52b8fdf514a"
                                }
                        ]
                }
        }
]

 

Each manifest entry contains different information including the image signature digest, the operating system and the supported architecture. Let’s pull the Centos:7 image:

$ docker pull centos:7
7: Pulling from library/centos
8ba884070f61: Pull complete
Digest: sha256:a799dd8a2ded4a83484bbae769d97655392b3f86533ceb7dd96bbac929809f3c
Status: Downloaded newer image for centos:7
docker.io/library/centos:7

 

Let’s have a look at the unique identifier of the centos:7 image:

$ docker inspect --format='{{.Id}}' centos:7sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1

 

It corresponds to the SchemaV2Manifest digest value of the manifest entry related to the x64 architecture (please refer to the docker manifest inspect output above). Another official way to query manifest list and architecture is to go through the mplatform/mquery container as follows:

$ docker run mplatform/mquery centos:7
Image: centos:7
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm/v7
   - linux/arm64
   - linux/386
   - linux/ppc64le

 

However, for a Linux Centos 6.6 image (used in my first demo) the architecture support seems to be limited to  the x64 architecture:

$ docker run mplatform/mquery centos:6.6
Image: centos:6.6
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64

 

Now we are aware of manifest lists and multi-architecture images let’s go back to the initial problem. The customer ran into an platform compatibility issue when trying to spin-up a the hello-world:nanoserver container on a Windows Server 2016 Docker host. As a reminder, the error message was:

no matching manifest for windows/amd64 10.0.14393 in the manifest list entries.

In the way, that may be surprising because Windows host and containers also share a single Kernel. That’s true and it was the root cause of my customer’s issue by the way. The image he wanted to pull supports only the following Windows architecture (queried from the manifest list):

> docker run mplatform/mquery hello-world:nanoserver
Image: hello-world:nanoserver
 * Manifest List: Yes
 * Supported platforms:
   - windows/amd64:10.0.17134.885
   - windows/amd64:10.0.17763.615

 

You may notice several supported Windows platforms but with different operating system versions. Let’s have look at the Docker host version in the context of my customer:

> [System.Environment]::OSVersion.Version
Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      14393  0

 

The tricky part is Windows Server 2016 comes with different branches – 1607/1709 and 1803 – which aren’t technically all the same Windows Server version. Each branch comes with a different build number. Referring to the Microsoft documentation when the build number (3rd column) is changing a new operating system version is published. What it means in that case is the OS version between the Windows Docker host and the Docker image we tried to pull are different hence we experienced this compatibility issue. However let’s precise that images and containers may run with newer versions on the host side but the opposite is not true obviously. You can refer to the same Microsoft link to get a picture of Windows container and host compatibility. 

How to fix this issue? Well, we may go two ways here. The first one consists in re-installing a Docker host platform compatible with the corresponding image. The second one consists in using an image compatible with the current architecture and referring to the hello-world image tags we have one. We may check the architecture compatibility by query the manifest file list as follows:

> docker run mplatform/mquery hello-world:nanoserver-sac2016
Image: hello-world:nanoserver-sac2016
 * Manifest List: Yes
 * Supported platforms:
   - windows/amd64:10.0.14393.2551

 

Let’s try to pull the image with the nanoserver-sac2016 tag:

> docker pull hello-world:nanoserver-sac2016
nanoserver-sac2016: Pulling from library/hello-world
bce2fbc256ea: Already exists
6f2071dcd729: Pull complete
909cdbafc9e1: Pull complete
a43e426cc5c9: Pull complete
Digest: sha256:878fd913010d26613319ec7cc83b400cb92113c314da324681d9fecfb5082edc
Status: Downloaded newer image for hello-world:nanoserver-sac2016
docker.io/library/hello-world:nanoserver-sac2016

 

Here we go!

See you!

By David Barbarin