Your Docker Image Is Not Your Running Container (And a Migration Will Prove It)

We were moving a live ERP off a cloud VM managed by another vendor onto infrastructure the client controlled. The application ran in Docker. So the plan was the obvious one: docker save the image, move it, docker load it on the target, restore the database and filestore, done.

The image loaded. The container started. The application then refused to serve the database at all, with this in the log:

openpyxl library is not installed

followed, on every request, by:

KeyError: 'Production'

The second error is the misleading one. It looks like the database is missing or misnamed. It is not. The registry failed to build because a module import failed, and a database whose registry cannot build looks exactly like a database that does not exist.

What Actually Happened

Somebody, at some point in the previous two years, had run pip install openpyxl inside the running container to make an Excel export work. It fixed the problem. It was never added to a Dockerfile, a requirements file, or any image.

This is the part worth internalising: docker save exports the image. It does not export the container. Anything installed into a running container after it started lives in that container's writable layer, and a saved image knows nothing about it. The old container had been running for a long time and had quietly accumulated state that existed nowhere else.

On the old host, that state was invisible, because the container that had it was the container that was running. The moment we tried to recreate the environment from its own image, the difference became the outage.

The Diff That Finds It in Thirty Seconds

You do not have to guess which packages drifted. Ask both containers what they have and compare:

# on the old host, against the LIVE container
docker exec <old-container> pip3 freeze | sort > /tmp/live.txt

# on the new host, against the freshly loaded image
docker exec <new-container> pip3 freeze | sort > /tmp/fresh.txt

# what the live container has that the image does not
comm -23 /tmp/live.txt /tmp/fresh.txt

In our case that produced five lines:

et-xmlfile==1.1.0
future==1.0.0
openpyxl==3.1.3
ply==3.11
pyzk==0.9

Only the first error message had mentioned openpyxl. Had we installed just that one and moved on, we would have hit the next missing package on a different code path days later, probably in front of the client. The diff is what turns one visible symptom into the complete list.

Two Gotchas When You Fix It

You must install as root. The obvious command does not work:

docker exec <container> pip3 install openpyxl==3.1.3
# Defaulting to user installation because normal site-packages is not writeable
# ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied

The container's default user is not root, so pip falls back to a user install, which then lands on a root-owned mount and fails. Force root:

docker exec -u root <container> pip3 install   openpyxl==3.1.3 et-xmlfile==1.1.0 future==1.0.0 ply==3.11 pyzk==0.9

Pin the exact versions from the diff. Not the latest. The live system had been running against those specific versions for years, and a migration is the worst possible moment to also find out whether the application still works against a newer openpyxl.

The Package Whose Import Name Is Not Its Name

One of the five was pyzk, a library for talking to ZKTeco biometric attendance devices. Worth knowing if you ever have to verify this by hand: you install pyzk, but you import zk.

python3 -c "import zk; print(zk.__file__)"   # correct
python3 -c "import pyzk"                      # ModuleNotFoundError

A verification script that checks import pyzk will report a correctly installed library as missing. We have seen people reinstall a working package three times over exactly this.

Making the Fix Survive a Restart

Installing into a running container puts you right back where the original problem came from: the fix lives in the writable layer and disappears the next time the container is recreated. Commit it:

docker commit <container> yourregistry/yourimage:v1.0

One caution people miss: by default docker commit pauses the container while it snapshots. On a large filesystem that pause is long enough to be user-visible on a live system. Either accept the pause during a planned window, or pass --pause=false and accept that you are snapshotting a moving filesystem. On a database container, take the pause.

The properly correct fix is a Dockerfile that installs these packages, so the image is reproducible from source rather than from a snapshot of a machine. Do that when there is time. During a migration window, commit the working container, get the client back online, and schedule the Dockerfile for the following week.

The General Rule

Before migrating any long-lived container, assume it has drifted from its image, and prove otherwise rather than hoping. The check costs one command:

docker diff <container>

That lists every file added, changed or deleted in the container relative to its image. On a container that has been running for two years, that output is rarely empty, and everything in it is state that a docker save will silently leave behind.

A container that has run for years is not a deployment artefact. It is a pet with undocumented medication.