Add checkpoint/restore to docker API by boucher · Pull Request #13602 · moby/moby

boucher · 2015-05-29T20:12:15Z

Adds support in the daemon and driver for checkpoint and restore, and exposes that in the API, but not in the CLI yet.

cpuguy83 · 2015-05-29T20:29:52Z

Can just be if !opts.LeaveRunning {
Also, shouldn't this opt be called Shutdown or something?

I actually think we should remove this. There's a related section in cleanup that's not releasing when checkpointed. I think that maybe can get removed.

jessfraz · 2015-05-29T20:36:26Z

I want the cli 😍

cpuguy83 · 2015-05-29T20:37:33Z

Are we planning to support migration with this?

boucher · 2015-05-29T20:44:18Z

What kind of migration support do you have in mind?

cpuguy83 · 2015-05-29T20:49:53Z

As in I've got a tarball of checkpoint stuff which I pulled from daemon A, let me post that to daemon B's restore endpoint... I guess like docker checkpoint giddy_torvalds | docker restore -

boucher · 2015-05-29T20:53:36Z

@jfrazelle added

boucher · 2015-05-29T20:53:41Z

@cpuguy83 You can do: docker checkpoint --image-path=/something; docker restore
--image-path=/something. Eventually, it might be nice to add a flag like
--commit=true to also checkpoint the filesystem at the same time.

runcom · 2015-05-30T18:19:06Z

isn't err == nil always since you checked for != nil before in the if and never reassigned?

runcom · 2015-05-30T18:51:42Z

Just wondering, should we have to update the hack container to have the criu exe as we have lxc

boucher · 2015-05-30T20:45:57Z

Yes definitely. It's on my todo list

On Saturday, May 30, 2015, Antonio Murdaca [email protected] wrote:

Just wondering, should we update the hack container to have the criu exe
(and kernel conf) as we have lxc

—
Reply to this email directly or view it on GitHub
#13602 (comment).

Sent from Gmail Mobile

boucher · 2015-06-01T07:48:55Z

I'm having a tough time figuring out how to get the tests running in my vm. I can run the tests on master, but for some reason this branch produces this error (which jenkins does not):

---> Making bundle: cross (in bundles/1.7.0-dev/cross)
Created symlinks: bundles/1.7.0-dev/cross/linux/amd64/docker bundles/1.7.0-dev/cross/linux/amd64/docker-1.7.0-dev bundles/1.7.0-dev/cross/linux/amd64/docker-1.7.0-dev.md5 bundles/1.7.0-dev/cross/linux/amd64/docker-1.7.0-dev.sha256
Building: bundles/1.7.0-dev/cross/linux/386/docker-1.7.0-dev
# github.com/docker/libcontainer/apparmor
vendor/src/github.com/docker/libcontainer/apparmor/setup.go:17: undefined: IsEnabled
make: *** [test] Error 1

boucher · 2015-06-02T22:46:06Z

@runcom I've added instructions to the container Dockerfile to build criu, and added a simple test

runcom · 2015-06-02T22:47:56Z

@boucher amazing, thanks! I'm trying right now

ankushagarwal · 2015-06-02T22:57:15Z

I am really excited about this feature! Just to be clear, criu is going to be an external dependency which needs to be installed on the host running docker, right?

runcom · 2015-06-02T23:03:38Z

@boucher I'm getting the following error while inside the docker dev container (and was the error I told you about on IRC last time we spoke):

Error response from daemon: Cannot checkpoint container c6d: criu failed: type NOTIFY errno 0
Error: failed to checkpoint one or more containers

Could it be some kernel configs missing in my kernel? (though I have checked against criu manual and I'm pretty sure I have all configs either as module or built in)

boucher · 2015-06-02T23:04:32Z

@ankushagarwal Yes, you'll need to have it available on the host. It should print an error message telling you that version 1.5.2 or greater is needed if you don't have it.

ankushagarwal · 2015-06-02T23:16:18Z

I could not checkpoint a running nginx container

I checked out boucher@d3f1d38 and built the binary using make BINDDIR=. binary

Ran the docker daemon inside a container

docker run --privileged --rm -ti -v `pwd`:/go/src/github.com/docker/docker docker-dev:cr-combined /bin/bash
docker -dD

Then ran an nginx container using docker run -d nginx
and tried to checkpoint using the command
docker checkpoint <container-name>
and I get the error :

Error response from daemon: Cannot checkpoint container sleepy_almeida: criu failed: type DUMP errno 0
Error: failed to checkpoint one or more containers

These are the logs on the daemon

DEBU[0076] Calling POST /containers/{name:.*}/checkpoint
INFO[0076] POST /v1.20/containers/sleepy_almeida/checkpoint
ERRO[0076] Handler for POST /containers/{name:.*}/checkpoint returned error: Cannot checkpoint container sleepy_almeida: criu failed: type DUMP errno 0
ERRO[0076] HTTP Error                                    err=Cannot checkpoint container sleepy_almeida: criu failed: type DUMP errno 0 statusCode=500

boucher · 2015-06-03T00:24:06Z

@ankushagarwal What filesystem are you using? There's a known kernel issue with overlayfs. I'm told that this patch should fix it, if you're interested in trying it out: https://lkml.org/lkml/2015/3/20/372

I also can't checkpoint nginx on aufs though, so I'll look into it. For future reference, it's really helpful to get the dump.log criu generates for debugging these issues. You should be able to find it at:

/var/run/docker/execdriver/native/<container_id>/criu.work/dump.log

(You can also pass in a custom path to the checkpoint command and store this log somewhere else). Here's my failure log for nginx:

https://gist.github.com/boucher/0db7cb44e90f7d3c62bc

tianon · 2015-06-03T00:27:45Z

A note about version requirements for CRIU will need to be added in "project/PACKAGERS.md" too. 👍

jessfraz · 2015-12-29T16:17:12Z

I'm curious do the apparmor profiles work then or the default one?

On Dec 29, 2015, 08:11 -0800, Ross [email protected], wrote:

It seems the introduction of a default seccomp profile is the issue. Things work now only if you provide--security-opt seccomp:unconfined.

@xemul(https://github.com/xemul), I believe that seccomp isn't really supported in CRIU right now?
@jfrazelle(https://github.com/jfrazelle)I'm guessing this default profile is going to ship in 1.10?

—
Reply to this email directly orview it on GitHub(#13602 (comment)).

boucher · 2015-12-29T16:20:43Z

The default apparmor profile works. I'm looking through CRIU now, it seems to have at least some seccomp support. The error message I'm getting is in ptrace: Error (ptrace.c:64): suspending seccomp failed: Invalid argument (full log: https://gist.github.com/boucher/a64b611229ebe1a117f0).

jessfraz · 2015-12-29T16:23:41Z

Ah I think I have an idea I'll try somethings :)

On Dec 29, 2015, 08:22 -0800, Ross [email protected], wrote:

The default apparmor profile works. I'm looking through CRIU now, it seems to have at least some seccomp support. The error message I'm getting is in ptrace:Error (ptrace.c:64): suspending seccomp failed: Invalid argument(full log:https://gist.github.com/boucher/a64b611229ebe1a117f0).

—
Reply to this email directly orview it on GitHub(#13602 (comment)).

xemul · 2015-12-29T19:59:12Z

@boucher seccomp is supported in criu master, but it needs kernel 4.3 or higher.

boucher · 2015-12-29T20:04:52Z

But not in CRIU 1.8?

On Tue, Dec 29, 2015 at 3:00 PM Pavel Emelyanov [email protected]
wrote:

@boucher https://github.com/boucher seccomp is supported in criu
master, but it needs kernel 4.3 or higher.

—
Reply to this email directly or view it on GitHub
#13602 (comment).

xemul · 2015-12-30T08:52:20Z

@boucher in criu 1.7 there's support for strict mode and in 1.8 there's support for filter mode. Since you see the EINVAL from ptrace call this means that the problem is in the kernel -- necessary support isn't there (either not there at all or just not compiled in).

Signed-off-by: Marcos Lilljedahl <[email protected]>

boucher · 2016-01-05T19:54:00Z

@jfrazelle Any further thoughts on the seccomp issue? Some discussion on the mailing list suggested that kernel 4.4 or newer may be needed in order to get the necessary support for suspend/resume in seccomp.

jessfraz · 2016-01-05T19:57:50Z

yeah thats a pretty high kernel version, seeing as it hasn't even been
released yet, I am unsure what to do, we could warn people that by using
criu you cannot use seccomp...?

On Tue, Jan 5, 2016 at 11:55 AM, Ross Boucher [email protected]
wrote:

@jfrazelle https://github.com/jfrazelle Any further thoughts on the
seccomp issue? Some discussion on the mailing list suggested that kernel
4.4 or newer may be needed in order to get the necessary support for
suspend/resume in seccomp.

—
Reply to this email directly or view it on GitHub
#13602 (comment).

marcosnils · 2016-01-05T20:02:01Z

@boucher @jfrazelle. @xemul mentioned that 4.3 or above was necessary.

boucher · 2016-01-05T20:04:01Z

Yeah, I guess if there's no way to convince you not to turn on seccomp by
default ;-), the next best thing would be to detect the options in the
checkpoint/restore command and print out a helpful error. (Though, I guess
in many instances people might realize too late they can't actually
checkpoint the container they are running)

On Tue, Jan 5, 2016 at 3:00 PM Jess Frazelle [email protected]
wrote:

yeah thats a pretty high kernel version, seeing as it hasn't even been
released yet, I am unsure what to do, we could warn people that by using
criu you cannot use seccomp...?

On Tue, Jan 5, 2016 at 11:55 AM, Ross Boucher [email protected]
wrote:

@jfrazelle https://github.com/jfrazelle Any further thoughts on the
seccomp issue? Some discussion on the mailing list suggested that kernel
4.4 or newer may be needed in order to get the necessary support for
suspend/resume in seccomp.

—
Reply to this email directly or view it on GitHub
#13602 (comment).

—
Reply to this email directly or view it on GitHub
#13602 (comment).

jessfraz · 2016-01-05T20:07:11Z

can you restore it without the seccomp rules applied? or is it just a no
across the bat?

UPDATE: yeah it probably doesnt even know how set bpf

On Tue, Jan 5, 2016 at 12:06 PM, Ross Boucher [email protected]
wrote:

Yeah, I guess if there's no way to convince you not to turn on seccomp by
default ;-), the next best thing would be to detect the options in the
checkpoint/restore command and print out a helpful error. (Though, I guess
in many instances people might realize too late they can't actually
checkpoint the container they are running)

On Tue, Jan 5, 2016 at 3:00 PM Jess Frazelle [email protected]
wrote:

yeah thats a pretty high kernel version, seeing as it hasn't even been
released yet, I am unsure what to do, we could warn people that by using
criu you cannot use seccomp...?

On Tue, Jan 5, 2016 at 11:55 AM, Ross Boucher [email protected]
wrote:

@jfrazelle https://github.com/jfrazelle Any further thoughts on the
seccomp issue? Some discussion on the mailing list suggested that
kernel
4.4 or newer may be needed in order to get the necessary support for
suspend/resume in seccomp.

—
Reply to this email directly or view it on GitHub
#13602 (comment).

—
Reply to this email directly or view it on GitHub
#13602 (comment).

—
Reply to this email directly or view it on GitHub
#13602 (comment).

jessfraz · 2016-01-05T20:20:14Z

I think for now we should just require people who want to use criu to have
kernel 4.3 :/, since it also will involve downloading experimental and criu
I think its not a huge ask, it is in some backports.

On Tue, Jan 5, 2016 at 12:06 PM, Jessie Frazelle [email protected] wrote:

can you restore it without the seccomp rules applied? or is it just a no
across the bat?

On Tue, Jan 5, 2016 at 12:06 PM, Ross Boucher [email protected]
wrote:

Yeah, I guess if there's no way to convince you not to turn on seccomp by
default ;-), the next best thing would be to detect the options in the
checkpoint/restore command and print out a helpful error. (Though, I guess
in many instances people might realize too late they can't actually
checkpoint the container they are running)

On Tue, Jan 5, 2016 at 3:00 PM Jess Frazelle [email protected]
wrote:

yeah thats a pretty high kernel version, seeing as it hasn't even been
released yet, I am unsure what to do, we could warn people that by using
criu you cannot use seccomp...?

On Tue, Jan 5, 2016 at 11:55 AM, Ross Boucher <[email protected]

wrote:

@jfrazelle https://github.com/jfrazelle Any further thoughts on the
seccomp issue? Some discussion on the mailing list suggested that
kernel
4.4 or newer may be needed in order to get the necessary support for
suspend/resume in seccomp.

—
Reply to this email directly or view it on GitHub
#13602 (comment).

—
Reply to this email directly or view it on GitHub
#13602 (comment).

—
Reply to this email directly or view it on GitHub
#13602 (comment).

SaiedKazemi · 2016-01-05T20:42:08Z

@boucher @jfrazelle I also think that the best way --at least for now-- would be to print a descriptive error message when the user is trying to checkpoint a container that has seccomp but the kernel is not 4.4 or newer. The user can easily restart the container without seccomp and do checkpoint restore. Requiring a kernel upgrade, especially to something as new as 4.4, has its own issues. It also kind of interferes with our ongoing C/R development.

cpuguy83 · 2016-01-12T16:18:18Z

What's the status on this? Should we remove from 1.10 milestone? I'm inclined to think so since this is going to experimental anyway + code freeze is days away.

icecrime · 2016-01-12T17:11:52Z

@cpuguy83 Agreed. This is still high in the priorities to get merged somewhere around the 1.10 timeframe.

southerngs · 2016-01-21T19:09:50Z

I'm running Arch Linux with a custom kernel and the linux-image-uname -r`` dependency breaks the build for me since there isn't a Ubuntu a package that matches my kernel. Without it the build is fine for me. I'm not sure what situation this is needed for, but it might be good to use a different technique that still works for users who aren't running a Ubuntu kernel on their host system.

Does it work if you don't use a custom kernel though?

@runcom @southerngs I supplied the patch that installs that dependency based on these findings:

#13602 (comment)

#13602 (comment)

In response to @runcom question, it doesn't work on stock ubuntu unless you install this package

I meant does it work on arch with custom kernel, not sure we should support custom kernel maybe :( but if it fixable without hacks I guess it's fine

Running CRIU on with an Arch Linux host currently requires a custom kernel because the default kernel build options don't support checkpoint/restore.

But building this branch doesn't work with the default Arch kernel either. The uname -r command returns 4.2.5-1-ARCH (even inside a Ubuntu container) and when I run make DOCKER_EXPERIMENTAL=1 build on an Arch Linux system with the default kernel I get an error:

E: Unable to locate package linux-image-4.2.5-1-ARCH E: Couldn't find any package by regex 'linux-image-4.2.5-1-ARCH'

I imagine that a similar problem could occur for other non-Ubuntu hosts too. I guess it would depend on if the kernel naming convention happens to match one used by Ubuntu (I don't have any other distros to test though).

@estesp agree. 👍

@estesp I don't want to make this a blocking issue, but I would like clarification regarding your statement:

This Dockerfile starts with FROM ubuntu:14.04 and so I think that the expectation is the RUN command installing pre-reqs is strongly tied to that distro's package set, both in name and content.

I thought one of the purposes of Docker is to insulate the application running inside a container from the host system that it is running on. But it seems like you are saying that to build Docker the user should be using a Ubuntu system because the Docker file says FROM ubuntu:14.04. Doesn't that defeat some of the purpose of encapsulating the Docker build process within a container? That means anyone using a non-Ubuntu host has to modify the Dockerfile in order to build Docker. Not just that they need to follow different steps to install CRIU and other dependencies (something I think is reasonable). But that they need to modify the Dockerfile source in order to build the docker static binary using a non-Ubuntu host. So basically anyone wanting to build the experimental version of Docker on a non-Ubuntu host would need to fork the code, even if they had no interest in experimenting with the checkpoint feature.

Again I don't want this to be a blocking issue since I'm very interested in using this feature and I don't mind having mine own fork to make it work for me on Arch Linux. But I wanted to comment since it seems like either my understanding of how the Docker build process is supposed to work is wrong, or this change might be more problematic for non-Ubuntu Linux users than initially imagined.

@southerngs I think the problem that is being conflated here is "official Docker builds" versus the main Dockerfile in the root of the repo which is meant for building Docker in one default context. What is more noticeable in recent times is that this specific Dockerfile's isolation is now imperfect regarding total separation from the host. However, prior to dynamic linking of the Docker executable, and prior to lots of recent features, the encapsulation was pretty reasonable for a lion's share of cases.

Since that isn't the case anymore for producing (dynamically linked) binaries for a cross-section of supported distros, I think the Dockerfiles that people should be looking at in this discussion are found in the docker/contrib/builder subtree of this repo.

If you follow that repository path through its "deb" and "rpm" subtrees, you will find many subdirectories for variants and versions of popular Linux distributions, each with a specific Dockerfile that builds the Docker binary against the proper library versions/packages for that distro. Obviously in these cases there is no sense in which Docker can "isolate" you from the host OS because the goal of each of these Dockerfiles is to create a runnable binary for proper operation in that specific distro's environment which includes it's own variants of needed libraries and/or package names. This is obviously a slightly different use case than a standard use of Docker as simply a way to run binaries isolated from the host OS and filesystem.

@estesp thanks for the clarification. I think my expectation was based on the old devenvironment documentation (https://docs.docker.com/v1.5/contributing/devenvironment/). I noticed that I can't find that file in current source tree (although there are still some references to it in a few comments). Anyway your explanation about dynamic linking makes sense as to why the generating the Docker binary can no longer be isolated from the environment where the binary will be used.

I noticed that the contrib directory has a note that says:

The contrib directory contains scripts, images, and other helpful things which are not part of the core docker distribution. Please note that they could be out of date, since they do not receive the same attention as the rest of the repository.

Does this mean that all of the development work for Docker Linux is done using Ubuntu 14.04 host systems (as referenced in the main Dockerfile), and then later ported to other systems later?

That note probably needs updated as contrib/ definitely has the usual sense of "contributed content" that may not be developed alongside the main body of code, but the Dockerfiles I pointed to happen to also live under that sub-tree and are used to create the official build packages that are used by the official install scripts, so they are definitely maintained appropriately. I'll ping @jfrazelle in case she wants to add any info about future ideas to keep all these Dockerfiles in sync as changes/features show up in the main development Dockerfile. No pressure @jfrazelle if there is nothing to share :) Just wanted to give you an opportunity to add anything of value about the process.

cpuguy83 · 2016-01-29T02:01:41Z

Code looks mostly straight forward other than the complications with networking.
Is there anything new on this front? I know the preferred IP stuff is in now.

amcaar · 2016-01-29T12:08:38Z

@boucher I don't know if this is the right place but I need to ask you a question. With the code provided in your branch "cr-combined" (Docker version 1.10.0-dev, build a6a4511, experimental), can I migrate a checkpointed container in one host to another and restore that container in the destination host?

Because I'm trying to do it but since now I'm not able to restore the container at the destination host (restore it in the origin host can be performed with no problem). I have copied the /var/lib/docker/0.0/containers/<container_id> and also all the files of the filesystem (AUFS) in /diff, /mnt and /layer with the id of the container to the destination host. I also have restarted the docker daemon and even copy the files in /var/run/docker/execdriver and /netns but I cannot realize what is missing here. I can give you more details if needed. Any suggestion would be appreciated, Thank you in advance!

marcosnils · 2016-01-29T15:26:41Z

@amcaar Which error are you getting when trying to restore?

amcaar · 2016-02-01T10:17:07Z

@marcosnils If I only copy the files of the container in /aufs, in /containers and restart the docker daemon, I obtain this error (note: in the destination host I already have downloaded the docker image that the container uses, in this case ubuntu-latest):

root@serverpublic:/var/lib/docker/0.0/containers# docker-1.10.0-dev restore 07a352069107 INFO[0237] POST /v1.22/containers/07a352069107/restore ERRO[0237] Handler for POST /v1.22/containers/07a352069107/restore returned error: Cannot restore container 07a352069107: Cannot find child for /jolly_swanson ERRO[0237] HTTP Error err=Cannot restore container 07a352069107: Cannot find child for /jolly_swanson statusCode=500 Error response from daemon: Cannot restore container 07a352069107: Cannot find child for /jolly_swanson Error: failed to restore one or more containers

Then, I'd try to copy the files in:
/var/run/docker/execdriver/native/<container_id>/
/var/run/docker/netns/<sandboxkey_config.json>
to the destination. If I try to restart again the container, I obtain:

`root@serverpublic:/home/ubuntu# docker-1.10.0-dev ps -a

INFO[0022] GET /v1.22/containers/json?all=1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
07a352069107 ubuntu:latest "/bin/sleep 1000" 17 minutes ago Checkpointed 13 minutes ago gigantic_booth
d98506fb4c2e ubuntu:latest "/bin/echo 'Hello Wor" 46 hours ago Exited (0) 46 hours ago drunk_goodall

root@serverpublic:/home/ubuntu# docker-1.10.0-dev restart 07a352069107
INFO[0046] POST /v1.22/containers/07a352069107/restart?t=10
WARN[0046] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/0.0/containers/07a3520691077e1e6c281077acd98e02596e8ac860ed1701a338284db3bc1de0/shm: invalid argument
failed to umount /var/lib/docker/0.0/containers/07a3520691077e1e6c281077acd98e02596e8ac860ed1701a338284db3bc1de0/mqueue: invalid argument
ERRO[0046] Handler for POST /v1.22/containers/07a352069107/restart returned error: Cannot restart container 07a352069107: Cannot find child for /jolly_swanson
ERRO[0046] HTTP Error err=Cannot restart container 07a352069107: Cannot find child for /jolly_swanson statusCode=500
Error response from daemon: Cannot restart container 07a352069107: Cannot find child for /jolly_swanson
Error: failed to restart containers: [07a352069107]`

It seems like there is a problem with the name of the container. The original was named "jolly_swanson" but now, the docker ps shows "gigantic_booth" as its name. I tried to change the name in config.json (field "Name"), but it seems that it didn't make effect.

Thank you for your help!

estesp · 2016-02-02T20:25:17Z

@boucher I know you are working on the somewhat-thorny rebase now given the move to engine-api since the last rebase.

While you are handling that, could you add a requirement on kernel 4.3 to the experimental docs with a comment about the seccomp issues in kernels < 4.3? That should handle the prior discussion on seccomp-criu incompatibility that kept us from getting this done earlier :( I'm trying to make sure now that the 1.10 release is well underway that we finally get this into experimental and save you more rebase headaches ASAP.

boucher · 2016-02-02T20:31:41Z

I'm happy to add some information to the README. I was going to attempt an actual warning or error message at runtime as well, but I won't block on that.

crosbymichael · 2016-02-08T23:12:16Z

We are going to work together on this after the containerd integration. This will allow us to merge in the underlying implementation for C/R in docker during the containerd merge then work on the external API for users after.

southerngs · 2016-02-10T19:17:22Z

@crosbymichael Is there another issue that users who are interested in checkpoint/restore can track to follow the status of when the checkpoint/restore feature will be added to Docker? For the moment I'm using the branch associated with this issue for a project I'm working on. But once containerd reaches feature parity I'd be interested in switching to that because in the long term I'd like to be able to use a Docker release version rather than a fork. And I'd be happy to switch to testing an alpha version once it's available. Now that this issue is closed I'd like to track an active issue if there is one, and I imagine that there could be other users who feel the same way.

kfox1111 · 2016-02-10T19:26:31Z

+1

rncry · 2016-02-11T09:03:40Z

+1

On Wed, Feb 10, 2016 at 7:28 PM, kfox1111 [email protected] wrote:

+1

—
Reply to this email directly or view it on GitHub
#13602 (comment).

thaJeztah · 2016-02-13T14:34:58Z

Opened #20300 for tracking

GordonTheTurtle added the status/0-triage label May 29, 2015

boucher mentioned this pull request May 29, 2015

Add checkpoint/restore support to the execdriver #13442

Closed

calavera added status/1-design-review and removed status/0-triage labels May 29, 2015

cpuguy83 reviewed May 29, 2015
View reviewed changes

runcom reviewed May 30, 2015
View reviewed changes

boucher force-pushed the cr-combined branch from 951be58 to ae2c402 Compare June 1, 2015 04:03

boucher mentioned this pull request Jun 1, 2015

Unable to build 'cross' target #13635

Closed

boucher force-pushed the cr-combined branch 5 times, most recently from 665ab78 to f01ef2a Compare June 1, 2015 22:15

Update to latest CRIU version

e2bd7db

Signed-off-by: Marcos Lilljedahl <[email protected]>

southerngs reviewed Jan 21, 2016
View reviewed changes

willseward mentioned this pull request Feb 4, 2016

Move containers between hosts rancher/rancher#438

Closed

thaJeztah mentioned this pull request Feb 13, 2016

Re-implement checkpoint/restore after containerd integration #20300

Closed

Uh oh!

Conversation

boucher commented May 29, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jessfraz commented May 29, 2015

Uh oh!

cpuguy83 commented May 29, 2015

Uh oh!

boucher commented May 29, 2015

Uh oh!

cpuguy83 commented May 29, 2015

Uh oh!

boucher commented May 29, 2015

Uh oh!

boucher commented May 29, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

runcom commented May 30, 2015

Uh oh!

boucher commented May 30, 2015

Uh oh!

boucher commented Jun 1, 2015

Uh oh!

boucher commented Jun 2, 2015

Uh oh!

runcom commented Jun 2, 2015

Uh oh!

ankushagarwal commented Jun 2, 2015

Uh oh!

runcom commented Jun 2, 2015

Uh oh!

boucher commented Jun 2, 2015

Uh oh!

ankushagarwal commented Jun 2, 2015

Uh oh!

boucher commented Jun 3, 2015

Uh oh!

tianon commented Jun 3, 2015 via email

Uh oh!

jessfraz commented Dec 29, 2015

Uh oh!

boucher commented Dec 29, 2015

Uh oh!

jessfraz commented Dec 29, 2015

Uh oh!

xemul commented Dec 29, 2015

Uh oh!

boucher commented Dec 29, 2015

Uh oh!

xemul commented Dec 30, 2015

Uh oh!

boucher commented Jan 5, 2016

Uh oh!

jessfraz commented Jan 5, 2016

Uh oh!

marcosnils commented Jan 5, 2016

Uh oh!

boucher commented Jan 5, 2016

Uh oh!

jessfraz commented Jan 5, 2016

Uh oh!

jessfraz commented Jan 5, 2016

Uh oh!

SaiedKazemi commented Jan 5, 2016

Uh oh!

cpuguy83 commented Jan 12, 2016

Uh oh!

icecrime commented Jan 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!