monitoring - Sidecar containers in Kubernetes Jobs?

Question

We use Kubernetes Jobs for a lot of batch computing here and I'd like to instrument each Job with a monitoring sidecar to update a centralized tracking system with the progress of a job.

The only problem is, I can't figure out what the semantics are (or are supposed to be) of multiple containers in a job.

I gave it a shot anyways (with an alpine sidecar that printed "hello" every 1 sec) and after my main task completed, the Jobs are considered Successful and the kubectl get pods in Kubernetes 1.2.0 shows:

NAME                                         READY     STATUS      RESTARTS   AGE
    job-69541b2b2c0189ba82529830fe6064bd-ddt2b   1/2       Completed   0          4m
    job-c53e78aee371403fe5d479ef69485a3d-4qtli   1/2       Completed   0          4m
    job-df9a48b2fc89c75d50b298a43ca2c8d3-9r0te   1/2       Completed   0          4m
    job-e98fb7df5e78fc3ccd5add85f8825471-eghtw   1/2       Completed   0          4m

And if I describe one of those pods

State:              Terminated
  Reason:           Completed
  Exit Code:        0
  Started:          Thu, 24 Mar 2016 11:59:19 -0700
  Finished:         Thu, 24 Mar 2016 11:59:21 -0700

Then GETing the yaml of the job shows information per container:

  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2016-03-24T18:59:29Z
      message: 'containers with unready status: [pod-template]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    containerStatuses:
    - containerID: docker://333709ca66462b0e41f42f297fa36261aa81fc099741e425b7192fa7ef733937
      image: luigi-reduce:0.2
      imageID: docker://sha256:5a5e15390ef8e89a450dac7f85a9821fb86a33b1b7daeab9f116be252424db70
      lastState: {}
      name: pod-template
      ready: false
      restartCount: 0
      state:
        terminated:
          containerID: docker://333709ca66462b0e41f42f297fa36261aa81fc099741e425b7192fa7ef733937
          exitCode: 0
          finishedAt: 2016-03-24T18:59:30Z
          reason: Completed
          startedAt: 2016-03-24T18:59:29Z
    - containerID: docker://3d2b51436e435e0b887af92c420d175fafbeb8441753e378eb77d009a38b7e1e
      image: alpine
      imageID: docker://sha256:70c557e50ed630deed07cbb0dc4d28aa0f2a485cf7af124cc48f06bce83f784b
      lastState: {}
      name: sidecar
      ready: true
      restartCount: 0
      state:
        running:
          startedAt: 2016-03-24T18:59:31Z
    hostIP: 10.2.113.74
    phase: Running

So it looks like my sidecar would need to watch the main process (how?) and exit gracefully once it detects it is alone in the pod? If this is correct, then are there best practices/patterns for this (should the sidecar exit with the return code of the main container? but how does it get that?)?

** Update ** After further experimentation, I've also discovered the following: If there are two containers in a pod, then it is not considered successful until all containers in the pod return with exit code 0.

Additionally, if restartPolicy: OnFailure is set on the pod spec, then any container in the pod that terminates with non-zero exit code will be restarted in the same pod (this could be useful for a monitoring sidecar to count the number of retries and delete the job after a certain number (to workaround no max-retries currently available in Kubernetes jobs)).

Answer 1

You can use the downward api to figure out your own podname from within the sidecar, and then retrieving your own pod from the apiserver to lookup exist status. Let me know how this goes.