Error when trying to run a job

[msalvaris]

I tried running a few different jobs and I get a message similar to the one below:

========================================================= ========================================================= ========================================================= logs from pod: c394fd78-1699-4d10-92ad-922f7bda2f37 ========================================================= ========================================================= ========================================================= failed to open log file "/var/log/pods/322513bd-524a-11e8-9e0d-000d3a0621a8/c394fd78-1699-4d10-92ad-922f7bda2f37_0.log": open /var/log/pods/322513bd-524a-11e8-9e0d-000d3a0621a8/c394fd78-1699-4d10-92ad-922f7bda2f37_0.log: no such file or directory ========================================================= end of logs from pod: c394fd78-1699-4d10-92ad-922f7bda2f37 =========================================================

I am using the alpha 1.5 release.
Is there a configuration I missed?

[hongzhili]

Can you check if the job is running in kubernetes:
./deploy.py kubectl get pods
And if you can get the job log from kubernetes:
./deploy.py kubectl logs [pod_name]

If you cannot get the pod logs from kubectl. Please check the docker configurations on your host machine. E.g. if folder /var/log/pods exists and is accessible by docker daemon.

[msalvaris]

Hongzhi Li (@hongzhili) Thanks for your help. Turned out to be an issue with the cuda drivers.