nvidia - CUDA GPU selected by position, but how to set default to be something other than device 0?

Question

I've recently installed a second GPU (Tesla K40) on my machine at home and my searches have suggested that the first PCI slot becomes the default GPU chosen for CUDA jobs. A great link is explaining it can be found here:

Default GPU Assignment

My original GPU is a TITAN X, also CUDA enabled, but it's really best for single precision calculations and the Tesla better for double precision. My question for the group is whether there is a way to set up my default CUDA programming device to be the second one always? Obviously I can specify in the code each time which device to use, but I'm hoping that I can configure my set such that it will always default to using the Tesla card.

Or is the only way to open the box up and physically swap positions of the devices? Somehow that seems wrong to me....

Any advice or relevant links to follow up on would be greatly appreciated.

Answer 1

As you've already pointed out, the cuda runtime has its own heuristic for ordering GPUs and assigning devices indices to them.

The CUDA_VISIBLE_DEVICES environment variable will allow you to modify this ordering.

For example, suppose that in ordinary use, my display device is enumerated as device 0, and my preferred CUDA GPU is enumerated as device 1. Applications written without any usage of cudaSetDevice, for example, will default to using the device enumerated as 0. If I want to change this, under linux I could use something like:

CUDA_VISIBLE_DEVICES="1" ./my_app

to cause the cuda runtime to enumerate the device that would ordinarily be device 1 as device 0 for this application run (and the ordinary device 0 would be "hidden" from CUDA, in this case). You can make this "permanent" for the session simply by exporting that variable (e.g., bash):

export CUDA_VISIBLE_DEVICES="1"
./my_app

If I simply wanted to reverse the default CUDA runtime ordering, but still make both GPUs available to the application, I could do something like:

CUDA_VISIBLE_DEVICES="1,0" ./deviceQuery

There are other specification options, such as using GPU UUID identifiers (instead of device indices) as provided by nvidia-smi.

Refer to the documentation or this writeup as well.