There is a variable called CUDA_TOOLKIT_DIR in the cmake configuration.
This variable is initialized by cmake with the location of the CUDA toolkit, by default /Developer/NVIDIA/CUDA-6.x/.
However, this settings yields error messages in the link step, claiming that ld can’t use the CUDA directory.
The thing is, the content of this CUDA_TOOLKIT_VARIABLE variable seems to passed directly to ld.
Thus, it needs to point to the actual location where ld will find the libraries to link OpenCV with, i.e., /Developer/NVIDIA/CUDA-6.x/lib (note the extra lib at the end).
Voilà !
Wow, that was easy, but took me a bit of time to figure it out. Hope this will help someone else!