Stop ignoring a failed cudaSetDevice in CudaAllocator

[shoumikhin]

Summary

CudaAllocator switches the current CUDA device before it allocates, frees or
copies, and switches back afterwards. The result of that switch was thrown away
in all three places:

(void)cudaSetDevice(index);

When the switch fails, for example because the requested device index does not
exist on this machine, the work still went ahead on whatever device happened to
be current. allocate then returned success with a pointer that lives on a
different device than the caller asked for. The caller stores the requested
index next to that pointer, so the pointer and its recorded device disagree
from then on, and the mistake surfaces much later as a wrong result or an
unrelated CUDA error somewhere else.

On a machine with one GPU:

auto r = CudaAllocator::instance().allocate(1024, /*index=*/1);
// before: r.ok() == true, and the pointer is really on device 0
// after:  r.ok() == false, r.error() == Error::MemoryAllocationFailed

What changes:

• allocate logs the CUDA error and returns Error::MemoryAllocationFailed
  instead of allocating on the wrong device.
• The host to device and device to host copy helpers log the CUDA error and
  return Error::Internal instead of copying against the wrong device.
• deallocate returns void, and cudaFree accepts a pointer from any device
  under unified addressing, so it logs the error and still frees the pointer
  rather than leaking it. The failure is no longer silent.

Only the switch to the requested device is checked. The switch back to the
previous device stays best effort, because there is nothing useful to do if
restoring fails.

Test plan

Three new tests in backends/cuda/runtime/test/test_cuda_allocator.cpp ask for
device index device_count, which is one past the last valid ordinal on any
machine, so the switch always fails:

• AllocateOnMissingDeviceFails
• CopyHostToDeviceOnMissingDeviceFails
• CopyDeviceToHostOnMissingDeviceFails

Built and ran the suite on one NVIDIA H100.

With this change:

[  PASSED  ] 9 tests.

Confirmed the tests really catch the bug. With the allocator reverted to the
old code and the same three tests:

[  FAILED  ] CudaAllocatorTest.AllocateOnMissingDeviceFails
[  FAILED  ] CudaAllocatorTest.CopyHostToDeviceOnMissingDeviceFails
[  FAILED  ] CudaAllocatorTest.CopyDeviceToHostOnMissingDeviceFails

The allocate failure shows the old behavior directly: the call reported success
for a device that does not exist.

The message the fixed build logs:

CudaAllocator::allocate: cudaSetDevice(1) failed: invalid device ordinal

The six tests already in that file still pass, so the change does not disturb
the normal single device path.

clang-format reports no changes needed on either touched file.

Not covered

deallocate only logs, so no test asserts on it. Everything here ran on a
single GPU machine, so the switch always failed with an invalid ordinal. A
device that exists but is unavailable, for example one in exclusive compute
mode, takes the same code path but was not exercised.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22093

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 3 Pending

As of commit 4b04163 with merge base 9a2d135 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.