This happens when the index tensor contains duplicate elements. If this is not allowed indexcopy should raise an exception when this happens and we should fix the test. Otherwise we need to fix the autograd op. Here's an example: cc
Why would you with repeated indices? Sounds for me like UB is a reasonable. We should check that in C
I ran into this in and fixed the test for now. I'm not sure how to interpret last comment: did you mean UB behavior is "unreasonable" rather than "a reasonable"? The last sentence reads that way.
I'd be ok with saying that it's UB in such case. There's no natural choice, and it can decrease performance.
Is this issue still worth working on? If adding the additional check can decrease performance, can we add another argument specifying that indexes contain duplicate elements?


As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10


Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).


This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?


We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.


Hi I think that in lines 28 and 30: if self.transA: a = a.transpose(2, 3) if self.transB: b = b.transpose(2, 3) should be: if self.transA: a = a.transpose(1, 2) if self.transB: b = b.transpose(1, 2) Indeed in that branch the tensor has 3 dimension and thus the code crashes. Maybe an indexing error when translating from lua ? Thank you very much,
Yes, you're right. Could you send a PR with a fix? Thanks!
Yes of course, just done, thank you very much.


as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.


Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


In we define a number of macros which are subsequently used in if statements: This is undefined. On clang 3.4.2, these warnings don't seem to actually get printed out () but on clang 4.0.0 they seem to always get emitted.


There is a possible memory leak in () This causes memory to blow up: When using () memory usage is ok There is no leak for unitary dimensions


A common issue - for both new and more seasoned users, including myself - is how to write nice device-agnostic code. There seems to be several potential solutions (using a global CUDA flag, always calling , using etc.), used by various people in various bits of code. Ideally, we should have "best practices" documented somewhere, like ; or at least some place where we can link to.
Alternative proposal - we provide a method that deals with the most common use cases (tensors, variables and networks). We could have something similar to Chainer's , something like a , etc. We're not going to be "write once, run everywhere", but we can probably catch most use cases. Pros: It makes it much simpler for people writing CPU/1 GPU code (presumably a majority of users). Depending on the approach it'll probably be fine for multiple GPUs too. Cons: An abstraction of this form isn't foolproof, but I can't immediately think of any major issues. Setting everything to CUDA by default is probably going to break things, but a wrapper, being more manual, can be used selectively enough.
I agree we should figure out a way to make writing device-agnostic code easier. I don't think fakecuda is a good idea, it's too magical and has global side effects. Here's a list of options I see, along with some comments. In general I'm not 100% happy with any of those, but I don't have any better ideas right now. Example: I think that sooner or later we should add arg to functions anyway, so it seems like a good enough extension. I think I'm fine with this option too, but I'm afraid you would either need a single global module captured inside functions, or use it everywhere inside. In general it can be simulated using the approach using partials (of course this is an inefficient implementation, but you get the idea): This is fine, except that you have to call it a lot and it makes the code super verbose. A similar alternative that people sometimes use now is this: This is really the best option, as it will give you a tensor of exactly the same type and on exactly the same device as the source. This is usually the desired behaviour, which is why we use it all over the place in the core library. On the other hand, it feels a bit unnatural, and is annoying when you do more complicated things, because you first construct the tensor and only then fill it with the data. and helps a lot with that
Thanks for more in-depth feedback on some options. For existing practices, highlighting , and are probably the way to go. In terms of what to do when there aren't existing tensors, what I see as the main target is the . Precisely, can we do and without using the if statement? The cast/dtype solution seems good for tensors, but anything we can do for modules?
I've put together some of this and other stuff I know into "best practices" documentation in I think we still want to think about future possibilities, but there's plenty that's worth noting down now for users.
It would indeed be amazing if PyTorch could make it simpler to write device agnostic code. I really like your first suggestion. Just a quick note from my (applied) perspective: maybe it would be possible to split up the and the into two attributes? First of all, I think this would make it easier to handle cases where tensors are on different gpus and second of all, it would allow to write code like this: ((5, 5), , ) # Returns on same device as x Don't know how practical this is though.
one thing I like about suggestion is that it makes it explicit and obvious when something is happening on cuda, rather than requiring users to know that non-negative device numbers refer to cuda. Maybe this is a bad idea, but what about also taking an optional kw in every function that takes a but to have both and non- dtypes? So passing would give you a cuda float tensor on the current GPU, passing would give you a CPU float, and passing would give you a cuda tensor on GPU 5. Obviously we'd need error checking to check things like non- devices don't have non-negative s specified (which is kind of silly because if the device is specified it tells you everything), but it does seem to make the common case of not specifying a device more clear.
Speaking only as an end-use with no particular insight into how difficult it would be to code: These suggestions make me feel like end users will end up having to think more about device semantics, not less. I'd like to see the API move away from having the tensor type imply the storage location entirely. Have a context manager that sets cpu vs. cuda, default device, etc. Something like: Then once this exists, class wraps its calls to the with an appropriate context, as do Modules. I think that this results in everything Just Working(tm).
I'm not convinced that making it easy for end users to avoid thinking about device semantics is the correct goal. The different devices have different performance characteristics, amount of memory/cache, incur data transfer costs/synchronizations, etc. My worry is that by papering over these issues, we make it really easy to write non-performant code and really difficult to figure out why. This discussion is similar to a discussion in the distributed systems community: is a good discussion of why trying to unify all interfaces might not be the best idea (there are admittedly additional complexities in distributed computing like partial failures). Anyway, that's why I like making it obvious when there are cuda storage / device transfers going on, because those are obvious places to look at performance.
I'm not an expert on any of this, but about the only two things that bug me about using is when I have to cast are I don't have a prototypical tensor at hand (e.g. in a module it feels unnatural to use so it might be neat to have a "canonical tensor" in modules that reacts to .cuda() and whose is exposed as or so. I want a different type is just a bit much. Other than that, I think using it is cool, I think that there is not much advantage in over But this is only from my limited experience.
Regarding the "canonical tensor" it's hard, because the module can have a lot of different parameters that have different types and are on different devices. I agree that chains are awful. I think (and similar for all other types) would be a convenient helper.
Oh yes, something like would be cool (or ). For the "canonical tensor", I imagine that, for basic applications, having something that "does the right thing" whether called before or after would be great. For more advanced applications, the user could create their own set of prototype tensors and just do , or so. But that is just pure imagination, as I don't have multiple GPUs. :slightlysmilingface:
there are several potential goals here, and they are somewhat conflicting. One is to make CPU/GPU code seamless for end users, with just one flag somewhere. On the other hand, that's likely to results in less-performant code and harder debugging. It seems like the conversation is going towards device-aware code, but making it a little less clunky in places. Sounds like is a good step forward, and was probably in the pipeline anyway?
a latecomer here: i agree that it's not a good idea to hide things behind the curtain from users. though, in many cases, it might be easier to to have a global behavior of ignoring, e.g., gpu (i.e., make .cuda() do nothing). could we at least give some kind of global switch allowing this behiavour?
Closing as 0.4 has plenty of options for this


It seems like autodiff of matrix terms of .mv() doesn't behave as I'd expect. But if we use .mm(), things work fine: python A = Variable((3,2), requiresgrad=True) x = Variable((2), requiresgrad=True) (A.mm(x[:,None])).sum().backward() print(A.grad) print(x.grad)
Hi, As a temporary fix, you can use . The problem is the behavior between the backward and the backward. The sum returns a strided gradient which is not supported by MKL/BLAS implementations of (used in the backward). not sure where this was introduced and how to fix this properly. Should the blas wrapper clone the input if it is strided? Or should it just raise an error (meaning that we need to add some s in some places in the code like the sum backward)?
It should clone the input. Still, this problem should have been caught. We must be missing some return code checks from BLAS
Ah apparently these functions have no return codes and the docs clearly state that vector strides can't be 0... We have to fix those bindings


A = (5, 4) B = (0, 9).view(3, 3) C = (0, 15).view(3, 5) idxs = torch.LongTensor([0, 2, 4]) A.indexadd(0, idxs, B) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [3] to have the same number of elements, but got 4, 4 and 3 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 A.indexadd(0, idxs, C) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [5] to have the same number of elements, but got 4, 4 and 5 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 So far so good. But if we use CUDA... A = (5, 4).cuda() B = (0, 9).view(3, 3).cuda() C = (0, 15).view(3, 5).cuda() idxs = torch.LongTensor([0, 2, 4]).cuda() A.indexadd(0, idxs, B) print(A) # 0 1 2 0 # 0 0 0 0 # 3 4 5 0 # 0 0 0 0 # 6 7 8 0 # [ of size 5x4 (GPU 0)] OK, this looks wrong... A.zero() A.indexadd(0, idxs, C) print(A) # 0 1 2 3 # 4 0 0 0 # 5 6 7 8 # 9 0 0 0 # 10 11 12 13 # [ of size 5x4 (GPU 0)] Now this looks definitely wrong. Increase C's dimension to something like (3, 500), and it overwrites other tensors or triggers asserts. Same thing happens with indexcopy_.
I'll take it if no one's looking at it yet.


Build error: According to , the should be modified as:
I the same problem when installing pytorch from source. You mean the file is wrong?
Although the PyYAML documentation site suggests that is possible, I could not perform it. That is a reason for the failing. This has been reported as well: I believe that the should have this instead: In fact when I did this: This indicates that the import didn't happen in PyYAML itself.
you guys are right, the case has to be modified to: . The original issue (my guess) is that your pyyaml had some compile error and the CLoader wasn't successfully installed.


Repro: Expect two small tensors of small integers to be printed. Instead, only the first case works. In the second case, you get a series of failed assertions starting with this: I wonder if it is related to
is this still an outstanding issue? I cannot reproduce this with an install from the master branch.
The size which causes failure doesn't seem deterministic. Using a pretty recent build (a few days old), I now have to increase the "9" to a larger number to make it fail. Sometimes it will fail at 10, sometimes at 11. Here's an updated case that seems to fail reliably at some size, for me:


If there was two libcudnn files ( and ) in the folder wouldn't it result in taking the older version ()? Maybe adding a warning or something to insure its the right lib?
I think and are quite possibly related. EDIT: also


If a test segfaults, it's hard to determine which test it is because the output looks like: ! I can work on this if we think this is a good idea.
Last time I asked about this, he didn't want to because it adds a lot of scrolling. I also did some work to move the CI tests to so that we can output XML status files, but (1) this is unlikely to help in case of segfault, and (2) you can't do this straightforwardly, because we run some test files multiple times with different environment variables, and the XML exporter isn't really set up to handle this case. For the record, I am personally OK with just slapping on the tests and calling it a day.
in the conbuild sounds good to me too
closed via


Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3


CC who was asking me about doc generation yesterday. Full log And CC
You probably know why this is happening, but just in case: fixes this. (as per: ) Matplotlib is trying to open a display window on what I assume is a screen-less server.


In PyTorch master:
Added clamp's output support in pre-template code.


I also found this issue when I tried to swap axes of a tensor. I googled an older doc in the source code. Hope it helps.


I think this may have been intentional, so I want to float this first before I apply the change. Historically, PyTorch did DEBUG builds with , because it prevents inlining, which means that you can inspect variables in all stack frames. However, in master, DEBUG mode appears to be compiled with optimizations, which means that any inlined functions are non-inspectable. The following patch will turn off optimizations when compiled in DEBUG mode. Is there any reason not to apply this?


yields irrelevant errors when passing wrong arguments First one: Second one: PyTorch or Caffe2: pytorch 0.4 How you installed PyTorch (conda, pip, source): pip Build command you used (if compiling from source): OS: ubuntu16.04 PyTorch version: 0.4 Python version: 3.6 CUDA/cuDNN version: GPU models and configuration: 1080 GCC version (if compiling from source): CMake version: Versions of any other relevant libraries:
Thank you for the report! I'll move them to the new check for same GPU (which includes reasonable CPU vs. GPU reporting).
In PyTorch master, the error message has already improved for the second case:
I am getting exactly the first error reported: and I don't understand what is the problem. Can someone help on this?
if you use PyTorch v0.4.1, you should get a better error message helping you on what's wrong.
Thanks for your reply I found out that the problem was the hidden state in my recurrent net, I did not use the .cuda()


Porting TH operators is essential for code simplicity and performance reasons. Porting guides and Q&A are available in umbrella issue: Feel free to add as a reviewer to get a prioritized review.


Kind of similar to Sometimes our Eigen build fails this way: It doesn't seem to reliably repro. cc
I got some sort of similar-ish error persistently on a test branch:
Not that persistently; a PR stacked on the failing one succeeded. So there is something nondeterministic going on here.
another instance: CI on
Looks like an NVCC bug. The workground is protecting eigen headers with Can we extract some basic code example and compile it against NVCC?
Another one in
Can you try with 10.2?
Mingbo, can you put general information on the issue you are seeing here in this issue?
latest word on this is from mingbo: "from what I can tell, it's a flaky one. might depend on which VS version. I hit the problem before and problem fixed (at least seem so) after upgrading VS 2019 latest version" and "the same error showed up once last week, and re-run the job finished without problem."
Hi. Did anyone have this issue persistently, ie that a rebuild did not "fix" it? We've had this issue in another (closed source) project. Exposing cuda to less includes (such as boost) seems to help but is not predictable.
It's never been persistent for us, but this is mostly from our Windows CI builds where we blow everything away and rebuild from scratch, which may help.
Yet another occurrence in the forum: It seems to be fixed by , which implies there may be a race condition in reading the include files or generating the intermediate source files.
I was able to reproduce the issue with a more verbose output with And then I inspected the variables carefully and found out that the VC env is activated twice. We should really avoid that.
You are SO COOL!!!
I tried to build from scratch several times with this PR and could not reproduce the issue anymore. Let's assume it's fixed.
Looks like it was not resolved.
Another occurrence: This time it seems to claim the error is around the usage of .
This sounds like a good old multiple-instances-of-nvcc-write-to-the-same-temp-file One can invoke to see all temporary filenames it creates. I guess on windows they are more likely to collide with each other than on linux. One solution to the problem would be to invoke nvcc sequentially. Another is to tweak environment (there must be something similar to TMPDIR on Windows, mustn't it?)
Yes, they are called TMP and TEMP.
Is there any example of the latter method on Unix systems?
I've written a utility . Hopefully it can be used to resolve this problem if that's the case.
New occurrence after the previous fix:
seems related to
If this is the case, maybe we could apply to fix that.
unfortunately, this introduces another thrust problem (e.g. ): D:/programs64/cuda101/includethrust/detail/complex/catrig.h(607): error: no instance of function template "thrust::atanh" matches the argument list argument types are: (double) with problematic include chain -thrust/complex.h -thrust/detail/complex/catrig.h (has cplusplus conditions) -thrust/detail/complex/c99math.h where "using ::atanh" is omitted for MSVC. Patching c99math.h (or even thrustdetailconfigcppdialect.h without using /Zc:_cplusplus) seems to work for me, but I'm not sure what's the best way to fix this within pytorch tree and whether c++ dialect is the only problem here.
Thanks for the investigation. BTW, could you please share with us how did you patch c99math.h?
Disregard that, still got same unreproducible C2993 error with /Zc:_cplusplus. Earlier, I had "build" directory reproducibly failing (at ), and hardcoding c++17 in thrustdetailconfigcppdialect.h fixed that. But looks like it is just a symptom of some inconsistence in environment/flags (across nvcc stages maybe?) just "using ::atanh"
Yeah, maybe. I am considering to add a wrapper to retry the build a few times.
I can now stably reproduce a related error by passing in .
By commenting out some part of the code, I find the code that causes the previous error. I don't know why it is executed in the device code.
Well, I figured out the reason. It seems that and are implicitly marked as and . This behavior can be disabled by adding .
Nice work!
Seems to be a nvcc bug
Thanks for reminding. I have commented there.
This is indeed an nvcc bug. There is no known workaround at the moment, but the next release of the CUDA toolkit will contain a fix. Ref thrust/thrust.
Met this problem:
Just retry a few times.
The CUDA TK 11.0 RC is now available: This issue has been fixed in that version of nvcc.
Thanks for the info. Actually, our release binaries rely on the cudatoolkit package in anaconda cloud. So we won't be able to publish the binaries with CUDA 11 now. I'm just curious whether that is maintained by Nvidia? Another question, any plan to backport the changes to exisiting CUDA versions (e.g. cuda 10.1, cuda 10.2)?
I don't have that information unfortunately -- it'd be best to contact the maintainer of that package and see when they plan to update. I'm not aware of any plans to backport this fix.
I have also obtained this error with CUDA 10.1/cuDNN 7.6.4 on Win Server 2019.
this is a cuda bug. Please see above discussion and upgrade to cuda 11.
I fixed the error by installing sscache for use with cuda 10.1 and cuda 10.2, as done here.
it's an intermittent error and the probability of occurrence varies depending on your system. So it's not really fixed, but you may have found a way to reduce the occurrence in your system, which may be sufficient in your case :)
Another instance, this time inside protobuf: Running with I found the syntax error is introduced by which is the cuda compiler front end. operates on the output from the compiler's pre-processor and that output barely even changed to trigger the build error, only line number metadata and some extra whitespace was changed in the pre-processed output. Here is the diff of the two inputs, one of which failed and the other didn't: And the corresponding change in the output: This failure reproduced reliably on the CI machine. Running nvcc manually multiple times, it always fails or passes given the same inputs. I think it's a deterministic failure, but is annoyingly sensitive to any changes in the input file whatsoever.
closing due to age
Wait, do we think this is fixed? It's not clear to me it is...
I think this was related to CUDA 10.2 builds on windows which we don't actually support anymore so it is probably safe to keep closed.


As it still using deprecated resource class, see See example of the failure in
Followup: we need to plan better --we knew it was gonna happen and none of us fixed it early enough. Action item: go through tasks about deprecation in sprint planning


When trying to install Pytorch on my Mac by following the instructions I get What I did: ` I also tried Both approaches gave the same error. System: xcode-select version 2395. Version: macOS Monterey 12.3.1 (21E258) MacBook Pro (16-inch, 2019) Processor: 2,6 GHz 6-Core Intel Core i7 memory: 16 GB 2667 MHz DDR4 PyTorch version: N/A Is debug build: N/A CUDA used to build PyTorch: N/A ROCM used to build PyTorch: N/A OS: macOS 12.3.1 (x8664) GCC version: Could not collect Clang version: 13.1.6 (clang-1316.0.21.2.3) CMake version: version 3.22.1 Libc version: N/A Python version: 3.9.12 (main, Apr 5 2022, 01:53:17) [Clang 12.0.0 ] (64-bit runtime) Python platform: macOS-10.16-x8664-i386-64bit Is CUDA available: N/A CUDA runtime version: Could not collect GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: N/A Versions of relevant libraries: [pip3] numpy==1.21.5 [conda] mkl 2022.0.0 hecd8cb5105 [conda] mkl-include 2022.0.0 hecd8cb5105 [conda] numpy 1.21.5 py39h9c3cb841 [conda] numpy-base 1.21.5 py39he782bc11 cc
I am running the same environment and get the same issue. Any insight would be very appreciated
Update: In another repo I get the same error when trying to link to pytorch etc. There I made a minimal case and managed to build when I removed linking to . I can see that we have in the script. Maybe that is the cause of the error?
Looks like they are built with the correct architecture.
More progress, from local minimal case: fails. is just a hello world program. If in is removed, then it builds.
The same story with PyTorch 1.10.0. The error appears when I'm trying to build with Apple clang 13.1.6 (Xcode Command Line Tools 13.3). But all works correctly if I build it with Apple clang 13.0 (Xcode Command Line Tools 13.2.1)
Nice, worked for me as well. Is this a bug somewhere or what is the exact problem? I drawback is that XCode needs to be up to date with new iOS versions.
Which version of Apple Clang worked? 13.0.0 or 13.0.1? Are you on Monterey 12.4?
Filed an issue: cc:
This issue has been fixed in PeachPy a while back by but pinned version of PeachPy that PyTorch is using has not been updated in a very long time


causes a host-device synchronization on CUDA. This is because it must first calculate the count of nonzero elements, and then copy this count to host. There are cases where I can easily calculate the count myself (from existing host data), or already know the count, which would make this unnecessary. For some given of shape : Instead of: Do this trickery: This avoids the host-device synchronization, but on the other hand, is more complicated, and a bit slower. I'm not sure if there are any other ways? Btw, I'm using to measure the speed. But actually I'm not sure if this really properly covers the problem of host-device synchronization with . Does it? E.g. when comparing this trickery code to , I just see that is slower. No response_
dupe of i think
Ah yes, I think you are right.