Question: def batch_norm(input, running_mean, running_var, weight=None, bias=None, training=False, momentum=0.1, eps=1e-5): <ins> size = list(input.size()) if reduce(mul, size[2:], size[0]) == 1: raise ValueError('Expected more than 1 value per channel, got input size {}'.format(size)) </ins> f = torch._C._functions.BatchNorm(running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled) return f(input, weight, bias)

Right now it makes everything .
To be clear, for a tensor, the problem is only when all of N, H, and W are 1. (So BatchNorm2d on batch size of 1 is OK as long as you don't have a 1x1 image). What's the desired behavior? The only reasonable behavior I can think of is: Raise an exception when the dimensions of which you are normalizing are one or Output zero (+ optional affine transform) I'm not sure the outputing zero is a good idea. I can't think of a case where that's what you want.


Question: sum  ); } <del> THArgCheckWithCleanup((sum > 0), THCleanup(THDoubleTensor_free(cum_dist);), 2, </del> <ins> THArgCheckWithCleanup((sum > 0), THCleanup(THDoubleTensor_free(cum_dist); if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, </ins> "invalid multinomial distribution (sum of probabilities <= 0)"); /* normalize cumulative probability distribution so that last val is 1 i.e. doesn't assume original prob_dist row sums to one */

I was reading the documentation of method in the . I tried this in IPython environment. Of course, error occurs because 5 is larger than the size of weights and I didn't set . The error message is like this: But after this, the dimensions of weights changed. If I run as following without runtime exception, then the dimensions of weights do not change. This is caused by not cleaning up after exception raised in this . Maybe the solution is adding cleaning up code in the .
The problem emerged in two other situations as well, here are the examples: I'm sending a pull request with the fix. Thanks for reporting the problem. :)


Question: assert data_tensor.size(0) == target_tensor.size(0) self.data_tensor = data_tensor self.target_tensor = target_tensor <del> if self.data_tensor.dim() == 1: self.data_tensor = self.data_tensor.view(-1, 1) if self.target_tensor.dim() == 1: self.target_tensor = self.target_tensor.view(-1, 1) </del> def __getitem__(self, index): return self.data_tensor[index], self.target_tensor[index]

TensorDataset adds casts 1D target tensor to 2D which is not acceptable by NLL loss layer.
Fixed in .


Question: def grid_sampler(input, grid, padding_mode): <del> if cudnn.is_acceptable(input.data) and padding_mode == 'zeros' and input.dim() == 4: </del> <ins> if (cudnn.is_acceptable(input.data) and padding_mode == 'zeros' and input.dim() == 4 and input.size(1) <= 1024):  # as of cudnn 7102, will not work for larger than 1024 </ins> return torch.cudnn_grid_sampler(input, grid) else: return GridSampler.apply(input, grid, padding_mode)

PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.0.176 OS: Ubuntu 16.04.3 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 CMake version: version 3.9.4 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 9.0.176 GPU models and configuration: GPU 0: TITAN X (Pascal) GPU 1: TITAN X (Pascal) Nvidia driver version: 384.98 cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux-gnu/libcudnnstaticv7.a /usr/local/lib/python2.7/dist- /usr/local/lib/python3.5/dist- Versions of relevant libraries: [pip3] numpy (1.14.1) [pip3] numpydoc (0.6.0) [pip3] torch (0.4.0) [pip3] torchvision (0.2.0) [conda] cuda90 1.0 h6433d270 pytorch [conda] magma-cuda90 2.3.0 1 soumith [conda] pytorch 0.4.0 py36cuda9.0.176cudnn7.1.21 [cuda90] pytorch [conda] torchvision 0.2.1 py361 pytorch
() returns 7102
Possibly related:
I'm not sure why, but I can't reproduce this on master. I can reproduce on 0.4 though.
Never mind, this error happens when CUDNN is updated regardless of pytorch version.


Question: #include <tuple> #include <TH/THMath.h> <del> #include "THP.h" #include "copy_utils.h" #include "DynamicTypes.h" </del> <ins> #include "torch/csrc/THP.h" #include "torch/csrc/copy_utils.h" #include "torch/csrc/DynamicTypes.h" </ins> <del> #include "generic/Tensor.cpp" #include <TH/THGenerateAllTypes.h> #include "generic/Tensor.cpp" #include <TH/THGenerateHalfType.h> </del> <ins> //generic_include TH torch/csrc/generic/Tensor.cpp </ins>

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: if [[ $WITH_CUDA -eq 1 ]]; then GLOO_FLAGS="-DUSE_CUDA=1 -DNCCL_ROOT_DIR=$INSTALL_DIR" fi <ins> CWRAP_FILES=" $BASE_DIR/torch/csrc/Declarations.cwrap; $BASE_DIR/torch/lib/ATen/Local.cwrap; $BASE_DIR/torch/lib/THNN/generic/THNN.h; $BASE_DIR/torch/lib/THCUNN/generic/THCUNN.h" </ins> # Used to build an individual library, e.g. build TH function build() {

There are previous issues addressing this, but I want to write this up with a plan in a new issue. We should make differentiable instead of having a wrapper class. The Variable class adds conceptual overhead without sufficient benefit. It makes it harder for new users to adopt PyTorch by adding a second API which is similar but not exactly the same as Tensor. There are few prerequisites: on s need to be at least as fast as operations on () need scalar support at the Python level () The constructor will be deprecated and become a no-op. The accessor will be deprecated and return a view that has requiresgrad=False. The constructors will take an optional keyword argument There will be a number of internal changes, but we shouldn't break most existing code: performance reasons, we will no longer build up the graph of operations by default when unless the operation uses stochastic variables. We'll use a context manager or global variable to enable graph creation. new differentiable Tensor will need support all the operations currently on Tensor, including in-place and operations. -place operations on leaf Variables which requiregrad are not currently supported. In the new API, the tensor's value will be updated and it will remain a leaf node. (This is important for optimizers, which update leaf Variables in-place). There are probably a few other subtle differences between Tensor and Variable APIs. I'll update this issue as we address them. See also: 23/738 # Greg will be fixing the tests, Tensor vs. Variable requiresgrad 2/1837 # Just testbroadcast from TestTorch OK OK # error at top level OK OK OK OK OK OK OK OK 10/120 # missing sspaddmm, dsmm/mm, addmm 1/257 TestTorch.testbroadcast 2OK
Properly handling in-place might not be as easy as it seems. Sometimes you actually need to move the Variable in the graph. Consider this case:
One of the cases of differences between variable and tensor is computation of var/std:
IMO Variable gives user a clear difference between graph node and data. .data attribute and Variable constructor provide convenient and explicit ways to take control of the graph (or graphs).
This is done. (Still some clean-up to do)
Is the conda PyTorch out of date? I'm getting the following error: .
Yes, we haven't released 0.4, which has the Variable Tensor merge, yet


Question: # Don't store the actual traceback to avoid memory cycle _queued_calls.append((callable, traceback.format_stack())) <ins> _lazy_call(_check_capability) </ins> class DeferredCudaCallError(Exception): pass

There's quite a few user-reports of installing the CUDA 7.5 binary of pytorch and then running on GTX 1080 / 1070, and there will be reports of installing 7.5 / 8.0 binaries and running on V100. Raising a user-warning in such situations and letting them know will fix the problem.
The version of CUDA used to compile the binary should be available thanks to this PR:
I think is a more robust way of getting the CUDA used to compile PyTorch, as currently if there are multiple CUDA installed in the machine, the heuristics used in might not match those used by CMake FindCUDA. I think that we should modify the to return the cuda version used during compilation. I have the impression that it will not require changing much because doesn't require cuda to execute. What do you think?


Question: __global__ void renormRowsL1(T* dist, long rows, long cols) { extern __shared__  unsigned char my_smem[]; T *smem = reinterpret_cast<T *>(my_smem); <ins> T zero = ScalarConvert<int, T>::to(0); T val; </ins> for (int64_t row = blockIdx.x; row < rows; row += gridDim.x) { T sum = ScalarConvert<int, T>::to(0); for (int64_t col = threadIdx.x; col < cols; col += blockDim.x) { <del> sum = THCNumerics<T>::add(sum, dist[row * cols + col]); </del> <ins> val = dist[row * cols + col]; assert(THCNumerics<T>::ge(val, zero)); sum = THCNumerics<T>::add(sum, val); </ins> } <del> sum = reduceBlock(smem, blockDim.x, sum, ReduceAdd<T, T>(), ScalarConvert<int, T>::to(0)); </del> <ins> sum = reduceBlock(smem, blockDim.x, sum, ReduceAdd<T, T>(), zero); </ins> if (threadIdx.x == 0) { <ins> assert(THCNumerics<T>::gt(sum, zero)); </ins> smem[0] = sum; } __syncthreads();

Looking at the values returned from tn.multinomial(1) and counting them, it seems like it returns the values of the multinomial distribution taken from t, i.e. that it has ignored the signs of the values. However, it seem like always returns a tensor filled with zeros.
looking into this


Question: if declaration['inplace']: env['return_value'] = 'self' <del> env['return_name'] = 'self' </del> env['result'] = 'static_cast<Variable&>(self)' <ins> env['trace_outputs'] = '{ self }' </ins> else: env['return_value'] = '{}(std::move(ret))'.format(declaration['return_type']) <del> env['return_name'] = 'ret' </del> env['result'] = 'std::get<0>(ret)' if len(declaration['returns']) > 1 else 'ret' <ins> if len(declaration['returns']) > 1: # NB: This won't work if we get heterogenous outputs outs = ['std::get<{}>(ret)'.format(i) for i, v in enumerate(declaration['returns']) if v['type'] == 'Tensor'] else: outs = ['ret'] env['trace_outputs'] = CodeTemplate("{ ${outs} }").substitute(outs=outs) </ins> if any(arg['simple_type'] in {'Generator', 'Storage'} for arg in arguments): env['record_trace'] = []

when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit


Question: } cudaError_t CudaStorageWeakRefAllocator::free(void* ptr) { <del> PyGILState_STATE gstate = PyGILState_Ensure(); PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; PyGILState_Release(gstate); </del> <ins> { AutoGIL gil; PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; } </ins> cudaError_t err = allocator->free(allocatorContext, ptr); delete this; return err;

When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1


Question: thisdir = path.dirname(__file__) libpaths = ['', path.join(thisdir, '../../lib')] if sys.platform.startswith('linux'): <del> libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3', 'libcudnn.so.5.0.5'] </del> <ins> libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3', 'libcudnn.so.5.0.5', 'libcudnn.so.5.1.10'] </ins> elif sys.platform == 'darwin': libnames = ['libcudnn.5.dylib'] else:

Kind of similar to Sometimes our Eigen build fails this way: It doesn't seem to reliably repro. cc
I got some sort of similar-ish error persistently on a test branch:
Not that persistently; a PR stacked on the failing one succeeded. So there is something nondeterministic going on here.
another instance: CI on
Looks like an NVCC bug. The workground is protecting eigen headers with Can we extract some basic code example and compile it against NVCC?
Another one in
Can you try with 10.2?
Mingbo, can you put general information on the issue you are seeing here in this issue?
latest word on this is from mingbo: "from what I can tell, it's a flaky one. might depend on which VS version. I hit the problem before and problem fixed (at least seem so) after upgrading VS 2019 latest version" and "the same error showed up once last week, and re-run the job finished without problem."
Hi. Did anyone have this issue persistently, ie that a rebuild did not "fix" it? We've had this issue in another (closed source) project. Exposing cuda to less includes (such as boost) seems to help but is not predictable.
It's never been persistent for us, but this is mostly from our Windows CI builds where we blow everything away and rebuild from scratch, which may help.
Yet another occurrence in the forum: It seems to be fixed by , which implies there may be a race condition in reading the include files or generating the intermediate source files.
I was able to reproduce the issue with a more verbose output with And then I inspected the variables carefully and found out that the VC env is activated twice. We should really avoid that.
You are SO COOL!!!
I tried to build from scratch several times with this PR and could not reproduce the issue anymore. Let's assume it's fixed.
Looks like it was not resolved.
Another occurrence: This time it seems to claim the error is around the usage of .
This sounds like a good old multiple-instances-of-nvcc-write-to-the-same-temp-file One can invoke to see all temporary filenames it creates. I guess on windows they are more likely to collide with each other than on linux. One solution to the problem would be to invoke nvcc sequentially.

Another is to tweak environment (there must be something similar to TMPDIR on Windows, mustn't it?)
Yes, they are called TMP and TEMP.
Is there any example of the latter method on Unix systems?
I've written a utility . Hopefully it can be used to resolve this problem if that's the case.
New occurrence after the previous fix:
seems related to
If this is the case, maybe we could apply to fix that.
unfortunately, this introduces another thrust problem (e.g. ): D:/programs64/cuda101/includethrust/detail/complex/catrig.h(607): error: no instance of function template "thrust::atanh" matches the argument list argument types are: (double) with problematic include chain -thrust/complex.h -thrust/detail/complex/catrig.h (has cplusplus conditions) -thrust/detail/complex/c99math.h where "using ::atanh" is omitted for MSVC. Patching c99math.h (or even thrustdetailconfigcppdialect.h without using /Zc:_cplusplus) seems to work for me, but I'm not sure what's the best way to fix this within pytorch tree and whether c++ dialect is the only problem here.
Thanks for the investigation. BTW, could you please share with us how did you patch c99math.h?
Disregard that, still got same unreproducible C2993 error with /Zc:_cplusplus. Earlier, I had "build" directory reproducibly failing (at ), and hardcoding c++17 in thrustdetailconfigcppdialect.h fixed that. But looks like it is just a symptom of some inconsistence in environment/flags (across nvcc stages maybe?) just "using ::atanh"
Yeah, maybe. I am considering to add a wrapper to retry the build a few times.
I can now stably reproduce a related error by passing in .
By commenting out some part of the code, I find the code that causes the previous error. I don't know why it is executed in the device code.
Well, I figured out the reason. It seems that and are implicitly marked as and . This behavior can be disabled by adding .
Nice work!
Seems to be a nvcc bug
Thanks for reminding. I have commented there.
This is indeed an nvcc bug.

There is no known workaround at the moment, but the next release of the CUDA toolkit will contain a fix. Ref thrust/thrust.
Met this problem:
Just retry a few times.
The CUDA TK 11.0 RC is now available: This issue has been fixed in that version of nvcc.
Thanks for the info. Actually, our release binaries rely on the cudatoolkit package in anaconda cloud. So we won't be able to publish the binaries with CUDA 11 now. I'm just curious whether that is maintained by Nvidia? Another question, any plan to backport the changes to exisiting CUDA versions (e.g. cuda 10.1, cuda 10.2)?
I don't have that information unfortunately -- it'd be best to contact the maintainer of that package and see when they plan to update. I'm not aware of any plans to backport this fix.
I have also obtained this error with CUDA 10.1/cuDNN 7.6.4 on Win Server 2019.
this is a cuda bug. Please see above discussion and upgrade to cuda 11.
I fixed the error by installing sscache for use with cuda 10.1 and cuda 10.2, as done here.
it's an intermittent error and the probability of occurrence varies depending on your system. So it's not really fixed, but you may have found a way to reduce the occurrence in your system, which may be sufficient in your case :)
Another instance, this time inside protobuf: Running with I found the syntax error is introduced by which is the cuda compiler front end. operates on the output from the compiler's pre-processor and that output barely even changed to trigger the build error, only line number metadata and some extra whitespace was changed in the pre-processed output. Here is the diff of the two inputs, one of which failed and the other didn't: And the corresponding change in the output: This failure reproduced reliably on the CI machine. Running nvcc manually multiple times, it always fails or passes given the same inputs. I think it's a deterministic failure, but is annoyingly sensitive to any changes in the input file whatsoever.
closing due to age
Wait, do we think this is fixed?

It's not clear to me it is...
I think this was related to CUDA 10.2 builds on windows which we don't actually support anymore so it is probably safe to keep closed.


Question: # This will segfault if things have been erroneously released out.backward(torch.randn(out.size())) <ins> def test_norm_subgradient(self): def run_test(input_size, norm_deg): input = Variable(torch.zeros(*input_size), requires_grad=True) out = input.norm(norm_deg) out.backward() self.assertEqual(input.grad.data.abs().sum(), 0) run_test((10,), 2) run_test((10, 10), 2) run_test((10,), 3) </ins> def index_variable(shape, max_indices): if not isinstance(shape, tuple):

If a norm is zero, its gradient returns nan: Obviously just happening because the gradient divides by the norm, but the (sub)gradient here should probably be zero, or at least not nan, since that will propagate to make all updates nan. Probably low priority, as it's not going to be an issue in 99% of cases, but we're doing a few things with (exact) line searches where this caused a nan to appear, breaking everything downstream.
I'm encountering exactly the same issue! Spent hours on debugging, just to find PyTorch has a bug on such basic thing.
+1 just found this bug too
+1 for this bug. Temporarily changing my code to something like the following for the sake of debugging. x = Variable((1), requires_grad=True) y = x + 1e-16 y.norm().backward() print x.grad
The thing is that in the 2 norm, there is a square root, which has a gradient of at 0. The gradient gives you because you then multiply 0 and an infinity during the backward pass.
For a scalar, norm 2 is basically abs. But x.abs().backward() gives you 0 gradient. In this sense, it's not coherent.
I found this error, too
Alban fixed this behavior in
Hi, the norm function can give use the 0 gradients now. However, the following code still has the nan gradient problem
Ho The square root has no gradient at 0. This is expected behavior.
Hi, but the sub-gradient of the square root should be zero? Also, y = ( x * x ) should equal to x.norm(), why they have different gradient ( 0 and nan )?
I think was right. The left-side derivative of sqrt(x) at x=0 is undefined, so it doesn't even have a subgradient at x=0.
square root has no subgradient at 0. You could define a gradient by continuity but then it would be ... Given that pytorch is using autograd, and (equivalent to your ) are completely different: The first one is a single function that is convex and defined on R, it has a subgradient of 0 at 0. The second one is composed of two function, the first function is the square function which is differentiable and outputs values in .

The second function is the square root that is not convex and even though it is defined on , it is only differentiable on and it's gradient in 0 in undefined. Given that, even though and will return the same value, their gradients may differ at points where it is not differentiable, this is because automatic differentiation looks at each step of the computation one by one and even though in some cases a subgradient exist (because we look at multiple operations as a single function), it is not always the case and the gradient remains undefined.
I think math is math. Any root's gradient at zero is either inf or undefined. This issue shall be handled by the users' themselves by adding a small value(as did), but an error(warning) message may be helpful since it is pretty hard to debug. Say: Infinite/Undefined gradient is detected at XfunctionX at line Y. Exit.
Agree, norm is not differentiable at 0 , the bandaid that Alban put there in is wrong (even in the limit sense the gradient at 0 should be 1 not 0), but it should not have been there at all. Norm is norm, if someone want to add epsilons to their norms (like batchrnorm, e.g.) they are welcome to do so in the user code. What would numpy do?
I've also run into a number of problems related to the change introduced in . Is there a reason the subgradient is set to 0, rather than the 1? (The limit as norm-0?) As a minimal example: Produces
Well any value between [-1, 1] is a valid subgradient for the 2-norm. More genereally, any vector in the 1 ball for the dual norm is a valid subgradient. This means that 0 is always going to be a subgradient, while 1 will not be for all p. Anyway, the theory says that any of them could be taken and subgradient descent will work. I'm sure that depending on the application, one will be better than the other. For example, the relu function will also give a 0 subgradient at 0, you could have given 1. The main point here was to remove nans that make your network give nan for everything which is not convenient.


Question: <ins> #include "torch/csrc/utils/object_ptr.h" #include <Python.h> template<> void THPPointer<PyObject>::free() { if (ptr) Py_DECREF(ptr); } template class THPPointer<PyObject>; </ins>

Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc


Question: def forward(input, weight, hidden): if batch_first: <del> input.transpose(0, 1) </del> <ins> input = input.transpose(0, 1) </ins> nexth, output = func(input, hidden, weight) if batch_first: <del> output.transpose(0, 1) </del> <ins> output = output.transpose(0, 1) </ins> return output, nexth

The last line here fails on CPU or when CUDNN is otherwise unavailable: This is because accidentally assumes 's in-place semantics rather than the functional semantics of ( gets it right): I can push a PR that fixes this, or one of the devs can put it in the next bugfix PR:
If you have a ready solution for your issues feel free to send the PR straight away. Contributions are welcome!
Thanks


Question: def fn_base(optimizer, weight, bias): optimizer.zero_grad() <del> loss = (weight.mv(input) + bias).pow(2).sum() </del> <ins> i = input_cuda if weight.is_cuda else input loss = (weight.mv(i) + bias).pow(2).sum() </ins> loss.backward() return loss

Hi, I encountered this bug: The code skeleton is like: It seems the loaded are , and I've tried a workaround to move to , but it still has the same bug.
Could you provide a full script to reproduce the problem?
maybe you can try like this, () (beta1).add_(1 - beta1, ())
Sorry, I missed the reply email. I am afraid that I am unable to provide a reproducer now. It is a work I am doing for the OpenNMT-py , trying to use for doing lr update. And I encoutered this problem when testing the case. So I factored out the code skeleton about this problem above. I've tried several methods, including tricks like what suggests, but it still happens. Per my analysis, it is because the previous training was done on gpu, so when saving the , the stored states(tensors) are of version. During resuming, when we load the saved optimizer, loads this version to cpu(the model() can be moved to gpu easily, but seems lacking this ability?) , so this problem emerges.
Try moving optimizer state to the GPU memory manually after loading it from the checkpoint. I agree that having an method for this operation would be nice.
thanks. My work was suspended due to other problems, when resumed, I will try your method.
I'm afraid solution won't work. It will make the error go away, but your optimizer will no longer be training the model. You should recreate optimizers after casting modules to a different type or device, and you can use to restore the state from a previous copy. This currently doesn't work, but we should fix it (by copying from the data from the state dict, instead of using the tensors directly - this allows for cross-device or cross-type updates).
, yep, your method is what I currently use, it works. But I will wait for upstream to fix this problem though. Thanks for your great works!
Ah, my bad. I forgot to update the line where the optimizer is recreated. But otherwise, the following should do the job, right?
ah, right. That should work
Except that you should use instead of
I had a similar problem. When I save the optimizer state from a GPU other than GPU 0 and then load the state it still loads everything to GPU 0.

Specifying in didn't work either. 's solution solves this though.
Hi guys, I have a very similar problem as the one in this thread, here's my code: And then once I resume, I got KeyErrors on my optimizer: Do you guys know how to fix this issue? BTW, I have 8 GPUs used, I'm guessing if this issue was because of that?
were you able to solve your issue with multiple-GPUs?
Thanks for asking. Yeah, I'm able to, turns out the issue is because I used an early version of PyTorch, after I updated the version, it works like a charm!
Just a comment, this problem is caused by makes all state tensor are moved into GPU0 so by moving the state of an optimizer to specific GPU will fix this problem.
Hi how do you know the specific GPU location of different tensors in advance?
Would a feature where all () calls always makes use of an automatically generated CPU version be feasible ? And then at resume the () would make use of the "current" device being used (or any better strategy). At the moment it seems we need a lot of boilerplater code to ensure saving and loading are consistents across devices for models/optimizer/scheduler/etc.
I have met similar problem, I recreated Adam optimizer without () after reloading model, () and DataParallel(model) according to solution.
thanks, it work!
Hi, as you say that: every time moving model to other device, we should build optimizer, but, if we move the model to other device and move back, should we build the optimizer again? here is an example code: does optimizer run as expected? also, if doing , should we rebuild optimizer ?
Is there a problem if you switch the order to something like this? Meaning moving the model to 'cuda' but only loading it's state dict from checkpoint after loading the optimizer's state dict first?
The problem can be concluded that the optimizer's state will be loaded to the device as same as the model. You must load the model to GPU at first, and then load the optimizer's state. So that both the model and the optimizer's state are loaded in GPU.

Instead of moving optimizer to cuda after loading it in cpu, you could load the checkpoint directly in cuda:
I've independently rediscovered that this works :) Should read until the end of the thread next time
I find my codes still have the problem. I tried my best to range the modules as the examples shown in the above. can anyone give me some hints? I also tried this
For me, doesn't solve the issue, but the one by works. Any idea why? I create the optimizer, load the state (with map location to CUDA), pass to the train loop, where the mode is pushed to device (though I assume it's not needed if the loaded model is pushed to cuda by maplocation). Specifically, I would get stuck at call, throwing the device mismatch error.
Still it doesn't work for me. Could you please help me?


Question: self._data_ptrs = [] return <ins> # If any parameters alias, we fall back to the slower, copying code path. This is # a sufficient check, because overlapping parameter buffers that don't completely # alias would break the assumptions of the uniqueness check in # Module.named_parameters(). unique_data_ptrs = set(p.data_ptr() for l in self.all_weights for p in l) if len(unique_data_ptrs) != sum(len(l) for l in self.all_weights): self._data_ptrs = [] return </ins> with torch.cuda.device_of(any_param): # This is quite ugly, but it allows us to reuse the cuDNN code without larger # modifications. It's really a low-level API that doesn't belong in here, but

While the fast path for cuDNN RNNs introduced in includes a check intended to fall back to copying weights if weights are moved or aliased, this only works if the aliasing happens after the weights are first compacted. But the most natural place to tie weights is inside a module's method, which runs before , the typical time when is first invoked. Here's an MWE. The three runs (CPU, weight tying before , and weight tying after ) should all give the same results, and they do before , but they have not since:


Question: (4.53, 3.83, -6.64, 2.06, -2.47, 4.70))).t() b = torch.Tensor(((8.58, 8.26, 8.48, -5.28, 5.72, 8.93), (9.35, -4.43, -0.70, -0.26, -7.36, -2.52))).t() <del> _test(a, b, expectedNorm) </del> <ins> _test_overdetermined(a, b, expectedNorm) </ins> # test underdetermined expectedNorm = 0

b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.


Question: ensure_attr('ClassNLLCriterion', 'weights') ensure_attr('ParallelCriterion', 'repeatTarget') ensure_attr('MultiMarginCriterion', 'weights') <ins> ensure_attr('SpatialConvolution', 'bias', 'finput', 'fgradInput', 'gradWeight', 'gradBias') </ins> attr_map('ReLU', {'val': 'value'}) attr_map('Threshold', {'val': 'value'}) attr_map('Unsqueeze', {'pos': 'dim'})

Subclassing torch.Tensor with fails due to a on the latest nightly builds, whereas it previously worked without issue. Run this snippet of code: And you'll recieve an error message like this: Changing the to results in the exact same error message. There should be no error message as it should be working like it was previously in Torch 1.8.1 and below. I think that the issue started on Torch version: , and I can confirm that it still exists as of , . The issue started on the nightly builds at least 2 days ago, if that helps resolve things. It's been causing some of Captum's CircleCI tests to fail, and the tests show the first occurrence of the error on . cc
This is due to either or
I think the problem is I'm not supposed to delete because that means the super() call doesn't work. Instead, I should just verify that the passed in class is a subclass, then this should work.


Question: ) @unittest.skipIf(BACKEND != 'gloo', "Only Gloo backend supports CUDA allReduce") <ins> @skip_if_no_cuda_distributed </ins> def test_all_reduce_sum_cuda(self): group, group_id, rank = self._init_global_test() self._test_all_reduce_helper(

is causing a crash when we try to build pytorch. Crash looks like this: (all times in US/West) Our 7AM, 8AM builds and earlier were all fine but 9AM, 10AM, 11AM builds all crashed with that same traceback. The new import was in which I think went in between 8AM and 9AM today See command below: See here if needed for : <!-- A clear and concise description of what you expected to happen. --Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): building from master OS (e.g., Linux): linux How you installed PyTorch (, , source): source Build command you used (if compiling from source): see above under 'reproduce' Python version: 3.6
maybe easier to read for build command used:
please try in the CircleCi environment?
in our build script seems to have fixed this. Thanks !


Question: # test is no more than 4 higher than the 10th available at the # start. This attempts to catch file descriptor leaks, but allows # one-off initialization that may use up a file descriptor <del> available_fds = self._get_next_fds(10) self.test_case.assertLessEqual( available_fds[-1] - self.next_fds[-1], 5) </del> <ins> # TODO: Disabled because this check is too flaky # available_fds = self._get_next_fds(10) # self.test_case.assertLessEqual( #     available_fds[-1] - self.next_fds[-1], 5) </ins> self.test_case.assertFalse(self.has_shm_files()) return False

causes a host-device synchronization on CUDA. This is because it must first calculate the count of nonzero elements, and then copy this count to host. There are cases where I can easily calculate the count myself (from existing host data), or already know the count, which would make this unnecessary. For some given of shape : Instead of: Do this trickery: This avoids the host-device synchronization, but on the other hand, is more complicated, and a bit slower. I'm not sure if there are any other ways? Btw, I'm using to measure the speed. But actually I'm not sure if this really properly covers the problem of host-device synchronization with . Does it? E.g. when comparing this trickery code to , I just see that is slower. No response_
dupe of i think
Ah yes, I think you are right.


Question: export CMAKE_PREFIX_PATH=${PYTORCH_ENV_DIR}/miniconda3/ # Test PyTorch <ins> if [[ "${JOB_BASE_NAME}" == *cuda9.2* ]]; then # Eigen gives "explicit specialization of class must precede its first use" error # when compiling with Xcode 9.1 toolchain, so we have to use Xcode 8.2 toolchain instead. export DEVELOPER_DIR=/Library/Developer/CommandLineTools else export DEVELOPER_DIR=/Applications/Xcode9.app/Contents/Developer fi </ins> export MACOSX_DEPLOYMENT_TARGET=10.9 export CXX=clang++ export CC=clang

Mac OS X builds failing this way. This appears to be correlated with worker-macos-high-sierra-2 and worker-macos-high-sierra-16.


Question: output (Tensor): output Tensor """ <ins> if mode != 'bilinear': raise NotImplementedError("nn.functional.grid_sample got unsupported mode: '{}'".format(mode)) </ins> return vision.grid_sampler(input, grid, padding_mode)

The only valid mode is 'bilinear', but one can pass anything to it and it will not complain:
Apparently, the is not passed to the backend function for . I think it should be removed.
Yes. I think the idea is that we'd like to support other modes at some point.


Question: python setup.py install cd test/ echo "Ninja version: $(ninja --version)" <del> sh run_test.sh </del> <ins> sh run_test.sh -- -v </ins> echo "BUILD PASSED"

If a test segfaults, it's hard to determine which test it is because the output looks like: ! I can work on this if we think this is a good idea.
Last time I asked about this, he didn't want to because it adds a lot of scrolling. I also did some work to move the CI tests to so that we can output XML status files, but (1) this is unlikely to help in case of segfault, and (2) you can't do this straightforwardly, because we run some test files multiple times with different environment variables, and the XML exporter isn't really set up to handle this case. For the record, I am personally OK with just slapping on the tests and calling it a day.
in the conbuild sounds good to me too
closed via


Question: are zeros. A sparse tensor is represented as a pair of dense tensors: a tensor <del> of values and a tensor of indices.  A sparse tensor can be constructed </del> <ins> of values and a 2D tensor of indices.  A sparse tensor can be constructed </ins> by providing these two tensors, as well as the size of the sparse tensor <del> (which cannot be inferred from these tensors!) </del> <ins> (which cannot be inferred from these tensors!)  Suppose we want to define a sparse tensor with the entry 3 at location (0, 2), entry 4 at location (1, 0), and entry 5 at location (1, 2).  We would then write: </ins> <del> >>> i = torch.LongTensor([[0, 1], [2, 0]]) >>> v = torch.FloatTensor([3, 4]) </del> <ins> >>> i = torch.LongTensor([[0, 1, 1], [2, 0, 2]]) >>> v = torch.FloatTensor([3, 4, 5]) </ins> >>> torch.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense() 0  0  3 <del> 4  0  0 </del> <ins> 4  0  5 [torch.FloatTensor of size 2x3] Note that the input to LongTensor is NOT a list of index tuples.  If you want to write your indices this way, you should transpose before passing them to the sparse constructor: >>> i = torch.LongTensor([[0, 2], [1, 0], [1, 2]]) >>> v = torch.FloatTensor([3,      4,      5    ]) >>> torch.sparse.FloatTensor(i.t(), v, torch.Size([2,3])).to_dense() 0  0  3 4  0  5 </ins> [torch.FloatTensor of size 2x3] You can also construct hybrid sparse tensors, where only the first n

The indices don't align with the values. In the example given on the website, by this logic, seems like it's indexing the columns first and that the indices are in reverse order to their values. Seems like corresponds to 4 and corresponds to 3. i = torch.LongTensor([[0, 1], [2, 0]]) v = torch.FloatTensor([3, 4]) (i, v, torch.Size([2,3])).todense() 0 0 3 4 0 0 [torch.FloatTensor of size 2x3] If I change the indices to and , I get i = torch.LongTensor([[0, 1], [0,1]]) v = torch.FloatTensor([3, 4]) (i, v, torch.Size([2,3])).todense() 3 0 0 0 4 0 [torch.FloatTensor of size 2x3] seems like unexpected behavior. Also if I try - should be intuitive - putting a value in position i = torch.LongTensor([[0, 1], [2, 0], [1,1]]) v = torch.FloatTensor([3, 4, 5]) (i, v, torch.Size([2,3])).to_dense() Traceback (most recent call last): File "<stdin", line 1, in <moduleRuntimeError: invalid argument 2: number of dimensions must be nDimI + nDimV at /pytorch/torch/lib/THS/generic/THSTensor.c:169 Help please?
This is a doc problem, thank you. The docs are misleading here, because it is not that is a coordinate, but 0 from the first tensor, and 2 from the second tensor. I'll try to rewrite these docs so they are clearer.
Basically, our constructor expects to be (transpose of what you gave it).
We should consider changing that


Question: 1.2469  0.0064 -1.6250 [torch.FloatTensor of size 3x3] <del> >>> torch.tril(a, k=1) </del> <ins> >>> torch.tril(a, diagonal=1) </ins> 1.3225  1.7304  0.0000 -0.3052 -0.3111 -0.1809 1.2469  0.0064 -1.6250 [torch.FloatTensor of size 3x3] <del> >>> torch.tril(a, k=-1) </del> <ins> >>> torch.tril(a, diagonal=-1) </ins> 0.0000  0.0000  0.0000 -0.3052  0.0000  0.0000

The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.


Question: // should have saved the grad accumulator. Even if the Variable no longer // alive, the accumulator should be kept alive by the references in the graph). if (requires_grad && !grad_fn && weak_grad_fn.expired() && grad_accumulator.expired()) <del> throw std::logic_error("No grad accumulator for a saved leaf!"); </del> <ins> throw std::logic_error("No grad accumulator for a saved leaf!"); </ins> new_var->grad_accumulator = grad_accumulator; return new_var;

there is a bug. the code is from The code for installing cuda10.2 should be corrected to The code for installing cuda10.2 should be corrected to cc
Thank you for reporting the typo

This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?


Question: self += modules def __getitem__(self, idx): <del> if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._modules[str(idx)] </del> <ins> if isinstance(idx, slice): return ModuleList(list(self._modules.values())[idx]) else: if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._modules[str(idx)] </ins> def __setitem__(self, idx, module): return setattr(self, str(idx), module)

it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.


Question: #include <initializer_list> #include <iostream> #include <functional> <ins> #include <cstddef> </ins> #ifdef _MSC_VER #ifdef Type

Hello, I'm trying to build pytorch (I tried both v3_buildfix and master branch) on a raspberry pi 3 but i'm getting that error: Note: I'm building without CUDA and DISTRIBUTED support. OS: Ubuntu 16.04 (Mate) gcc: arm-linux-gnueabihf-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.5) 5.4.0 cmake: v3.5.1
Can you try building with and let me know if that fixes it?
Thanks, it compiles now ;)


Question: call %CONDA_PARENT_DIR%Miniconda3Scriptsactivate.bat %CONDA_PARENT_DIR%Miniconda3 if "%INSTALL_FRESH_CONDA%"=="1" ( <del> call conda install -y -q python=%PYTHON_VERSION% numpy cffi pyyaml boto3 libuv </del> <ins> call conda install -y -q python=%PYTHON_VERSION% numpy"<1.23" cffi pyyaml boto3 libuv </ins> if errorlevel 1 exit /b if not errorlevel 0 exit /b call conda install -y -q -c conda-forge cmake=3.22.3

Numpy shouldn't require a pin with PyTorch. This issue is to ensure that should get reverted and all issues addressed (issues look like ) CI
Never mind, we apparently should have been pinning it.


Question: l = nn.Linear(10, 5).float().cuda() i = Variable(torch.randn(20, 10).float().cuda(1)) l.cuda(1) <del> expected_out = l(i).data l.cuda(0) out = dp.data_parallel(l, i, (0, 1)) self.assertEqual(out.get_device(), 0) self.assertEqual(out.data, expected_out) </del> <ins> expected_out = l(i) loss = expected_out.sum() loss.backward() expected_grads = [] for param in l.parameters(): expected_grads.append(param.grad.clone()) dev_ids_list = [(0, 1), (1, 0)] for dev_id in dev_ids_list: with torch.cuda.device(dev_id[0]): l.cuda() l.zero_grad() out = dp.data_parallel(l, i, dev_id) loss = out.sum() loss.backward() self.assertEqual(out.get_device(), dev_id[0]) self.assertEqual(out.data, expected_out.data) for expected, param in zip(expected_grads, l.parameters()): self.assertEqual(param.grad.data, expected.data) </ins> # Check for None device_ids <ins> l = l.cuda() </ins> out = dp.data_parallel(l, i) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported")

The loss will never decrease when I train a model using multi-gpus but without gpu0. This is an example I modified from the : If the result after one epoch is If the result after one epoch is
should always contain . If you want to specify which GPUs to use, you can set the environment variable , either by running your script like this: or set it programmatically in your script like this: So that deviceids 0, 1 will map to GPUs 1, 2, respectively. (Note that for the second method, you should put that line before any cuda relevant function call.)
Thank you.
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: - name: adaptive_max_pool3d(Tensor self, IntList[3] output_size) cname: VolumetricAdaptiveMaxPooling <del> - name: avg_pool2d(Tensor self, IntList[2] kernel_size, IntList[2] stride={}, IntList[2] padding=0, bool ceil_mode=false, bool count_include_pad=false) </del> <ins> - name: avg_pool2d(Tensor self, IntList[2] kernel_size, IntList[2] stride={}, IntList[2] padding=0, bool ceil_mode=false, bool count_include_pad=true) </ins> cname: SpatialAveragePooling default_init: stride: kernel_size <del> - name: avg_pool3d(Tensor self, IntList[3] kernel_size, IntList[3] stride={}, IntList[3] padding=0, bool ceil_mode=false, bool count_include_pad=false) </del> <ins> - name: avg_pool3d(Tensor self, IntList[3] kernel_size, IntList[3] stride={}, IntList[3] padding=0, bool ceil_mode=false, bool count_include_pad=true) </ins> cname: VolumetricAveragePooling default_init: stride: kernel_size

, thank you! The documentation for avgpool2d/3d says Indeed, the default was just that up to PyTorch 0.2. In 0.3-0.4 the default is False for 2d/3d and True 1d and the doc says it was True for all. The change in default happened when replaced the previous mechanism of calling THNN. Now I was about to submit a PR reverting the default to True, but the subtle twist is that TF uses False by default. avgpool2d is used in the wild, e.g. by ResNet. Should I change the docstring or should I change the default in Do you have an opinion ( maybe)? Best regards Thomas
So after submitting this, I saw that the module interface uses countincludepad=True all along, so the right option seems to be to have the same in the functional interface. I'll submit a PR.
we should indeed default functional interface to True as well. Thanks for the PR


Question: self.assertEqual(reference[None, 2:5, None, None], reference.unsqueeze(0)[:, 2:5].unsqueeze(2).unsqueeze(2)) # indexing with step <del> reference = self._consecutive((10, 10, 10)) </del> <ins> reference = consec((10, 10, 10)) </ins> self.assertEqual(reference[1:5:2], torch.stack([reference[1], reference[3]], 0)) self.assertEqual(reference[1:6:2], torch.stack([reference[1], reference[3], reference[5]], 0)) self.assertEqual(reference[1:9:4], torch.stack([reference[1], reference[5]], 0))

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.


Question: } #define RUN_T(INDEX_T)                                                   <del> TensorInfo<real, INDEX_T> inputInfo =                                 getTensorInfo<THCTensor, INDEX_T>(state, input);                  TensorInfo<real, INDEX_T> topKInfo =                                  getTensorInfo<THCTensor, INDEX_T>(state, topK);                   </del> <ins> TensorInfo<real, INDEX_T> inputInfo =                                  getTensorInfo<THCTensor, INDEX_T>(state, input);                     TensorInfo<real, INDEX_T> topKInfo =                                   getTensorInfo<THCTensor, INDEX_T>(state, topK);                      </ins> TensorInfo<int64_t, INDEX_T> indicesInfo =                             getTensorInfo<THCudaLongTensor, INDEX_T>(state, indices);

Repro: Expect two small tensors of small integers to be printed. Instead, only the first case works. In the second case, you get a series of failed assertions starting with this: I wonder if it is related to
is this still an outstanding issue? I cannot reproduce this with an install from the master branch.
The size which causes failure doesn't seem deterministic. Using a pretty recent build (a few days old), I now have to increase the "9" to a larger number to make it fail. Sometimes it will fail at 10, sometimes at 11. Here's an updated case that seems to fail reliably at some size, for me:


Question: {"_write_file", (PyCFunction)THPStorage_(writeFile), METH_O, NULL}, {"_new_with_file", (PyCFunction)THPStorage_(newWithFile), METH_O | METH_STATIC, NULL}, #ifndef THC_GENERIC_FILE <ins> {"from_buffer", (PyCFunction)THPStorage_(fromBuffer), METH_VARARGS | METH_KEYWORDS | METH_STATIC, NULL}, </ins> {"_share", (PyCFunction)THPStorage_(_share), METH_NOARGS, NULL}, {"_new_shared", (PyCFunction)THPStorage_(_newShared), METH_VARARGS | METH_STATIC, NULL}, #endif

<!-- A clear and concise description of what the bug is. --The target name is used by a test executable in mkldnn. Steps to reproduce the behavior: BUILDTEST=1 build <!-- If you have a code sample, error messages, stack traces, please provide it here as well --<!-- A clear and concise description of what you expected to happen. --CMake configure passes. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): v1.5.0-rc2 OS (e.g., Linux): Windows How you installed PyTorch (, , source): source Build command you used (if compiling from source): python build Python version: 3.x CUDA/cuDNN version: 9.2 / 10.1 / 10.2 / NA GPU models and configuration: Tesla M60 Any other relevant information: <!-- Add any other context about the problem here. --We should not build tests for mkldnn.
I have encountered the same error, how do I resolve it?
It is actually mkldnn that is out of sync. So running the following script should help.
Thanks a lot for this I posted this on the PyTorch Discussions but to no avail, and now I've finally understood the reason of this error. Thanks a lot. Surprisingly, for me the build works out if I build from the master repo, but changing to a different version always fails.


Question: .format(torch.typename(value), name)) modules[name] = value else: <del> object.__setattr__(self, name, value) </del> <ins> buffers = self.__dict__.get('_buffers') if buffers is not None and name in buffers: if value is not None and not torch.is_tensor(value): raise TypeError("cannot assign '{}' as buffer '{}' " "(torch.Tensor or None expected)" .format(torch.typename(value), name)) buffers[name] = value else: object.__setattr__(self, name, value) </ins> def __delattr__(self, name): if name in self._parameters: del self._parameters[name] <ins> elif name in self._buffers: del self._buffers[name] </ins> elif name in self._modules: del self._modules[name] else:

They do one pair of these calls for each slicing operation.
Does this actually have a performance impact? These calls tend to be very quick
Yes, it's a major reason why small unrolled LSTMs (which involve in addition to two GEMMs: a chunk into four gates, for 4 pairs of calls, five unary pointwise ops, for 5 more, and five binary pointwise ops, for another 5) are much slower than they are in Theano and TensorFlow.
Also, there are excessive cudaGetDevice calls for each pointwise operation (9 cudaGetDevice) even without chunking, as in the following repro code: As for small unrolled LSTMs it does show up.
This should no longer be an issue, now that is implemented as a native ATen function. I think we're still using the Python version for tensors, but this will be resolved soon when we stop using tensors altogether.


Question: namespace torch { namespace autograd { <ins> extern const char* ERR_BACKWARD_TWICE; </ins> struct Variable : std::enable_shared_from_this<Variable> { struct SavedVariable {

Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)


Question: INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}) ADD_LIBRARY(THPP SHARED ${all_cpp}) <del> SET_PROPERTY(TARGET THPP PROPERTY CXX_STANDARD 11) </del> SET_TARGET_PROPERTIES(THPP PROPERTIES VERSION 1 SOVERSION 1) <ins> if(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") SET_PROPERTY(TARGET THPP PROPERTY CXX_STANDARD 11) endif(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") </ins> TARGET_LINK_LIBRARIES(THPP PRIVATE ${TH_LIBRARIES}) INSTALL(TARGETS THPP

Currently, CMake 3.0 or higher is required for THPP. This is annoying for building from source on Ubuntu 14.04 systems, as default CMake is 2.8, and updating it requires either compiling from source or adding some random repos:
it should work fine if you change it to 2.8. I'm testing this out in a sec, and will push it into master
Nope, it does not for me.


Question: <ins> #pragma once #include <Python.h> template<class T> class THPPointer { public: THPPointer(): ptr(nullptr) {}; THPPointer(T *ptr): ptr(ptr) {}; THPPointer(THPPointer &&p) { free(); ptr = p.ptr; p.ptr = nullptr; }; ~THPPointer() { free(); }; T * get() { return ptr; } const T * get() const { return ptr; } T * release() { T *tmp = ptr; ptr = NULL; return tmp; } operator T*() { return ptr; } THPPointer& operator =(T *new_ptr) { free(); ptr = new_ptr; return *this; } THPPointer& operator =(THPPointer &&p) { free(); ptr = p.ptr; p.ptr = nullptr; return *this; } T * operator ->() { return ptr; } operator bool() { return ptr != nullptr; } private: void free(); T *ptr = nullptr; }; typedef THPPointer<PyObject> THPObjectPtr; </ins>

Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc


Question: add_docstr(torch._C.tril, """ <del> tril(input, k=0, out=None) -> Tensor </del> <ins> tril(input, diagonal=0, out=None) -> Tensor </ins> Returns the lower triangular part of the matrix (2D Tensor) :attr:`input`, the other elements of the result Tensor :attr:`out` are set to 0.

The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.


Question: fi # sccache will fail for CUDA builds if all cores are used for compiling <del> # gcc 7.2 with sccache seems to have intermittent OOM issue if all cores are used if ([[ "$BUILD_ENVIRONMENT" == *cuda* ]] || [[ "$BUILD_ENVIRONMENT" == *gcc7.2* ]]) && which sccache > /dev/null; then </del> <ins> # gcc 7 with sccache seems to have intermittent OOM issue if all cores are used if ([[ "$BUILD_ENVIRONMENT" == *cuda* ]] || [[ "$BUILD_ENVIRONMENT" == *gcc7* ]]) && which sccache > /dev/null; then </ins> export MAX_JOBS=`expr $(nproc) - 1` fi

We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in


Question: from torch.autograd import gradcheck from torch.autograd.function import once_differentiable <del> from common import TestCase, run_tests </del> <ins> from common import TestCase, run_tests, skipIfNoLapack </ins> from torch.autograd._functions import * from torch.autograd import Variable, Function

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: // Each block handles one distribution // First pass, find the total sum of the distribution AccT sum = accZero; <ins> T val; </ins> for (int cat = threadIdx.x; cat < categories; cat += blockDim.x) { <del> sum = THCNumerics<AccT>::add( sum, ScalarConvert<T, AccT>::to(dist[curDist * categories + cat])); </del> <ins> val = dist[curDist * categories + cat]; assert(THCNumerics<T>::ge(val, zero)); sum = THCNumerics<AccT>::add(sum, ScalarConvert<T, AccT>::to(val)); </ins> } // threadIdx.x == 0 has the sum value from this

Looking at the values returned from tn.multinomial(1) and counting them, it seems like it returns the values of the multinomial distribution taken from t, i.e. that it has ignored the signs of the values. However, it seem like always returns a tensor filled with zeros.
looking into this


Question: self.assertEqual(hy.data[0][0][0], 10) self.assertEqual(hy.data[1][0][0], output_val) <del> @unittest.skipIf(not TEST_CUDNN, "needs cudnn") </del> <ins> @unittest.skipIf(not (TEST_CUDNN and TEST_CUDNN_VERSION >= 5103), "needs cudnn >= 5.1") </ins> def test_RNN_dropout_state(self): import sys if sys.version_info[0] == 2:

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: modules = list(self._modules.keys()) buffers = list(self._buffers.keys()) keys = module_attrs + attrs + parameters + modules + buffers <ins> # Eliminate attrs that are not legal Python variable names keys = [key for key in keys if not key[0].isdigit()] </ins> return sorted(keys)

Hello, In the case where these classes sequentially append the received elements with , the modules become inaccessible since a variable name starting with a number is forbidden in Python. I attached a patch that names the modules as instead of just numbers.
Use syntax for Sequential and ModueList.
possible but the attributes are there and they are obviously shown in autocompletion suggestions in IPython for example and when you try to access them you got an error because their naming does not conform to Python rules.
Perhaps we should exclude them from


Question: <del> FROM nvidia/cuda:8.0-devel-ubuntu16.04 </del> <ins> FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04 </ins> RUN echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list <del> ENV CUDNN_VERSION 6.0.20 </del> RUN apt-get update && apt-get install -y --no-install-recommends  build-essential  cmake  git  curl  <ins> vim  </ins> ca-certificates  libjpeg-dev  <del> libpng-dev  libcudnn6=$CUDNN_VERSION-1+cuda8.0  libcudnn6-dev=$CUDNN_VERSION-1+cuda8.0 &&  </del> <ins> libpng-dev && </ins> rm -rf /var/lib/apt/lists/* RUN curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh  &&

Torchvision is considered a basic requirement of the tutorials. Perhaps it makes sense to include it in the docker build.
It's easy to install it by command ;)
It's easy to install torch itself using a similar command? :). I was thinking a docker image should have torchvision included.
Runtime docker file already has torchvision.


Question: [[17, 7], [18, 2], [19, 7], [20, 1]], ]) res = dense._sparse_mask(x) <del> exp_v = self.ValueTensor([[7, 9], [14, 1], [14, 1], [3, 3], [20, 1]]) </del> <ins> exp_v = self.ValueTensor([[7, 9], [14, 1], [3, 3], [20, 1]]) </ins> expected = self.SparseTensor(i, exp_v, torch.Size([5, 4, 2])) self.assertEqual(res, expected)

and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.


Question: pytorch-linux-xenial-cuda9-cudnn7-py2-test pytorch-linux-xenial-cuda9-cudnn7-py3-build pytorch-linux-xenial-cuda9-cudnn7-py3-test <ins> pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7-build pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7-test </ins> pytorch-linux-xenial-py3-clang5-asan-build pytorch-linux-xenial-py3-clang5-asan-test pytorch-linux-trusty-py2.7.9-build

We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in


Question: x_clone ^= y self.assertEqual(x_clone, xor_result) <ins> def test_apply(self): x = torch.range(1, 5) res = x.clone().apply_(lambda k: k + k) self.assertEqual(res, x * 2) self.assertRaises(RuntimeError, lambda: x.apply_(lambda k: "str")) </ins> if __name__ == '__main__': unittest.main()

On Python 3.5.2 errors out with . map and map2 seem to be work just fine.


Question: def test_broadcast_batched_matmul(self): TestTorch._test_broadcast_batched_matmul(self, lambda t: t.cuda()) <ins> def test_index(self): TestTorch._test_index(self, lambda t: t.cuda()) </ins> def test_advancedindex(self): TestTorch._test_advancedindex(self, lambda t: t.cuda())

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.


Question: if probs is not None: self.probs = probs / probs.sum(-1, keepdim=True) else: <del> self.logits = logits - log_sum_exp(logits) </del> <ins> self.logits = logits - _log_sum_exp(logits) </ins> self._param = self.probs if probs is not None else self.logits self._num_events = self._param.size()[-1] batch_shape = self._param.size()[:-1] if self._param.ndimension() > 1 else torch.Size()

Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!


Question: y._backward_hooks['test'] = error b.backward(torch.ones(5, 5)) <ins> def test_requires_grad_inplace(self): a = Variable(torch.randn(5, 5)) b = Variable(torch.randn(5, 5), requires_grad=True) a += b self.assertTrue(a.requires_grad) # non-leaf Variable a = Variable(torch.randn(5, 5)) + 0 b = Variable(torch.randn(5, 5), requires_grad=True) a += b self.assertTrue(a.requires_grad) </ins> def test_previous_functions(self): x = Variable(torch.randn(5, 5), requires_grad=True) y = Variable(torch.randn(5, 5), requires_grad=True)

import torch as th from torch.autograd import Variable x = Variable(th.Tensor([1 , 2 , 3]) , requiresgrad = False) y = Variable(th.Tensor([100]) , requiresgrad = True) x = x.cuda(0) y = y.cuda(0) x[0] = y print(x.requiresgrad) If ((0) (0)) is executed then the output is False, True otherwise. I think the result that x.requiresgrad = True is more natural.
I further checked some in-place operator in GPU mode(say, x = Variable(...).cuda(0) ), and find out that after the operation is done, the requiresgrad of the Variable doesn't change. For example: import torch as th from torch.autograd import Variable x = Variable(torch.Tensor([1]) , requiresgrad = False) y = Variable(torch.Tensor([2]) , requiresgrad = True) x = x.cuda(0) y = y.cuda(0) x.add(y) print(x.requires_grad) the output is False
It doesn't have to do with Variable holding a CUDA tensor, but rather with the fact that it's no longer a leaf. I've sent a fix in .
Thanks for the report!


Question: SET_PROPERTY(TARGET ATen PROPERTY CXX_STANDARD 11) endif(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") <del> FIND_PACKAGE(BLAS) </del> IF(BLAS_FOUND) <del> SET(USE_BLAS 1) </del> IF ($ENV{TH_BINARY_BUILD}) MESSAGE(STATUS "TH_BINARY_BUILD detected. Enabling special linkage.") TARGET_LINK_LIBRARIES(ATen "${BLAS_LIBRARIES};${BLAS_LIBRARIES};${BLAS_LIBRARIES}") ELSE ($ENV{TH_BINARY_BUILD}) TARGET_LINK_LIBRARIES(ATen ${BLAS_LIBRARIES}) ENDIF ($ENV{TH_BINARY_BUILD}) <del> IF(BLAS_INFO STREQUAL "mkl") ADD_DEFINITIONS(-DTH_BLAS_MKL) ENDIF() </del> ENDIF(BLAS_FOUND) IF(LAPACK_FOUND)

currently compiled master does not use any blas for any ops and uses our lame fallback C code. I'm looking into it. Basically USE_BLAS is not being set in


Question: Args: input: Variable :math:`(N, C)` where `C = number of classes` target: Variable :math:`(N)` where each value is `0 <= targets[i] <= C-1` <del> weight (Variable, optional): a manual rescaling weight given to each class. If given, has to be a Variable of size "nclasses" </del> <ins> weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size "nclasses" </ins> size_average (bool, optional): By default, the losses are averaged over observations for each minibatch. However, if the field sizeAverage is set to False, the losses are instead summed

The of nn.functional.crossentropy seems wrong. The parameter weight should be a tensor, rather than a Variable.


Question: fn.w_desc = init_weight_descriptor(fn, fn.weight_buf) w = fn.weight_buf <del> if tuple(hx.size()) != hidden_size: raise RuntimeError('Expected hidden size {}, got {}'.format( hidden_size, tuple(hx.size()))) </del> if cx is not None and tuple(cx.size()) != hidden_size: raise RuntimeError('Expected cell size {}, got {}'.format( hidden_size, tuple(cx.size())))

I run RNN sucessfully with the code. and I run embedding example. Then, got RuntimeError. I set to use nothing seems wrong. And the error info seems not the dim problem? The error is so strange.
What exactly do you mean by "embedding example"? Could you provide a code snippet that causes the RuntimeError?
To better reproduce the error, I use the official code example of The origin runs well: Due to use , I have to make batch first. And then , it raises error.
Thank you for the example! The say that setting means that the first dim of input is the batch, ie, input has size . however is unchanged, so the following should work: I do agree the error message could be better, though


Question: return storage <ins> def rebuild_storage_empty(cls): return cls() </ins> def reduce_storage(storage): from . import get_sharing_strategy if storage.is_cuda:

PyTorch version: 0.4.0a0+ Repro: Result:
Emm. I fail to reproduce this bug in Python 3.5.0 and Mac OS 10.13.3. What's your system environment?
The bug is probably specific to the sharing mode. Mac OS uses by default.


Question: pass def run(self): <ins> # Check if you remembered to check out submodules def check_file(f): if not os.path.exists(f): print("Could not find {}".format(f)) print("Did you run 'git submodule update --init'?") sys.exit(1) check_file(os.path.join(lib_path, "gloo", "CMakeLists.txt")) check_file(os.path.join(lib_path, "nanopb", "CMakeLists.txt")) check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) </ins> libs = [] if WITH_NCCL and not WITH_SYSTEM_NCCL: libs += ['nccl']

It seems like running has a dependency on first running . This command also seems to also error out when installing via a wheel. This creates a problem for tools like  which rely on the output of to examine dependencies. Is there anyway to allow this command to not error out when installing via a wheel? Full stack trace: I understand the preferred package manager for this project is conda but being officially endorsed from , pipenv has a fairly large audience:
thanks do you know when's the next planned release?
Hopefully within a month


Question: torch.stack([reference[:, 2, 1], reference[:, 2, 3], reference[:, 2, 5]], 1)) lst = [list(range(i, i + 10)) for i in range(0, 100, 10)] <del> tensor = torch.DoubleTensor(lst) </del> <ins> tensor = conv_fn(torch.DoubleTensor(lst)) </ins> for _i in range(100): idx1_start = random.randrange(10) idx1_end = idx1_start + random.randrange(1, 10 - idx1_start + 1)

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.


Question: <del> From PyTorch: Copyright (c) 2016-     Facebook, Inc            (Adam Paszke) Copyright (c) 2014-     Facebook, Inc            (Soumith Chintala) Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU                      (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006      Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) From Caffe2: Copyright (c) 2016-present, Facebook Inc. All rights reserved. All contributions by Facebook: Copyright (c) 2016 Facebook Inc. All contributions by Google: Copyright (c) 2015 Google Inc. All rights reserved. All contributions by Yangqing Jia: Copyright (c) 2015 Yangqing Jia All rights reserved. All contributions from Caffe: Copyright(c) 2013, 2014, 2015, the respective contributors All rights reserved. All other contributions: Copyright(c) 2015, 2016 the respective contributors All rights reserved. Caffe2 uses a copyright model similar to Caffe: each contributor holds copyright over their contributions to Caffe2. The project versioning records all such contribution and copyright details. If a contributor wants to further mark their specific copyright on a particular contribution, they should indicate their copyright solely in the commit message of the change when it is committed. </del> ======================================================================= Software under third_party =======================================================================

The three-clause BSD license in file LICENSE says in clause 1 "Redistributions of source code must retain the above copyright notice..." However, there is no longer a copyright notice in the file; it appears to have been removed in commit (see ).
cc


Question: "torch/csrc/Generator.cpp", "torch/csrc/Size.cpp", "torch/csrc/Exceptions.cpp", <del> "torch/csrc/Tensor.cpp", </del> "torch/csrc/Storage.cpp", "torch/csrc/DynamicTypes.cpp", "torch/csrc/byte_order.cpp",

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: is_master_only=True, requires=["binary_linux_manywheel_3_7m_cu102_devtoolset7_build"], extra_props={ <del> "resource_class": "gpu.medium", </del> <ins> "resource_class": "gpu.nvidia.small", </ins> "use_cuda_docker_runtime": miniutils.quote((str(1))), }, ),

As it still using deprecated resource class, see See example of the failure in
Followup: we need to plan better --we knew it was gonna happen and none of us fixed it early enough. Action item: go through tasks about deprecation in sprint planning


Question: { Py_VISIT(self->data); Py_VISIT(self->backward_hooks); <ins> // We don't want to traverse the grad_fn, even if the Variable owns it and the // shared pointer's use count is 1. This is because we would need to treat // the grad_fn as part of the Python state and hold the GIL sometimes when // grad_fn's shared_ptr is copied, otherwise a race condition with the Python // GC could occur. Holding the GIL when the shared_ptr is copied adds // undesirable complexity/overhead. // // When hooks, a Variable, and its grad_fn are involved in a Python reference // cycle, because we're not traversing the grad_fn, the reference cycle will // in fact leak. // // See https://gist.github.com/zou3519/7ac92b84dd7d206dcc6eae55fee8372c // for more details about the race condition involving traversing the grad_fn // and the python GC. </ins> if (self->cdata.defined()) { <del> // Only visit this if we actually own it (no one else use the shared pointer) auto& grad_fn = self->cdata.grad_fn(); if (grad_fn.use_count() == 1) { if (auto fn = dynamic_cast<PyFunction*>(grad_fn.get())) { Py_VISIT(fn->obj); } else { // visit hooks in C++ implemented autograd functions for (auto& hook : grad_fn->pre_hooks) { if (auto pyhook = dynamic_cast<PyFunctionPreHook*>(hook.get())) { Py_VISIT(pyhook->dict); } } for (auto& hook : grad_fn->post_hooks) { if (auto pyhook = dynamic_cast<PyFunctionPostHook*>(hook.get())) { Py_VISIT(pyhook->dict); } } } } </del> for (auto& hook : self->cdata.hooks()) { if (auto pyhook = dynamic_cast<PyFunctionPreHook*>(hook.get())) { Py_VISIT(pyhook->dict);

My code runs well for a while , it occurs randomly..... Traceback (most recent call last): File "", line 236, in <moduletrainvgg("",dset='luna16') File "", line 54, in trainvgg lossvalue.backward() File "/home/uih/anaconda2/lib/python2.7/site-", line 169, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/home/uih/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion "pos = 0 && pos < ()" failed. Any idea what's happend? Thanks in advance~
Yes, it looks like we have a bug somewhere in the code. Can you please try to isolate a small snippet that still triggers the same error so we can look into it? Thank you!
It's hard for me to isolate, the error occurs when I do the train and validation in turn, if I just do either the train or val, everthing runs well.
The problem is that we won’t be able to fix it, until we actually see under what conditions does the error happen
I got the same error in v0.3.0 today. when I trained my model with single GPU, it worked perfectly;with multi-gpu the error occured. RuntimeError: add: Assertion failed.
We've also been struggling with this issues for a long time. Unfortunately, finding which part of our code specifically is triggering the assert (which used to be a segfault before ) has proven extremely hard and, as of now, unsuccessful. In fact, we never submitted a bug report as we couldn't create a self-contained piece of code to trigger it. I'll try to summarize all the information we've gathered: We know for a fact that the issue appeared sometime after this commit: . All our code works perfectly fine on or v0.2. The assert is always triggered during the backward pass, but the exact moment when it happens seems to be completely random: when training over Imagenet we've seen this after 50 iterations as well as after 5000.

This makes it super-hard to understand whether a particular piece of code is triggering the assert or not, as you can never be sure how many iterations are enough to exclude that the assert will ever trigger! The issue seems to be somehow related to the use of multiple GPUs (via ): we've never observed it while training with a single GPU. The issue seems also to be related to the use of user-defined classes: we've never seen it when training networks composed of "standard" Pytorch functions / modules only... ... however, simply running forward / backward passes with our custom functions doesn't seem to be enough to trigger the assertion. In our experience, working with "complex" computational graphs increases the likelihood of encountering this issue. Recently we've been able to re-create the issue using and a modified version of , where we replace some of the layers with our custom-made ones. Unfortunately I can't share the code for now (our paper describing the new layers is still not out), but I'll come back to you as soon as I can! Some additional information: all our experiments are on Ubuntu 16.04 using CUDA 8, CUDNN 7 and TITAN X and Xp GPUs.
almost the same condition as us
Encounter the same condition after serveral imagenet epochs on a NasNet after update to 3.0. Pytorch 2.0 is working well.
I am seeing this too -- I have two runs of the same model running with different hyperparameters, one is running smoothly, this one quit with the exact same error avgloss = self.trainepoch(trainingdata,validationdata,batchsize,saveprefix,epoch) File "", line 759, in trainepoch gradients = loss.backward() File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 167, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion failed. Running PyTorch v0.3 and training on 4 K-80 GPUs. My other run with different parameters is also running on the exact same setup but its still running.

Both runs differ only in shuffle of data and a different batch size (the one that failed had a slightly larger batch size)
: I'm not using any user define autograd.Function but I see the error too, so maybe thats not the primary place with a bug. Using nn.DataParallel though.
I'll try again to train a network using standard layers only, maybe I'll also encounter the error in the no-custom-functions case if I let it run a little bit longer. If this is the case, then I guess we have some strong indication that (or multi-GPU computation in general) could indeed be the source of the error!
: I dont think the duration has got anything to do with it --- One model run has been running without a problem for the last 18 hours, the other one had quit last night pretty early on (Crashed after the backward pass of the 8th batch). I have just restarted it and watching it.
I agree that the bug is probably not related to training length. My guess is that each backward pass has a certain probability of randomly failing with this assert, so running for longer just increases the overall chances of encountering the bug.
Thanks for the information. We'd really like to fix this bug, and so there are a few things that would help us out a lot (in order of difficulty): you run your training process with gdb attached with a breakpoint, so that we can get a backtrace when the assertion fails? Also, if you can go one frame up and print the name of the function (run in gdb) that would also be helpful. We basically want to know if it's either a C++ function or a Python function. soon as you can publish your latest layers, uploading a script that reproduces (even if it takes a while to reproduce); we would be happy to run the script to try to reproduce, which would be very helpful for us. We'd love a bisection, but given that it can sometimes take 18hrs+ to repro, it doesn't sound like that would be easy to do.
This is the backtrace: For the function name I'll probably have to recompile pytorch in debug mode, since in gdb returns . Edit - I managed to get the name of the function by catching the exception raised by the assert: . Edit 2 - I did a second run and the name of the function is still the same.

Actually I just realized that what we really need is Can you please try printing that (might require a debug build as well)?
Printing instead of I get , which is the name of our custom function's backward pass. More importantly, our code and paper are now public so I can share this with you: As I mentioned in a previous comment, this code is just the Pytorch ImageNet + ResNet examples, except that standard BN is replaced with our custom layer. When running this on our machines with 4 GPUs we always encounter the assert within the first epoch. Please feel free to ask if you need help running the code.
I also encountered this assert. I have two machines, one with 2, and other one with 4 GPUs. Both machines are Ubuntu 16.04, python 2.7, cuda 9.0, cudnn 7, and pytorch v0.3.0. I never encountered this assert on my 2 GPUs machine. This assert always appeared on my 4 GPUs machine.
same as here. I have python 2.7, cuda 8.0, cudnn 7 and pytorch v0.3.0 installed from binaries. The assertion was triggered when I was training networks on 4 GPUs. It happened at a random time during training.
Same here, with multiple pixel-wise prediction models with just CNNs. Seems to occur only with multiple GPUs, has not happened so far with single GPU. python 2.7, CUDA 8.0, CUDNN 7, PyTorch v0.3.0.
can you take a look at this asap
edit: resolved, I ended up adding a compilation flag for compute capacity 3.5 to pytorchbugreport's building scripts (to work with my gpus) I'm getting the following after trying to run pytorchbugreport. Have you seen this before? !
you have to change to cover your GPU, which is compute35,code=sm_35 i think (if it is K40)
I ran your code for six epochs on a machine with 4 gpus on pytorch 0.3.0 but haven't been able to trigger the assert. I'll try letting it run for longer and see if anything pops up
For us the assert almost always triggers during the first epoch.

There might be some important difference in our setups. This is the exact sequence of steps we have used to reproduce the error across several machines: from a clean Ubuntu 16.04 installation with CUDA 8.0, CUDNN 7.0, pip, virtualenv and the compilers. and activate a new python2 virtualenv. necessary python packages . the pytorch repository and checkout and install with our repository and compile our native module with the training as described before. Our machines all have either Titan X or Titan Xp GPUs, so a different architecture from the one you are using, I don't know if this could have an influence on the error.
My machine has 4 GTX 1080 Ti which triggers the error.
we (finally) found a different repro that triggers this issue internally on one of our machines, a fix will be issued soon.
Quick update: we're still looking into this, but so far, I've been able to get this bug to trigger with two machines using python 2.7 but never with python 3. If you're in need of a quick fix using python 3 might help.
I also encountered this propblem, with pytorch 0.3.0, python2.7, CUDA8.0, Titan X GPUs. My session always crashed around 8 epochs. And I degrade pytorch to 0.2.0. now the training seems work well. Someone in emergency can try this.
Thank you , I'll try to switch to python 3 for now!
let me know if the bug comes back in python 3. As far as I can tell the python version shouldn't matter in the triggering of this bug... but it's never happened to me with python 3 yet.
Crashed in Python2.7 RuntimeError: add: Assertion failed Absolutely random crash using multi gpu.
This has been fixed in HEAD but there has not been a release of 0.3 with the fix yet.
Oh, Thanks for your reply. I will try the HEAD version. Thank you .
i have encountered this error , but change the pytorch to the latest version(0.4) will solve it.
Encountered same error when I add dropout layer to a Resnet which I modified. Trained on multi-GPU. Does latest conda release fix the problem?
it is not yet on latest conda release.

to get the fix at the moment you will have to install pytorch from source:
The same bug ! !
did you install pytorch from source?
Is there a simple way to block this error without changingthe version of python or pyotrch?
You could disable the python reference cycle . This would probably lead to memory leaks though.
It should be safe if you do this once all the graphs are deleted, and there will be no leaks unless you create cycles in your data structures
After 2 days of struggle, I finally changed back to 0.2.0 and expects the release of binary 0.4.0. Anyway, thank you for your advices. :)
I encountered the same error with version 0.3.0.post4. What was the proper fix here?
upgrade to version 0.3.1 or newer.


Question: class Dropout(Module): <del> r"""Randomly zeroes some of the elements of the input tensor. </del> <ins> r"""During training, randomly zeroes some of the elements of the input tensor with probability *p* using samples from a bernoulli distribution. </ins> The elements to zero are randomized on every forward call. <ins> This has proven to be an effective technique for regularization and preventing the co-adaptation of neurons as described in the paper `Improving neural networks by preventing co-adaptation of feature detectors`_ . Furthermore, the outputs are scaled by a factor of *1/(1-p)* during training. This means that during evaluation the module simply computes an identity function. </ins> Args: p: probability of an element to be zeroed. Default: 0.5 inplace: If set to True, will do this operation in-place. Default: false

"Randomly zeroes some of the elements of the input tensor. The elements to zero are randomized on every forward call." This is incorrect; the function also scales up by 1/(1-p), which the implementation correctly does.


Question: del l1[i] return l1 <del> def _backward(self, method, input, gradOutput, scale): </del> <ins> def _backward(self, method, input, gradOutput, scale=1): </ins> isTable = isinstance(input, list) wasTable = isinstance(self.gradInput, list) if isTable: for i, module in enumerate(self.modules): <del> currentGradInput = getattr(module, method)(input, gradOutput[i], scale) </del> <ins> if method == 'updateGradInput': currentGradInput = module.updateGradInput(input, gradOutput[i]) elif method == 'backward': currentGradInput = module.backward(input, gradOutput[i], scale) </ins> if not isinstance(currentGradInput, list): raise RuntimeError("currentGradInput is not a table!")

Subclassing torch.Tensor with fails due to a on the latest nightly builds, whereas it previously worked without issue. Run this snippet of code: And you'll recieve an error message like this: Changing the to results in the exact same error message. There should be no error message as it should be working like it was previously in Torch 1.8.1 and below. I think that the issue started on Torch version: , and I can confirm that it still exists as of , . The issue started on the nightly builds at least 2 days ago, if that helps resolve things. It's been causing some of Captum's CircleCI tests to fail, and the tests show the first occurrence of the error on . cc
This is due to either or
I think the problem is I'm not supposed to delete because that means the super() call doesn't work. Instead, I should just verify that the passed in class is a subclass, then this should work.


Question: -DCMAKE_CXX_FLAGS="$C_FLAGS $CPP_FLAGS" make install cp "lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so.1" <del> ln -s "${INSTALL_DIR}/lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so" </del> <ins> if [ ! -f "${INSTALL_DIR}/lib/libnccl.so" ]; then ln -s "${INSTALL_DIR}/lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so" fi </ins> cd ../.. }

there is a bug. the code is from The code for installing cuda10.2 should be corrected to The code for installing cuda10.2 should be corrected to cc
Thank you for reporting the typo

This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?


Question: For summation index :math:`j` given by `dim` and other indices :math:`i`, the result is <del> :math:`text{logsumexp}(x)_{i} = log sum_j exp(x_ij).` </del> <ins> .. math:: text{logsumexp}(x)_{i} = log sum_j exp(x_{ij}) </ins> If :attr:`keepdim` is ``True``, the output tensor is of the same size as :attr:`input` except in the dimension :attr:`dim` where it is of size 1.

Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!


Question: return g.op("Undefined") <ins> def Constant(g, value): return g.op("Constant", value_t=value) </ins> def add(g, self, other, alpha): if _scalar(alpha) != 1: return _unimplemented("add", "alpha != 1")

Hi! OS: Linux PyTorch version: 0.4.0a0+ How you installed PyTorch (conda, pip, source): source Python version: 3.6Context around what you are trying to do I am trying to convert layer with torch.indexselect() operation to onnx A script to reproduce the bug. Please try to provide as minimal of a test case as possible.Error messages and/or stack traces of the bug What is the right way to export torch.indexselect() operation?
It kind of sounds like we forgot to implement constant export, which is embarrassing and should be fixed.
Are there any chances it will be fixed soon?
I took a look, and actually, it's just a spurious warning, the model seems to export successfully into


Question: return THC_float2half(0); #else THError("Cublas_Hdot requires CUDA 8.0+"); <del> return THC_half2float(0); </del> <ins> return THC_float2half(0); </ins> #endif } #endif

Hi, I was trying to update from source but I get an error "no suitable constructor exists to convert from "int" to "__half"" when trying to build I am working with python 2.7 cuda 7.5 and GCC 4.8.3 in openSUSE. Any suggest? Here is the output error when building from source:
ATen Excluded: set(['bernoulli', 'bernoulli']) [ 0%] Built target atenfilesaregenerated [ 0%] Building NVCC (Device) object [ 1%] Building NVCC (Device) object [ 1%] Building NVCC (Device) object [ 1%] Building NVCC (Device) object [ 1%] Building NVCC (Device) object [ 2%] Building NVCC (Device) object [ 2%] Building NVCC (Device) object [ 2%] Building NVCC (Device) object (77): error: no suitable constructor exists to convert from "int" to "half" (77): error: no suitable constructor exists to convert from "float" to "half" 2 errors detected in the compilation of "/tmp/tmpxft0000587a00000000-". CMake Error at (message): Error generating file recipe for target '' failed make[2]: [] Error 1 make[2]: Waiting for unfinished jobs.... CMakeFiles/Makefile2:194: recipe for target '' failed make[1]: [] Error 2 Makefile:129: recipe for target 'all' failed make: [all] Error 2
NVIDIA cuDNN v6.x or above
same problem here. gcc (Ubuntu 4.8.4), python 2.7, cuda 7.5, cuDNN 6.0.21
Same problem here. CUDA: v7.5 CUDNN: v6.0