Question: """ # TODO: try to find an input on another gpu, copy it, # and accumulate into the copy <ins> if destination is None: destination = torch.cuda.current_device() </ins> input_size = inputs[0].size() <ins> nccl_root = None </ins> for i, inp in enumerate(inputs): assert inp.is_cuda, "reduce_add expects all inputs to be on GPUs" <ins> if inp.get_device() == destination: nccl_root = i </ins> if inp.size() != input_size: got = 'x'.join(str(x) for x in inp.size()) expected = 'x'.join(str(x) for x in input_size) raise ValueError("input {} has invalid size: got {}, but expected " "{}".format(i, got, expected)) <del> if destination is None: destination = torch.cuda.current_device() </del> <ins> assert nccl_root is not None, "reduce_add expects destination to be on the same GPU with one of the tensors" </ins> with torch.cuda.device(destination): result = type(inp)(input_size).zero_() if nccl.is_available(inputs) and inputs[0].get_device() == destination: outputs = [result] + [t.new(t.size()) for t in inputs[1:]] <del> nccl.reduce(inputs, outputs, root=destination) </del> <ins> nccl.reduce(inputs, outputs, root=nccl_root) </ins> return result for inp in inputs:

The loss will never decrease when I train a model using multi-gpus but without gpu0. This is an example I modified from the : If the result after one epoch is If the result after one epoch is
should always contain . If you want to specify which GPUs to use, you can set the environment variable , either by running your script like this: or set it programmatically in your script like this: So that deviceids 0, 1 will map to GPUs 1, 2, respectively. (Note that for the second method, you should put that line before any cuda relevant function call.)
Thank you.
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: :members: <ins> Padding Layers -------------- :hidden:`ReflectionPad2d` ~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReflectionPad2d :members: :hidden:`ReplicationPad2d` ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReplicationPad2d :members: :hidden:`ReplicationPad3d` ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReplicationPad3d :members: :hidden:`ZeroPad2d` ~~~~~~~~~~~~~~~~~~~ .. autoclass:: ZeroPad2d :members: :hidden:`ConstantPad2d` ~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ConstantPad2d :members: </ins> Non-linear Activations ----------------------------------

After merging cuda 12.4 for nightly. I observe following failures: libtorch-cuda124-shared-with-deps-cxx11-abi-build: Build Official Docker Images 124: We need to resolve these before continuing adding CUDA 12.4 CI workflows 0 cc
attempt 3 worked for libtorch build issue:
For Docker image, related discussion:
I will instance a new bug for the docker job failure.

Is there a way to use reflection padding instead of zero padding with nn.Conv2d ? I see there is a but the option is not in nn.Conv2d. Thanks,
are in , but they're not exposed in the docs.
Thanks!


Question: self.assertEqual(inp_i.grad, i.grad) assert not hasattr(TestAutograd, test_name), 'Two tests have the same name: ' + test_name <ins> for skip in skipTestIf: do_test = skip(do_test) </ins> setattr(TestAutograd, test_name, do_test)

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: return r; } <del> variable_list flatten(const Tensor& x, const TensorList& y, const Tensor& z) { </del> <ins> static variable_list flatten(const Tensor& x, const TensorList& y, const Tensor& z) { </ins> std::vector<Variable> r; r.reserve(2 + y.size()); r.emplace_back(x);

See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:


Question: arguments(), is_variadic(is_variadic), has_out(has_out) {}; Option(const Option&) = delete; Option(Option&& other): <del> arguments(std::move(other.arguments)), is_variadic(other.is_variadic) {}; </del> <ins> arguments(std::move(other.arguments)), is_variadic(other.is_variadic), has_out(other.has_out) {}; </ins> std::vector<Argument> arguments; bool is_variadic;

! Maybe we should change it to `? No response cc
are these old installation commands also auto-tested with CI?
Looks no longer relevant.


Question: auto m = self.size(0); auto n = self.size(1); auto k = sigma.size(0); <ins> auto gsigma = grads[1]; auto u = raw_u; auto v = raw_v; auto gu = grads[0]; auto gv = grads[2]; </ins> <del> Tensor u, v; </del> if (!some) { <del> // ignore the free subspace </del> <ins> // We ignore the free subspace here because possible base vectors cancel // each other, e.g., both -v and +v are valid base for a dimension. // Don't assume behavior of any particular implementation of svd. </ins> u = raw_u.narrow(1, 0, k); v = raw_v.narrow(1, 0, k); <del> } else { u = raw_u; v = raw_v; </del> <ins> if (gu.defined()) { gu = gu.narrow(1, 0, k); } if (gv.defined()) { gv = gv.narrow(1, 0, k); } </ins> } <del> auto gu = grads[0]; auto gsigma = grads[1]; auto gv = grads[2]; </del> auto vt = v.t(); Tensor sigma_term;

I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.


Question: raise ValueError("output_size should be a sequence containing " "2 or 4 elements, but it has a length of {}".format( len(output_size))) <del> out_sizew, out_sizeh = output_size sizew = ((input.size(3) - 1) * self.dw - 2 * self.padw + self.kw) </del> <ins> out_sizeh, out_sizew = output_size </ins> sizeh = ((input.size(2) - 1) * self.dh - 2 * self.padh + self.kh) <del> out_padw = out_sizew - sizew </del> <ins> sizew = ((input.size(3) - 1) * self.dw - 2 * self.padw + self.kw) </ins> out_padh = out_sizeh - sizeh <del> out_padw_ok = 0 <= out_padw < self.dw </del> <ins> out_padw = out_sizew - sizew </ins> out_padh_ok = 0 <= out_padh < self.dh <del> if not out_padw_ok or not out_padh_ok: </del> <ins> out_padw_ok = 0 <= out_padw < self.dw if not out_padh_ok or not out_padw_ok: </ins> raise ValueError(("requested an output size of {}x{}, but " "valid sizes range from {}x{} to {}x{} (for an input of " "{}x{})").format(out_sizeh, out_sizew, sizeh, sizew,

Repro is torchtrain w/ it happens more often if in my experience but it still happens either way. Logically its clear we have a race: rank0 is free to exit and shut down TCPStore before or after other ranks shut themselves down. we have to consider either a) rank 0 waits for all other ranks to exit before it exists b) we make other ranks tolerant of tcpstore going away (A) has potential downsides in efficiency even in the best case. But also practically, we could use tcpstore barrier, but it wont work reliably at large scales. we'd have to consider something like 'scalable coordinator' to go this route (B) we could probably implement w/ a try/catch in the heartbeat monitor for the connection error, similar to how we catch cuda driver exit. This feels a little hacky. Its important we somehow distinguish between the cases when we are actually shutting down and want to swallow the error, from cases where the is a tcpstore/network error during runtime and we actually want to raise the error. This points to (a) as a more correct direction. cc
do you have any update on "make torchrun share TCPStore instance with pytorch"? I think that is our "proper" fix for this problem, and in the meantime people can work around it by 1) sleeping on rank0 for a sufficiently long time (between 5-10s seemed plenty for 128rank jobs) simplest possible to not sleep long enough in edge cases 2) using a barrier at job shutdown such that rank0 (and tcsptore) are gauranteed alive.
was landed as a fix


Question: self.assertIn('buf', l.state_dict()) self.assertIs(l.state_dict()['buf'], buf) <ins> def test_Conv2d_inconsistent_types(self): inputs = Variable(torch.randn(4, 1, 7, 7).float()) weights = Variable(torch.randn(1, 1, 3, 3).double()) # inconsistent types should raise an exception self.assertRaises(RuntimeError, lambda: nn.functional.conv2d(inputs, weights)) # but it should work with the same type nn.functional.conv2d(inputs.float(), weights.float()) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_Conv2d_large_workspace(self): # These sizes require huge cuDNN workspaces. Make sure we choose a

the error only happens in very specific conditions. (change the view of input tensor and use weight converted from numpy array.) The output of functional conv2d is far off in scale (e12 or so.)
this happens because generates double precision tensor by default, whereas torch generates single precision. We need to add guards for tensor types in THNN and THCUNN.
It's a problem with function, that only . Should be a simple fix.
Whoops! Yeah it just ignores the "thpp::Type type" parameter
what type is expected here?
a is a FloatTensor but weights is a DoubleTensor. Change: to: (or change a to (...).double())
cool thanks!


Question: THArgCheckWithCleanup(n_sample > 0, THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, <del> "cannot sample n_sample < 0 samples"); </del> <ins> "cannot sample n_sample <= 0 samples"); </ins> if (!with_replacement) {

Looking at the values returned from tn.multinomial(1) and counting them, it seems like it returns the values of the multinomial distribution taken from t, i.e. that it has ignored the signs of the values. However, it seem like always returns a tensor filled with zeros.
looking into this


Question: def in_cooldown(self): return self.cooldown_counter > 0 <del> def _init_is_better(self, mode, threshold, threshold_mode): if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') </del> <ins> def _cmp(self, mode, threshold_mode, threshold, a, best): </ins> if mode == 'min' and threshold_mode == 'rel': rel_epsilon = 1. - threshold <del> self.is_better = lambda a, best: a < best * rel_epsilon self.mode_worse = float('Inf') </del> <ins> return a < best * rel_epsilon </ins> elif mode == 'min' and threshold_mode == 'abs': <del> self.is_better = lambda a, best: a < best - threshold self.mode_worse = float('Inf') </del> <ins> return a < best - threshold </ins> elif mode == 'max' and threshold_mode == 'rel': rel_epsilon = threshold + 1. <del> self.is_better = lambda a, best: a > best * rel_epsilon self.mode_worse = -float('Inf') </del> <ins> return a > best * rel_epsilon </ins> else:  # mode == 'max' and epsilon_mode == 'abs': <del> self.is_better = lambda a, best: a > best + threshold self.mode_worse = -float('Inf') </del> <ins> return a > best + threshold def _init_is_better(self, mode, threshold, threshold_mode): if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') if mode == 'min': self.mode_worse = float('inf') else:  # mode == 'max': self.mode_worse = (-float('inf')) self.is_better = partial(self._cmp, mode, threshold_mode, threshold) </ins>

I'm working on checkpointing in our nn trainer. When saving the state objects, I get an error: PicklingError: Can't pickle <type 'function': attribute lookup builtin.function failed I tracked it down to trying to save the scheduler ReduceLROnPlateau. I think the issue is due to the scheduler referencing the optimizer, which has to be saved with statedict(). I presume the same error will occur with other schedulers too. It seems like the problem could be solved by providing a statedict() and loadstatedict() in the scheduler classes. pytorch master rev (Jan 22) installed from source x86_64 on centos 7.4 cuda 9.0 cudnn 7 k80 gcc 5.4 python 2.7.14
The following script reproduces the crash: import torch import from import ReduceLROnPlateau if name == 'main': model = (10,10) optim = (model.parameters()) sched = ReduceLROnPlateau(optim, mode='max', factor=0.5) with open('','wb') as f: # Works # ((model.statedict(), optim.statedict()), f) # Crashes ((model.statedict(), optim.state_dict(), sched), f)


Question: check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) check_file(os.path.join('aten', 'src', 'ATen', 'cpu', 'cpuinfo', 'CMakeLists.txt')) check_file(os.path.join('aten', 'src', 'ATen', 'cpu', 'tbb', 'tbb_remote', 'Makefile')) <ins> check_file(os.path.join('aten', 'src', 'ATen', 'utils', 'catch', 'CMakeLists.txt')) </ins> check_pydep('yaml', 'pyyaml') check_pydep('typing', 'typing')

Trying to build master (), I'm seeing I tried in to no avail.
NM, also needed a .
Need a check to see if this is folder is checked out in , like the other submodules. Should be a simple patch.


Question: r__->stride[(transpose_r == 'n' ? 1 : 0)]); /* free intermediate variables */ <del> if(m1_ != m1) </del> <ins> if(free_m1) </ins> THTensor_(free)(m1_); <del> if(m2_ != m2) </del> <ins> if(free_m2) </ins> THTensor_(free)(m2_); if(r__ != r_)

Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.


Question: #   method name, #   input size/constructing fn, #   args (tuple represents shape of a tensor arg), <del> #   test variant name (will be used at test name suffix),  // optional #   indices for possible dim arg,                          // optional #   output indices that should be gradcheck'ed,            // optional </del> <ins> #   test variant name (will be used at test name suffix),    // optional #   indices for possible dim arg,                            // optional #   fn mapping output to part that should be gradcheck'ed,   // optional </ins> # ) method_tests = [ ('add', (S, S, S), ((S, S, S),)),

I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.


Question: TensorTriOp(T *start_, int64_t stride0_, int64_t stride1_, int64_t k_) : start(start_), stride0(stride0_), stride1(stride1_), k(k_) {} <del> __device__ __forceinline__ int mask(T *in) { ptrdiff_t n = in - start; </del> <ins> __device__ __forceinline__ int mask(T *out) { ptrdiff_t n = out - start; </ins> int64_t row, col; if (stride0 > stride1) {

On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)


Question: LDFLAGS="-L$INSTALL_DIR/lib " LD_POSTFIX=".so.1" LD_POSTFIX_UNVERSIONED=".so" <del> if [[ $(uname) == 'Darwin' ]]; then </del> <ins> if [[ $(uname) == 'Darwin' ]]; then </ins> LDFLAGS="$LDFLAGS -Qunused-arguments -Wl,-rpath,@loader_path" LD_POSTFIX=".1.dylib" LD_POSTFIX_UNVERSIONED=".dylib"

there is a bug. the code is from The code for installing cuda10.2 should be corrected to The code for installing cuda10.2 should be corrected to cc
Thank you for reporting the typo

This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?


Question: return t(*sizes).copy_(torch.randn(*sizes)) return tmp <ins> # Content of each tuple: # - function name # - constructor for the tensor,    signature: fn(tensor_type) -> tensor # - constructor for the arguments, signature: fn(tensor_type) -> list # - postfix name for the test (must be unique for a given function) (default='') # - tensor types to use (default=types) # - disable inplace test, if set to True, no inplace test will be done (default=False) </ins> tests = [ ('add', small_3d, lambda t: [number(3.14, 3, t)]), ('add', small_3d, lambda t: [small_3d_positive(t)], 'tensor'),

On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)


Question: } void THCTensor_(sparseMask)(THCState *state, THCSTensor *r_, THCTensor *t, THCSTensor *mask) { <ins> THArgCheck(mask->coalesced, 2, "mask is uncoalesced"); </ins> THCAssertSameGPU(THCSTensor_(checkGPU)(state, 2, 3, r_, mask, t)); if(!THCSTensor_(isSameSizeAsDense)(state, mask, t)) { THError("sparseMask operands have incompatible sizes");

and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.


Question: (Addcdiv,       (),                 ((S, S), (S, S), torch.rand(S, S) + 1e-2)   ), (Addcdiv,       (0.6,),             ((S, S), (S, S), torch.rand(S, S) + 1e-2), 'scale'), (IndexAdd,      (0,),               ((S, S), index_variable(2, S), (2, S))      ), <del> (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </del> <ins> # (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </ins> (IndexFill,     (0, 2),             ((S, S), index_variable(2, S))              ), (IndexSelect,   (0,),               ((S, S), index_variable(2, S))              ), (Gather,        (0,),               ((M, S), gather_variable((S, S), 1, M))     ),

This happens when the index tensor contains duplicate elements. If this is not allowed indexcopy should raise an exception when this happens and we should fix the test. Otherwise we need to fix the autograd op. Here's an example: cc
Why would you with repeated indices? Sounds for me like UB is a reasonable. We should check that in C
I ran into this in and fixed the test for now. I'm not sure how to interpret last comment: did you mean UB behavior is "unreasonable" rather than "a reasonable"? The last sentence reads that way.
I'd be ok with saying that it's UB in such case. There's no natural choice, and it can decrease performance.
Is this issue still worth working on? If adding the additional check can decrease performance, can we add another argument specifying that indexes contain duplicate elements?


Question: #include <Python.h> #include "THP.h" <ins> #include "torch/csrc/utils/auto_gil.h" </ins> // Adapted from fblualib void* ObjectPtrAllocator::malloc(ptrdiff_t size) {

When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1


Question: #include <vector> #include <string> <ins> #include "torch/csrc/utils/object_ptr.h" </ins> #define THPUtils_(NAME) TH_CONCAT_4(THP,Real,Utils_,NAME) #define THPUtils_typename(obj) (Py_TYPE(obj)->tp_name)

Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc


Question: name: binary_linux_manywheel_3_7m_cu102_devtoolset7_test requires: - binary_linux_manywheel_3_7m_cu102_devtoolset7_build <del> resource_class: gpu.medium </del> <ins> resource_class: gpu.nvidia.small </ins> use_cuda_docker_runtime: "1" - binary_linux_test: build_environment: libtorch 3.7m cpu devtoolset7

As it still using deprecated resource class, see See example of the failure in
Followup: we need to plan better --we knew it was gonna happen and none of us fixed it early enough. Action item: go through tasks about deprecation in sprint planning


Question: optimizer (Optimizer): Wrapped optimizer. step_size (int): Period of learning rate decay. gamma (float): Multiplicative factor of learning rate decay. <del> Default: -0.1. </del> <ins> Default: 0.1. </ins> last_epoch (int): The index of last epoch. Default: -1. Example:

as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.


Question: from .serialization import save, load <ins> from ._tensor_str import set_printoptions </ins> ################################################################################ # Define Storage and Tensor classes

It seems that the following link that can be reached from here (check the screenshot) is broken: Page: Broken link: Screenshot: ! cc
this should be transferred to , but I don't have permissions to do that. Fix is available at pytorch/hub.


Question: "torch/csrc/Generator.cpp", "torch/csrc/Tensor.cpp", "torch/csrc/Storage.cpp", <ins> "torch/csrc/byte_order.cpp", </ins> "torch/csrc/utils.cpp", "torch/csrc/allocators.cpp", "torch/csrc/serialization.cpp",

<!-- A clear and concise description of what the bug is. --The target name is used by a test executable in mkldnn. Steps to reproduce the behavior: BUILDTEST=1 build <!-- If you have a code sample, error messages, stack traces, please provide it here as well --<!-- A clear and concise description of what you expected to happen. --CMake configure passes. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): v1.5.0-rc2 OS (e.g., Linux): Windows How you installed PyTorch (, , source): source Build command you used (if compiling from source): python build Python version: 3.x CUDA/cuDNN version: 9.2 / 10.1 / 10.2 / NA GPU models and configuration: Tesla M60 Any other relevant information: <!-- Add any other context about the problem here. --We should not build tests for mkldnn.
I have encountered the same error, how do I resolve it?
It is actually mkldnn that is out of sync. So running the following script should help.
Thanks a lot for this I posted this on the PyTorch Discussions but to no avail, and now I've finally understood the reason of this error. Thanks a lot. Surprisingly, for me the build works out if I build from the master repo, but changing to a different version always fails.


Question: grouped_args += [group(t) for t in args] res.append(impl[fn_name](self, self._bufs[g], *grouped_args)) if fn_name == 'grad_params': <del> return [torch.cat(t, 0) for t in zip(*res)] </del> <ins> return [torch.cat(t, 0) if t[0] is not None else None for t in zip(*res)] </ins> else: return torch.cat(res, 1)

x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.


Question: self.assertEqual(counter[0], 1, 'bw_hook not called') self.assertEqual(x.grad.data, torch.ones(5, 5) * 2) <del> @unittest.skipIf(sys.version_info[0] == 2, "Python 2 doesn't collect cycles involving __del__") def test_hooks_cycle(self): import gc counter = [0] class GradHook(object): def __init__(self, var): self.var = var def __del__(self): counter[0] += 1 def __call__(self, *args): pass def run_test(): x = Variable(torch.ones(5, 5), requires_grad=True) y = x * 2 x.register_hook(GradHook(x)) y.register_hook(GradHook(y)) y._backward_hooks[1] = GradHook(y) run_test() gc.collect() self.assertEqual(counter[0], 3) </del> def test_hook_none(self): # WARNING: this is a test for autograd internals. # You should never have to use such things in your code.

My code runs well for a while , it occurs randomly..... Traceback (most recent call last): File "", line 236, in <moduletrainvgg("",dset='luna16') File "", line 54, in trainvgg lossvalue.backward() File "/home/uih/anaconda2/lib/python2.7/site-", line 169, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/home/uih/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion "pos = 0 && pos < ()" failed. Any idea what's happend? Thanks in advance~
Yes, it looks like we have a bug somewhere in the code. Can you please try to isolate a small snippet that still triggers the same error so we can look into it? Thank you!
It's hard for me to isolate, the error occurs when I do the train and validation in turn, if I just do either the train or val, everthing runs well.
The problem is that we won’t be able to fix it, until we actually see under what conditions does the error happen
I got the same error in v0.3.0 today. when I trained my model with single GPU, it worked perfectly;with multi-gpu the error occured. RuntimeError: add: Assertion failed.
We've also been struggling with this issues for a long time. Unfortunately, finding which part of our code specifically is triggering the assert (which used to be a segfault before ) has proven extremely hard and, as of now, unsuccessful. In fact, we never submitted a bug report as we couldn't create a self-contained piece of code to trigger it. I'll try to summarize all the information we've gathered: We know for a fact that the issue appeared sometime after this commit: . All our code works perfectly fine on or v0.2. The assert is always triggered during the backward pass, but the exact moment when it happens seems to be completely random: when training over Imagenet we've seen this after 50 iterations as well as after 5000.

This makes it super-hard to understand whether a particular piece of code is triggering the assert or not, as you can never be sure how many iterations are enough to exclude that the assert will ever trigger! The issue seems to be somehow related to the use of multiple GPUs (via ): we've never observed it while training with a single GPU. The issue seems also to be related to the use of user-defined classes: we've never seen it when training networks composed of "standard" Pytorch functions / modules only... ... however, simply running forward / backward passes with our custom functions doesn't seem to be enough to trigger the assertion. In our experience, working with "complex" computational graphs increases the likelihood of encountering this issue. Recently we've been able to re-create the issue using and a modified version of , where we replace some of the layers with our custom-made ones. Unfortunately I can't share the code for now (our paper describing the new layers is still not out), but I'll come back to you as soon as I can! Some additional information: all our experiments are on Ubuntu 16.04 using CUDA 8, CUDNN 7 and TITAN X and Xp GPUs.
almost the same condition as us
Encounter the same condition after serveral imagenet epochs on a NasNet after update to 3.0. Pytorch 2.0 is working well.
I am seeing this too -- I have two runs of the same model running with different hyperparameters, one is running smoothly, this one quit with the exact same error avgloss = self.trainepoch(trainingdata,validationdata,batchsize,saveprefix,epoch) File "", line 759, in trainepoch gradients = loss.backward() File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 167, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion failed. Running PyTorch v0.3 and training on 4 K-80 GPUs. My other run with different parameters is also running on the exact same setup but its still running.

Both runs differ only in shuffle of data and a different batch size (the one that failed had a slightly larger batch size)
: I'm not using any user define autograd.Function but I see the error too, so maybe thats not the primary place with a bug. Using nn.DataParallel though.
I'll try again to train a network using standard layers only, maybe I'll also encounter the error in the no-custom-functions case if I let it run a little bit longer. If this is the case, then I guess we have some strong indication that (or multi-GPU computation in general) could indeed be the source of the error!
: I dont think the duration has got anything to do with it --- One model run has been running without a problem for the last 18 hours, the other one had quit last night pretty early on (Crashed after the backward pass of the 8th batch). I have just restarted it and watching it.
I agree that the bug is probably not related to training length. My guess is that each backward pass has a certain probability of randomly failing with this assert, so running for longer just increases the overall chances of encountering the bug.
Thanks for the information. We'd really like to fix this bug, and so there are a few things that would help us out a lot (in order of difficulty): you run your training process with gdb attached with a breakpoint, so that we can get a backtrace when the assertion fails? Also, if you can go one frame up and print the name of the function (run in gdb) that would also be helpful. We basically want to know if it's either a C++ function or a Python function. soon as you can publish your latest layers, uploading a script that reproduces (even if it takes a while to reproduce); we would be happy to run the script to try to reproduce, which would be very helpful for us. We'd love a bisection, but given that it can sometimes take 18hrs+ to repro, it doesn't sound like that would be easy to do.
This is the backtrace: For the function name I'll probably have to recompile pytorch in debug mode, since in gdb returns . Edit - I managed to get the name of the function by catching the exception raised by the assert: . Edit 2 - I did a second run and the name of the function is still the same.

Actually I just realized that what we really need is Can you please try printing that (might require a debug build as well)?
Printing instead of I get , which is the name of our custom function's backward pass. More importantly, our code and paper are now public so I can share this with you: As I mentioned in a previous comment, this code is just the Pytorch ImageNet + ResNet examples, except that standard BN is replaced with our custom layer. When running this on our machines with 4 GPUs we always encounter the assert within the first epoch. Please feel free to ask if you need help running the code.
I also encountered this assert. I have two machines, one with 2, and other one with 4 GPUs. Both machines are Ubuntu 16.04, python 2.7, cuda 9.0, cudnn 7, and pytorch v0.3.0. I never encountered this assert on my 2 GPUs machine. This assert always appeared on my 4 GPUs machine.
same as here. I have python 2.7, cuda 8.0, cudnn 7 and pytorch v0.3.0 installed from binaries. The assertion was triggered when I was training networks on 4 GPUs. It happened at a random time during training.
Same here, with multiple pixel-wise prediction models with just CNNs. Seems to occur only with multiple GPUs, has not happened so far with single GPU. python 2.7, CUDA 8.0, CUDNN 7, PyTorch v0.3.0.
can you take a look at this asap
edit: resolved, I ended up adding a compilation flag for compute capacity 3.5 to pytorchbugreport's building scripts (to work with my gpus) I'm getting the following after trying to run pytorchbugreport. Have you seen this before? !
you have to change to cover your GPU, which is compute35,code=sm_35 i think (if it is K40)
I ran your code for six epochs on a machine with 4 gpus on pytorch 0.3.0 but haven't been able to trigger the assert. I'll try letting it run for longer and see if anything pops up
For us the assert almost always triggers during the first epoch.

There might be some important difference in our setups. This is the exact sequence of steps we have used to reproduce the error across several machines: from a clean Ubuntu 16.04 installation with CUDA 8.0, CUDNN 7.0, pip, virtualenv and the compilers. and activate a new python2 virtualenv. necessary python packages . the pytorch repository and checkout and install with our repository and compile our native module with the training as described before. Our machines all have either Titan X or Titan Xp GPUs, so a different architecture from the one you are using, I don't know if this could have an influence on the error.
My machine has 4 GTX 1080 Ti which triggers the error.
we (finally) found a different repro that triggers this issue internally on one of our machines, a fix will be issued soon.
Quick update: we're still looking into this, but so far, I've been able to get this bug to trigger with two machines using python 2.7 but never with python 3. If you're in need of a quick fix using python 3 might help.
I also encountered this propblem, with pytorch 0.3.0, python2.7, CUDA8.0, Titan X GPUs. My session always crashed around 8 epochs. And I degrade pytorch to 0.2.0. now the training seems work well. Someone in emergency can try this.
Thank you , I'll try to switch to python 3 for now!
let me know if the bug comes back in python 3. As far as I can tell the python version shouldn't matter in the triggering of this bug... but it's never happened to me with python 3 yet.
Crashed in Python2.7 RuntimeError: add: Assertion failed Absolutely random crash using multi gpu.
This has been fixed in HEAD but there has not been a release of 0.3 with the fix yet.
Oh, Thanks for your reply. I will try the HEAD version. Thank you .
i have encountered this error , but change the pytorch to the latest version(0.4) will solve it.
Encountered same error when I add dropout layer to a Resnet which I modified. Trained on multi-GPU. Does latest conda release fix the problem?
it is not yet on latest conda release.

to get the fix at the moment you will have to install pytorch from source:
The same bug ! !
did you install pytorch from source?
Is there a simple way to block this error without changingthe version of python or pyotrch?
You could disable the python reference cycle . This would probably lead to memory leaks though.
It should be safe if you do this once all the graphs are deleted, and there will be no leaks unless you create cycles in your data structures
After 2 days of struggle, I finally changed back to 0.2.0 and expects the release of binary 0.4.0. Anyway, thank you for your advices. :)
I encountered the same error with version 0.3.0.post4. What was the proper fix here?
upgrade to version 0.3.1 or newer.


Question: if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: <del> raise ValueError('threshold mode ' + mode + ' is unknown!') </del> <ins> raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') </ins> if mode == 'min' and threshold_mode == 'rel': rel_epsilon = 1. - threshold self.is_better = lambda a, best: a < best * rel_epsilon

Trivial report, unfortunately I can't make a PR at the moment, but I can submit a report. line 329 in should replace with


Question: .. automethod:: log2 .. automethod:: log2_ .. automethod:: log_normal_ <ins> .. automethod:: logsumexp </ins> .. automethod:: long .. automethod:: lt .. automethod:: lt_

Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!


Question: ] extra_compile_args += ['-DWITH_DISTRIBUTED_MW'] include_dirs += [tmp_install_path + "/include/THD"] <del> main_link_args += [THD_LIB] </del> <ins> main_link_args += [THD_LIB, THPP_LIB] </ins> if platform.system() == 'Linux': main_link_args += [GLOO_LIB]

It seems that the shared object file is not being built correctly and this is leading to build errors on Travis.
this is weird, i saw this intermittently on travis as well. But couldn't reproduce on subsequent builds on Travis or locally.
I ran a clean build on the latest commit locally just now and I can't reproduce it either. Is it that ccache is preserving some build files and those are interfering with subsequent builds?
Actually, I apologize, I am able to reproduce. Importing torch in an ipython shell gives the same issue.
I meet the same problem. I am using the newest version "commit ". And I checked all dependent libraries of , all of them are available. Any advices are welcome. Much thanks.
I have reproduced something similar. It seems to only happen when you do . EDIT: No, my problem was actually that I was running Python inside my PyTorch checkout.
It also happens when you do
Varun, i cant reproduce this. What's your OS version and python version?
I have met the same problem as is mentioned above. I'm using Ubuntu 16.04.3 and Anaconda python 2.7.13.
Ubuntu 16.04.2 Python 3.6.2 Cuda 8.0
one last thing, what's your gcc version, and did you do python clean, or clone a fresh new folder
I did a , but am running it again to make sure.
On definitively running , I am not getting the error.
thanks everyone, should be fixed on master now.


Question: num_devices = 0; } #endif <del> ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_devices + 1); for (auto& queue : ready_queues) { </del> <ins> int num_threads = num_devices + 1; ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_threads); for (int i = 0; i < num_threads; ++i) { auto& queue = ready_queues[i]; </ins> queue.reset(new ReadyQueue()); <del> std::thread t(&Engine::thread_main, this, queue); </del> <ins> std::thread t(&Engine::thread_main, this, queue, i - 1); </ins> t.detach(); } }

Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: return make_tensor(t, M, M) <ins> def medium_2d_expanded(t): return t(1).expand(M, M) </ins> def medium_2d_scaled(t, scale=10): return make_tensor(t, M, M).mul(scale)

On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)


Question: ctx.keepdim = False if keepdim is None else keepdim if dim is None: <del> ctx.norm = input.norm(p) ctx.save_for_backward(input) return input.new((ctx.norm,)) </del> <ins> norm = input.norm(p) output = input.new((norm,)) </ins> else: if keepdim is not None: output = input.norm(p, dim, keepdim=keepdim) else: output = input.norm(p, dim) <del> ctx.save_for_backward(input, output) return output </del> <ins> ctx.save_for_backward(input, output) return output </ins> @staticmethod def backward(ctx, grad_output): <del> if ctx.dim is None: input, = ctx.saved_variables if ctx.p == 2: scale_v = (grad_output / ctx.norm).expand_as(input) return input.mul(scale_v), None, None, None else: pow = input.abs().pow(ctx.p - 2) scale_v = (grad_output / ctx.norm ** (ctx.p - 1)).expand_as(input) return input.mul(pow).mul(scale_v), None, None, None </del> <ins> input, output = ctx.saved_variables if ctx.dim is not None and ctx.keepdim is False and input.dim() != 1: grad_output = grad_output.unsqueeze(ctx.dim) output = output.unsqueeze(ctx.dim) if ctx.p == 2: grad_input = input.mul(grad_output).div(output) </ins> else: <del> input, output = ctx.saved_variables </del> <ins> input_pow = input.abs().pow(ctx.p - 2) output_pow = output.pow(ctx.p - 1) grad_input = input.mul(input_pow).mul(grad_output).div(output_pow) </ins> <del> if ctx.keepdim is False and input.dim() != 1: grad_output = grad_output.unsqueeze(ctx.dim) output = output.unsqueeze(ctx.dim) </del> <ins> # Special case at 0 where we return a subgradient containing 0 grad_input.masked_fill_(output == 0, 0) </ins> <del> big_grad_output = grad_output.expand_as(input) if ctx.p == 2: big_output = output.expand_as(input) return input.mul(big_grad_output).div(big_output), None, None, None else: pow = input.abs().pow(ctx.p - 2) big_output = output.pow(ctx.p - 1).expand_as(input) return input.mul(pow).mul(big_grad_output).div(big_output), None, None, None </del> <ins> return grad_input, None, None, None </ins> # TODO: renorm

If a norm is zero, its gradient returns nan: Obviously just happening because the gradient divides by the norm, but the (sub)gradient here should probably be zero, or at least not nan, since that will propagate to make all updates nan. Probably low priority, as it's not going to be an issue in 99% of cases, but we're doing a few things with (exact) line searches where this caused a nan to appear, breaking everything downstream.
I'm encountering exactly the same issue! Spent hours on debugging, just to find PyTorch has a bug on such basic thing.
+1 just found this bug too
+1 for this bug. Temporarily changing my code to something like the following for the sake of debugging. x = Variable((1), requires_grad=True) y = x + 1e-16 y.norm().backward() print x.grad
The thing is that in the 2 norm, there is a square root, which has a gradient of at 0. The gradient gives you because you then multiply 0 and an infinity during the backward pass.
For a scalar, norm 2 is basically abs. But x.abs().backward() gives you 0 gradient. In this sense, it's not coherent.
I found this error, too
Alban fixed this behavior in
Hi, the norm function can give use the 0 gradients now. However, the following code still has the nan gradient problem
Ho The square root has no gradient at 0. This is expected behavior.
Hi, but the sub-gradient of the square root should be zero? Also, y = ( x * x ) should equal to x.norm(), why they have different gradient ( 0 and nan )?
I think was right. The left-side derivative of sqrt(x) at x=0 is undefined, so it doesn't even have a subgradient at x=0.
square root has no subgradient at 0. You could define a gradient by continuity but then it would be ... Given that pytorch is using autograd, and (equivalent to your ) are completely different: The first one is a single function that is convex and defined on R, it has a subgradient of 0 at 0. The second one is composed of two function, the first function is the square function which is differentiable and outputs values in .

The second function is the square root that is not convex and even though it is defined on , it is only differentiable on and it's gradient in 0 in undefined. Given that, even though and will return the same value, their gradients may differ at points where it is not differentiable, this is because automatic differentiation looks at each step of the computation one by one and even though in some cases a subgradient exist (because we look at multiple operations as a single function), it is not always the case and the gradient remains undefined.
I think math is math. Any root's gradient at zero is either inf or undefined. This issue shall be handled by the users' themselves by adding a small value(as did), but an error(warning) message may be helpful since it is pretty hard to debug. Say: Infinite/Undefined gradient is detected at XfunctionX at line Y. Exit.
Agree, norm is not differentiable at 0 , the bandaid that Alban put there in is wrong (even in the limit sense the gradient at 0 should be 1 not 0), but it should not have been there at all. Norm is norm, if someone want to add epsilons to their norms (like batchrnorm, e.g.) they are welcome to do so in the user code. What would numpy do?
I've also run into a number of problems related to the change introduced in . Is there a reason the subgradient is set to 0, rather than the 1? (The limit as norm-0?) As a minimal example: Produces
Well any value between [-1, 1] is a valid subgradient for the 2-norm. More genereally, any vector in the 1 ball for the dual norm is a valid subgradient. This means that 0 is always going to be a subgradient, while 1 will not be for all p. Anyway, the theory says that any of them could be taken and subgradient descent will work. I'm sure that depending on the application, one will be better than the other. For example, the relu function will also give a 0 subgradient at 0, you could have given 1. The main point here was to remove nans that make your network give nan for everything which is not convenient.


Question: else: sizes = torch.LongStorage(args) sizes = _infer_sizes(sizes, self.nelement()) <del> if reduce(lambda a,b: a * b, sizes) != self.nelement(): raise RuntimeError('Invalid size for view. Input size: ' + 'x'.join(map(lambda v: str(v), self.size())) + ', output size: ' + 'x'.join(map(lambda v: str(v), sizes)) + '.') assert self.is_contiguous(), "expecting a contiguous tensor" dst.set_(self.storage(), self.storage_offset(), sizes) </del> <ins> numel = reduce(lambda a,b: a * b, sizes) if len(sizes) > 0 else 0 if numel != self.nelement(): def format_size(size): return 'x'.join(str(v) for v in size) if len(size) > 0 else '0' raise ValueError( "view of size '{0}' is invalid for input of size '{1}'" .format(format_size(sizes), format_size(self.size()))) if not self.is_contiguous(): raise ValueError("input should be contiguous") if self.storage() is not None: dst.set_(self.storage(), self.storage_offset(), sizes) </ins> return dst def view_as(self, tensor):

Please check and fix all the PyTorch web links format from to , like in the official site.
closed via


Question: -DNO_NNPACK=%NO_NNPACK% ^ -DCUDNN_INCLUDE_DIR="%CUDNN_INCLUDE_DIR%" ^ -DCUDNN_LIB_DIR="%CUDNN_LIB_DIR%" ^ <del> -DCMAKE_BUILD_TYPE=Release </del> <ins> -DCMAKE_BUILD_TYPE=%BUILD_TYPE% </ins> %MAKE_COMMAND% cd ../..

The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.


Question: for i, (sample, target) in enumerate(loader): idx = i * batch_size self.assertEqual(sample, self.data[idx:idx + batch_size]) <del> self.assertEqual(target, self.labels[idx:idx + batch_size].view(-1, 1)) </del> <ins> self.assertEqual(target, self.labels[idx:idx + batch_size]) </ins> self.assertEqual(i, math.floor((len(self.dataset) - 1) / batch_size)) def _test_shuffle(self, loader):

TensorDataset adds casts 1D target tensor to 2D which is not acceptable by NLL loss layer.
Fixed in .


Question: CUDNN_LIB_DIR = path break else: <del> libraries = sorted(glob.glob(os.path.join(path, 'libcudnn*'))) </del> <ins> libraries = sorted(glob.glob(os.path.join(path, 'libcudnn*' + str(CUDNN_INCLUDE_VERSION) + "*"))) </ins> if libraries: CUDNN_LIBRARY = libraries[0] CUDNN_LIB_DIR = path break <del> for path in include_paths: if path is None or not os.path.exists(path): continue if os.path.exists((os.path.join(path, 'cudnn.h'))): CUDNN_INCLUDE_DIR = path break </del> # Specifying the library directly will overwrite the lib directory library = os.getenv('CUDNN_LIBRARY') if library is not None and os.path.exists(library):

Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/abc/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/abc/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/abc/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 216, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 123, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 65, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 82, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110 How to solve the problem?
could you pull the latest master branch and recompile. I pushed a change yesterday that fixes this:
I remove the old code and pull new code, then compile, install.It still fails. /usr/local/lib/python2.7/dist- UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.

('No training configuration found in save file: ' Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/qianbb/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/qianbb/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/qianbb/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 216, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 100, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 66, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 87, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110 The library of cuda is shown as following screenshot. !
I found the issue, I fixed it in which we will merge shortly.
ThanksDoes it have been merged?
yes. it is merged.
It seems the fix doesn't work.

The log is shown as following Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/qianbb/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/qianbb/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/qianbb/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 222, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 100, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 66, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 87, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110
that is weird. Can I see the full log of in pytorch directory. Can you make sure you have latest commits with .
I am sure the code is up-to-date. First I uninstall the code using the command pip unisntall torch. Then I remove the old code using the command rm -r pytorch and git clone again. The log of install is
Sorry to trouble you. Is there any new information on this issue?
I spent some time resolving this issue and I don't see too much about it online. Try ensuring that your Pytorch build is correct for your specific versions of Cuda and Cudnn. While I had the correct version of Pytorch, I resolved the issue by installing a different build of the same version.

The build I installed is . Here are the list of versions and builds: You can download these files locally and install with conda: (assumes command is being run in a directory containing , where this file will be specific to your Cuda and Cudnn version.
feedback from this code PackagesNotFoundError
thank you - I have edited my comment to reflect that I installed from a local zip.


Question: (hx + cx).sum().backward() <del> @unittest.skipIf(not TEST_CUDNN, "needs cudnn") @default_tensor_type(torch.FloatTensor)  # FIXME: just until torch.cuda.DoubleTensor.sum() implemented def test_RNN_cpu_vs_cudnn(self): </del> <ins> def _test_RNN_cpu_vs_cudnn(self, dropout): </ins> def forward_backward(cuda, rnn, input_val, hx_val, weights_val): is_lstm = type(rnn) == nn.LSTM

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: # Make sure state dict wasn't modified self.assertEqual(state_dict, state_dict_c) <ins> # Check that state dict can be loaded even when we cast parameters # to a different type and move to a different device. if not torch.cuda.is_available(): return input_cuda = Variable(input.data.float().cuda()) weight_cuda = Variable(weight.data.float().cuda(), requires_grad=True) bias_cuda = Variable(bias.data.float().cuda(), requires_grad=True) optimizer_cuda = constructor(weight_cuda, bias_cuda) fn_cuda = functools.partial(fn_base, optimizer_cuda, weight_cuda, bias_cuda) state_dict = deepcopy(optimizer.state_dict()) state_dict_c = deepcopy(optimizer.state_dict()) optimizer_cuda.load_state_dict(state_dict_c) # Make sure state dict wasn't modified self.assertEqual(state_dict, state_dict_c) for i in range(20): optimizer.step(fn) optimizer_cuda.step(fn_cuda) self.assertEqual(weight, weight_cuda) self.assertEqual(bias, bias_cuda) </ins> def _test_basic_cases(self, constructor, ignore_multidevice=False): self._test_state_dict( torch.randn(10, 5),

Hi, I encountered this bug: The code skeleton is like: It seems the loaded are , and I've tried a workaround to move to , but it still has the same bug.
Could you provide a full script to reproduce the problem?
maybe you can try like this, () (beta1).add_(1 - beta1, ())
Sorry, I missed the reply email. I am afraid that I am unable to provide a reproducer now. It is a work I am doing for the OpenNMT-py , trying to use for doing lr update. And I encoutered this problem when testing the case. So I factored out the code skeleton about this problem above. I've tried several methods, including tricks like what suggests, but it still happens. Per my analysis, it is because the previous training was done on gpu, so when saving the , the stored states(tensors) are of version. During resuming, when we load the saved optimizer, loads this version to cpu(the model() can be moved to gpu easily, but seems lacking this ability?) , so this problem emerges.
Try moving optimizer state to the GPU memory manually after loading it from the checkpoint. I agree that having an method for this operation would be nice.
thanks. My work was suspended due to other problems, when resumed, I will try your method.
I'm afraid solution won't work. It will make the error go away, but your optimizer will no longer be training the model. You should recreate optimizers after casting modules to a different type or device, and you can use to restore the state from a previous copy. This currently doesn't work, but we should fix it (by copying from the data from the state dict, instead of using the tensors directly - this allows for cross-device or cross-type updates).
, yep, your method is what I currently use, it works. But I will wait for upstream to fix this problem though. Thanks for your great works!
Ah, my bad. I forgot to update the line where the optimizer is recreated. But otherwise, the following should do the job, right?
ah, right. That should work
Except that you should use instead of
I had a similar problem. When I save the optimizer state from a GPU other than GPU 0 and then load the state it still loads everything to GPU 0.

Specifying in didn't work either. 's solution solves this though.
Hi guys, I have a very similar problem as the one in this thread, here's my code: And then once I resume, I got KeyErrors on my optimizer: Do you guys know how to fix this issue? BTW, I have 8 GPUs used, I'm guessing if this issue was because of that?
were you able to solve your issue with multiple-GPUs?
Thanks for asking. Yeah, I'm able to, turns out the issue is because I used an early version of PyTorch, after I updated the version, it works like a charm!
Just a comment, this problem is caused by makes all state tensor are moved into GPU0 so by moving the state of an optimizer to specific GPU will fix this problem.
Hi how do you know the specific GPU location of different tensors in advance?
Would a feature where all () calls always makes use of an automatically generated CPU version be feasible ? And then at resume the () would make use of the "current" device being used (or any better strategy). At the moment it seems we need a lot of boilerplater code to ensure saving and loading are consistents across devices for models/optimizer/scheduler/etc.
I have met similar problem, I recreated Adam optimizer without () after reloading model, () and DataParallel(model) according to solution.
thanks, it work!
Hi, as you say that: every time moving model to other device, we should build optimizer, but, if we move the model to other device and move back, should we build the optimizer again? here is an example code: does optimizer run as expected? also, if doing , should we rebuild optimizer ?
Is there a problem if you switch the order to something like this? Meaning moving the model to 'cuda' but only loading it's state dict from checkpoint after loading the optimizer's state dict first?
The problem can be concluded that the optimizer's state will be loaded to the device as same as the model. You must load the model to GPU at first, and then load the optimizer's state. So that both the model and the optimizer's state are loaded in GPU.

Instead of moving optimizer to cuda after loading it in cpu, you could load the checkpoint directly in cuda:
I've independently rediscovered that this works :) Should read until the end of the thread next time
I find my codes still have the problem. I tried my best to range the modules as the examples shown in the above. can anyone give me some hints? I also tried this
For me, doesn't solve the issue, but the one by works. Any idea why? I create the optimizer, load the state (with map location to CUDA), pass to the train loop, where the mode is pushed to device (though I assume it's not needed if the loaded model is pushed to cuda by maplocation). Specifically, I would get stuck at call, throwing the device mismatch error.
Still it doesn't work for me. Could you please help me?


Question: std::vector<at::Tensor> unpack(at::TensorList tl, const char *name, int pos) const; std::vector<at::Tensor> unpack_idxs(at::TensorList tl, const char *name, int pos) const; <del> Variable as_variable(Tensor tensor) const; std::tuple<Variable, Variable> as_variable(std::tuple<Tensor, Tensor> tensor) const; std::tuple<Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor> tensor) const; std::tuple<Variable, Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor, Tensor> tensor) const; std::vector<Variable> as_variable(TensorList tensor) const; Variable maybe_wrap(Tensor data, const Variable & self, bool inplace) const; </del> private: at::Type* baseType; std::string str;

See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:


Question: python ..ci_scriptsdelete_image.py 7z x %IMAGE_COMMIT_TAG%.7z <del> sh run_test.sh </del> <ins> sh run_test.sh -- -v </ins> EOL

If a test segfaults, it's hard to determine which test it is because the output looks like: ! I can work on this if we think this is a good idea.
Last time I asked about this, he didn't want to because it adds a lot of scrolling. I also did some work to move the CI tests to so that we can output XML status files, but (1) this is unlikely to help in case of segfault, and (2) you can't do this straightforwardly, because we run some test files multiple times with different environment variables, and the XML exporter isn't really set up to handle this case. For the record, I am personally OK with just slapping on the tests and calling it a day.
in the conbuild sounds good to me too
closed via


Question: int collapseIndicesDim = indicesInfo.collapseDims(dim);                 int64_t inputSlices = 1;                                               <del> int64_t topKSlices = 1;                                                for (int i = 0; i < numDims; ++i) {                                    </del> <ins> for (int i = 0; i < inputInfo.dims; ++i) {                             </ins> inputSlices *= inputInfo.sizes[i];                                   <ins> }                                                                      int64_t topKSlices = 1;                                                for (int i = 0; i < topKInfo.dims; ++i) {                              </ins> topKSlices *= topKInfo.sizes[i];                                     }

Repro: Expect two small tensors of small integers to be printed. Instead, only the first case works. In the second case, you get a series of failed assertions starting with this: I wonder if it is related to
is this still an outstanding issue? I cannot reproduce this with an install from the master branch.
The size which causes failure doesn't seem deterministic. Using a pretty recent build (a few days old), I now have to increase the "9" to a larger number to make it fail. Sometimes it will fail at 10, sometimes at 11. Here's an updated case that seems to fail reliably at some size, for me:


Question: >>> hx = Variable(torch.randn(3, 20)) >>> output = [] >>> for i in range(6): <del> ...     hx = rnn(input, hx) ...     output[i] = hx </del> <ins> ...     hx = rnn(input[i], hx) ...     output.append(hx) </ins> """ def __init__(self, input_size, hidden_size, bias=True, nonlinearity="tanh"):

Previously, worked to install CPU version explicitly. However, does not work. Are the whl files for 2.4 missing? N/A cc
Hi, this link is deprecated since 2.4. Please use the official installation guide instead.
Still it is not available for macosx x86 platform
Hi and Please use official install commands from For linux its: MacOS x86 binaries where deprecated since release 2.3.0.
Closing as resolved. We previously used the command in original post for building dockerfiles for specific versions. I have now replaced it with:
This still installs torch 2.2.2 but not 2.4.0 On Monday, August 19, 2024 at 10:41:25 PM GMT+5:30, Mihir Patel Closing as resolved. We previously used the command in original post for building dockerfiles for specific versions. I have now replaced it with: pip${PYTHONVERSION} install --no-cache-dir --find-links  torch=={CUDAVERSIONTAG} — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID:
This still installs torch 2.2.2 but not 2.4.0
python --version Python 3.12.5 pip3 install torch torchvision torchaudio --index-url Collecting torch Downloading (151.0 MB) 151.0/151.0 MB 11.3 MB/s eta 0:00:00 Successfully installed torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2 but not 2.4.0


Question: cache_key = metadata[1] rebuild = rebuild_storage_filename storage._shared_incref() <ins> elif storage.size() == 0: # This is special cased because Empty tensors # (with size 0) cannot be mmapped. return (rebuild_storage_empty, (type(storage),)) </ins> else: fd, size = storage._share_fd_() if sys.version_info[0] == 2:

PyTorch version: 0.4.0a0+ Repro: Result:
Emm. I fail to reproduce this bug in Python 3.5.0 and Mac OS 10.13.3. What's your system environment?
The bug is probably specific to the sharing mode. Mac OS uses by default.


Question: from tools.setup_helpers.env import check_env_flag from tools.setup_helpers.cuda import WITH_CUDA, CUDA_HOME from tools.setup_helpers.cudnn import WITH_CUDNN, CUDNN_LIB_DIR, CUDNN_INCLUDE_DIR <ins> from tools.setup_helpers.split_types import split_types </ins> DEBUG = check_env_flag('DEBUG') WITH_DISTRIBUTED = check_env_flag('WITH_DISTRIBUTED') WITH_DISTRIBUTED_MW = WITH_DISTRIBUTED and check_env_flag('WITH_DISTRIBUTED_MW')

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: """Fills the input Tensor or Variable with values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from :math:`N(0, std)` where <del> :math:`std = sqrt{2 / ((1 + a^2) times fan_in)}`. Also known as He initialisation. </del> <ins> :math:`std = sqrt{2 / ((1 + a^2) times fan_in)}`. Also known as He initialisation. </ins> Args: tensor: an n-dimensional torch.Tensor or autograd.Variable

why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:


Question: PyObject *THSPTensorClass = NULL; <del> static void THSPTensor_(initStorage)(THSTensor* tensor) </del> <ins> static void THSTensor_(initStorage)(THSTensor* tensor) </ins> { // Ensure that PyTorch's "storage is not NULL" invariant is upheld. // See Note [Storage is not NULL]

This causes to not work properly on newly created empty sparse tensors. In the example below, both should print 1 instead.


Question: def test_view(self): tensor = torch.rand(15) template = torch.rand(3, 5) <ins> empty = torch.Tensor() </ins> target = template.size().tolist() self.assertEqual(tensor.view_as(template).size().tolist(), target) self.assertEqual(tensor.view(3, 5).size().tolist(), target)

Please check and fix all the PyTorch web links format from to , like in the official site.
closed via


Question: std::unique_ptr<Tensor> save_std(output->newTensor()); save_std->resizeAs(*running_var); <del> if (use_cudnn) { </del> <ins> if (use_cudnn && eps >= CUDNN_BN_MIN_EPSILON) { </ins> #ifdef WITH_CUDNN torch::cudnn::cudnn_batch_norm_forward( state,

Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me.

As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .


Question: #define IS_CUDA false #define CUDA_FLOAT false #else #define IS_CUDA true <del> #define CUDA_BYTE defined(THC_REAL_IS_BYTE) #define CUDA_CHAR defined(THC_REAL_IS_CHAR) #define CUDA_SHORT defined(THC_REAL_IS_SHORT) #define CUDA_INT defined(THC_REAL_IS_INT) #define CUDA_LONG defined(THC_REAL_IS_LONG) #define CUDA_FLOAT defined(THC_REAL_IS_FLOAT) #define CUDA_DOUBLE defined(THC_REAL_IS_DOUBLE) #define CUDA_HALF defined(THC_REAL_IS_HALF) </del> <ins> #if defined(THC_REAL_IS_BYTE) #define CUDA_BYTE 1 #else #define CUDA_BYTE 0 </ins> #endif <ins> #if defined(THC_REAL_IS_CHAR) #define CUDA_CHAR 1 #else #define CUDA_CHAR 0 #endif #if defined(THC_REAL_IS_SHORT) #define CUDA_SHORT 1 #else #define CUDA_SHORT 0 #endif #if defined(THC_REAL_IS_INT) #define CUDA_INT 1 #else #define CUDA_INT 0 #endif #if defined(THC_REAL_IS_LONG) #define CUDA_LONG 1 #else #define CUDA_LONG 0 #endif #if defined(THC_REAL_IS_FLOAT) #define CUDA_FLOAT 1 #else #define CUDA_FLOAT 0 #endif #if defined(THC_REAL_IS_DOUBLE) #define CUDA_DOUBLE 1 #else #define CUDA_DOUBLE 0 #endif #if defined(THC_REAL_IS_HALF) #define CUDA_HALF 1 #else #define CUDA_HALF 0 #endif #endif // ifndef THC_GENERIC_FILE </ins> #if IS_CUDA #define THIndexTensor THCudaLongTensor #define THIndexTensor_(NAME) TH_CONCAT_2(THCudaLongTensor_,NAME)

In we define a number of macros which are subsequently used in if statements: This is undefined. On clang 3.4.2, these warnings don't seem to actually get printed out () but on clang 4.0.0 they seem to always get emitted.


Question: END_HANDLE_TH_ERRORS } <ins> // The type for PyInt_FromLong varies between Windows (int64_t) and // Linux (long).  Unfortunately, on Clang, if the template type argument // does not *exactly* line up, Clang discards the template // specialization.  So we write a little stub function which does have // the right type. static PyObject* PortablePyInt_FromLong(int64_t ival) { return PyInt_FromLong(ival); } </ins> static struct PyGetSetDef batch_norm_forward_properties[] = { THP_FUNCTION_DEFAULT_PROPERTIES, {(char*)"running_mean", (getter)getTensorAttr<BatchNormForward, BatchNormParams,

One representative error: The cause of the problem is : It seems that on clang, changing the type parameter here is sufficient to cause template instantiation to fail. Maybe the easiest way to fix this is to write a more portable version of PyIntFromLong (and friends) which always returns .


Question: ('ones', small_3d, lambda t: [1, 2, 3, 4, 5],), ('permute', new_t(1, 2, 3, 4), lambda t: [2, 1, 3, 0],), ('put_', new_t(2, 5, 3), lambda t: [long_type(t)([[0], [-2]]), t([[3], [4]])],), <ins> ('put_', new_t(2, 3), lambda t: [long_type(t)([]), t([])], 'empty'), </ins> ('put_', new_t(2, 2), lambda t: [long_type(t)([[1], [-3]]), t([[1], [2]]), True], 'accumulate'), ('prod', small_2d_oneish, lambda t: [],), ('prod', small_3d, lambda t: [1], 'dim'),

odd difference between cuda Tensor and Variable: encountered in torch.distributions.Categorical, built yesterday from master
related to hmm, here's the issue is for variables, so probably not (immediately) related
different code path but similar issue
This is a regression from the latest 0.3 binaries, which don't have this error.


Question: endif() # ---[ Exract versions <del> message(STATUS "Caffe2: CUDA detected: " ${CUDA_VERSION}) </del> if (CAFFE2_FOUND_CUDNN) # Get cuDNN version file(READ ${CUDNN_INCLUDE_DIR}/cudnn.h CUDNN_HEADER_CONTENTS)

Here's the situation: Ideally, you'd say something like to make a build of PyTorch with a non-standard version of CUDA. When this happens, needs to find the correct copy of nvcc to use. If you are using ccache as per the instructions in you'll have set to your ccache binary, so that ccache happens. cmake will correctly respect this variable. However, here's the problem: ccache's nvcc will use your path to determine what the real nvcc to use is, this means that you'll pick up whatever nvcc is in your PATH, which is probably not the one you want, because you're trying to use a different nvcc. So in sum, you have to do the non-intuitive to get the whole shebang to work. The worst part is that it won't tell you if you've got it wrong, it will happily use the wrong nvcc. I haven't gone through user submitted issues to see if anyone has succumbed to this particular bug.
A long-term solution might be getting to work
works now.


Question: from functools import reduce from ._utils import _range <ins> class __PrinterOptions(object): precision = 4 threshold = 1000 edgeitems = 3 linewidth = 80 PRINT_OPTS = __PrinterOptions() </ins> SCALE_FORMAT = '{:.5e} *n' <del> def _number_format(tensor): min_sz = 2 </del> <ins> # We could use **kwargs, but this will give better docs def set_printoptions( precision=None, threshold=None, edgeitems=None, linewidth=None, profile=None, ): """Set options for printing. Items shamelessly taken from Numpy Args: precision: Number of digits of precision for floating point output (default 8). threshold: Total number of array elements which trigger summarization rather than full repr (default 1000). edgeitems: Number of array items in summary at beginning and end of each dimension (default 3). linewidth: The number of characters per line for the purpose of inserting line breaks (default 80). Thresholded matricies will ignore this parameter. profile: Sane defaults for pretty printing. Can override with any of the above options. (default, short, full) """ if profile is not None: if profile == "default": PRINT_OPTS.precision = 4 PRINT_OPTS.threshold = 1000 PRINT_OPTS.edgeitems = 3 PRINT_OPTS.linewidth = 80 elif profile == "short": PRINT_OPTS.precision = 2 PRINT_OPTS.threshold = 1000 PRINT_OPTS.edgeitems = 2 PRINT_OPTS.linewidth = 80 elif profile == "full": PRINT_OPTS.precision = 4 PRINT_OPTS.threshold = float('inf') PRINT_OPTS.edgeitems = 3 PRINT_OPTS.linewidth = 80 if precision is not None: PRINT_OPTS.precision = precision if threshold is not None: PRINT_OPTS.threshold = threshold if edgeitems is not None: PRINT_OPTS.edgeitems = edgeitems if linewidth is not None: PRINT_OPTS.linewidth = linewidth def _number_format(tensor, min_sz=-1): min_sz = max(min_sz, 2) </ins> tensor = torch.DoubleTensor(tensor.nelement()).copy_(tensor).abs_() pos_inf_mask = tensor.eq(float('inf'))

It seems that the following link that can be reached from here (check the screenshot) is broken: Page: Broken link: Screenshot: ! cc
this should be transferred to , but I don't have permissions to do that. Fix is available at pytorch/hub.


Question: msg='LR is wrong in epoch {}: expected {}, got {}'.format( epoch, target[epoch], param_group['lr']), delta=1e-5) if __name__ == '__main__': run_tests()

I'm working on checkpointing in our nn trainer. When saving the state objects, I get an error: PicklingError: Can't pickle <type 'function': attribute lookup builtin.function failed I tracked it down to trying to save the scheduler ReduceLROnPlateau. I think the issue is due to the scheduler referencing the optimizer, which has to be saved with statedict(). I presume the same error will occur with other schedulers too. It seems like the problem could be solved by providing a statedict() and loadstatedict() in the scheduler classes. pytorch master rev (Jan 22) installed from source x86_64 on centos 7.4 cuda 9.0 cudnn 7 k80 gcc 5.4 python 2.7.14
The following script reproduces the crash: import torch import from import ReduceLROnPlateau if name == 'main': model = (10,10) optim = (model.parameters()) sched = ReduceLROnPlateau(optim, mode='max', factor=0.5) with open('','wb') as f: # Works # ((model.statedict(), optim.statedict()), f) # Crashes ((model.statedict(), optim.state_dict(), sched), f)


Question: python_name: apply_ defined_if: "!IS_CUDA" only_register: True <ins> override_method_flags: METH_O </ins> ]] static PyObject * THPTensor_(apply)(THPTensor *self, PyObject *arg) {

On Python 3.5.2 errors out with . map and map2 seem to be work just fine.


Question: import os import torch import traceback <ins> import warnings </ins> from torch._six import raise_from from subprocess import Popen, PIPE from multiprocessing.util import register_after_fork as _register_after_fork

There's quite a few user-reports of installing the CUDA 7.5 binary of pytorch and then running on GTX 1080 / 1070, and there will be reports of installing 7.5 / 8.0 binaries and running on V100. Raising a user-warning in such situations and letting them know will fix the problem.
The version of CUDA used to compile the binary should be available thanks to this PR:
I think is a more robust way of getting the CUDA used to compile PyTorch, as currently if there are multiple CUDA installed in the machine, the heuristics used in might not match those used by CMake FindCUDA. I think that we should modify the to return the cuda version used during compilation. I have the impression that it will not require changing much because doesn't require cuda to execute. What do you think?


Question: composed of several input planes. This module can be seen as the gradient of Conv1d with respect to its input. <del> It is sometimes (but incorrectly) refered to as a deconvolutional operation. </del> <ins> It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation). | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides for :attr:`padding` number of points. | If :attr:`output_padding` is non-zero, then the output is implicitly zero-padded on one side for :attr:`output_padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> .. note::

Hi, I have installed two versions of CUDA. But I want to use one of it, can I specify it during setup process?
Try running
I have tried it, () still returns False. Thx for your reply!
Do you have the cuda driver installed?
Of course I do. It raise assertion error which shows "The NVIDIA driver on your system is too old(found version 8000)".
that's the answer to your problem. upgrade your driver.
Thx!

When building from source, I got this error. My system is OSX 10.12.4, conda, CUDA 8.0. Any thoughts? Thanks!
can you try adding to the file and see if that fixes things? if so i'll commit it.
actually, that file already has memory included. Are you following instructions from Especially: This part is important:
works nicely! Thanks a lot!

The suggest that takes a as a parameter, but it actually must be wrapped in a Variable first.
you are looking at the Tensor docs. When operating on Variable, it is implied from the tutorials that all functions operating on Variables have to use Variables. We didn't write separate docs for Variable itself.

something wrong with import torchvision Traceback (most recent call last): File "", line 3, in <moduleimport torchvision File "", line 4, in <moduleimport torchvision.transforms as transforms ImportError: No module named transforms
you have a file called for some reason. delete it or change your pythonpath to ignore it.
Hey, I have the same problem persisting when I import torchvision. I have no other file by the same name. ImportError Traceback (most recent call last) <ipython-input-6-3baf6837d8f1in <module() 8 from import Dataset 9 from import DataLoader ---10 from torchvision import transforms ImportError: No module named torchvision Any idea why this is happening?
Do you have torchvision installed (it is a separate package)?
pip Install torchvision try this
from data import BaseTransform,VOC_CLASSES as labelmap this is not working
pip install torchvision

Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

def inithidden(self): h0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) c0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) return (h0, c0) def forward(self, sequences): lstmout, = ((1, self.batchlen, self.inputdim), ) ... = nn.LSTM(inputdim, hiddendim, numlayers, dropout = dropout, batchfirst=True) = self.inithidden() output = model(traininput) produces the following error although I defined weight and input dimensions as in the documentation: File "/usr/local/lib/python2.7/dist-", line 202, in call result = self.forward(input, kwargs) File "/usr/local/lib/python2.7/dist-", line 91, in forward output, hidden = func(input, self.allweights, hx) File "/usr/local/lib/python2.7/dist-", line 327, in forward return func(input, fargs, fkwargs) File "/usr/local/lib/python2.7/dist-", line 227, in forward nexth, output = func(input, hidden, weight) File "/usr/local/lib/python2.7/dist-", line 67, in forward hy, output = inner(input, hidden[l], weight[l]) File "/usr/local/lib/python2.7/dist-", line 96, in forward hidden = inner(input[i], hidden, *weight) File "/usr/local/lib/python2.7/dist-", line 22, in LSTMCell gates = F.linear(input, wih, bih) + F.linear(hx, whh, bhh) File "/usr/local/lib/python2.7/dist-", line 744, in add return (other) File "/usr/local/lib/python2.7/dist-", line 293, in add return (other, False) File "/usr/local/lib/python2.7/dist-", line 287, in add return Add(inplace)(self, other) File "/usr/local/lib/python2.7/dist-", line 20, in forward return a.add(b) RuntimeError: inconsistent tensor size at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:827
I can't reproduce your problem. Could you include a short runnable example that exhibits this behavior?

You also don't need to manually init hidden anymore, the following would work as well...
Thanks! After removing problem solved.

Hi, I would like to implement a sharing weight 'alexnet' with two input batches of images, and the fully connected layers are concatenated together like this: However, when I run it, I come across the following error: Does anyone know about it? Thank you very much.
You forgot to call on the output of . In the future please remember that we're using the issues for bug reports and feature requests only. If you have questions or need help, please post .


Question: @wraps(fn) def wrapper(self): if self.rank == self.MANAGER_PROCESS_RANK: <del> self._join_and_reduce() </del> <ins> self._join_and_reduce(fn) </ins> else: fn(self) return wrapper

is causing a crash when we try to build pytorch. Crash looks like this: (all times in US/West) Our 7AM, 8AM builds and earlier were all fine but 9AM, 10AM, 11AM builds all crashed with that same traceback. The new import was in which I think went in between 8AM and 9AM today See command below: See here if needed for : <!-- A clear and concise description of what you expected to happen. --Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): building from master OS (e.g., Linux): linux How you installed PyTorch (, , source): source Build command you used (if compiling from source): see above under 'reproduce' Python version: 3.6
maybe easier to read for build command used:
please try in the CircleCi environment?
in our build script seems to have fixed this. Thanks !


Question: def _initialize_backend(): from .._functions.thnn import _all_functions as _thnn_functions from .._functions.linear import Linear <del> from .._functions.batchnorm import BatchNorm </del> from .._functions.conv import ConvNd from .._functions.rnn import RNN,  RNNTanhCell, RNNReLUCell, GRUCell, LSTMCell

As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10


Question: .. automethod:: fmod_ .. automethod:: frac .. automethod:: frac_ <del> .. automethod:: free </del> .. automethod:: gather .. automethod:: ge .. automethod:: ge_

Dataloader call getitem as many times as indices in the current batch. In case datasets support a list of indices in one call, or a native python slice object, add a getbatch (optional) to pytorch dataset class. If dataloader sees the dataset has such a method implemented, make it fetch the batch corresping to the list of indices with one getbatch call rather than many getitem calls. This saves the trouble of calling getitem many times in cases where for example a disk file has to be opened then closed for every call and the underlying data structure supports a slicing or lists of indices for example hdf5 files.
will support something similar
is the latest version of this.