Question: # static library only NANOPB_STATIC_LIB = os.path.join(lib_path, 'libprotobuf-nanopb.a') <del> if WITH_NCCL and (subprocess.call('ldconfig -p | grep libnccl >/dev/null', shell=True) == 0 or subprocess.call('/sbin/ldconfig -p | grep libnccl >/dev/null', shell=True) == 0): SYSTEM_NCCL = True </del> main_compile_args = ['-D_THP_CORE'] main_libraries = ['shm'] main_link_args = [TH_LIB, THS_LIB, THNN_LIB, ATEN_LIB, NANOPB_STATIC_LIB]

Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).


Question: CUDA_INC ?= $(CUDA_HOME)/include NVCC ?= $(CUDA_HOME)/bin/nvcc <del> NVCC_GENCODE ?= -gencode=arch=compute_35,code=sm_35  </del> <ins> NVCC_GENCODE ?= -gencode=arch=compute_30,code=sm_30  -gencode=arch=compute_35,code=sm_35  </ins> -gencode=arch=compute_50,code=sm_50  -gencode=arch=compute_52,code=sm_52  -gencode=arch=compute_52,code=compute_52

I'm running PyTorch on a system with 4 Nvidia K10 GPUs and can't get the Multi-GPU code to work. I attempted to modify the first neural net in the 60-minute tutorial to run on all my GPUs, by adding this line to the cell defining the network: If I then attempt to use the net as follows , Jupyter waits a few seconds and reports the kernel has died. Following along in I see all GPUs encountering an at the same moment. Am I using the API wrong, could there be a configuration issue in my system, or is this a bug in the DataParallel implementation? I'm using CUDA 8 and PyTorch 0.1.6. Edit: my Nvidia driver version is 367.48 I also forgot to add, executing the above code on CUDA without parallelization (i.e. ) works fine.
Is there a stack-trace that you see appear on the command-line of the Jupyter process? Is this a cloud provider box? (I ask to see if I can reproduce it myself)
This is the only command-line output Jupyter gives related to the error: I'm running this on an in-house compute node.
thanks. this is helpful, I'll look further. does nccl have cuda code that we need to build for specific archs? K10 is 3.0 and i build 3.0 code for thc and thcunn
Yes, that's the problem: The README says "Kepler or newer". does NCCL work on K10s?
It should (once you add the necessary compilation flags).


Question: 'clang_version', 'cmake_version', 'os', <ins> 'libc_version', </ins> 'python_version', <ins> 'python_platform', </ins> 'is_cuda_available', 'cuda_runtime_version', 'nvidia_driver_version',

TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC


Question: namespace torch { namespace autograd { <ins> struct FunctionTask { BackwardTask* base; std::shared_ptr<Function> fn; GradBuffer grad; FunctionTask(BackwardTask* base, std::shared_ptr<Function> fn, GradBuffer grad) : base(base) , fn(fn) , grad(std::move(grad)) {} }; </ins> struct ReadyQueue { std::deque<FunctionTask> queue; std::condition_variable not_empty;

Version include <stringin some cpp files: define order in The pre-defined struct is not passed by clang in macOS, so we should reorder them.
Simple fix


Question: from .module import Module from .utils import _quadruple, _ntuple <del> from .._functions.padding import ConstantPad2d as F_ConstantPad2d </del> <ins> from .. import functional as F </ins> # TODO: grad_output size asserts in THNN class ReflectionPad2d(Module): <ins> r"""Pads the input tensor using the reflection of the input boundary. Args: padding (int, tuple): the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (paddingLeft, paddingRight, paddingTop, paddingBottom) Shape: - Input: :math:`(N, C, H_{in}, W_{in})` - Output: :math:`(N, C, H_{out}, W_{out})` where :math:`H_{out} = H_{in} + paddingTop + paddingBottom` :math:`W_{out} = W_{in} + paddingLeft + paddingRight` Examples:: >>> m = nn.ReflectionPad2d(3) >>> input = autograd.Variable(torch.randn(16, 3, 320, 480)) >>> output = m(input) >>> # using different paddings >>> m = nn.ReflectionPad2d((3, 3, 6, 6)) >>> output = m(input) """ </ins> def __init__(self, padding): super(ReflectionPad2d, self).__init__() self.padding = _quadruple(padding) def forward(self, input): <del> return self._backend.ReflectionPad2d(*self.padding)(input) </del> <ins> return F.pad(input, self.padding, 'reflect') </ins> def __repr__(self): return self.__class__.__name__ + ' ' + str(self.padding) class ReplicationPad2d(Module): <ins> r"""Pads the input tensor using replication of the input boundary. Args: padding (int, tuple): the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (paddingLeft, paddingRight, paddingTop, paddingBottom) Shape: - Input: :math:`(N, C, H_{in}, W_{in})` - Output: :math:`(N, C, H_{out}, W_{out})` where :math:`H_{out} = H_{in} + paddingTop + paddingBottom` :math:`W_{out} = W_{in} + paddingLeft + paddingRight` Examples:: >>> m = nn.ReplicationPad2d(3) >>> input = autograd.Variable(torch.randn(16, 3, 320, 480)) >>> output = m(input) >>> # using different paddings >>> m = nn.ReplicationPad2d((3, 3, 6, 6)) >>> output = m(input) """ </ins> def __init__(self, padding): super(ReplicationPad2d, self).__init__() self.padding = _quadruple(padding) def forward(self, input): <del> return self._backend.ReplicationPad2d(*self.padding)(input) </del> <ins> return F.pad(input, self.padding, 'replicate') </ins> def __repr__(self): return self.__class__.__name__ + ' ' + str(self.padding) class ReplicationPad3d(Module): <ins> r"""Pads the input tensor using replication of the input boundary. Args: padding (int, tuple): the size of the padding. If is int, uses the same padding in all boundaries. If a 6-tuple, uses (paddingLeft, paddingRight, paddingTop, paddingBottom, paddingFront, paddingBack) Shape: - Input: :math:`(N, C, D_{in}, H_{in}, W_{in})` - Output: :math:`(N, C, D_{out}, H_{out}, W_{out})` where :math:`D_{out} = D_{in} + paddingFront + paddingBack` :math:`H_{out} = H_{in} + paddingTop + paddingBottom` :math:`W_{out} = W_{in} + paddingLeft + paddingRight` Examples:: >>> m = nn.ReplicationPad3d(3) >>> input = autograd.Variable(torch.randn(16, 3, 8, 320, 480)) >>> output = m(input) >>> # using different paddings >>> m = nn.ReplicationPad3d((3, 3, 6, 6, 1, 1)) >>> output = m(input) """ </ins> def __init__(self, padding): super(ReplicationPad3d, self).__init__() self.padding = _ntuple(6)(padding) def forward(self, input): <del> return self._backend.ReplicationPad3d(*self.padding)(input) </del> <ins> return F.pad(input, self.padding, 'replicate') </ins> def __repr__(self): return self.__class__.__name__ + ' ' + str(self.padding) class ZeroPad2d(Module): <ins> r"""Pads the input tensor boundaries with zero. Args: padding (int, tuple): the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (paddingLeft, paddingRight, paddingTop, paddingBottom) Shape: - Input: :math:`(N, C, H_{in}, W_{in})` - Output: :math:`(N, C, H_{out}, W_{out})` where :math:`H_{out} = H_{in} + paddingTop + paddingBottom` :math:`W_{out} = W_{in} + paddingLeft + paddingRight` Examples:: >>> m = nn.ZeroPad2d(3) >>> input = autograd.Variable(torch.randn(16, 3, 320, 480)) >>> output = m(input) >>> # using different paddings >>> m = nn.ZeroPad2d((3, 3, 6, 6)) >>> output = m(input) """ </ins> def __init__(self, padding): super(ZeroPad2d, self).__init__() self.padding = _quadruple(padding) def forward(self, input): <del> return F_ConstantPad2d(pad=self.padding, value=0)(input) </del> <ins> return F.pad(input, self.padding, 'constant', 0) </ins> def __repr__(self): return self.__class__.__name__ + ' ' + str(self.padding) class ConstantPad2d(Module): <ins> r"""Pads the input tensor boundaries with a constant value. Args: padding (int, tuple): the size of the padding. If is int, uses the same padding in all boundaries. If a 4-tuple, uses (paddingLeft, paddingRight, paddingTop, paddingBottom) Shape: - Input: :math:`(N, C, H_{in}, W_{in})` - Output: :math:`(N, C, H_{out}, W_{out})` where :math:`H_{out} = H_{in} + paddingTop + paddingBottom` :math:`W_{out} = W_{in} + paddingLeft + paddingRight` Examples:: >>> m = nn.ConstantPad2d(3, 3.5) >>> input = autograd.Variable(torch.randn(16, 3, 320, 480)) >>> output = m(input) >>> # using different paddings >>> m = nn.ConstantPad2d((3, 3, 6, 6), 3.5) >>> output = m(input) """ </ins> def __init__(self, padding, value): super(ConstantPad2d, self).__init__()

After merging cuda 12.4 for nightly. I observe following failures: libtorch-cuda124-shared-with-deps-cxx11-abi-build: Build Official Docker Images 124: We need to resolve these before continuing adding CUDA 12.4 CI workflows 0 cc
attempt 3 worked for libtorch build issue:
For Docker image, related discussion:
I will instance a new bug for the docker job failure.

Is there a way to use reflection padding instead of zero padding with nn.Conv2d ? I see there is a but the option is not in nn.Conv2d. Thanks,
are in , but they're not exposed in the docs.
Thanks!


Question: if fn.batch_first and not is_input_packed: input = input.transpose(0, 1) <del> if (not is_input_packed and input.dim() != 3) or (is_input_packed and input.dim() != 2): raise RuntimeError( 'input must have 3 dimensions, got {}'.format(input.dim())) if fn.input_size != input.size(-1): raise RuntimeError('input.size(-1) must be equal to input_size. Expected {}, got {}'.format( fn.input_size, input.size(-1) )) </del> if fn.dropout != 0 and cudnn.version() < 5103: raise RuntimeError('dropout supported only in cudnn v5.1 and above')

I run RNN sucessfully with the code. and I run embedding example. Then, got RuntimeError. I set to use nothing seems wrong. And the error info seems not the dim problem? The error is so strange.
What exactly do you mean by "embedding example"? Could you provide a code snippet that causes the RuntimeError?
To better reproduce the error, I use the official code example of The origin runs well: Due to use , I have to make batch first. And then , it raises error.
Thank you for the example! The say that setting means that the first dim of input is the batch, ie, input has size . however is unchanged, so the following should work: I do agree the error message could be better, though


Question: // Kernel for fast unfold+copy // (borrowed from Caffe: https://github.com/BVLC/caffe/blob/master/src/caffe/layers/conv_layer.cu) template <typename Dtype> <ins> __launch_bounds__(CUDA_NUM_THREADS) </ins> __global__ void im2col_kernel(const int n, const Dtype* data_im, const int height, const int width, const int ksize_h, const int ksize_w,

I'm trying to run a variant of ERFNet on an NVIDIA TX-2 running Jetpack 3.2 (CUDA 9.0 and CuDNN 7). I get the following error: RuntimeError: cuda runtime error (7) : too many resources requested for launch at ../../../pytorch/torch/lib/THCUNN/im2col.h:120 This error indicates the size of the model + overhead is too large for the GPU? But this is roughly a 700mb model trying to perform inference on a single 640x512 grayscale image on a GPU that has roughly 6.5Gb of free space. I even tried training a new model on images at half that resolution and get the same error. Any tips/feedback is appreciated. PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): Source Build command you used (if compiling from source): Followed jetson-reinforcement github OS: Ubuntu 16.04 on Nvidia TX2 PyTorch version: 0.3.0 Python version: 3.6 CUDA/cuDNN version: CUDA 9.0 , CUDNN 7 GPU models and configuration: Tegra X2 GCC version (if compiling from source): 5.4 CMake version: 3.5.1 Versions of any other relevant libraries:
we should fix the launch parameters. I presume we cant use as many threads per block on the TX2 as we use on desktop GPUs. can you tell us what the limits of TX2 GPUs are for the fix to im2col
Threads per block and maximum blocks in the grid are actually the same for TX2 as they are for desktop GPUs , only number of registers is smaller. do you know what might be causing this?
as said, the number of threads per block is constant across different NVIDIA GPUs, the Tegra series included. Is it possible that it was compiled to require more registers than is available to the TX2, and maybe the kernel invocation of im2col requires some sort of launch_bounds() command?
TX2 is arm64 platform. I presume pytorch is compiled from source. In this case, there isn't a chance that it was compiled to requre more registers than available on TX2 I think. The build log will be helpful to see.
thanks for the quick response.

I will try dig up the build log tomorrow and post it here.
if there are no launch bounds, it is in fact possible that a kernel is compiled to request more registers than available. At the compile time, compiler does not know how many threads you'll want to launch with, so, potentially, it can use too many registers per thread to later satisfy runtime requirements, and, e.g. trying to launch with 512 or 1024 threads could fail (not even a single block can be put on an SM), whereas launching with 256 would succeed.
is there a way where we can audit that GPU constraints on our server setting (i.e. not actually sitting and compiling TX2 PyTorch)
Adding launch_bounds with the max number of threads the kernel is going to be launched with will cause compiler not to overuse registers. We had to do it e.g. for interp kernels when cuda 9 started using more registers
Ahh, that was as I suspected Thank you for clearing this up. do you guys think a fix will be released in the near future (week?). Thanks for all the help
Works now, thanks!
hello, I met the same error on tx2, complied pytorch 0.3.0.. But the error is this: 'cuda runtime error(7): too many resources requested for launch at '. The file is different with '', is that I need add this line 'launch_bounds(1024)' to every function in file ''? Thank you!
Hello I am also facing a similar issue: Here's my setup: How you installed PyTorch (conda, pip, source): Source Build command you used (if compiling from source): Followed jetson-reinforcement github OS: Ubuntu 16.04 on Nvidia TX2 PyTorch version: 0.5.0a0+ (torchvision 0.2.1) Python version: 3.5 CUDA/cuDNN version: CUDA 9.0 , CUDNN 7 GPU models and configuration: Tegra X2 GCC version (if compiling from source): 5.4 CMake version: 3.12.2 Source code: Full log: Do you know what it could be?

I know that '' without launch_bounds(1024) leads to this error, but I don't know how to fix it... Tried this solution , but still not working (after compilation)
My issue was solved by following
Hi, I have the same problem. Can this problem solved by gitting clone the new pytorch, than changing CUDANUMTHREADS =256 in the below two file and compiling it? !
that appears to fix some of the functions, but not perhaps all - not sure. For more info, see:


Question: - THTensor* output - std::vector<int> pad - std::vector<int> stride <del> - std::vector<int> dilation </del> <ins> - std::vector<int> dilation </ins> - int groups - bool benchmark ]]

We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.


Question: dim_args_idx = test[4] if len(test) == 5 else [] <ins> skipTestIf = test[5] if len(test) == 6 else [] </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name new_args = [arg * dim_perm[dim_args_idx.index(i)] if i in dim_args_idx else arg for i, arg in enumerate(args)]

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: grad_input = torch.cross(other, grad_output, self.dim) grad_other = torch.cross(grad_output, input, self.dim) return grad_input, grad_other <ins> class Inverse(Function): @staticmethod def forward(ctx, input): inverse = torch.inverse(input) ctx.save_for_backward(inverse) return inverse @staticmethod def backward(ctx, grad_output): inverse, = ctx.saved_variables return -torch.mm(inverse.t(), torch.mm(grad_output, inverse.t())) </ins>

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: template<typename FnType, FnType fn, typename ...Args> static PyObject* wrap_tuple_fn(Args ... args) { <del> PyObject *result = (*fn)(std::forward<Args>(args)...); </del> <ins> THPObjectPtr result((*fn)(std::forward<Args>(args)...)); </ins> if (!result) return NULL; <del> if (PyTuple_Check(result)) { return PyObject_CallFunctionObjArgs((PyObject*)&THPSizeType, result, NULL); </del> <ins> if (PyTuple_Check(result.get())) { return PyObject_CallFunctionObjArgs((PyObject*)&THPSizeType, result.get(), NULL); </ins> } <del> Py_INCREF(result); return result; </del> <ins> return result.release(); </ins> } static auto sq_concat = PyTuple_Type.tp_as_sequence->sq_concat;

I think I encountered a memory leak situation potentially due to . I create a to reproduce my observations. Basically, for three different (meaningless) models, once the function is used, the memory usage will keep increasing. To run the code, where the flag controls whether is used. Please run it long enough. I'm on Ubuntu 14.04 with python3.6. I tried two versions of pytorch, 0.1.122 and 0.1.12+. Side Note: Even without using , the memory usauge can also increase for a while, but it will finally stablize at a level. What's more, when (i.e. ) is used, it can take very long before the memory usage stablizes. I'm not sure whether this is a problem or not.
I think I'm sure there is a memory leak due to , and I found a weird way to do a temporary fix under pytorch version "0.1.122". The simplest example is as follows: Basically, if we replace with , and pass in an unpacked list of numbers, (i.e., instead of ), the memory leak is gone. Using will also have the leak issue. Unfortunately, I don't know why this is working. Finally, since there are so many in pytorch code, to do a quick hacky fix, one can change the function in the file as follows:
After some more research, I think we have a much bigger problem here. My current conjecture is that: whenever we use ( object) as an argument to a function, there will be a memory leak. An example is the function in , where of the code is which uses . Again, a minimal example for reproduction is as follows: As in the case, once I change the code to , memory leak is gone. So, in short, the problem is on the usage of .
The same issue. +1
Thanks for tracking this down and for the repro script!
Thanks for the quick fix! BTW, will the fix be included in the release version (0.1.12_?)?
it'll be released in
I see. Thanks again.


Question: if momentum != 0: param_state = self.state[p] if 'momentum_buffer' not in param_state: <del> buf = param_state['momentum_buffer'] = d_p.clone() </del> <ins> buf = param_state['momentum_buffer'] = p.data.new().resize_as_(p.data).zero_() buf.mul_(momentum).add_(d_p) </ins> else: buf = param_state['momentum_buffer'] buf.mul_(momentum).add_(1 - dampening, d_p)

When using classic SGD optimizer with momentum with sparse embeddings the memory keeps garbage collecting / allocating leading to slow down and out of memory error eventually. ! ! The issue dissapears when momentum is not used ! or when embeddings are not sparse ! I'm using the last pytorch version on conda:
I tried out your script with momentum 0.1 on master, it takes roughly 10800mb gpu memory max. This is caused by using sparse buffer. I'm sending out a PR for this.


Question: input_size=(1, 2, 4, 4, 4)), OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2), <del> input_size=(2, 3, 5, 5, 5)), </del> <ins> input=(torch.randn(2, 3, 5, 5, 5) * 1000)), </ins> OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2, 2, 2, 2), input_size=(2, 3, 5, 5, 5),

A few times, VolumetricMaxPooling (in ) kept failing on the continuous builds, and needed to investigate that. So it turns out, there were these one set of indices that were in the same Pooling window that differed by less than epsilon. So, the numeric gradient was hitting boundary conditions (max-pooling is discontinuous of course). ! ! The obvious fix is to modify the test to not have the input tensor have these boundary conditions. However, as an aside, I am not sure why the boundary condition is only hit sometimes and not always, because the same random numbers are generated always, and division is unstable but it is deterministic (right? or have i always got this wrong)
Repeat running will produce error occasional , more detail:


Question: above the diagonal. The argument :attr:`diagonal` controls which diagonal to consider. If <del> :attr:`diagonal` = 0, all elements on and below the main diagonal are </del> <ins> :attr:`diagonal` = 0, all elements on and above the main diagonal are </ins> retained. A positive value excludes just as many diagonals above the main diagonal, and similarly a negative value includes just as many diagonals below the main diagonal. The main diagonal are the set of indices

! It seems that the highlighted "below" should be "upper“. <!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --
Hi, thank you for bringing this up. I'll send in a patch to fix it.


Question: composed of several input planes. This module can be seen as the gradient of Conv2d with respect to its input. <del> It is sometimes (but incorrectly) refered to as a deconvolutional operation. </del> <ins> It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation). </ins> | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides <del> for :attr:`padding` number of points </del> <ins> for :attr:`padding` number of points. </ins> | If :attr:`output_padding` is non-zero, then the output is implicitly zero-padded on one side <del> for :attr:`output_padding` number of points | :attr:`dilation` controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. </del> <ins> for :attr:`output_padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. </ins> |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. <ins> At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> The parameters :attr:`kernel_size`, :attr:`stride`, :attr:`padding`, :attr:`output_padding` can either be: <del> - a single ``int`` -- in which case the same value is used for the height and width dimension </del> <ins> - a single ``int`` -- in which case the same value is used for the height and width dimensions </ins> - a ``tuple`` of two ints -- in which case, the first `int` is used for the height dimension, and the second `int` for the width dimension

Hi, I have installed two versions of CUDA. But I want to use one of it, can I specify it during setup process?
Try running
I have tried it, () still returns False. Thx for your reply!
Do you have the cuda driver installed?
Of course I do. It raise assertion error which shows "The NVIDIA driver on your system is too old(found version 8000)".
that's the answer to your problem. upgrade your driver.
Thx!

When building from source, I got this error. My system is OSX 10.12.4, conda, CUDA 8.0. Any thoughts? Thanks!
can you try adding to the file and see if that fixes things? if so i'll commit it.
actually, that file already has memory included. Are you following instructions from Especially: This part is important:
works nicely! Thanks a lot!

The suggest that takes a as a parameter, but it actually must be wrapped in a Variable first.
you are looking at the Tensor docs. When operating on Variable, it is implied from the tutorials that all functions operating on Variables have to use Variables. We didn't write separate docs for Variable itself.

something wrong with import torchvision Traceback (most recent call last): File "", line 3, in <moduleimport torchvision File "", line 4, in <moduleimport torchvision.transforms as transforms ImportError: No module named transforms
you have a file called for some reason. delete it or change your pythonpath to ignore it.
Hey, I have the same problem persisting when I import torchvision. I have no other file by the same name. ImportError Traceback (most recent call last) <ipython-input-6-3baf6837d8f1in <module() 8 from import Dataset 9 from import DataLoader ---10 from torchvision import transforms ImportError: No module named torchvision Any idea why this is happening?
Do you have torchvision installed (it is a separate package)?
pip Install torchvision try this
from data import BaseTransform,VOC_CLASSES as labelmap this is not working
pip install torchvision

Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

def inithidden(self): h0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) c0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) return (h0, c0) def forward(self, sequences): lstmout, = ((1, self.batchlen, self.inputdim), ) ... = nn.LSTM(inputdim, hiddendim, numlayers, dropout = dropout, batchfirst=True) = self.inithidden() output = model(traininput) produces the following error although I defined weight and input dimensions as in the documentation: File "/usr/local/lib/python2.7/dist-", line 202, in call result = self.forward(input, kwargs) File "/usr/local/lib/python2.7/dist-", line 91, in forward output, hidden = func(input, self.allweights, hx) File "/usr/local/lib/python2.7/dist-", line 327, in forward return func(input, fargs, fkwargs) File "/usr/local/lib/python2.7/dist-", line 227, in forward nexth, output = func(input, hidden, weight) File "/usr/local/lib/python2.7/dist-", line 67, in forward hy, output = inner(input, hidden[l], weight[l]) File "/usr/local/lib/python2.7/dist-", line 96, in forward hidden = inner(input[i], hidden, *weight) File "/usr/local/lib/python2.7/dist-", line 22, in LSTMCell gates = F.linear(input, wih, bih) + F.linear(hx, whh, bhh) File "/usr/local/lib/python2.7/dist-", line 744, in add return (other) File "/usr/local/lib/python2.7/dist-", line 293, in add return (other, False) File "/usr/local/lib/python2.7/dist-", line 287, in add return Add(inplace)(self, other) File "/usr/local/lib/python2.7/dist-", line 20, in forward return a.add(b) RuntimeError: inconsistent tensor size at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:827
I can't reproduce your problem. Could you include a short runnable example that exhibits this behavior?

You also don't need to manually init hidden anymore, the following would work as well...
Thanks! After removing problem solved.

Hi, I would like to implement a sharing weight 'alexnet' with two input batches of images, and the fully connected layers are concatenated together like this: However, when I run it, I come across the following error: Does anyone know about it? Thank you very much.
You forgot to call on the output of . In the future please remember that we're using the issues for bug reports and feature requests only. If you have questions or need help, please post .


Question: unset(CUDA_ARCH_PTX CACHE) endif() <del> if(DEFINED ENV{TORCH_CUDA_ARCH_LIST}) </del> <ins> if($ENV{TORCH_CUDA_ARCH_LIST}) </ins> # Pass CUDA architecture directly set(__cuda_arch_bin $ENV{TORCH_CUDA_ARCH_LIST}) message(STATUS "Set CUDA arch from TORCH_CUDA_ARCH_LIST: ${__cuda_arch_bin}")

This is most obvious by observing binary sizes, for example on pypi: (6.0 MB ) (33.8 MB ) Also, from the build log it's clear that 3.8 for some reason build with flag: CI cc
Which is a good old

Einsum currently modifies variables in-place (without an explicit indication that it does so), which prevents pytorch from automatically backpropagating. Results in the following runtime error: Demonstration and discussion can also be found . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): pip OS: PyTorch version: 0.4.0 Python version: 3.6.1
Oops. Thank you for reporting and the minimal example. I'll see to get that fixed.
Saw the same issue over here. Thank you !
Is cloning the tensor before passing to einsum a valid workaround?
Yes, but a somewhat expensive one. If you have the ability to recompile, you could also apply the PR fixing this. Somehow it seems to be stuck in the review queue because it hit an unrelated CI problem...
Okay so how to use it then?
Just get a recent enough master and it will work.


Question: output_padding (int or tuple, optional): Zero-padding added to one side of the output groups (int, optional): Number of blocked connections from input channels to output channels bias (bool, optional): If True, adds a learnable bias to the output <ins> dilation (int or tuple, optional): Spacing between kernel elements </ins> Shape: - Input: :math:`(N, C_{in}, L_{in})`

Hi, I have installed two versions of CUDA. But I want to use one of it, can I specify it during setup process?
Try running
I have tried it, () still returns False. Thx for your reply!
Do you have the cuda driver installed?
Of course I do. It raise assertion error which shows "The NVIDIA driver on your system is too old(found version 8000)".
that's the answer to your problem. upgrade your driver.
Thx!

When building from source, I got this error. My system is OSX 10.12.4, conda, CUDA 8.0. Any thoughts? Thanks!
can you try adding to the file and see if that fixes things? if so i'll commit it.
actually, that file already has memory included. Are you following instructions from Especially: This part is important:
works nicely! Thanks a lot!

The suggest that takes a as a parameter, but it actually must be wrapped in a Variable first.
you are looking at the Tensor docs. When operating on Variable, it is implied from the tutorials that all functions operating on Variables have to use Variables. We didn't write separate docs for Variable itself.

something wrong with import torchvision Traceback (most recent call last): File "", line 3, in <moduleimport torchvision File "", line 4, in <moduleimport torchvision.transforms as transforms ImportError: No module named transforms
you have a file called for some reason. delete it or change your pythonpath to ignore it.
Hey, I have the same problem persisting when I import torchvision. I have no other file by the same name. ImportError Traceback (most recent call last) <ipython-input-6-3baf6837d8f1in <module() 8 from import Dataset 9 from import DataLoader ---10 from torchvision import transforms ImportError: No module named torchvision Any idea why this is happening?
Do you have torchvision installed (it is a separate package)?
pip Install torchvision try this
from data import BaseTransform,VOC_CLASSES as labelmap this is not working
pip install torchvision

Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

def inithidden(self): h0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) c0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) return (h0, c0) def forward(self, sequences): lstmout, = ((1, self.batchlen, self.inputdim), ) ... = nn.LSTM(inputdim, hiddendim, numlayers, dropout = dropout, batchfirst=True) = self.inithidden() output = model(traininput) produces the following error although I defined weight and input dimensions as in the documentation: File "/usr/local/lib/python2.7/dist-", line 202, in call result = self.forward(input, kwargs) File "/usr/local/lib/python2.7/dist-", line 91, in forward output, hidden = func(input, self.allweights, hx) File "/usr/local/lib/python2.7/dist-", line 327, in forward return func(input, fargs, fkwargs) File "/usr/local/lib/python2.7/dist-", line 227, in forward nexth, output = func(input, hidden, weight) File "/usr/local/lib/python2.7/dist-", line 67, in forward hy, output = inner(input, hidden[l], weight[l]) File "/usr/local/lib/python2.7/dist-", line 96, in forward hidden = inner(input[i], hidden, *weight) File "/usr/local/lib/python2.7/dist-", line 22, in LSTMCell gates = F.linear(input, wih, bih) + F.linear(hx, whh, bhh) File "/usr/local/lib/python2.7/dist-", line 744, in add return (other) File "/usr/local/lib/python2.7/dist-", line 293, in add return (other, False) File "/usr/local/lib/python2.7/dist-", line 287, in add return Add(inplace)(self, other) File "/usr/local/lib/python2.7/dist-", line 20, in forward return a.add(b) RuntimeError: inconsistent tensor size at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:827
I can't reproduce your problem. Could you include a short runnable example that exhibits this behavior?

You also don't need to manually init hidden anymore, the following would work as well...
Thanks! After removing problem solved.

Hi, I would like to implement a sharing weight 'alexnet' with two input batches of images, and the fully connected layers are concatenated together like this: However, when I run it, I come across the following error: Does anyone know about it? Thank you very much.
You forgot to call on the output of . In the future please remember that we're using the issues for bug reports and feature requests only. If you have questions or need help, please post .


Question: #include "THHalf.h" <del> /* * Copyright 1993-2014 NVIDIA Corporation.  All rights reserved. * * NOTICE TO LICENSEE: * * This source code and/or documentation ("Licensed Deliverables") are * subject to NVIDIA intellectual property rights under U.S. and * international Copyright laws. * * These Licensed Deliverables contained herein is PROPRIETARY and * CONFIDENTIAL to NVIDIA and is being provided under the terms and * conditions of a form of NVIDIA software license agreement by and * between NVIDIA and Licensee ("License Agreement") or electronically * accepted by Licensee.  Notwithstanding any terms or conditions to * the contrary in the License Agreement, reproduction or disclosure * of the Licensed Deliverables to any third party without the express * written consent of NVIDIA is prohibited. * * NOTWITHSTANDING ANY TERMS OR CONDITIONS TO THE CONTRARY IN THE * LICENSE AGREEMENT, NVIDIA MAKES NO REPRESENTATION ABOUT THE * SUITABILITY OF THESE LICENSED DELIVERABLES FOR ANY PURPOSE.  IT IS * PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND. * NVIDIA DISCLAIMS ALL WARRANTIES WITH REGARD TO THESE LICENSED * DELIVERABLES, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY, * NONINFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE. * NOTWITHSTANDING ANY TERMS OR CONDITIONS TO THE CONTRARY IN THE * LICENSE AGREEMENT, IN NO EVENT SHALL NVIDIA BE LIABLE FOR ANY * SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, OR ANY * DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, * WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS * ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE * OF THESE LICENSED DELIVERABLES. * * U.S. Government End Users.  These Licensed Deliverables are a * "commercial item" as that term is defined at 48 C.F.R. 2.101 (OCT * 1995), consisting of "commercial computer software" and "commercial * computer software documentation" as such terms are used in 48 * C.F.R. 12.212 (SEPT 1995) and is provided to the U.S. Government * only as a commercial end item.  Consistent with 48 C.F.R.12.212 and * 48 C.F.R. 227.7202-1 through 227.7202-4 (JUNE 1995), all * U.S. Government End Users acquire the Licensed Deliverables with * only those rights set forth herein. * * Any use of the Licensed Deliverables in individual and commercial * software must include, in the user documentation and internal * comments to the code, the above Disclaimer and U.S. Government End * Users Notice. */ </del> <ins> /* Copyright 1993-2014 NVIDIA Corporation.  All rights reserved. */ </ins> // Host functions for converting between FP32 and FP16 formats <del> // Paulius Micikevicius (pauliusm@nvidia.com) </del> float TH_half2float(THHalf h) {

When performing license check on Pytorch debian package I found this license problem: License of Pytorch is BSD-3-Clause, however THHalf.c is non-free according to its header.
A possible replacement can be found in in NNPACK (BSD-3 licensed).
Actually NVIDIA's copyright declaration in THHalf.c makes it impossible for Pytorch to enter the main section of Debian archive. To promote to official package, we should have copyright problems cleared at first.
We're working on getting this cleared -- sorry for the inconvenience.
Marking as done since was merged.
It's my bad. The fix entered torch/torch7 but is still not synced into pytorch.
Did you want me to patch pytorch separately? I figured the file can now be copied over from torch by anyone.
i'll sync the subtrees now!
Thank you guys.


Question: 'Cumprod', 'Gather', 'MaskedScatter', <del> 'Sort', 'Topk', </del> 'Norm', 'Prod', ))

Just a very small grammatical issue. ! Suggested fix: "mutates" to "mutate" <!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --
Thanks for pointing this out. Please feel free to open a pull request for this.
Thanks for adding labels. Forgot to do that. I will make a pull request


Question: self += parameters def __getitem__(self, idx): <del> if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._parameters[str(idx)] </del> <ins> if isinstance(idx, slice): return ParameterList(list(self._parameters.values())[idx]) else: if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._parameters[str(idx)] </ins> def __setitem__(self, idx, param): return self.register_parameter(str(idx), param)

it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.


Question: upstream="$1" pr="$2" git diff --name-only "$upstream" "$pr" <del> git diff --name-only "$upstream" "$pr" | grep -Eq '^(aten/|caffe2/|.jenkins/pytorch|docs/(make.bat|Makefile|requirements.txt|source)|mypy|requirements.txt|setup.py|test/|third_party/|tools/|.gitmodules|torch/)' </del> <ins> # Now that PyTorch build depends on Caffe2, unconditionally trigger # for any changes. # TODO: Replace this with a NEGATIVE regex that allows us to blacklist # files (letting us skip builds when they are unnecessary) #git diff --name-only "$upstream" "$pr" | grep -Eq '^(aten/|caffe2/|.jenkins/pytorch|docs/(make.bat|Makefile|requirements.txt|source)|mypy|requirements.txt|setup.py|test/|third_party/|tools/|.gitmodules|torch/)' </ins>

This is most obvious by observing binary sizes, for example on pypi: (6.0 MB ) (33.8 MB ) Also, from the build log it's clear that 3.8 for some reason build with flag: CI cc
Which is a good old

Einsum currently modifies variables in-place (without an explicit indication that it does so), which prevents pytorch from automatically backpropagating. Results in the following runtime error: Demonstration and discussion can also be found . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): pip OS: PyTorch version: 0.4.0 Python version: 3.6.1
Oops. Thank you for reporting and the minimal example. I'll see to get that fixed.
Saw the same issue over here. Thank you !
Is cloning the tensor before passing to einsum a valid workaround?
Yes, but a somewhat expensive one. If you have the ability to recompile, you could also apply the PR fixing this. Somehow it seems to be stuck in the review queue because it hit an unrelated CI problem...
Okay so how to use it then?
Just get a recent enough master and it will work.


Question: - **c_n** (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len Attributes: <del> weight_ih_l[k] : the learnable input-hidden weights of the k-th layer `(W_ir|W_ii|W_in)`, of shape `(input_size x 3*hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the k-th layer `(W_hr|W_hi|W_hn)`, of shape `(hidden_size x 3*hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the k-th layer `(b_ir|b_ii|b_in)`, of shape `(3*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the k-th layer `(W_hr|W_hi|W_hn)`, of shape `(3*hidden_size)` </del> <ins> weight_ih_l[k] : the learnable input-hidden weights of the k-th layer `(W_ii|W_if|W_ig|W_io)`, of shape `(input_size x 4*hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the k-th layer `(W_hi|W_hf|W_hg|W_ho)`, of shape `(hidden_size x 4*hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the k-th layer `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the k-th layer `(W_hi|W_hf|W_hg|b_ho)`, of shape `(4*hidden_size)` </ins> Examples::

From : Happened because of some change in the environment as it is also affecting historical builds. cc
This issue seems to come from within libuv, a dependency of TensorPipe. First guess: there's two conflicting versions of libuv (e.g., a preinstalled one and the one shipped as a submodule of TensorPipe), and we build the one shipped by TensorPipe but somehow it ends up including the headers of the other one? In particular, the constant it's complaining about was in , which became part of libuv 1.41.0, which is the version that TensorPipe bundles. But it seems those jobs are installing version 1.40.0 via conda: So the question is why are the conda headers picked up and prioritized? I feel this will require some CMake digging, and it's possible that the fix will have to land in libuv's upstream CMake files. Ugh.
BTW, if you want to validate this theory you can try setting this flag to : Which will cause TensorPipe to use the preinstalled libuv instead of building its own one. But please don't land this "workaround", as it will only lead to users coming to complain that they couldn't build PyTorch because their system has a libuv version that is too old for what TensorPipe needs.
this is definitely what is happening: libuv-1.40 is installed from conda, which does not have in the header: But I'm still confused why it started to happen just now [Edit] Can it be update that happened 4 hours ago?
Finally found a PR that caused a regression: (and testing were clearly regressed in CI: )


Question: // a test for Py_None here; instead, you need to mark the argument // as *allowing none*; you can do this by writing 'Tensor?' instead // of 'Tensor' in the ATen metadata. <del> throw TypeError("expected Variable as argument %d, but got %s", i, </del> <ins> throw TypeError("expected Tensor as argument %d, but got %s", i, </ins> Py_TYPE(args[i])->tp_name); } return reinterpret_cast<THPVariable*>(args[i])->cdata;

Considering the merge of and , and the "removal" of from the API, there are still some messages that reference . This might prove to be confusing for new users starting off from , as they probably might not have heard of . I did a code search to find out multiple instances of this in . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): source (@ ) Build command you used (if compiling from source): OS: Ubuntu 16.04.2 PyTorch version: Python version: 3.5 CUDA/cuDNN version: 8.0.44 GPU models and configuration: GeForce 940M GCC version (if compiling from source): 5.4.0 CMake version: 3.5.1


Question: self.assertEqual(torch.mm(flattened_tensor, flattened_tensor.t()), torch.eye(rows) * gain ** 2, prec=1e-6) <ins> # Generates rand tensor with non-equal values. This ensures that duplicate # values won't be causing test failure for modules like MaxPooling. # size should be small, otherwise randperm fails / long overflows. def _rand_tensor_non_equal(*size): total = reduce(mul, size, 1) return torch.randperm(total).view(*size).double() </ins> def add_test(test): test_name = test.get_name()

Intermittent test failures occurring on AdaptiveMaxPool3d. and I have seen this in the past where Max pooling has exactly the same values in the same window, then it's differently resolved on CPU/GPU. For example [0 2.5 2.5 3] can give a max index of or on CUDA depending on the runtime. So generating input values that are within epsilon of each other is important (especially at half precision). fix that one.
Will do. Thanks for identifying the issue.


Question: @skipIfNoLapack def test_gels(self): <del> def _test(a, b, expectedNorm): </del> <ins> def _test_underdetermined(a, b, expectedNorm): m = a.size()[0] n = a.size()[1] assert(m <= n) </ins> a_copy = a.clone() b_copy = b.clone() res1 = torch.gels(b, a)[0]

b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.


Question: return pad_bias<mem_eff_alignment>(attn_mask); } // Check and make the tensor contiguous if needed <del> if (attn_mask.sym_stride(0) % 16 != 0 || attn_mask.sym_stride(1) % 16 != 0 || attn_mask.sym_stride(2) % 16 != 0) { </del> <ins> auto needs_contig = [](const c10::SymInt& stride) { return (stride % 16 != 0) || (stride == 0); }; if (needs_contig(attn_mask.sym_stride(0)) || needs_contig(attn_mask.sym_stride(1)) || needs_contig(attn_mask.sym_stride(2)) || needs_contig(attn_mask.sym_stride(3))) { </ins> return attn_mask.contiguous(); }

Hi after more hours of debugging than I am comfortable to admit, I noticed the following breaking change between PyTorch 2.0.1 and PyTorch 2.1. The issue can be reproduced both with & . There is no issue on 2.0.1 For fp32 inputs to SDPA on CUDA device & passing a custom , we have the following: PyTorch 2.0.1: dispatches to math ( for other backends) PyTorch 2.1: dispatches to mem-efficient attention (likely following ). And it appears that the mem-efficient attention backend outputs wrong results when passing non-contigous key/value, while the math backend goes just fine. Reproduction: The result is: Note that these metrics are not super representative of how the drift is bad locally: ! The relevant tensors are attached in a zip here. Thank you! cc
So for this case the mask passed in is all Trues which will get converted to a "zeros" additive attention_bias mask.. in theory a no-op. When I run the same q,k,v input with no mask I get which leads me to believe that non-contiguous inputs alone are not the problem but non-contiguous + masking is no bueno
If I update the script to only call contiguous on either key or value but not both I still see the same results
Okayy I got it, so this happens when bias_mask is not contiguous
Weird - so not being contiguous was responsible of the issue?
I can confirm this issue is fixed on
Unfortunately this solutions makes the (1, 1, seqlenq, seqlenkv case) have to fully materialize the attention mask. I think that a third component of this bug is that it is occuring when seqlenq = 1, I tried with seqlen_q = 2 and it appears to not occur anymore
I was acting somewhat rash with the impeding 2.1.1 deadline and I am unhappy with my solution, and I didn't fully think through its ramifications. For a very hot path I.e. a mask of (1, 1, seqlenq, seqlenkv) this causes a substantial memory regression.

I am having a hard time figure out whats goin on with the repro mentioned here: The results of the repo appear to be dependent on order in which the ops are ran, which makes me thing there is some IMA happening, but alas I haven't found it.
Thanks a lot for looking into it sorry for having been a bit pushy about having this in the patch! I appear not to be able to reproduce the logits difference in using and A100-SXM4-80GB, is the output difference appearing using the memory patch
Tentatively marking hi-pri for the regression
Could you help double check if the memory issue is fixed in 2.1.2? Here are the results and myself gets using the reproduction example in the issue: This is the same as what we get from 2.1.0
So the memory issue was "fixed" in 2.1.1 via this PR: While this did change the result of the repro, it caused a memory usage "regression" for a very common hot path for SDPA. This PR: Was used to fix the memory regression. I did further analysis on the initial correctness report. I determined that the repro wasn't behaving as I expected.. e.g the order with which aten ops were called impacted. I am also not entirely sure that a contiguous vs non contiguous input should produce identical results (it likely should but I need to reason if there is some potential float addmulitple ops that get reordered cc ) I was leaving this open in case we need to revisit this but the the memory reduction should be visible via usual memory profiling techniques
I can also confirm that this issue is reintroduced in the 2.1.2 release, while it was fixed in 2.1.1. This is a bit bad for us as following 2.1.1 we SDPA by default in Transformers and a very common execution path has non-contiguous inputs: We'll likely have to do a patch and manually call , I guess. In 2.1.2, this codepath starts erroring out with . Here, silently SDPA starts outputting different results when inputs are not contiguous, which I think is not wishable. If the kernel requires contiguous inputs, why not check it on the C++ side and call if needed (maybe with a warning)?
do you have a repro for the illegal memory access?

I wasn't able to re-repro this with the existing one in this issue.
I don't have a minimal reproduction unfortunately, but you can try: and run with the following script: which errors at: This corresponds to this code: Note that on of Transformers you will not be able to reproduce as for now I just disabled the mem-efficient path for GPT BigCode due to this issue (fix at , removing the expand).
Is it possible to have minimal repro for this ? Running using your instructions I get following:
Oh sorry. You can try with instead that is non-gated. I'll try to come up with a minimal repro ASAP. I edited my message above. For the record using
Running above command produced for me: Notable Environment differences: CUDA-12.1 PyTorch: Main GPU: H100 `
While the repro isn't throwing a IMA compute-sanitizer is reporting errors:
Current gists with compute-sanitizer outputs:
I have a fix:
I rerun the script to confirm that this is fixed in 2.2.0
Okay I am going to close this issue for now


Question: self.assertEqual(reference[2, 2, 2, ...], 27, 0) self.assertEqual(reference[...], reference, 0) <del> reference_5d = self._consecutive((3, 3, 3, 3, 3)) </del> <ins> reference_5d = conv_fn(consec((3, 3, 3, 3, 3))) </ins> self.assertEqual(reference_5d[..., 1, 0], reference_5d[:, :, :, 1, 0], 0) self.assertEqual(reference_5d[2, ..., 1, 0], reference_5d[2, :, :, 1, 0], 0) self.assertEqual(reference_5d[2, 1, 0, ..., 1], reference_5d[2, 1, 0, :, 1], 0) self.assertEqual(reference_5d[...], reference_5d, 0) # LongTensor indexing <del> reference = self._consecutive((5, 5, 5)) idx = torch.LongTensor([2, 4]) </del> <ins> reference = conv_fn(consec((5, 5, 5))) idx = conv_fn(torch.LongTensor([2, 4])) </ins> self.assertEqual(reference[idx], torch.stack([reference[2], reference[4]])) # TODO: enable one indexing is implemented like in numpy # self.assertEqual(reference[2, idx], torch.stack([reference[2, 2], reference[2, 4]]))

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.


Question: return linearIndex; } <ins> static bool hasEmptyTensor(TensorList tensors) { for (auto& tensor : tensors) { if (tensor.defined() && tensor.numel() == 0) { return true; } } return false; } </ins> static std::tuple<Tensor, Tensor> makeLinearIndex(Tensor self, TensorList orig) { <ins> checkIndexTensorTypes(orig); </ins> // first expand ByteTensor (boolean masks) into 1 or more LongTensors auto indices = expandByteTensors(self, orig); <ins> if (hasEmptyTensor(indices)) { return std::make_tuple(self, self.type().toScalarType(kLong).tensor()); } </ins> // next broadcast all index tensors together indices = expand_outplace(indices); // add missing null Tensors so that it matches self.dim()

In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:


Question: return() endif() set(CAFFE2_FOUND_CUDA TRUE) <ins> message(STATUS "Caffe2: CUDA detected: " ${CUDA_VERSION}) message(STATUS "Caffe2: CUDA nvcc is: " ${CUDA_NVCC_EXECUTABLE}) message(STATUS "Caffe2: CUDA toolkit directory: " ${CUDA_TOOLKIT_ROOT_DIR}) if(CUDA_FOUND) # Sometimes, we may mismatch nvcc with the CUDA headers we are # compiling with, e.g., if a ccache nvcc is fed to us by CUDA_NVCC_EXECUTABLE # but the PATH is not consistent with CUDA_HOME.  It's better safe # than sorry: make sure everything is consistent. set(file "${PROJECT_BINARY_DIR}/detect_cuda_version.c") file(WRITE ${file} "" "#include <cuda.h>n" "#include <stdio.h>n" "int main() {n" "  printf("%d.%d", CUDA_VERSION / 1000, (CUDA_VERSION / 10) % 100);n" "  return 0;n" "}n" ) try_run(run_result compile_result ${PROJECT_BINARY_DIR} ${file} CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CUDA_INCLUDE_DIRS}" LINK_LIBRARIES ${CUDA_LIBRARIES} RUN_OUTPUT_VARIABLE cuda_version_from_header COMPILE_OUTPUT_VARIABLE output_var ) if(NOT compile_result) message(FATAL_ERROR "Caffe2: Couldn't determine version from header: " ${output_var}) endif() message(STATUS "Caffe2: Header version is: " ${cuda_version_from_header}) if(NOT ${cuda_version_from_header} STREQUAL ${CUDA_VERSION}) # Force CUDA to be processed for again next time # TODO: I'm not sure if this counts as an implementation detail of # FindCUDA set(${cuda_version_from_findcuda} ${CUDA_VERSION}) unset(CUDA_TOOLKIT_ROOT_DIR_INTERNAL CACHE) # Not strictly necessary, but for good luck. unset(CUDA_VERSION CACHE) # Error out message(FATAL_ERROR "FindCUDA says CUDA version is ${cuda_version_from_findcuda} (usually determined by nvcc), " "but the CUDA headers say the version is ${cuda_version_from_header}.  This often occurs " "when you set both CUDA_HOME and CUDA_NVCC_EXECUTABLE to " "non-standard locations, without also setting PATH to point to the correct nvcc.  " "Perhaps, try re-running this command again with PATH=${CUDA_TOOLKIT_ROOT_DIR}/bin:$PATH.  " "See above log messages for more diagnostics, and see https://github.com/pytorch/pytorch/issues/8092 for more details.") endif() endif() </ins> # Find cuDNN. if(CAFFE2_STATIC_LINK_CUDA)

Here's the situation: Ideally, you'd say something like to make a build of PyTorch with a non-standard version of CUDA. When this happens, needs to find the correct copy of nvcc to use. If you are using ccache as per the instructions in you'll have set to your ccache binary, so that ccache happens. cmake will correctly respect this variable. However, here's the problem: ccache's nvcc will use your path to determine what the real nvcc to use is, this means that you'll pick up whatever nvcc is in your PATH, which is probably not the one you want, because you're trying to use a different nvcc. So in sum, you have to do the non-intuitive to get the whole shebang to work. The worst part is that it won't tell you if you've got it wrong, it will happily use the wrong nvcc. I haven't gone through user submitted issues to see if anyone has succumbed to this particular bug.
A long-term solution might be getting to work
works now.


Question: (4.53, 3.83, -6.64))).t() b = torch.Tensor(((8.58, 8.26, 8.48), (9.35, -4.43, -0.70))).t() <del> _test(a, b, expectedNorm) </del> <ins> _test_underdetermined(a, b, expectedNorm) </ins> # test reuse expectedNorm = 0

b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.


Question: n_dist = THTensor_(size)(prob_dist, 0); n_categories = THTensor_(size)(prob_dist, 1); <del> THArgCheck(n_sample > 0, 2, "cannot sample n_sample < 0 samples"); </del> <ins> THArgCheckWithCleanup(n_sample > 0, THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, "cannot sample n_sample < 0 samples"); </ins> if (!with_replacement) { <del> THArgCheck((!with_replacement) && (n_sample <= n_categories), 2,  "cannot sample n_sample > prob_dist:size(1) samples without replacement"); </del> <ins> THArgCheckWithCleanup((!with_replacement) && (n_sample <= n_categories), THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, "cannot sample n_sample > prob_dist:size(1) samples without replacement"); </ins> } /* cumulative probability distribution vector */

I was reading the documentation of method in the . I tried this in IPython environment. Of course, error occurs because 5 is larger than the size of weights and I didn't set . The error message is like this: But after this, the dimensions of weights changed. If I run as following without runtime exception, then the dimensions of weights do not change. This is caused by not cleaning up after exception raised in this . Maybe the solution is adding cleaning up code in the .
The problem emerged in two other situations as well, here are the examples: I'm sending a pull request with the fix. Thanks for reporting the problem. :)


Question: return r; } <del> std::vector<Tensor> as_tensor_list(std::vector<Variable> &vars) { </del> <ins> static std::vector<Tensor> as_tensor_list(std::vector<Variable> &vars) { </ins> std::vector<Tensor> tensors; for (auto& v : vars) { tensors.emplace_back(std::move(v));

See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:


Question: }; struct PythonEngine : public Engine { <del> virtual void thread_main(std::shared_ptr<ReadyQueue> queue) override { </del> <ins> virtual void thread_main(std::shared_ptr<ReadyQueue> queue, int device) override { </ins> // Create a PyThreadState, but release the GIL. This lets AutoGIL calls // inside thread_main acquire the GIL without having to create a new // PyThreadState each time. AutoGIL gil; AutoNoGIL no_gil; <del> Engine::thread_main(queue); </del> <ins> Engine::thread_main(queue, device); </ins> } virtual void thread_on_exception(FunctionTask& task, std::exception& e) override {

Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: print('-- Detected CUDA at ' + CUDA_HOME) else: print('-- Not using CUDA') <del> if WITH_NCCL and SYSTEM_NCCL: print('-- Using system provided NCCL library') </del> <ins> if WITH_NCCL and WITH_SYSTEM_NCCL: print('-- Using system provided NCCL library at ' + NCCL_LIB_DIR + ', ' + NCCL_INCLUDE_DIR) </ins> elif WITH_NCCL: print('-- Building NCCL library') else:

Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).


Question: going to be volatile. Volatility spreads accross the graph much easier than non-requiring gradient - you only need a **single** volatile leaf to have a volatile output, while you need **all** leaves to not require gradient to <del> have an output the doesn't require gradient. Using volatile flag you don't </del> <ins> have an output that doesn't require gradient. Using volatile flag you don't </ins> need to change any settings of your model parameters to use it for inference. It's enough to create a volatile input, and this will ensure that no intermediate states are saved.

Platforms: mac, macos This test was disabled because it is failing on master (). This fails again , so I disable this test pending the discussion in on how best to clean up these runners. cc
<!-- validation-comment-start --<bodyHello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disable a test in PyTorch CI. The information I have parsed is below: Test name: Platforms for which to skip the test: mac, macos Within ~15 minutes, will be disabled in PyTorch CI for these platforms: mac, macos. Please verify that your test name looks correct, e.g., . To modify the platforms list, please include a line in the issue body, like below. The default action will disable the test for all platforms if no platforms list is specified. We currently support the following platforms: asan, linux, mac, macos, rocm, win, windows.</body<!-- validation-comment-end --
I don't think this is really sustainable (nor productive) as all tests will eventually fail with no space left on device error... Instead, we should be reclaiming space on those nodes...
Yeah, this is only a temporary measure while we continue the investigation in This test writes a 2GB file to disk I think. I'll bring up the issue again tomorrow to get help from the team. At the moment, I have no idea which one is responsible for filling up the disk.
mac M1 runners has been cleaned up, resolving this one
failing again
Got a fix here to increase the minimum space required on MacOS to be 6GB. The current threshold of 4GB is not big enough


Question: IF ($ENV{PYTORCH_BINARY_BUILD}) MESSAGE(STATUS "PYTORCH_BINARY_BUILD detected. Statically linking libstdc++") SET(CMAKE_CXX_FLAGS "-static-libstdc++ ${CMAKE_CXX_FLAGS}") <ins> IF (UNIX AND NOT APPLE) # hiding statically linked library symbols, this flag is not available for the linker under MACOSX SET(CMAKE_CXX_FLAGS "-Wl,--exclude-libs,ALL ${CMAKE_CXX_FLAGS}") ENDIF(UNIX AND NOT APPLE) </ins> ENDIF() ADD_LIBRARY(shm SHARED core.cpp)

see the commit comment: To reproduce install ros-kinetic-tf 1.11.8-0xenial---0800 package from the deb xenial main repository. my system is: lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial cuda version is 8.0


Question: self.assertRaisesRegex(RuntimeError, 'modified by an inplace operation', lambda: z.backward()) <ins> def test_no_grad_input(self): class MyFunction(Function): @staticmethod def forward(self, x): return x @staticmethod def backward(self, grad_output): return grad_output x = torch.randn(5, requires_grad=True) with torch.no_grad(): y = MyFunction.apply(x) self.assertTrue(x.requires_grad) self.assertIsNone(y.grad_fn) </ins> def test_backward_copy(self): # This tests checks backward engine for a very subtle bug that appreared # in one of the initial versions of autograd. Gradients tensors were

OS: ubuntu 16.04 PyTorch version: 0.4.0a0+ How you installed PyTorch (conda, pip, source): source Python version: 3.6.3 CUDA/cuDNN version: v9.1, cudnn-9.1-linux-x64-v7 GPU models and configuration: 4x Tesla M60 GCC version (if compiling from source): 5.4.0 When using context () on a model with dataparallel operation (same with ) on 1 GPU the flag requires_grad does not come back to True. If only one GPU is used then everything works fine. Script: On 2 GPUs, this outputs which is wrong in the last line. On 1 GPU (set ) the output is correct.
is looking into it


Question: dim_args_idx = test[4] if len(test) == 5 else [] <ins> skipTestIf = test[5] if len(test) == 6 else [] </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name new_constructor_args = [arg * dim_perm[dim_args_idx.index(i)] if i in dim_args_idx else arg

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: { char transpose_r, transpose_m1, transpose_m2; THTensor *r__, *m1_, *m2_; <ins> int free_m1 = 0; int free_m2 = 0; </ins> if( (m1->nDimension != 2) || (m2->nDimension != 2)) THError("matrices expected, got %dD, %dD tensors", m1->nDimension, m2->nDimension);

Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.


Question: raise assert not hasattr(TestAutograd, test_name), 'Two tests have the same name: ' + test_name <ins> for skip in skipTestIf: do_test = skip(do_test) </ins> setattr(TestAutograd, test_name, do_test)

Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc


Question: np.int64, np.int32, np.int16, <del> np.uint8 </del> <ins> np.uint8, np.longlong, </ins> ] for dtype in dtypes: array = np.array([1, 2, 3, 4], dtype=dtype)

Minimal example: Torch initializes the tensor with values which are different from the numpy array. I've seen issues for similar kinds of problems (see ). Maybe the easiest solution to all these initialization problems might be, to check for the same values of the original and initialized array/tensor instead of initializing tensors silently with wrong numbers. What do you think? Furthermore, I've opened an issue which is also related this one in terms of initializing a numpy array with a number of type and then initializing a torch tensor from that array, see Feel free to merge them if you think that both issues can be solved together.
Could you please provide some information about what version of pytorch and numpy you're using? ! It works as expected for me on master
I'm using torch 0.3.0.post4 (latest one on website of PyTorch) and numpy 1.13.3.
I'm running the same thing, this is strange. In the first line , I assumed that came from . Is this the case?
Nope, it's indeed the python built-in type. Actually this issue came up when I tried to get the shape of (i.e. , , returns ).
Thanks for the clarification, I'm now seeing your results. Looks like the tensor constructor doesn't work well with python2.7 long types in this case
Sure, no problem. Thanks for looking into it!
The problem seems to be that np.longlong and np.int64 are the same thing (at least on a 64bit system, I think), but the NPYLONGLONG and NPYINT64 macros are different... I'll put up a pr for this soon.

A minimal example: Although has and this type is listed as supported, the array can't be converted. Consider that is initialized with an upper bound of type . I'm not completely sure, whether this issue is related to torch or numpy, but numpy initializes the array with values as expected, whereas torch throws an inconsistent error message.
Edit: nevermind, I didn't run the same exact thing ! Works for me on master and 0.3. Seems like it could be related to


Question: (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), (IndexFill,     (0, 2),             ((S, S), index_variable(2, S))              ), (IndexSelect,   (0,),               ((S, S), index_variable(2, S))              ), <ins> (Gather,        (0,),               ((M, S), gather_variable((S, S), 1, M))     ), (Gather,        (1,),               ((M, S), gather_variable((M, S//2), 0, S)), 'dim1'), (Scatter,       (0,),               ((M, S), gather_variable((S, S), 1, M), (S, S))), (Scatter,       (1,),               ((M, S), gather_variable((M, S//2), 0, S), (M, S//2)), 'dim1'), </ins> (Concat,        (0,),               ((1, S, S), (2, S, S), (3, S, S))           ), (Resize,        (S*S, S),           ((S, S, S),)                                ), (Diag,          (),                 ((S, S),),                  '2d'            ),

<!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --In the getting started section section the following quote appears: I guess this is from previous versions where CUDA support was not built-in and this can be removed, correct? cc
This seems to be easy fixable docs issue
If gets merged this is fixed.
Thanks for the PR Just merged it.


Question: lib_path = os.path.join(cwd, "torch", "lib") <del> # Check if you remembered to check out submodules def check_file(f): if not os.path.exists(f): print("Could not find {}".format(f)) print("Did you run 'git submodule update --init'?") sys.exit(1) check_file(os.path.join(lib_path, "gloo", "CMakeLists.txt")) check_file(os.path.join(lib_path, "nanopb", "CMakeLists.txt")) check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) </del> tmp_install_path = lib_path + "/tmp_install" include_dirs += [ cwd,

It seems like running has a dependency on first running . This command also seems to also error out when installing via a wheel. This creates a problem for tools like  which rely on the output of to examine dependencies. Is there anyway to allow this command to not error out when installing via a wheel? Full stack trace: I understand the preferred package manager for this project is conda but being officially endorsed from , pipenv has a fairly large audience:
thanks do you know when's the next planned release?
Hopefully within a month


Question: THCDeviceTensor<Dtype, 1> gradOutput, THCDeviceTensor<Dtype, 2> gradInput, Dtype *weights, <ins> int n_classes, </ins> int ignore_index) { CUDA_KERNEL_LOOP(index, batch_size) {

Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3


Question: if (is_padding_neg()) throw std::runtime_error("negative padding is not supported"); if (is_output_padding_neg()) throw std::runtime_error("negative output_padding is not supported"); <del> AutoGPU guard(input_.data->getDevice()); auto input = input_.unpack_data()->contiguous(); </del> <ins> auto input = input_.unpack_data(); AutoGPU guard(input->getDevice()); input = input->contiguous(); </ins> std::unique_ptr<Tensor> weight(weight_.unpack_data()->clone_shallow()); auto bias = bias_.unpack_data(); auto grad_output = grad_outputs[0]->data->contiguous();

Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)


Question: {"_cuda_getDevice",   (PyCFunction)THCPModule_getDevice_wrap,   METH_NOARGS,  NULL}, {"_cuda_getDeviceCount", (PyCFunction)THCPModule_getDeviceCount_wrap, METH_NOARGS, NULL}, {"_cuda_getDeviceName", (PyCFunction)THCPModule_getDeviceName_wrap, METH_O,   NULL}, <ins> {"_cuda_getDeviceCapability", (PyCFunction)THCPModule_getDeviceCapability_wrap, METH_O,   NULL}, </ins> {"_cuda_getCurrentStream", (PyCFunction)THCPModule_getCurrentStream_wrap, METH_NOARGS, NULL}, {"_cuda_getCurrentBlasHandle", (PyCFunction)THCPModule_getCurrentBlasHandle_wrap, METH_NOARGS, NULL}, {"_cuda_setStream",    (PyCFunction)THCPModule_setStream_wrap,  METH_O, NULL}, {"_cuda_isDriverSufficient", (PyCFunction)THCPModule_isDriverSufficient, METH_NOARGS, NULL}, {"_cuda_getDriverVersion", (PyCFunction)THCPModule_getDriverVersion, METH_NOARGS, NULL}, <ins> {"_cuda_getCompiledVersion", (PyCFunction)THCPModule_getCompiledVersion, METH_NOARGS, NULL}, </ins> {"_cuda_getRNGState", (PyCFunction)THCPModule_getRNGState,      METH_NOARGS,  NULL}, {"_cuda_setRNGState", (PyCFunction)THCPModule_setRNGState,      METH_O,       NULL}, {"_cuda_emptyCache", (PyCFunction) THCPModule_emptyCache,       METH_NOARGS,  NULL},

There's quite a few user-reports of installing the CUDA 7.5 binary of pytorch and then running on GTX 1080 / 1070, and there will be reports of installing 7.5 / 8.0 binaries and running on V100. Raising a user-warning in such situations and letting them know will fix the problem.
The version of CUDA used to compile the binary should be available thanks to this PR:
I think is a more robust way of getting the CUDA used to compile PyTorch, as currently if there are multiple CUDA installed in the machine, the heuristics used in might not match those used by CMake FindCUDA. I think that we should modify the to return the cuda version used during compilation. I have the impression that it will not require changing much because doesn't require cuda to execute. What do you think?


Question: extra_link_args = [] if IS_WINDOWS: <del> extra_compile_args = ['/Z7', '/EHa', '/DNOMINMAX' </del> <ins> extra_compile_args = ['/Z7', '/EHa', '/DNOMINMAX', '/wd4267', '/wd4251', '/wd4522', '/wd4522', '/wd4838', '/wd4305', '/wd4244', '/wd4190', '/wd4101', '/wd4996', '/wd4275' </ins> # /Z7 turns on symbolic debugging information in .obj files # /EHa is about native C++ catch support for asynchronous # structured exception handling (SEH) # /DNOMINMAX removes builtin min/max functions <ins> # /wdXXXX disables warning no. XXXX </ins> ] if sys.version_info[0] == 2: # /bigobj increases number of sections in .obj file, which is needed to link

That's pretty big! In contrast, an OS X build log is only 1.3M. Is there a reason they have to be this big? CC
My guess is that it's because of warnings. I remember having to get through thousands of them when looking for the error that failed the build.
Yes, the major part of them are warnings. And some of them is not fixable. We can hide them using some flags on the compilers.
And this build configuration actually contains both the CPU and GPU build, which may also be a possible reason.
The reduced log size after PR is about 1.2M.


Question: 0.0000  0.0000 -1.6250 [torch.FloatTensor of size 3x3] <del> >>> torch.triu(a, k=1) </del> <ins> >>> torch.triu(a, diagonal=1) </ins> 0.0000  1.7304  1.4573 0.0000  0.0000 -0.1809 0.0000  0.0000  0.0000 [torch.FloatTensor of size 3x3] <del> >>> torch.triu(a, k=-1) </del> <ins> >>> torch.triu(a, diagonal=-1) </ins> 1.3225  1.7304  1.4573 -0.3052 -0.3111 -0.1809

The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.


Question: torch.mm(a, b, out=self.output) else: if self.transA: <del> a = a.transpose(2, 3) </del> <ins> a = a.transpose(1, 2) </ins> if self.transB: <del> b = b.transpose(2, 3) </del> <ins> b = b.transpose(1, 2) </ins> self.output.resize_(a.size(0), a.size(1), b.size(2)) torch.bmm(a, b, out=self.output)

Hi I think that in lines 28 and 30: if self.transA: a = a.transpose(2, 3) if self.transB: b = b.transpose(2, 3) should be: if self.transA: a = a.transpose(1, 2) if self.transB: b = b.transpose(1, 2) Indeed in that branch the tensor has 3 dimension and thus the code crashes. Maybe an indexing error when translating from lua ? Thank you very much,
Yes, you're right. Could you send a PR with a fix? Thanks!
Yes of course, just done, thank you very much.


Question: last_epoch (int): The index of last epoch. Default: -1. Example: <del> >>> # Assuming optimizer uses lr = 0.5 for all groups </del> <ins> >>> # Assuming optimizer uses lr = 0.05 for all groups </ins> >>> # lr = 0.05     if epoch < 30 >>> # lr = 0.005    if 30 <= epoch < 60 >>> # lr = 0.0005   if 60 <= epoch < 90

Hi, In the is written: "lr values are [0.05, 0.005, ...]." If I understand it correctly, it should be [0.5, 0.05, 0.005, ...].
That is correct. Please feel free to send a PR to fix it! (otherwise, I can)
If you don't mind to change it yourself, I'm not contributor yet and don't want to start just for one comment. I guess is the same case.


Question: else: if exp_max > prec + 1 or exp_max < 0: sz = max(min_sz, 7) <del> scale = math.pow(10, exp_max - 1) </del> <ins> scale = math.pow(10, max(exp_max - 1, __MIN_LOG_SCALE)) </ins> else: if exp_max == 0: sz = 7

on print(torch.DoubleTensor(1)) I got ZeroDivisionError: float division by zero but torch.DoubleTensor(0)) or torch.DoubleTensor(2)) work just fine. I work in Jupyter notebook pytorch 0.1.12
Note that returns undefined memory. Could you find the specific content of the tensor that gave this error, by printing ? I'm not sure if this bug has been fixed in the latest release already.
the error is pretty interesting. I restarted Jupyter notebook. # many freaking signatures: print(torch.FloatTensor(1)) print(torch.IntTensor(1)) print(torch.DoubleTensor(1)) # no error right after initial run. #print(torch.DoubleTensor(0)) print(torch.DoubleTensor(2)) print(torch.ByteTensor(1)) print(torch.CharTensor(1)) print(torch.ShortTensor(1)) print(torch.LongTensor(1)) steps to reproduce the error: 1). run, no errors 2) repeat, no error 3) uncomment ..DoubleTensor(0), run, got an error (see below): 4) comment ..DouleTensor(0), run still have this error 2.+24 [torch.FloatTensor of size 1] 7.+08 [torch.IntTensor of size 1] ZeroDivisionError Traceback (most recent call last) <ipython-input-5-3629c250c505in <module() 2 print(torch.FloatTensor(1)) 3 print(torch.IntTensor(1)) ----4 print(torch.DoubleTensor(1)) # ZeroDivisionError: float division by zero 5 #print(torch.DoubleTensor(0)) 6 print(torch.DoubleTensor(2)) C:UsersleogAppDataLocalcondacondaenvspy35libsite-packagestorch in str(self) 126 # characters to replace unicode characters with.

127 if sys.versioninfo (3,): --128 return (self) 129 else: 130 if hasattr(, 'encoding'): C:UsersleogAppDataLocalcondacondaenvspy35libsite-packagestorch in str(self) 284 return '[{} with no dimension]n'.format(torch.typename(self)) 285 elif self.ndimension() == 1: --286 strt = vectorstr(self) 287 elif self.ndimension() == 2: 288 strt = matrixstr(self) C:UsersleogAppDataLocalcondacondaenvspy35libsite-packagestorch in vectorstr(self) 270 if () < PRINTOPTS.threshold: 271 return (strt + --272 'n'.join(ident + (val / scale) for val in self) + 273 'n') 274 else: C:UsersleogAppDataLocalcondacondaenvspy35libsite-packagestorch in <genexpr(.0) 270 if () < PRINT_OPTS.threshold: 271 return (strt + --272 'n'.join(ident + (val / scale) for val in self) + 273 'n') 274 else: ZeroDivisionError: float division by zero
Ok, I got a repro. Here is the failing case: Fails with the error you mentioned. This seems to be an edge case though, as both and works. The issue is in , where becomes zero because is a too large negative number. Maybe we should clip to be positive?


Question: CMAKE_POLICY(SET CMP0026 OLD) ENDIF() <ins> IF(MSVC AND NOT "${CMAKE_BUILD_TYPE}" MATCHES "Debug") SET(MSVC_OPT_FLAG "/Ox ") ELSE() SET(MSVC_OPT_FLAG "") ENDIF() </ins> ######################## # SET_SOURCE_FILES_PROPERTIES must be in the same CMakeLists.txt file as the target that includes the file # so we need to set these commands here rather than in src/TH IF(C_SSE4_1_FOUND AND C_SSE4_2_FOUND) IF(MSVC) <del> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_sse.c PROPERTIES COMPILE_FLAGS "/Ox /fp:fast") </del> <ins> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_sse.c PROPERTIES COMPILE_FLAGS "${MSVC_OPT_FLAG}/fp:fast") </ins> ELSE(MSVC) SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_sse.c PROPERTIES COMPILE_FLAGS "-O3 -ffast-math") ENDIF(MSVC) ENDIF(C_SSE4_1_FOUND AND C_SSE4_2_FOUND) IF(C_AVX_FOUND) IF(MSVC) <del> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_avx.c PROPERTIES COMPILE_FLAGS "/Ox /fp:fast ${C_AVX_FLAGS}") SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX.c PROPERTIES COMPILE_FLAGS "/Ox /arch:AVX ${C_AVX_FLAGS}") </del> <ins> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_avx.c PROPERTIES COMPILE_FLAGS "${MSVC_OPT_FLAG}/fp:fast ${C_AVX_FLAGS}") SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX.c PROPERTIES COMPILE_FLAGS "${MSVC_OPT_FLAG}/arch:AVX ${C_AVX_FLAGS}") </ins> ELSE(MSVC) SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/generic/simd/convolve5x5_avx.c PROPERTIES COMPILE_FLAGS "-O3 -ffast-math ${C_AVX_FLAGS}") SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX.c PROPERTIES COMPILE_FLAGS "-O3 ${C_AVX_FLAGS}")

The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.