Question: import math from bisect import bisect_right <ins> from functools import partial </ins> from .optimizer import Optimizer

I'm working on checkpointing in our nn trainer. When saving the state objects, I get an error: PicklingError: Can't pickle <type 'function': attribute lookup builtin.function failed I tracked it down to trying to save the scheduler ReduceLROnPlateau. I think the issue is due to the scheduler referencing the optimizer, which has to be saved with statedict(). I presume the same error will occur with other schedulers too. It seems like the problem could be solved by providing a statedict() and loadstatedict() in the scheduler classes. pytorch master rev (Jan 22) installed from source x86_64 on centos 7.4 cuda 9.0 cudnn 7 k80 gcc 5.4 python 2.7.14
The following script reproduces the crash: import torch import from import ReduceLROnPlateau if name == 'main': model = (10,10) optim = (model.parameters()) sched = ReduceLROnPlateau(optim, mode='max', factor=0.5) with open('','wb') as f: # Works # ((model.statedict(), optim.statedict()), f) # Crashes ((model.statedict(), optim.state_dict(), sched), f)


Question: ctx.num_inputs = len(inputs) ctx.input_device = inputs[0].get_device() outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus) <ins> non_differentiables = [] for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]): if not input_requires_grad: for output in outputs: non_differentiables.append(output[idx]) ctx.mark_non_differentiable(*non_differentiables) </ins> return tuple([t for tensors in outputs for t in tensors]) @staticmethod

PyTorch version: 0.3.0 Python version: 3.5 GPU models and configuration: 8x NVIDIA Titan I found that requiresgrad setting in module is ignored when module is wrapped with nn.DataParallel. For example, This code should be executed because very small part of network activation (only fc layer) is stored for backward computation, but it results to run-time error: Internally, I found a crack in function which is in . In replicate function, it copies all parameter in module (# of replica times) with . In broadcasting code, it just defines new with default constructor parameter, which is always set to True. I think there can be some choices to fix this issue. is intended behavior for DataParallel, so we should use volatile (or ) to implement transfer learning in DataParallel model should be fixed to handle requires_grad when copying parameters. should be synchronizing in I think any of solution doesn't need lots of effort to fix it.


Question: THCDeviceTensor<THCIndex_t, 1> target, THCDeviceTensor<Dtype, 1> output, Dtype *weights, <ins> int n_classes, </ins> int ignore_index) { CUDA_KERNEL_LOOP(index, batch_size) {

Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3


Question: skipTestIf = test[5] if len(test) >= 6 else [] <del> test_output_indices = test[6] if len(test) >= 7 else None </del> <ins> output_process_fn = test[6] if len(test) >= 7 else lambda x: x </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name

I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.


Question: y.sum().backward() self.assertEqual(x.grad.data, torch.ones(5, 5) * 2) <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") def test_backward_device(self): # check that current device matches the variable's device device = [None] class Identity(torch.autograd.Function): @staticmethod def forward(ctx, x): return x.clone() @staticmethod def backward(ctx, grad_output): device[0] = torch.cuda.current_device() return grad_output.clone() v = Variable(torch.randn(1).cuda(1), requires_grad=True) Identity.apply(v).backward() self.assertEqual(device[0], 1) </ins> def test_detach(self): x = Variable(torch.randn(10, 10), requires_grad=True) y = x + 2

Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: with values from the interval ``[start, end)`` taken with common difference :attr:`step` beginning from `start`. <ins> Note that non-integer `step` is subject to floating point rounding errors when comparing against `end`; to avoid inconsistency, we advise adding a small epsilon to `end` in such cases. </ins> .. math:: out_{i+1} = out_{i} + mathrm{step}

! OS: Ubuntu 17.10 PyTorch version: 0.3.1 How you installed PyTorch (conda, pip, source): conda Python version: 3.6.4 CUDA/cuDNN version: - GPU models and configuration: -
This is a floating point problem that has been mitigated in master with this commit: But even with that fix, arange doesn't always give the expected result. You can avoid this issue by doing
It’s impossible to have arange work in all kinda of possible conditions due to rounding errors in floating point math. What we did in master was to make it consistent with numpy. An alternate solution would be to avoid having the length of the interval be exactly divisible by its length (you can make it a tiny bit larger).
The numpy docs have a note saying "When using a non-integer step, such as 0.1, the results will often not be consistent." Maybe this could be to the doc as well?


Question: torch/csrc/nn/THNN_generic.cwrap torch/csrc/nn/THNN_generic.cpp torch/csrc/nn/THNN_generic.h <ins> torch/csrc/generated </ins> docs/src/**/* test/data/legacy_modules.t7 test/data/gpu_tensors.pt

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: :math:`p_n > 1` increases the recall, :math:`p_n < 1` increases the precision. For example, if a dataset contains 100 positive and 300 negative examples of a single class, <del> then `pos_weight` for the class should be equal to math:`frac{300}{100}=3`. </del> <ins> then `pos_weight` for the class should be equal to :math:`frac{300}{100}=3`. </ins> The loss would act as if the dataset contains math:`3times 100=300` positive examples. Args:

is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.


Question: { transpose_m1 = (transpose_r == 'n' ? 't' : 'n'); m1_ = THTensor_(newContiguous)(m1); <ins> free_m1 = 1; </ins> } /* m2 */

Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.


Question: l.param_attr = None self.assertEqual(num_params(), 3) <ins> def test_assignment(self): l = nn.Module() a = nn.Parameter(torch.randn(2)) b = nn.Parameter(torch.randn(3)) c = nn.Parameter(torch.randn(4)) q = nn.Linear(4, 4) r = nn.Linear(5, 5) w = nn.Linear(6, 6) def test_assignments(get_list, a, b, c): # Check that None can be shadowed l.a = None self.assertIsNone(l.a) self.assertIn('a', l.__dict__) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [a]) self.assertNotIn('a', l.__dict__) # Assign second object l.b = None self.assertIsNone(l.b) self.assertIn('b', l.__dict__) l.b = b self.assertIs(l.b, b) self.assertEqual(get_list(), [a, b]) self.assertNotIn('b', l.__dict__) # Remove and add the object back. Order should be unchanged. l.a = None self.assertIsNone(l.a) self.assertEqual(get_list(), [b]) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [a, b]) # Replace object with another one. Order should be unchanged. l.a = c self.assertIs(l.a, c) self.assertEqual(get_list(), [c, b]) # Remove and reassign an attribute. It should appear at the end of the list now. del l.a self.assertFalse(hasattr(l, 'a')) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [b, a]) test_assignments(lambda: list(l.parameters()), a, b, c) del l.a, l.b self.assertEqual(list(l.parameters()), []) test_assignments(lambda: list(l.children()), q, r, w) del l.a, l.b self.assertEqual(list(l.children()), []) buf = torch.randn(10) l.register_buffer('buf', buf) self.assertIs(l.buf, buf) l.buf = None self.assertIs(l.buf, None) self.assertNotIn('buf', l.__dict__)  # should be stored in l._buffers l.buf = buf self.assertIn('buf', l.state_dict()) self.assertIs(l.state_dict()['buf'], buf) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_Conv2d_large_workspace(self): # These sizes require huge cuDNN workspaces. Make sure we choose a

They do one pair of these calls for each slicing operation.
Does this actually have a performance impact? These calls tend to be very quick
Yes, it's a major reason why small unrolled LSTMs (which involve in addition to two GEMMs: a chunk into four gates, for 4 pairs of calls, five unary pointwise ops, for 5 more, and five binary pointwise ops, for another 5) are much slower than they are in Theano and TensorFlow.
Also, there are excessive cudaGetDevice calls for each pointwise operation (9 cudaGetDevice) even without chunking, as in the following repro code: As for small unrolled LSTMs it does show up.
This should no longer be an issue, now that is implemented as a native ATen function. I think we're still using the Python version for tensors, but this will be resolved soon when we stop using tensors altogether.


Question: <ins> import os this_file = os.path.dirname(os.path.abspath(__file__)) generated_dir = os.path.abspath(os.path.join(this_file, '..', '..', 'torch', 'csrc', 'generated')) line_start = '//generic_include ' types = [ 'Double', 'Float', 'Half', 'Long', 'Int', 'Short', 'Char', 'Byte' ] generic_include = '#define {lib}_GENERIC_FILE "{path}"' generate_include = '#include "{lib}/{lib}Generate{type}Type.h"' def split_types(file_name): assert file_name.startswith('torch/csrc/') if not os.path.exists(generated_dir): os.makedirs(generated_dir) with open(file_name, 'r') as f: lines = f.read().split('n') # Find //generic_include for i, l in enumerate(lines): if l.startswith(line_start): args = l[len(line_start):] lib_prefix, generic_file = filter(bool, args.split()) break else: raise RuntimeError("generic include not found") gen_name_prefix = file_name[len('torch/csrc/'):].replace('/', '_').replace('.cpp', '') gen_path_prefix = os.path.join(generated_dir, gen_name_prefix) prefix = 'n'.join(lines[:i]) suffix = 'n'.join(lines[i + 1:]) to_build = [] g_include = generic_include.format(lib=lib_prefix, path=generic_file) for t in types: t_include = generate_include.format(lib=lib_prefix, type=t) gen_path = gen_path_prefix + t + '.cpp' to_build.append(gen_path) with open(gen_path, 'w') as f: f.write(prefix + 'n' + g_include + 'n' + t_include + 'n' + suffix) return to_build </ins>

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: self.add_module(str(idx), module) def __getitem__(self, idx): <del> if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) it = iter(self._modules.values()) for i in range(idx): next(it) return next(it) </del> <ins> if isinstance(idx, slice): return Sequential(OrderedDict(list(self._modules.items())[idx])) else: if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) it = iter(self._modules.values()) for i in range(idx): next(it) return next(it) </ins> def __len__(self): return len(self._modules)

it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.


Question: cudnn=True, ), dict( <ins> fullname='Conv2d_groups_thnn', constructor=lambda: nn.Conv2d(4, 6, (3, 2), groups=2), input_size=(2, 4, 6, 5), ), dict( </ins> module_name='ConvTranspose2d', constructor_args=(3, 4, 3, (3, 2), 1, (1, 1)), cudnn=True,

x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.


Question: :members: :hidden:`InstanceNorm1d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm1d :members: :hidden:`InstanceNorm2d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm2d :members: :hidden:`InstanceNorm3d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm3d :members:

why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:


Question: for t in types: tensor = t() gpu_tensor = get_gpu_type(t)() <ins> # Default values desc = '' type_subset = types no_inplace = False </ins> if len(decl) == 3: name, constr, arg_constr = decl <del> desc = '' </del> elif len(decl) == 4: name, constr, arg_constr, desc = decl elif len(decl) == 5: name, constr, arg_constr, desc, type_subset = decl <del> if t not in type_subset: continue </del> <ins> elif len(decl) == 6: name, constr, arg_constr, desc, type_subset, no_inplace = decl if t not in type_subset: continue </ins> precision = custom_precision.get(name, TestCuda.precision) for inplace in (True, False): <ins> if inplace and no_inplace: continue </ins> if inplace: name_inner = name + '_' else:

On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)


Question: | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides <del> for :attr:`padding` number of points | :attr:`dilation` controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. </del> <ins> for :attr:`padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. </ins> |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. <ins> At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> The parameters :attr:`kernel_size`, :attr:`stride`, :attr:`padding`, :attr:`dilation` can either be:

Hi, I have installed two versions of CUDA. But I want to use one of it, can I specify it during setup process?
Try running
I have tried it, () still returns False. Thx for your reply!
Do you have the cuda driver installed?
Of course I do. It raise assertion error which shows "The NVIDIA driver on your system is too old(found version 8000)".
that's the answer to your problem. upgrade your driver.
Thx!

When building from source, I got this error. My system is OSX 10.12.4, conda, CUDA 8.0. Any thoughts? Thanks!
can you try adding to the file and see if that fixes things? if so i'll commit it.
actually, that file already has memory included. Are you following instructions from Especially: This part is important:
works nicely! Thanks a lot!

The suggest that takes a as a parameter, but it actually must be wrapped in a Variable first.
you are looking at the Tensor docs. When operating on Variable, it is implied from the tutorials that all functions operating on Variables have to use Variables. We didn't write separate docs for Variable itself.

something wrong with import torchvision Traceback (most recent call last): File "", line 3, in <moduleimport torchvision File "", line 4, in <moduleimport torchvision.transforms as transforms ImportError: No module named transforms
you have a file called for some reason. delete it or change your pythonpath to ignore it.
Hey, I have the same problem persisting when I import torchvision. I have no other file by the same name. ImportError Traceback (most recent call last) <ipython-input-6-3baf6837d8f1in <module() 8 from import Dataset 9 from import DataLoader ---10 from torchvision import transforms ImportError: No module named torchvision Any idea why this is happening?
Do you have torchvision installed (it is a separate package)?
pip Install torchvision try this
from data import BaseTransform,VOC_CLASSES as labelmap this is not working
pip install torchvision

Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

def inithidden(self): h0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) c0 = Variable(FloatTensor(self.numlayers, self.batchlen, self.hiddendim).zero(),  requiresgrad=True) return (h0, c0) def forward(self, sequences): lstmout, = ((1, self.batchlen, self.inputdim), ) ... = nn.LSTM(inputdim, hiddendim, numlayers, dropout = dropout, batchfirst=True) = self.inithidden() output = model(traininput) produces the following error although I defined weight and input dimensions as in the documentation: File "/usr/local/lib/python2.7/dist-", line 202, in call result = self.forward(input, kwargs) File "/usr/local/lib/python2.7/dist-", line 91, in forward output, hidden = func(input, self.allweights, hx) File "/usr/local/lib/python2.7/dist-", line 327, in forward return func(input, fargs, fkwargs) File "/usr/local/lib/python2.7/dist-", line 227, in forward nexth, output = func(input, hidden, weight) File "/usr/local/lib/python2.7/dist-", line 67, in forward hy, output = inner(input, hidden[l], weight[l]) File "/usr/local/lib/python2.7/dist-", line 96, in forward hidden = inner(input[i], hidden, *weight) File "/usr/local/lib/python2.7/dist-", line 22, in LSTMCell gates = F.linear(input, wih, bih) + F.linear(hx, whh, bhh) File "/usr/local/lib/python2.7/dist-", line 744, in add return (other) File "/usr/local/lib/python2.7/dist-", line 293, in add return (other, False) File "/usr/local/lib/python2.7/dist-", line 287, in add return Add(inplace)(self, other) File "/usr/local/lib/python2.7/dist-", line 20, in forward return a.add(b) RuntimeError: inconsistent tensor size at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:827
I can't reproduce your problem. Could you include a short runnable example that exhibits this behavior?

You also don't need to manually init hidden anymore, the following would work as well...
Thanks! After removing problem solved.

Hi, I would like to implement a sharing weight 'alexnet' with two input batches of images, and the fully connected layers are concatenated together like this: However, when I run it, I come across the following error: Does anyone know about it? Thank you very much.
You forgot to call on the output of . In the future please remember that we're using the issues for bug reports and feature requests only. If you have questions or need help, please post .


Question: RECORD_TRACE = CodeTemplate(""" if (jit::tracer::isTracing({ ${tensor_args} })) { <del> jit::Node *n = jit::tracer::recordTrace( "${api_name}", { ${tensor_args} }, ${return_name} ); </del> <ins> jit::Node *n = jit::tracer::recordTrace( "${api_name}", ${trace_inputs}, ${trace_outputs} ); </ins> ${record_attributes} } """)

when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit


Question: #include "tuple_parser.h" <ins> #include <string> </ins> #include "torch/csrc/DynamicTypes.h" #include "torch/csrc/autograd/python_variable.h" #include "python_numbers.h"

Version include <stringin some cpp files: define order in The pre-defined struct is not passed by clang in macOS, so we should reorder them.
Simple fix


Question: #ifndef TH_GENERIC_FILE #define TH_GENERIC_FILE "generic/Tensor.cpp" #else

When I don't have a working numpy dependency detected (due to unrelated shenanigans), the build fails with this: And then a big pile of related errors. CC
This is most likely the problem:
CC
Ok, should probably read
Should I go ahead and make the change? I'll also remove the at line 193. The rest looks ok to me.
go for it.
Actually, the change I proposed is incorrect (I misread the second defined), but I'm on it.


Question: if (free_b) THTensor_(free)(b);), "gels", info,""); <del> /* rb__ is currently ldb by nrhs; resize it to n by nrhs */ rb__->size[0] = n; if (rb__ != rb_) </del> <ins> /* * In the m < n case, if the input b is used as the result (so b == _rb), * then rb_ was originally m by nrhs but now should be n by nrhs. * This is larger than before, so we need to expose the new rows by resizing. */ if (m < n && b == rb_) { </ins> THTensor_(resize2d)(rb_, n, nrhs); <ins> } </ins> THTensor_(freeCopyTo)(ra__, ra_); THTensor_(freeCopyTo)(rb__, rb_);

b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.


Question: gtest_reports_dir="${TEST_DIR}/cpp" pytest_reports_dir="${TEST_DIR}/python" <ins> # This is needed to work around ROCm using old docker images until # the transition to new images is complete. # TODO: Remove once ROCm CI is using new images. if [[ $BUILD_ENVIRONMENT == py3.6-devtoolset7-rocmrpm-centos* ]]; then # This file is sourced multiple times, only install conda the first time. # We must install conda where we have write access. CONDA_DIR="$ROOT_DIR/conda" if [[ ! -d $CONDA_DIR ]]; then ANACONDA_PYTHON_VERSION=3.6 BASE_URL="https://repo.anaconda.com/miniconda" CONDA_FILE="Miniconda3-latest-Linux-x86_64.sh" mkdir $CONDA_DIR pushd /tmp wget -q "${BASE_URL}/${CONDA_FILE}" chmod +x "${CONDA_FILE}" ./"${CONDA_FILE}" -b -f -p "$CONDA_DIR" popd export PATH="$CONDA_DIR/bin:$PATH" # Ensure we run conda in a directory that jenkins has write access to pushd $CONDA_DIR # Track latest conda update conda update -n base conda # Install correct Python version conda install python="$ANACONDA_PYTHON_VERSION" conda_install() { # Ensure that the install command don't upgrade/downgrade Python # This should be called as #   conda_install pkg1 pkg2 ... [-c channel] conda install -q -y python="$ANACONDA_PYTHON_VERSION" $* } # Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README conda_install numpy pyyaml mkl mkl-include setuptools cffi typing future six # TODO: This isn't working atm conda_install nnpack -c killeent # Install some other packages # Need networkx 2.0 because bellmand_ford was moved in 2.1 . Scikit-image by # defaults installs the most recent networkx version, so we install this lower # version explicitly before scikit-image pulls it in as a dependency pip install networkx==2.0 # TODO: Why is scipy pinned # numba & llvmlite is pinned because of https://github.com/numba/numba/issues/4368 # scikit-learn is pinned because of # https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5 # only) pip install --progress-bar off pytest scipy==1.1.0 scikit-learn==0.20.3 scikit-image librosa>=0.6.2 psutil numba==0.46.0 llvmlite==0.30.0 # click - onnx # hypothesis - tests # jupyter - for tutorials pip install --progress-bar off click hypothesis jupyter protobuf tabulate virtualenv mock typing-extensions popd else export PATH="$CONDA_DIR/bin:$PATH" fi fi </ins> # Figure out which Python to use PYTHON="$(which python)" if [[ "${BUILD_ENVIRONMENT}" =~ py((2|3).?[0-9]?.?[0-9]?) ]]; then

See e.g.: cc
Can we close it?
The build is passing now, although the tests might be failing (although it might have been an environment failure).


Question: compare_cpu_gpu(outputs_cpu, outputs_gpu) @unittest.skipIf(not TEST_CUDNN, "needs cudnn") <ins> @default_tensor_type(torch.FloatTensor)  # FIXME: just until torch.cuda.DoubleTensor.sum() implemented def test_RNN_cpu_vs_cudnn_no_dropout(self): self._test_RNN_cpu_vs_cudnn(0) @unittest.skipIf(not (TEST_CUDNN and TEST_CUDNN_VERSION >= 5103), "needs cudnn >= 5.1") @default_tensor_type(torch.FloatTensor)  # FIXME: just until torch.cuda.DoubleTensor.sum() implemented def test_RNN_cpu_vs_cudnn_with_dropout(self): # Because of dropout randomness, can only compare dropout=0 and dropout=1 self._test_RNN_cpu_vs_cudnn(1) @unittest.skipIf(not (TEST_CUDNN and TEST_CUDNN_VERSION >= 5103), "needs cudnn >= 5.1") </ins> def test_RNN_dropout(self): # checking the assumption that cuDNN sticks dropout in between # RNN layers

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: >>> m = nn.Dropout(p=0.2) >>> input = autograd.Variable(torch.randn(20, 16)) >>> output = m(input) <ins> .. _Improving neural networks by preventing co-adaptation of feature detectors: https://arxiv.org/abs/1207.0580 </ins> """ def __init__(self, p=0.5, inplace=False):

"Randomly zeroes some of the elements of the input tensor. The elements to zero are randomized on every forward call." This is incorrect; the function also scales up by 1/(1-p), which the implementation correctly does.


Question: torch.Size([3, 2, 2, 1]) >>> torch.unflatten(torch.randn(3, 4, 1), 1, (-1, 2)).shape torch.Size([3, 2, 2, 1]) <del> >>> torch.unflatten(torch.randn(5, 12, 3), -1, (2, 2, 3, 1, 1)).shape </del> <ins> >>> torch.unflatten(torch.randn(5, 12, 3), -2, (2, 2, 3, 1, 1)).shape </ins> torch.Size([5, 2, 2, 3, 1, 1, 3]) """.format( **common_args

Link In the third example the doc shows torch.unflatten((5, 12, 3), -1, (2, 2, 3, 1, 1)).shape torch.Size([5, 2, 2, 3, 1, 1, 3]) which should be incorrect, the correct code should be torch.unflatten((5, 12, 3), -2, (2, 2, 3, 1, 1)).shape torch.Size([5, 2, 2, 3, 1, 1, 3]) No response cc
Thank you for reporting. Next time you spot a problem like that please do not hesitate to propose a PR with the fix.


Question: from torch.autograd import Variable <ins> def get_a_var(obj): if isinstance(obj, Variable): return obj if isinstance(obj, list) or isinstance(obj, tuple): results = map(get_a_var, obj) for result in results: if isinstance(result, Variable): return result if isinstance(obj, dict): results = map(get_a_var, obj.items()) for result in results: if isinstance(result, Variable): return result return None </ins> def parallel_apply(modules, inputs, kwargs_tup=None): assert len(modules) == len(inputs) if kwargs_tup:

is the culprit. Assumes that the first input is a variable or a sequence containing a variable in index 0 (recursively indexed). Workaround is to pass a dummy variable as first input, as suggested in
Could you provide a short reproducer for testing? Think this shouldn't be too hard to fix.
Sure, here's a gist along with commands/outputs:
fixed in master via


Question: scale = 1 exp_max = int(exp_max) <ins> prec = PRINT_OPTS.precision </ins> if int_mode: <del> if exp_max > 9: format = '{:11.4e}' sz = max(min_sz, 11) </del> <ins> if exp_max > prec + 1: format = '{{:11.{}e}}'.format(prec) sz = max(min_sz, 7 + prec) </ins> else: sz = max(min_sz, exp_max + 1) format = '{:' + str(sz) + '.0f}' else: <del> if exp_max - exp_min > 4: sz = 11 </del> <ins> if exp_max - exp_min > prec: sz = 7 + prec </ins> if abs(exp_max) > 99 or abs(exp_min) > 99: sz = sz + 1 sz = max(min_sz, sz) <del> format = '{:' + str(sz) + '.4e}' </del> <ins> format = '{{:{}.{}e}}'.format(sz, prec) </ins> else: <del> if exp_max > 5 or exp_max < 0: </del> <ins> if exp_max > prec + 1 or exp_max < 0: </ins> sz = max(min_sz, 7) scale = math.pow(10, exp_max-1) else:

It seems that the following link that can be reached from here (check the screenshot) is broken: Page: Broken link: Screenshot: ! cc
this should be transferred to , but I don't have permissions to do that. Fix is available at pytorch/hub.


Question: for (int idx = 0; idx < size; idx++) { PyObject* obj = tuple ? PyTuple_GET_ITEM(arg, idx) : PyList_GET_ITEM(arg, idx); if (!THPVariable_Check(obj)) { <del> throw TypeError("expected Variable as element %d in argument %d, but got %s", </del> <ins> throw TypeError("expected Tensor as element %d in argument %d, but got %s", </ins> idx, i, Py_TYPE(args[i])->tp_name); } res[idx] = reinterpret_cast<THPVariable*>(obj)->cdata;

Considering the merge of and , and the "removal" of from the API, there are still some messages that reference . This might prove to be confusing for new users starting off from , as they probably might not have heard of . I did a code search to find out multiple instances of this in . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): source (@ ) Build command you used (if compiling from source): OS: Ubuntu 16.04.2 PyTorch version: Python version: 3.5 CUDA/cuDNN version: 8.0.44 GPU models and configuration: GeForce 940M GCC version (if compiling from source): 5.4.0 CMake version: 3.5.1


Question: - bool training - double epsilon ]] <ins> [[ name: cudnn_version only_register: True ]] static PyObject * THCUDNN_cudnn_version(PyObject *self, PyObject *args) { return PyLong_FromLong(CUDNN_VERSION); } </ins>

We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.


Question: { transpose_m2 = (transpose_r == 'n' ? 't' : 'n'); m2_ = THTensor_(newContiguous)(m2); <ins> free_m2 = 1; </ins> } #pragma omp critical(blasgemm)

Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.


Question: setattr(self, name, param) self._all_weights.append(param_names) <del> self._param_buf_size += sum(p.numel() for p in layer_params) </del> self.flatten_parameters() self.reset_parameters()

Since the new pytorch version, I have a warning when I that says: When I do as suggested, and then try to encode again, I get the following error though: Any idea?
do you have code the reproduces the issue?
What was the fix? I still see the error on Pytorch Version: 0.2.03 AttributeError: 'LSTM' object has no attribute 'parambufsize'


Question: from copy import deepcopy from collections import OrderedDict from itertools import product <del> from operator import mul </del> <ins> from operator import mul, itemgetter </ins> from functools import reduce, wraps from torch.autograd.gradcheck import gradgradcheck, gradcheck from torch.autograd.function import once_differentiable

I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.


Question: output[index] = ScalarConvert<int, Dtype>::to(0); continue; } <ins> assert(cur_target  >= 0 && cur_target  < n_classes); </ins> Dtype weight = weights ? weights[cur_target] : ScalarConvert<int, Dtype>::to(1); output[index] = -weight * input[index][cur_target];

Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3


Question: Writes all values from the tensor :attr:`src` into :attr:`self` at the indices specified in the :attr:`index` tensor. For each value in :attr:`src`, its output <del> index is specified by its index in :attr:`src` for dimension != :attr:`dim` and by the corresponding value in :attr:`index` for dimension = :attr:`dim`. </del> <ins> index is specified by its index in :attr:`src` for ``dimension != dim`` and by the corresponding value in :attr:`index` for ``dimension = dim``. </ins> For a 3-D tensor, :attr:`self` is updated as::

is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.


Question: Args: input (Tensor): the input 2-D tensor <del> some (bool, optional): controls the number of singular values to be computed </del> <ins> some (bool, optional): controls the shape of returned `U` and `V` </ins> out (tuple, optional): the output tuple of tensors Example::

The documentation of the parameter to is one of the strangest I've seen... what does (the default) do???
Ha it is somewhat weird indeed. When , the returned U and V matrices are square orthonormal matrices. I will send out a PR to clarify this.