Question: if IS_CONDA: lib_paths.append(os.path.join(CONDA_DIR, 'lib')) include_paths.append(os.path.join(CONDA_DIR, 'include')) <ins> for path in include_paths: if path is None or not os.path.exists(path): continue include_file_path = os.path.join(path, 'cudnn.h') if os.path.exists(include_file_path): CUDNN_INCLUDE_DIR = path CUDNN_INCLUDE_VERSION = -1 with open(include_file_path) as f: for line in f: if "#define CUDNN_MAJOR" in line: CUDNN_INCLUDE_VERSION = int(line.split()[-1]) break if CUDNN_INCLUDE_VERSION == -1: raise AssertionError("Could not find #define CUDNN_MAJOR in " + include_file_path) break </ins> for path in lib_paths: if path is None or not os.path.exists(path): continue

Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/abc/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/abc/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/abc/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 216, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 123, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 65, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 82, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110 How to solve the problem?
could you pull the latest master branch and recompile. I pushed a change yesterday that fixes this:
I remove the old code and pull new code, then compile, install.It still fails. /usr/local/lib/python2.7/dist- UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.

('No training configuration found in save file: ' Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/qianbb/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/qianbb/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/qianbb/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 216, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 100, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 66, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 87, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110 The library of cuda is shown as following screenshot. !
I found the issue, I fixed it in which we will merge shortly.
ThanksDoes it have been merged?
yes. it is merged.
It seems the fix doesn't work.

The log is shown as following Traceback (most recent call last): File "", line 3, in <moduleimport model File "/home/qianbb/github/chinese-", line 5, in <modulefrom import crnnOcr File "/home/qianbb/github/chinese-", line 72, in <modulemodel,converter = crnnSource() File "/home/qianbb/github/chinese-", line 64, in crnnSource model = crnn.CRNN(32, 1, len(alphabet)+1, 256, 1).cuda() File "/usr/local/lib/python2.7/dist-", line 222, in cuda return (lambda t: t.cuda(device)) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 146, in apply (fn) File "/usr/local/lib/python2.7/dist-", line 100, in apply self.flattenparameters() File "/usr/local/lib/python2.7/dist-", line 66, in flattenparameters if not or not (anyparam): File "/usr/local/lib/python2.7/dist-", line 87, in isacceptable if libcudnn() is None: File "/usr/local/lib/python2.7/dist-", line 58, in libcudnn 'but linked against {}'.format(compileversion, _cudnnversion)) RuntimeError: cuDNN version mismatch: PyTorch was compiled against 6021 but linked against 5110
that is weird. Can I see the full log of in pytorch directory. Can you make sure you have latest commits with .
I am sure the code is up-to-date. First I uninstall the code using the command pip unisntall torch. Then I remove the old code using the command rm -r pytorch and git clone again. The log of install is
Sorry to trouble you. Is there any new information on this issue?
I spent some time resolving this issue and I don't see too much about it online. Try ensuring that your Pytorch build is correct for your specific versions of Cuda and Cudnn. While I had the correct version of Pytorch, I resolved the issue by installing a different build of the same version.

The build I installed is . Here are the list of versions and builds: You can download these files locally and install with conda: (assumes command is being run in a directory containing , where this file will be specific to your Cuda and Cudnn version.
feedback from this code PackagesNotFoundError
thank you - I have edited my comment to reflect that I installed from a local zip.


Question: char *bytes = (char *) data; int64_t remaining = sizeof(real) * storage->size; while (remaining > 0) { <del> ssize_t result = read(fd, bytes, remaining); </del> <ins> // we write and read in 1GB blocks to avoid bugs on some OSes ssize_t result = read(fd, bytes, THMin(remaining, 1073741824)); </ins> if (result <= 0) // 0 means EOF, which is also an error throw std::system_error(result, std::system_category()); bytes += result;

Running this on OSX: More details of failure in Ubuntu 14.04.1 in
Maybe it's caused by a ? I just test this code and it doesn't work (on Mac OS X 10.12.3 with Python 3.6):
That thread links to this one , which says it’s an OS bug and proposes a workaround patch that hasn’t been merged. There’s also a numpy issue thread, which links to this Torch7 commit that implements a workaround
We're doing a loop in case it writes less, but we always pass the full amount of remaining bytes. Putting a cap (std::min with INT_MAX) on the size of a single write might help.
I can't reproduce it on CentOS. I think it might have been a report for a PyTorch version prior to fixing .


Question: input2 = torch.randn(input_size, requires_grad=True) self.assertEqual(F.cosine_similarity(input1, input2, dim=1).size(), expected_size) <ins> def test_grid_sample_unsupported_mode(self): with self.assertRaisesRegex(NotImplementedError, "nn.functional.grid_sample got unsupported mode: 'garbage'"): F.grid_sample(torch.tensor([]), torch.tensor([]), mode='garbage') </ins> def test_grid_sample(self): def test_cpu_against_cuda(N, C, H, W, padding_mode): def test_shape(N, C, IH, IW, H, W, padding_mode):

The only valid mode is 'bilinear', but one can pass anything to it and it will not complain:
Apparently, the is not passed to the backend function for . I think it should be removed.
Yes. I think the idea is that we'd like to support other modes at some point.


Question: self.value = value def forward(self, input): <del> return F_ConstantPad2d(pad=self.padding, value=self.value)(input) </del> <ins> return F.pad(input, self.padding, 'constant', self.value) </ins> def __repr__(self): return self.__class__.__name__ + ' ' + str(self.padding)

After merging cuda 12.4 for nightly. I observe following failures: libtorch-cuda124-shared-with-deps-cxx11-abi-build: Build Official Docker Images 124: We need to resolve these before continuing adding CUDA 12.4 CI workflows 0 cc
attempt 3 worked for libtorch build issue:
For Docker image, related discussion:
I will instance a new bug for the docker job failure.

Is there a way to use reflection padding instead of zero padding with nn.Conv2d ? I see there is a but the option is not in nn.Conv2d. Thanks,
are in , but they're not exposed in the docs.
Thanks!


Question: padding_idx (int, optional): If given, pads the output with zeros whenever it encounters the index. max_norm (float, optional): If given, will renormalize the embeddings to always have a norm lesser than this norm_type (float, optional): The p of the p-norm to compute for the max_norm option <del> scale_grad_by_freq (boolean, optional): if given, this will scale gradients by the frequency of </del> <ins> scale_grad_by_freq (bool, optional): if given, this will scale gradients by the frequency of </ins> the words in the mini-batch. <del> sparse (boolean, optional): if ``True``, gradient w.r.t. weight matrix will be a sparse tensor. See Notes for </del> <ins> sparse (bool, optional): if ``True``, gradient w.r.t. weight matrix will be a sparse tensor. See Notes for </ins> more details regarding sparse gradients. Attributes: weight (Tensor): the learnable weights of the module of shape (num_embeddings, embedding_dim) Shape: <del> - Input: LongTensor `(N, W)`, N = mini-batch, W = number of indices to extract per mini-batch - Output: `(N, W, embedding_dim)` </del> <ins> - Input: LongTensor of arbitrary shape containing the indices to extract - Output: `(*, embedding_dim)`, where `*` is the input shape </ins> Notes: Keep in mind that only a limited number of optimizers support

I have a straightforward use case. Given a token (or word), lookup the embedding of the word, pass that through a linear layer and perform L2 regression. If the token corresponds to the of the embedding layer, throws an error saying OS: macOS High Sierra PyTorch version: 0.3.1 How you installed PyTorch (conda, pip, source): Tried both conda and pip Python version: 3.6.4 CUDA/cuDNN version: CPU GPU models and configuration: NA GCC version (if compiling from source): NA A script to reproduce the bug:
I got the same trouble, and it throw this error after some epochs of training. Does anyone have some solutions?


Question: # caffe2::cudart is dealt with separately, due to CUDA_ADD_LIBRARY # design reason (it adds CUDA_LIBRARIES itself). set(Caffe2_PUBLIC_CUDA_DEPENDENCY_LIBS <del> caffe2::cuda caffe2::cufft caffe2::curand caffe2::nvrtc) </del> <ins> caffe2::cufft caffe2::curand) if(BUILD_CAFFE2) # Don't be deceived!  caffe2::cuda is the low-level DRIVER API, # not the actual CUDA library.  Caffe2 depends directly on nvrtc # and needs it, but ATen doesn't use it at all, and so the # dependency is unnecessary. # # BTW, if you change this so that PyTorch has this dependency # again, make sure Mac OS X GPU builds still work; there's # a decent chance the library finding algorithm picked up # on cuda.framework, which is totally not going to work when # linking. list(APPEND Caffe2_PUBLIC_CUDA_DEPENDENCY_LIBS caffe2::cuda caffe2::nvrtc) endif() </ins> if(CAFFE2_FOUND_CUDNN) LIST(APPEND Caffe2_PUBLIC_CUDA_DEPENDENCY_LIBS caffe2::cudnn) else()

error occurs when linking the cuda library ld: can't map file, errno=22 file '/usr/local/cuda/lib/stubs/cuda.framework' for architecture x8664 Provide a short description. I have tracked down in findlibrary(CUDACUDALIB cuda PATHS ${CUDATOOLKITROOTDIR} PATHSUFFIXES lib lib64 lib/stubs lib64/stubs lib/x64) finds the cuda.framework instead of Have confirm it was broken by this commit () Any idea why it is linking the cuda.framework? CUDA used to build PyTorch: 9.2 OS: Mac OSX 10.13.4 GCC version: Could not collect CMake version: version 3.11.1 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 9.2.64 GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Probably one of the following: /usr/local/cuda/lib/libcudnn_static.a
cc
From cmake doc , CMake will use a -framework A, and a -F<fullPathto link the framework to the target, however it is not. I have tried separating ::cuda from Caffe2PUBLICCUDADEPENDENCY_LIBS, still not linking properly. Removing ::cuda passes the build though.
can you post your cmake output that contains the list of CUDA-related libraries that cmake found? Is this statically linking CUDA? What's the actual linking command that is failing (probably need to pass VERBOSE=1 to the make command to see this)? None of us will be able to repro on our end as we do not have macs with GPUs :( , but we can try to answer questions and give pointers as to where to look.
thanks for reporting this and helping reproduce. Can you try and add this to the root around line 162? You'll then want to run to clear the directory and try again.

[ 94%] Linking CXX shared library cd /Users/hongbinliu/pytorch/build/ && /usr/local/miniconda3/bin/cmake -E cmakelinkscript --verbose=1 /Library/Developer/CommandLineTools/usr/bin/clang++ -msse4.1 -msse4.2 --std=c++11 -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DONNXNAMESPACE=onnxc2 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-unused-parameter -Wno-unknown-warning-option -Wno-unknown-pragmas -DHAVEAVXCPUDEFINITION -DHAVEAVX2CPUDEFINITION -O3 -isysroot -mmacosx-version-min=10.10 -dynamiclib -Wl,-headerpadmaxinstallnames -compatibilityversion 1.0.0 -currentversion 1.0.0 -o -installname -Wl,-rpath,/usr/local/miniconda3/lib -Wl,-rpath,/usr/local/cuda/lib -Wl,-rpath,/Users/hongbinliu/pytorch/build/lib /usr/local/cuda/lib/libcudartstatic.a -Wl,-rpath,/usr/local/cuda/lib /usr/local/cuda/lib/libcudartstatic.a -Wl,-rpath,/usr/local/cuda/lib /usr/local/cuda/lib/stubs/cuda.framework /usr/local/cuda/lib/libcublasdevice.a ld: can't map file, errno=22 file '/usr/local/cuda/lib/stubs/cuda.framework' for architecture x8664 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: [] Error 1 make[1]: [] Error 2 make: * [all] Error 2
Thx, adding that did work. However, when I run training with cuda, it throws an RuntimeError: cannot initialize CUDA without ATen_cuda library. I guess this is another issue anyway.
Same issue.

<img width="545" alt="downloads" src="https://user-"Also, I found a small problem when copying the dylibs to torch/lib, it is replacing the symbolic link with the actual file (screenshot above). Replacing cp -r to cp -R does the trick, please let me know if we want a pr.
Have tracked down the problem was in , always return null because in Registry.h line 77 the (key) is always 0; ObjectPtrType Create(const SrcType& key, Args... args) { ` Suspect CUDAHooksRegistry() is creating new instance, because of namespace? Is it working in Linux? Any help will be appreciated.
I think Mac OS X with CUDA is broken at the moment. We think the problem has something to do with not getting a dependency on (or, I guess, , now)
Hi ezyang, thanks for your hint. I just tried before . It gives me "Key already registered". I will leave it to you guys, and I can help with testing, please @ me.
Thanks for giving it a try. Sounds like is being loaded correctly, so my initial guess was wrong. Would it be possible to print the keys of the map to see what is in it when you get the key already registered error message? And maybe also, you noted that there was never anything at the key "CUDAHooks" when you didn't have the dlopen line. Is the map always empty in that case, or is there something registered, perhaps at a different name? (I made this bug when I developed the patch and figured it out by seeing that there was another CUDAHooks thing but named slightly differently). I'd like to debug this myself but I can't get PyTorch to build with Mac OS X and CUDA at the moment. (See also the bugs I've recently filed)
Yes, the map was always empty when calling CUDAHooksRegistry(). Loading the dylib works as expected that it is reusing the same static Registry and registers the key. However calling CUDAHooksRegistry() doesn't get the same static Registry instance, instead it creates a new one. Could it be relating to how mac handles dynamic loading? e.g different memory space?
Successfully reproed.

And yes, it looks like the dynamic dependencies are setup correctly:
BTW, fix for the cuda.framework error didn't work for me. Where exactly did you end up placing the block in the OTOH, this patch worked for me: (It's totally the wrong way to solve the problem but it might help someone)
and I diagnosed the problem, fix coming.
That is great!
Go try . You'll need to do a clean build from scratch. Also, you'll still need a cuda.framework fix of some sort or another. Working on a more robust version of this fix.
Thanks ezyang, it is working.


Question: weight_data[:] = 4 self.assertEqual(weight_data, all_vars[4].data) <ins> @unittest.skipIf(not TEST_CUDNN, 'CUDNN not available') def test_cudnn_weight_tying(self): rnns = [ nn.LSTM(10, 20, batch_first=True, bidirectional=True), nn.GRU(10, 20, batch_first=True, bidirectional=True), nn.RNN(10, 20, batch_first=True, bidirectional=True) ] for rnn in rnns: rnn.bias_ih_l0_reverse = rnn.bias_ih_l0 rnn.cuda() input = Variable(torch.randn(5, 4, 10).cuda(), requires_grad=True) hx = Variable(torch.randn(2, 5, 20).cuda(), requires_grad=True) all_vars = [input, hx] + list(rnn.parameters()) opt = torch.optim.SGD(rnn.parameters(), lr=0.1) opt.zero_grad() if isinstance(rnn, nn.LSTM): cx = Variable(torch.randn(2, 5, 20).cuda(), requires_grad=True) all_vars[2:2] = [cx] hx = (hx, cx) with warnings.catch_warnings(record=True) as w: output = rnn(input, hx) self.assertEqual(len(w), 1) self.assertIn('weights are not part of single contiguous chunk of memory', w[0].message.args[0]) output[0].sum().backward() opt.step() with warnings.catch_warnings(record=True) as w: output_cuda = rnn(input, hx) rnn.cpu() hx = (hx[0].cpu(), hx[1].cpu()) if isinstance(rnn, nn.LSTM) else hx.cpu() output_cpu = rnn(input.cpu(), hx) self.assertEqual(output_cuda, output_cpu) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_cuda_rnn_fused(self): def copy_rnn(rnn1, rnn2):

While the fast path for cuDNN RNNs introduced in includes a check intended to fall back to copying weights if weights are moved or aliased, this only works if the aliasing happens after the weights are first compacted. But the most natural place to tie weights is inside a module's method, which runs before , the typical time when is first invoked. Here's an MWE. The three runs (CPU, weight tying before , and weight tying after ) should all give the same results, and they do before , but they have not since:


Question: #ifndef TH_GENERAL_INC #define TH_GENERAL_INC <ins> #define __STDC_FORMAT_MACROS </ins> #include <stdint.h> #include <stdlib.h> #include <stdio.h>

I am installing PyTorch from source. Python version: 2.7.14. CUDA: 8.0. GPU: gtx1080. GCC: 5.4. Processor: AMD Opteron(tm) Processor 6128 While executing "python install" build completes till 87% but after that I get the following errors And many more messages of the same type regarding errors with 'PRId64' I looked at similar issues online and most of the people facing this issue had an old version of gcc, so I'm a bit confused about what my exact problem is. Also, when I'm installing via conda, the installation completes, but when I run the following bit of code, I get an illegal instruction at the print command That I guess is because of the fact I'm using an AMD processor? Any help will be appreciated. Thanks
looks like the same issue as

I am not able to build the current version. Getting the following error: pytorch/aten/src/TH/generic/THTensorLapack.c:860:7: warning: unused variable ‘n’ [-Wunused-variable] int n = ra-size[1]; ^ make[2]: [] Error 1 make[1]: [] Error 2 make: * [all] Error 2 I am using Anaconda as directed in the readme. I have created a conda environment and have set: export CMAKEPREFIXPATH="/u/anaconda2/envs/pyt4/" As specified here: Thanks
What you posted here is only a warning, and it won't stop the build. Please paste the full log into a GitHub gist and link it here
Hi, Here are the logs: I see it says Could NOT find CUDNN, I am installing it in an env which submit jobs to a cluster, do I need to install CUDNN locally?
what compiler do you have? and what OS are you using? The issue is this:
lsbrelease -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.2 (Maipo) Release: 7.2 Codename: Maipo uname -a Linux 1 SMP Thu Oct 29 17:29:29 EDT 2015 x8664 x8664 x8664 GNU/Linux
- We are seeing the same error above with the build scripts that you provided in , on PyTorch master. I can bisect and let you know the commit that is breaking the build.
The CPU build script fails with this commit () -
Got the same error when trying to compile under CentOS 7.2.1511 with GCC 4.8 and GCC 5.4 (no newer gcc available on that server).

CC
Searching issues for "PRId64" shows that this error popped up and has been fixed in various places in the pytorch codebase, e.g. The problem seems to be that this format specifier is not defined by default in C++ and/or old gcc versions. A hacky way to fix this is replacing with in the two files currently causing this error on master. A better way is to find the proper place for addding as described in the issue linked above.
Could you try and see if the branch in fixes the issue? I wasn't able to reproduce the compile error locally, but based on the past issues it should be the right fix. Thanks!
your branch compiles without error with GCC 5.4


Question: PyObject *THPFunction_saved_variables(THPFunction *self, void *_unused) { <del> THPUtils_assert(!self->has_freed_buffers, "Trying to backward through the " "graph second time, but the buffers have already been freed. Please " "specify retain_variables=True when calling backward for the first time."); </del> <ins> THPUtils_assert(!self->has_freed_buffers, ERR_BACKWARD_TWICE); </ins> if (!self->saved_variables) return PyTuple_New(0);

Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)


Question: tasks or pull requests with your environment details is helpful and appreciated. -  **CI tests failed, what does it mean?** Maybe your PR is based <del> off a broken main bracnh? You can try to rebase your change on top </del> <ins> off a broken main branch? You can try to rebase your change on top </ins> of the latest main branch. You can also see the current status of main branch's CI at https://hud.pytorch.org/. -  **What are the most high risk changes?** Anything that touches build

There is a typo on the , in the Frequently Asked Questions. It says: Is that mean to be a joke (broken branch), or an actual typo? If it is meant to be a joke, please ignore this issue. I will be submitting a PR with the fix for the typo.
A typo for sure. I've force-merged your PR, thanks!


Question: l1=l, l2=l ) <ins> self.param = Variable(torch.Tensor(3, 5)) </ins> l = nn.Linear(10, 20) n = Net() <del> s = nn.Sequential(l, l, l, l) </del> <ins> s = nn.Sequential(n, n, n, n) </ins> self.assertEqual(num_params(l), 2) <del> self.assertEqual(num_params(n), 2) self.assertEqual(num_params(s), 2) </del> <ins> self.assertEqual(num_params(n), 3) self.assertEqual(num_params(s), 3) def test_modules(self): class Net(nn.Container): def __init__(self): super(Net, self).__init__() self.l1 = l self.l2 = l self.param = Variable(torch.Tensor(3, 5)) l = nn.Linear(10, 20) n = Net() s = nn.Sequential(n, n, n, n) self.assertEqual(list(s.modules()), [s, n, l]) </ins> def test_Sequential_getitem(self): l1 = nn.Linear(10, 20)

While using torch on a Macbook pro with M1 ship, using python 3.8.10 and torch 1.12.1, I came across a weird bug that happened because I forgot to put a tensor on MPS. The reproducible code is the following: As you can see, if is on MPS and on CPU, the line wrongly replaces the index in place of the index . I would expect to have an error saying both tensors should be on the same device instead. I get an error: cc
The error seems to happen somewhere within Also I'm getting a warning "The operator 'aten::nonzero' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications" when printing tensors on MPS.
Hi - thanks for reporting this issue. This is actually an allowed usecase (copying CPU tensors into MPS tensors / MPS -CPU / MPS -MPS): The issue is on MPS side when copying a CPU tensor into a strided MPS tensor. I'll take a look.


Question: main_sources += split_types("torch/csrc/cuda/Tensor.cpp") if WITH_NCCL: <del> if SYSTEM_NCCL: </del> <ins> if WITH_SYSTEM_NCCL: </ins> main_libraries += ['nccl'] <ins> include_dirs.append(NCCL_INCLUDE_DIR) library_dirs.append(NCCL_LIB_DIR) </ins> else: main_link_args += [NCCL_LIB] extra_compile_args += ['-DWITH_NCCL']

Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).


Question: r""" expm1_() -> Tensor <del> In-place version of :meth:`~Tensor.exp` </del> <ins> In-place version of :meth:`~Tensor.expm1` </ins> """) add_docstr_all('exponential_',

Build error: According to , the should be modified as:
I the same problem when installing pytorch from source. You mean the file is wrong?
Although the PyYAML documentation site suggests that is possible, I could not perform it. That is a reason for the failing. This has been reported as well: I believe that the should have this instead: In fact when I did this: This indicates that the import didn't happen in PyYAML itself.
you guys are right, the case has to be modified to: . The original issue (my guess) is that your pyyaml had some compile error and the CLoader wasn't successfully installed.


Question: if modules is None: raise AttributeError( "cannot assign module before Module.__init__() call") <ins> remove_from(self.__dict__, self._parameters, self._buffers) </ins> modules[name] = value elif modules is not None and name in modules: if value is not None:

They do one pair of these calls for each slicing operation.
Does this actually have a performance impact? These calls tend to be very quick
Yes, it's a major reason why small unrolled LSTMs (which involve in addition to two GEMMs: a chunk into four gates, for 4 pairs of calls, five unary pointwise ops, for 5 more, and five binary pointwise ops, for another 5) are much slower than they are in Theano and TensorFlow.
Also, there are excessive cudaGetDevice calls for each pointwise operation (9 cudaGetDevice) even without chunking, as in the following repro code: As for small unrolled LSTMs it does show up.
This should no longer be an issue, now that is implemented as a native ATen function. I think we're still using the Python version for tensors, but this will be resolved soon when we stop using tensors altogether.


Question: fi if [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-xenial-cuda9-cudnn7-py3 ]] ||  <del> [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-trusty-py3.6-gcc7.2 ]]; then </del> <ins> [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-trusty-py3.6-gcc7* ]]; then </ins> BUILD_TEST_LIBTORCH=1 else BUILD_TEST_LIBTORCH=0

We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in


Question: This is the reverse operation of the manner described in :meth:`~Tensor.gather`. <del> :attr:`self`, :attr:`index` and :attr:`src` should have same number of dimensions. It is also required that `index.size(d) <= src.size(d)` for all dimensions `d`, and that `index.size(d) <= self.size(d)` for all dimensions `d != dim`. </del> <ins> :attr:`self`, :attr:`index` and :attr:`src` (if it is a Tensor) should have same number of dimensions. It is also required that ``index.size(d) <= src.size(d)`` for all dimensions ``d``, and that ``index.size(d) <= self.size(d)`` for all dimensions ``d != dim``. </ins> Moreover, as for :meth:`~Tensor.gather`, the values of :attr:`index` must be <del> between `0` and `(self.size(dim) -1)` inclusive, and all values in a row along the specified dimension :attr:`dim` must be unique. </del> <ins> between ``0`` and ``self.size(dim) - 1`` inclusive, and all values in a row along the specified dimension :attr:`dim` must be unique. </ins> Args: dim (int): the axis along which to index

is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.


Question: *tempValues__data = *t_data; *tempIndices__data = *tempIndices__dimOffset; }); <ins> THTensor_(free)(tempValues_); THLongTensor_free(tempIndices_); </ins> } if (!keepdim) {

There is a possible memory leak in () This causes memory to blow up: When using () memory usage is ok There is no leak for unitary dimensions


Question: import ctypes import warnings import torch.cuda <ins> import sys </ins> import os.path as path lib = None <del> # TODO: fix libname for OSX / Windows </del> <ins> # TODO: fix libname for Windows </ins> # TODO: dynamic version checks via cudnnGetVersion # TODO: load 5.1.3 if using CUDA 7.5 and 5.1.5 if using CUDA 8.0 thisdir = path.dirname(__file__) libpaths = ['', path.join(thisdir, '../../lib')] <del> libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3'] </del> <ins> if sys.platform.startswith('linux'): libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3'] elif sys.platform == 'darwin': libnames = ['libcudnn.5.dylib'] else: libnames = [] </ins> def _loadlib(): global lib

The official website does not provide the torchvison version installation method. Here is the error message when torchvison 0.3.0 is used. torchvision/C.cpython-36m-x8664-linux- undefined symbol: ZN2at7getTypeERKNS6TensorE That should be caused by the mismatch version between pytorch-nightly and torchvision 0.3.0.
It should be that matches with , but they are not provided currently.
I tried installing it with pip as per , but now I get when importing . I did a similar thing on June 25th. It worked then. Any known fix for this?
do we document what to do here?
I finally solved this by building from source with Before running that make sure to upgrade and to use . Note: this will not work if you have on your system when building since you'd get version mismatch between and . Tested only on .
you build pytorch from source or just torchvision? or both of them.
I installed from conda with: I then cloned the repo and ran inside the repo. I guess this would also work if you do , but I haven't tried it.
Just a reminder (what I encountered just now): one should uninstall previous ` from source, or they can mix up and mess up everything.
There are nightlies for torchvision under Windows now.
I have problems using Pytorch with my Geforce RTX 3060, (sm86). I tried to install with help of : Create a new venv, based on python 3.7.10. Activate it. conda install pytorch-nightly cudatoolkit=10.0 -c pytorch (I also did it without the "=10.0") After this, Pytorch detects the sm86 with no errors, problem sm86 solved with pytorch-nightly Now, I have a problem with torchvision. tried installing torchvision with File "C:repovisiontorchvisionmodels", line 195, in Inception3 AttributeError: module '' has no attribute 'unused' tried installing torchvision version 0.3.0, from , but an error rose: tried installing torchvision at origin/master, from , but another error rose: Here is the list of packages installed:


Question: def _test_sparse_mask_fixed(self): i = self.IndexTensor([ <del> [1, 3, 3, 0, 4], [2, 1, 1, 2, 3], </del> <ins> [1, 3, 0, 4], [2, 1, 2, 3], </ins> ]) <del> v = self.ValueTensor([1, 2, 3, 4, 5]) x = self.SparseTensor(i, v, torch.Size([5, 4])) </del> <ins> v = self.ValueTensor([1, 2, 3, 4]) x = self.SparseTensor(i, v, torch.Size([5, 4])).coalesce() </ins> dense = self.ValueTensor([ [1, 2, 3, 4], [5, 6, 7, 8],

and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.


Question: from torch.autograd import Variable from torch.nn import Parameter from common_nn import NNTestCase, ModuleTest, CriterionTest, TestBase,  <del> module_tests, criterion_tests, TEST_CUDA, TEST_MULTIGPU, TEST_CUDNN, PRECISION </del> <ins> module_tests, criterion_tests, TEST_CUDA, TEST_MULTIGPU, TEST_CUDNN,  TEST_CUDNN_VERSION, PRECISION </ins> from common import freeze_rng_state, run_tests

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: } std::string FunctionSignature::toString() const { <ins> // TODO: consider printing more proper schema strings with defaults, optionals, etc. </ins> std::ostringstream ss; <ins> bool keyword_already = false; </ins> ss << "("; int i = 0; for (auto& param : params) { if (i != 0) { ss << ", "; } <ins> if (param.keyword_only && !keyword_already) { ss << "*, "; keyword_already = true; } </ins> ss << param.type_name() << " " << param.name; i++; }

Consider the following: A reasonable reading of this message is that the order of the parameters should be reversed, but this isn't equivalent: The problem is that the second signature has a keyword-only argument but it's printed. It also doesn't include defaults and such, but that's probably secondary. In this case the problem is at least flagged, but it's not clear to me whether there are other deprecated signatures that silently suggest the wrong thing. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: Collecting environment information... PyTorch version: 1.5.0a0+ Is debug build: Yes CUDA used to build PyTorch: Could not collect OS: CentOS Linux 7 (Core) GCC version: (GCC) 7.4.0 CMake version: version 3.14.0 Python version: 3.7 Is CUDA available: No CUDA runtime version: 9.2.88 GPU models and configuration: GPU 0: Tesla M40 GPU 1: Tesla M40 Nvidia driver version: 396.69 cuDNN version: /usr/local/cuda-9.2/targets/x8664- Versions of relevant libraries: [pip] numpy==1.17.2 [pip] torch==1.5.0a0+ [conda] blas 1.0 mkl [conda] mkl 2019.4 243 [conda] mkl-include 2019.4 243 [conda] mkl-service 2.3.0 py37he904b0f0 [conda] mklfft 1.0.14 py37ha843d7b0 [conda] mklrandom 1.1.0 py37hd6b4f25_0 [conda] torch 1.5.0a0+ <pip<!-- Add any other context about the problem here. --


Question: gI = apply_fn<Transpose>(0, 1)(gIt); } } <ins> if (should_compute_output(0) && !ggO.defined()) ggO = at::zeros_like(gO); if (should_compute_output(1) && !gI.defined()) gI = at::zeros_like(input); if (should_compute_output(2) && !gW.defined()) gW = at::zeros_like(weight); </ins> return {ggO, gI, gW}; }

The following code: produces this error Also, removing the ReLU from the network: causes the error to change to: However, when the second convolution is removed as well: then the code runs error free I'm on a ppc64le architecture with a P100 GPU, CUDA 8.0 and PyTorch version '0.4.0a0+'
I'm also having this problem using branch v0.3.0
i could reproduce this as well. we'll look into this.
I'm having the same error :/
same error with gradient penalty for WGAN
When you execute , are you cause memory leak
no there shouldn't be a memory leak.
I am having an out of memory error as well, only when applying backward to the output of this method. I am using 3 titans GPUs so memory is not a problem I would assume. def computeGP(netD, realdata, realembed, fakedata, LAMBDA): BATCHSIZE = (0) alpha = (BATCHSIZE, 1) alpha = (BATCHSIZE, int(realdata.nelement() / BATCHSIZE)).contiguous().view(BATCHSIZE, 3, 64, 64) alpha = () interpolates = alpha realdata + ((1 - alpha) fakedata) interpolates = () interpolates = autograd.Variable(interpolates, requiresgrad=True) discinterpolates, = netD(interpolates, realembed) gradients = (outputs=discinterpolates, inputs=interpolates, (()).cuda(), creategraph=True, retaingraph=True, onlyinputs=True)[0] gradientpenalty = (((2, dim=1) - 1) * 2).mean() LAMBDA return gradient_penalty `
I am also using it with gradient penalty for WGAN, But When I use gradient_penalty.backward(), the memory leak is happen, My pytorch is 0.4.0a0+.
I can reproduce the memory leak with this: However the leak disappears when only one convolution is used: Also, if nn.Linear is used, there is no leak. Torch version: 0.4.0a0+
I've opened a separate issue for that memory leak:


Question: results = {} def _worker(i, module, input, kwargs, results, lock): <del> var_input = input while not isinstance(var_input, Variable): var_input = var_input[0] </del> <ins> var_input = get_a_var(input) </ins> try: with torch.cuda.device_of(var_input): output = module(*input, **kwargs)

is the culprit. Assumes that the first input is a variable or a sequence containing a variable in index 0 (recursively indexed). Workaround is to pass a dummy variable as first input, as suggested in
Could you provide a short reproducer for testing? Think this shouldn't be too hard to fix.
Sure, here's a gist along with commands/outputs:
fixed in master via


Question: PyObject *THPFunction_saved_tensors(THPFunction *self, void *_unused) { <del> THPUtils_assert(!self->has_freed_buffers, "Trying to backward through the " "graph second time, but the buffers have already been freed. Please " "specify retain_variables=True when calling backward for the first time."); </del> <ins> THPUtils_assert(!self->has_freed_buffers, ERR_BACKWARD_TWICE); </ins> if (!self->saved_variables) return PyTuple_New(0);

Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)


Question: import torch.optim as optim import torch.legacy.optim as old_optim from torch.autograd import Variable <ins> from torch import sparse </ins> from common import TestCase, run_tests

and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.


Question: PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (!array_ptr || ptr != PyArray_DATA(array_ptr)) throw std::logic_error("invalid call to NumpyArrayAllocator::free()"); <del> object = nullptr; </del> <ins> { AutoGIL gil; object = nullptr; } </ins> delete this; } #endif

When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1


Question: using TensorRef = std::reference_wrapper<const Tensor>; using TensorRefList = std::initializer_list<TensorRef>; <ins> // ArrayRef is not covariant, which means there is no // implicit conversion between TensorList (aka ArrayRef<Tensor>) // and ArrayRef<Variable>.  What we do instead is manually // construct a variable_list, which itself is implicitly convertible // into an ArrayRef<Variable> (but don't return an ArrayRef<Variable>; // ArrayRef is non-owning!) static variable_list cast_tensor_list(const TensorList& tensors) { // TODO: Eliminate the intermediate vector allocation return variable_list(tensors.begin(), tensors.end()); } </ins> static VariableFlags compute_flags(const TensorRefList& tensors) { return compute_flags_tmpl(tensors); }

when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit


Question: The lower triangular part of the matrix is defined as the elements on and below the diagonal. <del> The argument :attr:`k` controls which diagonal to consider. </del> <ins> The argument :attr:`diagonal` controls which diagonal to consider. </ins> <del> - :attr:`k` = 0, is the main diagonal. - :attr:`k` > 0, is above the main diagonal. - :attr:`k` < 0, is below the main diagonal. </del> <ins> - :attr:`diagonal` = 0, is the main diagonal. - :attr:`diagonal` > 0, is above the main diagonal. - :attr:`diagonal` < 0, is below the main diagonal. </ins> Args: input (Tensor): the input `Tensor` <del> k (int, optional): the diagonal to consider </del> <ins> diagonal (int, optional): the diagonal to consider </ins> out (Tensor, optional): The result `Tensor` Example::

The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.


Question: #ifdef USE_MAGMA THArgCheck(a_->nDimension == 2, 1, "A should be 2 dimensional"); THArgCheck(b_->nDimension == 2, 1, "b should be 2 dimensional"); <del> THArgCheck(a_->size[0] == b_->size[0], 2, "size incompatible A,b"); THArgCheck(a_->size[0] >= a_->size[1], 2, "A should have m >= n"); </del> <ins> THArgCheck(a_->size[0] == b_->size[0], 2, "Expected A and b to have same size " "at dim 0, but they have incompatible sizes"); THArgCheck(a_->size[0] >= a_->size[1], 2, "Expected A with shape (m x n) to have " "m >= n. The case for m < n is not implemented yet."); </ins> THCTensor *a = THCTensor_(newColumnMajor)(state, ra_, a_); THCTensor *b = THCTensor_(newColumnMajor)(state, rb_, b_);

I am currently using PyTorch 0.3.0 and I am noticing some (possibly strange) differences between the behavior of when operating on CPU tensors versus GPU ones, in particular relating to whether the number of rows of A is allowed to be larger than the number of columns or not. Here is some minimal code that reproduces my problem: The first solve on the CPU works fine; however, the second solve gives me the error: Is this a limitation of the CUDA implementation of that PyTorch uses, or otherwise intended? If so, any guidance about a possible work-around would be greatly appreciated. Our only thought on a work-around was to manually QR decompose A with and then use , but doesn't support CUDA tensors.
MAGMA gels doesn't implement the m < n case.


Question: (Cross, (), ((S, 3), (S, 3))), (Cross, (), ((S, 3, S), (S, 3, S), 1), 'dim'), (Inverse, (), ((S, S),), '', (), [skipIfNoLapack]), <ins> (Gesv, (), ((S, S), (S, S)), '', (), [skipIfNoLapack]), </ins> (Clone, (), ((S, M, S),)), (Squeeze, (), ((S, 1, M, 1),)), # TODO: enable neg dim checks

There seems to be regression in the dynamo export of the MONAI VNet model as regards . Export script: The versions I'm using is torch 2.5.0.dev20240625+cu121 Previous torch versions such as were able to export this model successfully. Perhaps dynamo was generating FX graph without the expm1 node previously? cc: PyTorch version: 2.5.0.dev20240625+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (x8664) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.29.0 Libc version: glibc-2.35 Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.5.0-35-generic-x8664-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.4.131 CUDAMODULELOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA RTX A5000 Nvidia driver version: 535.171.04 cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i9-9820X CPU @ 3.

30GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 Stepping: 4 CPU max MHz: 4200.0000 CPU min MHz: 1200.0000 BogoMIPS: 6599.98 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constanttsc art archperfmon pebs bts repgood nopl xtopology nonstoptsc cpuid aperfmperf pni pclmulqdq dtes64 monitor dscpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse41 sse42 x2apic movbe popcnt tscdeadlinetimer aes xsave avx f16c rdrand lahflm abm 3dnowprefetch cpuidfault epb catl3 cdpl3 invpcidsingle pti ssbd mba ibrs ibpb stibp tprshadow flexpriority ept vpid eptad fsgsbase tscadjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdta avx512f avx512dq rdseed adx smap clflushopt clwb intelpt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqmllc cqmoccupllc cqmmbmtotal cqmmbmlocal dtherm ida arat pln pts hwp hwpactwindow hwpepp hwppkgreq vnmi mdclear flushl1d archcapabilities Virtualization: VT-x L1d cache: 320 KiB (10 instances) L1i cache: 320 KiB (10 instances) L2 cache: 10 MiB (10 instances) L3 cache: 16.5 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-19 Vulnerability Gather data sampling: Mitigation;

Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and _user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Versions of relevant libraries: [pip3] numpy==1.24.4 [pip3] onnx==1.16.0 [pip3] onnxscript==0.1.0.dev20240625 [pip3] optree==0.11.0 [pip3] pytorch-quantization==2.1.2 [pip3] pytorch-triton==3.0.0+ [pip3] torch==2.5.0.dev20240625+cu121 [pip3] torch-tensorrt==2.3.0a0 [pip3] torchaudio==2.4.0.dev20240625+cu121 [pip3] torchdata==0.7.1a0 [pip3] torchtext==0.17.0a0 [pip3] torchvision==0.20.0.dev20240625+cu121 [conda] Could not collect
I am not very sure, but this looks like the kind of ops that should get decomp'ed. do you know? could you help take a look as part of the op support work you are doing? Thanks!
Yes, it looks like there is a graph-changed in dynamo, which dynamo now decomposes elu for some reasons. Using can preserve elu and then successfully export though.

Yes can run but we would need to be resolved first since the weights are still not exported properly.
I can take a look at this.
should fix this.

Here, I can't export trivial net with sort() using dynamoexport(). Legacy () works fine, dynamoexport errors out : Unsupported FX nodes: {'callfunction': ['']}. Pytorch nightly 06/03 cc
a missing op, could you take this?
Not sure if related:
Hmmm. Ideally, ONNX should add a sort op. It doesn't have one. I wonder what the legacy exporter converts it to.
should fix this issue.

Are there any plans to implement something like tensorflow's and ? exists but it doesn't have the backward step implemented.
One workaround for the moment would be to write as , which should support backprop since . Adding support for should be trivial, the gradient for it can be found in
I'll add backward support for , which should be equivalent to . I'll try to do it later today.
How difficult would it be to add in batched versions (where only the final two dimensions are square) of GESV and inverse? Are there any plans for this?
If we have batched version of , then adapting the autograd part is very simple. There is an open issue already that discuss implementing batched gesv
Actually, there is an open PR for that, so it might be coming soon
I'll rebase the batched version of gesv. It'll come in soon
May I ask why gesv only support . As I saw from , it support that A is a MxN matrix, and B is a NxK matrix, then solve a matrix X (NxK). If I want to use a non-square matrix A in gesv, would you mind to let me know how to implement in PyTorch?
this doesn't exactly answer your question but you can add zero-rows to the A matrix until it becomes square.
thanks a lot, I find the solution, can solve the problem.
Any plan to add gradient calculation for gels?
I am also wondering if will have differentiation support. My current workaround is to use instead
I actually implemented overdetermined case following tensorflow implementation. For backward pass I only did basic grad test. On Monday, March 18, 2019, Johnson Zhong <:
Thanks, seems to work fine!


Question: return None return match.group(1) <ins> def run_and_return_first_line(run_lambda, command): """Runs command using run_lambda and returns first line if output is not empty""" rc, out, _ = run_lambda(command) if rc != 0: return None return out.split('n')[0] </ins> def get_conda_packages(run_lambda): if get_platform() == 'win32':

TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC


Question: from tools.setup_helpers.env import check_env_flag from tools.setup_helpers.cuda import WITH_CUDA, CUDA_HOME from tools.setup_helpers.cudnn import WITH_CUDNN, CUDNN_LIB_DIR, CUDNN_INCLUDE_DIR <ins> from tools.setup_helpers.nccl import WITH_NCCL, WITH_SYSTEM_NCCL, NCCL_LIB_DIR, NCCL_INCLUDE_DIR, NCCL_ROOT_DIR </ins> from tools.setup_helpers.split_types import split_types DEBUG = check_env_flag('DEBUG') WITH_DISTRIBUTED = not check_env_flag('NO_DISTRIBUTED') WITH_DISTRIBUTED_MW = WITH_DISTRIBUTED and check_env_flag('WITH_DISTRIBUTED_MW') <del> WITH_NCCL = WITH_CUDA and platform.system() != 'Darwin' SYSTEM_NCCL = False </del> ################################################################################

Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).


Question: export PATH="$PWD:$PATH" popd <del> time test/run_test.sh </del> <ins> time test/run_test.sh -- -v </ins> rm -rf ninja

If a test segfaults, it's hard to determine which test it is because the output looks like: ! I can work on this if we think this is a good idea.
Last time I asked about this, he didn't want to because it adds a lot of scrolling. I also did some work to move the CI tests to so that we can output XML status files, but (1) this is unlikely to help in case of segfault, and (2) you can't do this straightforwardly, because we run some test files multiple times with different environment variables, and the XML exporter isn't really set up to handle this case. For the record, I am personally OK with just slapping on the tests and calling it a day.
in the conbuild sounds good to me too
closed via


Question: std::unique_ptr<npy_intp[]> sizes; std::unique_ptr<npy_intp[]> strides; <ins> THPUtils_assert(PyTuple_Size(args) == 0, "numpy() takes no positional arguments"); </ins> // Numpy and Torch disagree on empty tensors. In Torch, an empty tensor // is a tensor with zero dimensions. In Numpy, a tensor with zero dimensions // is a scalar (with one element). So we'll convert an empty Torch tensor

Currently due to the way cwrap generates wrapping code for the c source, passing an argument to .numpy() would not result in any error. I think probably this should not be the desired behaviour for the API?
fixed via


Question: // See Note [Numpy memory management] void* NumpyArrayAllocator::realloc(void* ptr, ptrdiff_t size) { <del> PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (array_ptr && ptr == PyArray_DATA(array_ptr)) { void* newPtr = this->malloc(size); memcpy(newPtr, ptr, std::min((size_t) size, (size_t) PyArray_NBYTES(array_ptr))); // Whee! We're done! object = nullptr; return newPtr; } return allocator->realloc(allocatorContext, ptr, size); </del> <ins> throw std::logic_error("NumpyArrayAllocator::realloc() not supported"); </ins> } // See Note [Numpy memory management]

When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1


Question: """Fills the input Tensor or Variable with values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from :math:`U(-bound, bound)` where <del> :math:`bound = sqrt{2 / ((1 + a^2) times fan_in)} times sqrt{3}`. Also known as He initialisation. </del> <ins> :math:`bound = sqrt{2 / ((1 + a^2) times fan_in)} times sqrt{3}`. Also known as He initialisation. </ins> Args: tensor: an n-dimensional torch.Tensor or autograd.Variable

why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:


Question: sequence.add_(start - 1) return sequence.resize_(*size) <del> def test_index(self): reference = self._consecutive((3, 3, 3)) </del> <ins> @staticmethod def _test_index(self, conv_fn): def consec(size, start=1): sequence = torch.ones(int(torch.Tensor(size).prod(0)[0])).cumsum(0) sequence.add_(start - 1) return sequence.view(*size) reference = conv_fn(consec((3, 3, 3))) </ins> # empty tensor indexing <del> self.assertEqual(reference[torch.LongTensor()], reference.new()) </del> <ins> self.assertEqual(reference[conv_fn(torch.LongTensor())], reference.new()) </ins> <del> self.assertEqual(reference[0], self._consecutive((3, 3)), 0) self.assertEqual(reference[1], self._consecutive((3, 3), 10), 0) self.assertEqual(reference[2], self._consecutive((3, 3), 19), 0) self.assertEqual(reference[0, 1], self._consecutive((3,), 4), 0) self.assertEqual(reference[0:2], self._consecutive((2, 3, 3)), 0) </del> <ins> self.assertEqual(reference[0], consec((3, 3)), 0) self.assertEqual(reference[1], consec((3, 3), 10), 0) self.assertEqual(reference[2], consec((3, 3), 19), 0) self.assertEqual(reference[0, 1], consec((3,), 4), 0) self.assertEqual(reference[0:2], consec((2, 3, 3)), 0) </ins> self.assertEqual(reference[2, 2, 2], 27, 0) <del> self.assertEqual(reference[:], self._consecutive((3, 3, 3)), 0) </del> <ins> self.assertEqual(reference[:], consec((3, 3, 3)), 0) </ins> # indexing with Ellipsis self.assertEqual(reference[..., 2], torch.Tensor([[3, 6, 9],

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.


Question: <del> import torch from torch.autograd.function import Function from torch._thnn import type2backend import torch.backends.cudnn as cudnn class BatchNorm(Function): def __init__(self, running_mean, running_var, training, momentum, eps): super(BatchNorm, self).__init__() self.running_mean = running_mean self.running_var = running_var self.training = training self.momentum = momentum self.eps = eps def forward(self, input, weight=None, bias=None): self.save_for_backward(input, weight, bias) # don't use cuDNN for half inputs because cuDNN requires the weight and # bias tensors to be floats, unlike THCUNN which requires half tensors. self.use_cudnn = (cudnn.is_acceptable(input) and cudnn.version() > 5110 and weight is not None and bias is not None and not isinstance(input, torch.cuda.HalfTensor)) # temporary buffers used in forward and backward num_features = input.size(1) _save_mean = input.new(num_features) _save_std = input.new(num_features) output = input.new(input.size()) if self.use_cudnn: torch._C._cudnn_batch_norm_forward( input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) else: backend = type2backend[type(input)] backend.BatchNormalization_updateOutput( backend.library_state, input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) if self.requires_grad: self._save_mean = _save_mean self._save_std = _save_std return output def backward(self, grad_output): input, weight, bias = self.saved_tensors grad_input, grad_weight, grad_bias = None, None, None if self.needs_input_grad[0] or self.use_cudnn: grad_input = input.new(input.size()) if (len(self.needs_input_grad) > 1 and self.needs_input_grad[1]) or self.use_cudnn: grad_weight = weight.new(weight.size()).zero_() if (len(self.needs_input_grad) > 1 and self.needs_input_grad[2]) or self.use_cudnn: grad_bias = bias.new(bias.size()).zero_() if self.use_cudnn and self.training: # cudnn does not support backward in evaluate mode torch._C._cudnn_batch_norm_backward( input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, self.eps) else: grad_output = grad_output.contiguous() backend = type2backend[type(input)] backend.BatchNormalization_backward( backend.library_state, input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, 1.0, self.eps) return grad_input, grad_weight, grad_bias </del>

As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10


Question: for i in range(10): Variable(torch.randn(10, 10), _grad_fn=CollectOnDelete()) <del> @unittest.skipIf(not torch.cuda.is_available() or torch.cuda.device_count() < 2, "CUDA not available or <2 GPUs detected") </del> <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") </ins> def test_unused_output_gpu(self): from torch.nn.parallel._functions import Broadcast x = Variable(torch.randn(5, 5).float().cuda(), requires_grad=True)

Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: res = [] for g in range(self.groups): def group(tensor, dim=None): <ins> if tensor is None: return None </ins> if dim is None: dim = 0 if tensor.dim() == 1 else 1 n = tensor.size(dim) // self.groups

x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.


Question: self.assertEqual(n[1], l2) self.assertEqual(n[2], l3) self.assertEqual(n[3], l4) <ins> self.assertEqual(n[1:], nn.Sequential(l2, l3, l4)) self.assertEqual(n[3:], nn.Sequential(l4)) self.assertEqual(n[:-1], nn.Sequential(l1, l2, l3)) self.assertEqual(n[:-3], nn.Sequential(l1)) self.assertEqual(n[::-1], nn.Sequential(l4, l3, l2, l1)) </ins> def test_ModuleList(self): modules = [nn.ReLU(), nn.Linear(5, 5)]

it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.


Question: id_map = {old_id: p for old_id, p in zip(chain(*(g['params'] for g in saved_groups)), chain(*(g['params'] for g in groups)))} <del> state = {id_map.get(k, k): v for k, v in state_dict['state'].items()} </del> <ins> state = defaultdict( dict, {id_map.get(k, k): v for k, v in state_dict['state'].items()}) </ins> # Update parameter groups, setting their 'params' value def update_group(group, new_group):

The state attribute in the opt. class is a defaultdict which helps to add new parameter states in the state dict as they are being used while training. Once the optimizer's statedict is saved and loaded, the state attribute is a dict and not a defaultdict which assumes that before saving, all the parameters in the network are present in the state dict. When loading the opt. using a saved statedict the state attribute should be a defaultdict. I currently do the following to have this behavior Corresponding
Hi all, it is my first time trying to contribute to an open-source project. I took a look on the bug. It seems that it is related to the function "statedict()" of class "". When the state is transformed to packedstate, it uses dict instead of defaultdict, which is inconsistent with the type of state. The solution I am proposing is to let the packedstate be defaultdict to maintain consistency with Here is my proposed change: packedstate = defaultdict(dict, {(id(k) if isinstance(k, Variable) else k): v for k, v in ()}) Is there anything I am missing? Is "" in the pytorch/test enough to validate my change? Where could I add a test point to ensure the type of state remain the same after saving and loading? Thanks a lot for your patience.
closed via thanks to !! Thanks a lot !


Question: std::tie(self, indices) = transposeToFront(self, indices); } auto linearIndex = computeLinearIndex(self, indices); <del> return std::make_tuple<>(self, linearIndex); </del> <ins> return std::make_tuple(self, linearIndex); </ins> } Tensor index(const Tensor & self, TensorList indices) {

In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:


Question: if not self.shutdown: self.shutdown = True self.done_event.set() <del> # if worker_manager_thread is waiting to put while not self.data_queue.empty(): self.data_queue.get() </del> <ins> # if worker_manager_thread is waiting to put, make place for it try: while not self.data_queue.empty(): self.data_queue.get() except FileNotFoundError: # FileNotFoundError can happen when we rebuild the fd # fetched from the queue but the socket is already closed # from the worker side (e.g. due to Python shutting down). pass </ins> for _ in self.workers: self.index_queue.put(None) # done_event should be sufficient to exit worker_manager_thread,

I'm creating a in my script ([]), and I get an error when this script is close to ending. I am using MPI through mpi4py, not the MPI PyTorch distributed backend. This error only happens when under , not under (which completes successfully). Nothing in the traceback reveals this is an issue with my code, and I haven't touched the DataLoader at all. I've been running this code for a couple months in the same environment, and I haven't seen this bug pop before. That is, this command does not raise an error: However, this command does raise an error: Traceback: PyTorch version: 0.3.1.post2 How you installed PyTorch (conda, pip, source): conda Python version: 3.6.1 OS/CUDA/cuDNN version: A modification of the Deep Learning AMI GPU models and configuration: Amazon instance []
I do not see this bug with Pytorch 0.3.0.post4`


Question: @property def saved_tensors(self): flat_tensors = super(NestedIOFunction, self).saved_tensors <del> return _map_tensor_fromiter(iter(flat_tensors))(self._to_save_nested) </del> <ins> return _unflatten(flat_tensors, self._to_save_nested) </ins> def mark_dirty(self, *args, **kwargs): self.dirty_tensors = tuple(_iter_tensors((args, kwargs)))

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: using thpp::Tensor; <ins> #ifndef CUDNN_BN_MIN_EPSILON #define CUDNN_BN_MIN_EPSILON 0 #endif </ins> auto BatchNormForward::apply(const variable_list& inputs) -> variable_list { check_input_variables("BatchNorm", inputs, 3, 1);

Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me.

As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .


Question: self.assertEqual(list(tensor), [4, 4, 4, 4]) p.join() <ins> def _test_empty_tensor_sharing(self, dtype): q = mp.Queue() empty = torch.tensor([], dtype=dtype) q.put(empty) out = q.get(timeout=1) self.assertEqual(out, empty) def test_empty_tensor_sharing(self): self._test_empty_tensor_sharing(torch.float32) self._test_empty_tensor_sharing(torch.int64) @unittest.skipIf(not torch.cuda.is_available(), 'CUDA not available') def test_empty_tensor_sharing_cuda(self): self._test_empty_tensor_sharing(torch.cuda.float32) self._test_empty_tensor_sharing(torch.cuda.int64) </ins> def _test_autograd_sharing(self, var): ready = mp.Event() master_modified = mp.Event()

PyTorch version: 0.4.0a0+ Repro: Result:
Emm. I fail to reproduce this bug in Python 3.5.0 and Mac OS 10.13.3. What's your system environment?
The bug is probably specific to the sharing mode. Mac OS uses by default.


Question: .. automethod:: repeat .. automethod:: resize_ .. automethod:: resize_as_ <del> .. automethod:: retain </del> .. automethod:: round .. automethod:: round_ .. automethod:: rsqrt

Dataloader call getitem as many times as indices in the current batch. In case datasets support a list of indices in one call, or a native python slice object, add a getbatch (optional) to pytorch dataset class. If dataloader sees the dataset has such a method implemented, make it fetch the batch corresping to the list of indices with one getbatch call rather than many getitem calls. This saves the trouble of calling getitem many times in cases where for example a disk file has to be opened then closed for every call and the underlying data structure supports a slicing or lists of indices for example hdf5 files.
will support something similar
is the latest version of this.


Question: // This Engine's ReadyQueues and their corresponding threads are leaked here Engine::~Engine() = default; <del> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue) -> void { </del> <ins> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue, int device) -> void { </ins> THInferNumThreads(); <ins> AutoGPU guard(device); </ins> while (1) { FunctionTask task = queue->pop_back(); if (!task.base->has_error.load()) {

Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --


Question: return _iter <ins> def _unflatten(input, proto): # unflatten a list or tuple input into a nested list/tuple structure # specified by proto def unflatten_helper(input, proto): res = [] if not isinstance(proto, (list, tuple)): return input[0], input[1:] for e in proto: res_e, input = unflatten_helper(input, e) res.append(res_e) return type(proto)(res), input return unflatten_helper(input, proto)[0] </ins> _iter_variables = _iter_filter(lambda o: isinstance(o, torch.autograd.Variable)) _iter_tensors = _iter_filter(torch.is_tensor) _iter_None_tensors = _iter_filter(lambda o: o is None or torch.is_tensor(o)) _map_variable_tensor = _nested_map(lambda o: isinstance(o, torch.autograd.Variable), lambda o: o.data) <del> def _map_tensor_fromiter(itr): return _nested_map(lambda o: torch.is_tensor(o), lambda o: next(itr)) </del> class NestedIOFunction(Function): def _do_forward(self, *input):

cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with


Question: - THTensor* tensor2 ]] <del> #ifndef THP_LAPACK_CONSTANTS #define THP_LAPACK_CONSTANTS </del> <ins> #if defined(TH_REAL_IS_FLOAT) || defined(TH_REAL_IS_DOUBLE) || CUDA_FLOAT || CUDA_DOUBLE </ins> // We need to pass pointers to chars to tensor lapack functions... static const char __U = 'U'; static const char __L = 'L';

Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something


Question: .. autofunction:: einsum .. autofunction:: flip .. autofunction:: histc <ins> .. autofunction:: meshgrid </ins> .. autofunction:: renorm .. autofunction:: trace .. autofunction:: tril

is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.


Question: [ 0.0000,  0.0000,  0.0000,  1.2300]]) """) <ins> add_docstr_all('scatter_add_', r""" scatter_add_(dim, index, other) -> Tensor Adds all values from the tensor :attr:`other` into :attr:`self` at the indices specified in the :attr:`index` tensor in a similar fashion as :meth:`~torch.Tensor.scatter_`. For each value in :attr:`other`, it is added to an index in :attr:`self` which is specified by its index in :attr:`other` for ``dimension != dim`` and by the corresponding value in :attr:`index` for ``dimension = dim``. For a 3-D tensor, :attr:`self` is updated as:: self[index[i][j][k]][j][k] += other[i][j][k]  # if dim == 0 self[i][index[i][j][k]][k] += other[i][j][k]  # if dim == 1 self[i][j][index[i][j][k]] += other[i][j][k]  # if dim == 2 :attr:`self`, :attr:`index` and :attr:`other` should have same number of dimensions. It is also required that ``index.size(d) <= other.size(d)`` for all dimensions ``d``, and that ``index.size(d) <= self.size(d)`` for all dimensions ``d != dim``. Moreover, as for :meth:`~Tensor.gather`, the values of :attr:`index` must be between ``0`` and ``self.size(dim) - 1`` inclusive, and all values in a row along the specified dimension :attr:`dim` must be unique. Args: dim (int): the axis along which to index index (LongTensor): the indices of elements to scatter and add other (Tensor): the source elements to scatter and add Example:: >>> x = torch.rand(2, 5) >>> x tensor([[0.7404, 0.0427, 0.6480, 0.3806, 0.8328], [0.7953, 0.2009, 0.9154, 0.6782, 0.9620]]) >>> torch.ones(3, 5).scatter_add_(0, torch.tensor([[0, 1, 2, 0, 0], [2, 0, 0, 1, 2]]), x) tensor([[1.7404, 1.2009, 1.9154, 1.3806, 1.8328], [1.0000, 1.0427, 1.0000, 1.6782, 1.0000], [1.7953, 1.0000, 1.6480, 1.0000, 1.9620]]) """) </ins> add_docstr_all('select', r""" select(dim, index) -> Tensor

is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.


Question: } } <del> if (use_cudnn) { </del> <ins> if (use_cudnn && eps >= CUDNN_BN_MIN_EPSILON) { </ins> #ifdef WITH_CUDNN torch::cudnn::cudnn_batch_norm_backward( state,

Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me.

As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .


Question: return ret; } <del> Variable VariableType::as_variable(Tensor tensor) const { </del> <ins> static Variable as_variable(Tensor tensor) { </ins> return make_variable(std::move(tensor)); } <del> std::tuple<Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable> as_variable(std::tuple<Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors)))); } <del> std::tuple<Variable, Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors))), make_variable(std::move(std::get<2>(tensors)))); } <del> std::tuple<Variable, Variable, Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor, Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors))),

See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:


Question: self.assertRaises(TypeError, lambda: reference[0.0, ..., 0.0:2.0]) self.assertRaises(TypeError, lambda: reference[0.0, :, 0.0]) <ins> def test_index(self): self._test_index(self, lambda x: x) </ins> @staticmethod def _test_advancedindex(self, conv_fn): # Tests for Integer Array Indexing, Part I - Purely integer array

Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.