25.01.2025, 09:40 (This post was last modified: 25.01.2025, 10:37 by Selur.)
Just as a heads-up: updating to Torch 2.7 and updating torchvision breaks ColorMNet)
Noticed this when updating for dfttest2 (https://github.com/AmusementClub/vs-dfttest2/issues/8).
=> would be nice if you could update ColorMNet (the spatial_correlation_sampler stuff) to work with newer torch versions. (All other addons seems to work fine, even BasicVSR++ since it dropped the mmc dependency)
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
which installs an even newer version. (see attachment for how I setup the torch add-on so far; still incomplete)
Quote:I cannot recompile the correlation sample at every torch build, please change it only if really necessary.
I guess there is no way to get rid of that dependency?
Since I from time to time need to update the torch-addon to support newer stuff, I need a way to get this working, ideally without you having to build it. If you could write down the instructions on how to build the correlation sample, I could do it myself whenever I need to setup a new torch-addon.
Alternatively, I would have to drop vsDeOldify since I can't install it.
I could set up a download for the last working Hybrid version with the torch-addon for that version on my filejump.com account, but I would not update vsDeOldify anymore.
(I could keep the current support for it in Hybrid, so users could still use newer versions, if the manage to add them and their dependencies to the Lib/site-packages as long as the vsDeOldify calls do not change.)
Cu Selur
Ps.: Please also check whether the Deoldify setup steps seem correct to you, thanks.
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
(25.01.2025, 12:43)Selur Wrote: Since I from time to time need to update the torch-addon to support newer stuff, I need a way to get this working, ideally without you having to build it. If you could write down the instructions on how to build the correlation sample, I could do it myself whenever I need to setup a new torch-addon.
I think that is better to explain how to build Pytorch-Correlation-extension so that you are free to install any pytorch version.
To build Pytorch-Correlation-extension it is necessary to have an installation of Microsoft Visual Studio.
I installed the version 2019. This software is necessary to compile the C/C++ code.
This is a zip archive that need to be extracted in the same folder of CUDA SDK.
The directories to extract are: lib, include and bin.
For example in my PC I extracted the cudnn directories above in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4.
in this way all the cudnn dlls are stored in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin.
After the installation you should end-up having the following environment variables defined:
Okay, plans changed and I ended setting up the VM.
Installed Win11, Visual Studio, the NVIDIA stuff (needed to add the LAZY loading to the environment) and Anaconda3.
I then started the anaconda_prompt from the Anaconda.Navigator and started calling the listed call:
Preparing transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate colormnet
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) C:\Users\selur>conda activate colormnet
(colormnet) C:\Users\selur>cd colormnet
Das System kann den angegebenen Pfad nicht finden.
Did something go wrong, or do I need to change to ' C:\ProgramData\anaconda3\envs\colormnet' ?
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Sorry I forgot the step of creating the working folder (don't forget to install also Visual Studio).
You can create a working folder with any name, in this case it is just necessary to install Pytorch-Correlation-extension source code to compile.
I suggested to call it with the name colorment since is used by colorment but any name is Ok.
I'm happy that you have discovered the location of colormnet environment on you Windows installation.
To be sure, after having installed the pytorch stuff, I override the conda environment (in this case C:\ProgramData\anaconda3\envs\colormnet')
with exactly the same pytorch packages installed in Hybrid, this should guarantee the maximun compatibility.
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\cmd.py:79: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
running bdist_egg
running egg_info
creating Correlation_Module\spatial_correlation_sampler.egg-info
writing Correlation_Module\spatial_correlation_sampler.egg-info\PKG-INFO
writing dependency_links to Correlation_Module\spatial_correlation_sampler.egg-info\dependency_links.txt
writing requirements to Correlation_Module\spatial_correlation_sampler.egg-info\requires.txt
writing top-level names to Correlation_Module\spatial_correlation_sampler.egg-info\top_level.txt
writing manifest file 'Correlation_Module\spatial_correlation_sampler.egg-info\SOURCES.txt'
C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py:537: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'Correlation_Module\spatial_correlation_sampler.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'Correlation_Module\spatial_correlation_sampler.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build\lib.win-amd64-cpython-312\spatial_correlation_sampler
copying Correlation_Module\spatial_correlation_sampler\spatial_correlation_sampler.py -> build\lib.win-amd64-cpython-312\spatial_correlation_sampler
copying Correlation_Module\spatial_correlation_sampler\__init__.py -> build\lib.win-amd64-cpython-312\spatial_correlation_sampler
running build_ext
C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py:422: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py:456: UserWarning: The detected CUDA version (12.4) has a minor version mismatch with the version that was used to compile PyTorch (12.6). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
building 'spatial_correlation_sampler_backend' extension
creating build\temp.win-amd64-cpython-312\Release\Correlation_Module
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DUSE_CUDA -IC:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\include -IC:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\include\torch\csrc\api\include -IC:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\include\TH -IC:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\include" -IC:\ProgramData\anaconda3\include -IC:\ProgramData\anaconda3\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /EHsc /TpCorrelation_Module\correlation.cpp /Fobuild\temp.win-amd64-cpython-312\Release\Correlation_Module\correlation.obj /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /wd4624 /wd4067 /wd4068 /EHsc -std=c++17 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=spatial_correlation_sampler_backend -D_GLIBCXX_USE_CXX11_ABI=0 /std:c++17
cl : Befehlszeile warning D9002 : Unbekannte Option "-std=c++17" wird ignoriert.
cl : Befehlszeile warning D9002 : Unbekannte Option "-fopenmp" wird ignoriert.
correlation.cpp
C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py:2069: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\colormnet\Pytorch-Correlation-extension\setup.py", line 69, in <module>
launch_setup()
File "C:\ProgramData\anaconda3\envs\colormnet\Pytorch-Correlation-extension\setup.py", line 37, in launch_setup
setup(
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\core.py", line 186, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\core.py", line 202, in run_commands
dist.run_commands()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\dist.py", line 983, in run_commands
self.run_command(cmd)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\dist.py", line 1002, in run_command
cmd_obj.run()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\install.py", line 109, in run
self.do_egg_install()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\install.py", line 167, in do_egg_install
self.run_command('bdist_egg')
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\cmd.py", line 339, in run_command
self.distribution.run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\dist.py", line 1002, in run_command
cmd_obj.run()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\bdist_egg.py", line 177, in run
cmd = self.call_command('install_lib', warn_dir=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\bdist_egg.py", line 163, in call_command
self.run_command(cmdname)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\cmd.py", line 339, in run_command
self.distribution.run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\dist.py", line 1002, in run_command
cmd_obj.run()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\install_lib.py", line 19, in run
self.build()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\command\install_lib.py", line 110, in build
self.run_command('build_ext')
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\cmd.py", line 339, in run_command
self.distribution.run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\dist.py", line 1002, in run_command
cmd_obj.run()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\build_ext.py", line 99, in run
_build_ext.run(self)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\command\build_ext.py", line 365, in run
self.build_extensions()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py", line 908, in build_extensions
build_ext.build_extensions(self)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\command\build_ext.py", line 481, in build_extensions
self._build_extensions_serial()
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\command\build_ext.py", line 507, in _build_extensions_serial
self.build_extension(ext)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\command\build_ext.py", line 264, in build_extension
_build_ext.build_extension(self, ext)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\command\build_ext.py", line 562, in build_extension
objects = self.compiler.compile(
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py", line 803, in win_wrap_single_compile
return original_compile(sources, output_dir, macros,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\setuptools\_distutils\_msvccompiler.py", line 455, in compile
self.spawn(args)
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py", line 784, in spawn
cflags = win_cuda_flags(cflags) + ['-std=c++17', '--use-local-env']
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py", line 738, in win_cuda_flags
cflags + _get_cuda_arch_flags(cflags))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\selur\AppData\Roaming\Python\Python312\site-packages\torch\utils\cpp_extension.py", line 2089, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
~~~~~~~~~^^^^
IndexError: list index out of range
Any idea what a sensibe value to set TORCH_CUDA_ARCH_LIST to would be? (Pascal?)
(Attached the created whl file)
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
)
