Hybrid job stalls at endcoding with Deoldify

[wpsandy]

I let it run for 1hr 20 mins until I killed it off.   Screen never came up, process didn't finish on its own, VRAM never moved up from the 1GB VRAM allocated.  Attached debug log.

[Selur]

Debug shows

2024.09.02 - 10:41:08_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9: Got unhandled IPC message:   C : \ P r o g r a m   F i l e s \ H y b r i d \ 6 4 b i t \ V a p o u r s y n t h \ L i b \ s i t e - p a c k a g e s \ t o r c h \ n n \ u t i l s \ w e i g h t _ n o r m . p y : 1 4 3 :   F u t u r e W a r n i n g :   ` t o r c h . n n . u t i l s . w e i g h t _ n o r m `   i s   d e p r e c a t e d   i n   f a v o r   o f   ` t o r c h . n n . u t i l s . p a r a m e t r i z a t i o n s . w e i g h t _ n o r m ` .
     W e i g h t N o r m . a p p l y ( m o d u l e ,   n a m e ,   d i m )

2024.09.02 - 12:09:38_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9: player finished,...
2024.09.02 - 12:09:38_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9: setting updatingScript to: false
2024.09.02 - 12:09:38_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9: kill process,...
2024.09.02 - 12:09:38_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9:  VapoursynthPreview disable show(3),..
2024.09.02 - 12:09:38_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9:  VapoursynthPreview disable updating(3),..

Sadly nothing informative, hopefully Dan64 has some time and can look into / help find the problem.

Just to make sure:
a. neither your gpu, cpu or ram are overclocked, right?
b. it doesn't make a difference if you disable whatever antivirus software you have running on your system, right?

Cu Selur

[wpsandy]

That's correct.   Candidly, I prefer stability these days on my computers.   Many years ago I would play with overclocking (think Celeron 300A era) but that's far in the past for me.

I'm only running the vanilla MS Virus and threat protection and have made runs with both the real time protection active and with it disabled and did not seem to make a difference.  The renderfactor=10 run was made with it deactivated.

---

Just to provide details... No overclock or undervolting/overvolting on anything.   

Motherboard: MSI B550 Tomahawk
CPU: AMD 3700X.   No overclock or undervolting.  
Memory:  2x16GB G.SKILL DDR4 3200
GPU: Nvidia 3070 FE
PSU= Corsair 650W
HD:  Crucial 1TB NVME

Currently Win 11 23H2

[wpsandy]

Ok.  I'm back since I went and acquired a 3090 FE that has 24GB of VRAM and I'm still seeing the same behaviors.   I haven't had a preview successfully come up with deoldify set as active.

What are some things I should check?

1. Should i go back to the release/stable version?   I'm currently working with the dev release.
2. Should I look at moving back to a prior NVidia driver release?  I am currently on 560.94
3. Could I have some permission issues?   What files/directories should I check? 
4. When placing the vsgan-models, vs-mirt and onnx_models, is something awry there?
5. When running deoldify, should I always be activating DeepEx?   I have not always with the intent of simplifying.

Some additionals things I have tried:
1.  Different source clip.  I pulled down a progressive mp4 black and white clip.   This removed de-interlacing from the equation
2. Set the configuration of the temp space to a specific folder and added that exclusion to the MS Anti-virus.   I haven't turn anti-virus on yet, so I do not know if this has had any impact.
3. Validated that the memory in the PC is actually running at 3200Mhz
4. Reset Hybrid back to its defaults, and tested with the B&W sample
5. Shutdown most of the things in the tray that aren't totally needed.
6. Tried running Hybrid with admin rights...it wouldn't let me do that 

Thoughts?

[Selur]

I'm using NVIDIA Studio drivers 560.83.

Try whether it changes anything if you:

• deinstall Hybrid with all its settings.
  
• install Hybrid dev version
  
• copy the content of the Hybrid folder somewhere out of Windows normal progam folders, i.e. d:\Hybrid
  
• deinstall Hybrid with all its settings.
  
• turn Hybrid into a portable version, see: https://forum.selur.net/thread-10.html
  
• install the torch-addon from the experimental folder
  
• Try that version.
  

Cu Selur

[wpsandy]

Preview worked with the steps above.   My D: is a SATA spinning hard drive so it takes a bit of churning for the preview to come up, but it did.   May have to source a SSD to put Hybrid on to get loads snappier.

I couldn't find 560.83 but I could download 560.81 and went with that.   One other thing was that I was using the game ready driver before and not the studio one.  

I'll take my test B&W clip and see if the full job will render now.   

Thx for all the help and advice.

[Selur]

Sorry, that was a typo meant 560.81
If the portable works but the normal not, then it's probably an issue with the engine location.
=> I will look into that.

[wpsandy]

To add on to above, I have been able to succesfully process a couple of jobs with just deoldify and one with deoldify/deepex.

[Selur]

To test whether this is related to the portable version, I installed the latest dev in normal location and added the latest (VapoursynthR68_torch_2024.08.27) torch-addon and then tried deoldify.
=> worked fine here, so it must be something with your setup, or I overlook something obvious.
No clue.
Sadly, no reply from Dan64, so far.

Cu Selur

[Dan64]

I'm using a RTX 3060 with 12GB and I never experienced such odd behaviour  
The only way to reduce the RAM usage is to lower the render factor, a value of 10 is very low and I will not use it for serious encoding.
Given that it seems that the problem is still present even with a RTX 3090 with 24GB I will exclude memory problems.

Unfortunately there is not a way to get a better understanding, since the logs are not reporting any useful message to debug this issue.

I'm sorry but I don't have any idea on how to fix this issue.

Dan