[HELP] Vapoursynth: core.imwri.Read on Windows 10 is extremely slow

[The_Tinkerer]

Edit to clarify - core.imwri.Read is extremely slow to start processing files... once it starts processing files, then it runs normally, at expected speed.

I don't think this is a problem with Hybrid specifically, but I'm not sure where else to start asking about this. I suspect it might not be solvable if it's only happening to me, since there's nothing crashing or any error messages.

The problem described below does not happen for me on Windows 7. It also does not happen for ImageReader on AviSynth on Windows 10 (in the same version of Hybrid).

In Vapoursynth on Windows 10, core.imwri.Read is extremely slow to start processing files. It appears to somehow be trying to "pre-process" in some way the entire source directory from which images are being loaded. It does not matter if the start frame is 000000.png and the end frame is 000001.png. vsViewer.exe and VSPipe.exe will not start processing files until this "preprocessing" is done for the ENTIRE source directory.

In Task Manager (I use the classic Task Manager on W10), I can see that "I/O Other" and "I/O Other Bytes" increase at about 1 KB/s for either vsViewer.exe or VSPipe.exe. If the source directory has 8,000 images, it takes about 3 minutes for this to get to 517 KB and start processing files. For a source directory that has ~150,000 png images for a full length movie, it takes around 20 minutes for vsViewer.exe or VSPipe.exe to begin processing files. I guess that could be tolerable for just encoding, but it's a problem when using the viewer. Waiting ~20 minutes for the viewer to appear doesn't make for a usable workflow! (I guess a workaround would be to copy a smaller number of files to a temporary directory and load them from there, but what a pain!)

On Windows 7, this "preprocessing" does not appear to happen. Processing / viewer loading happens immediately. ("I/O Other" and "I/O Other Bytes" in Task Manager do not increase before it starts processing files for Windows 7.)

I tried on older versions of Hybrid going back about a year, and this problem is still present. It will take me much more time keep testing older and older versions.

Have you ever heard of anything like this? If you don't think this is Hybrid specific, do you have an idea on where I can ask about this?

If you want to try testing this on a source movie that is over 1 hour long, you can run the following command on any source file to extract all frames (be sure it is in its own folder first):

ffmpeg -r 1 -i "source.mp4" -r 1 %06d.png

[Selur]

Have you ever heard of anything like this? If you don't think this is Hybrid specific, do you have an idea on where I can ask about this?

Nope.
What does the Vapoursynth script used look like? (does chaning the thread count change anything?)
What resolution do the images have?

General side notes:
- I hope your images ly on a local hdd/ssd and not on a network drive.
- When handling tons of images make sure that your anti virus tools are not slowing down processing.


-> I will try to reproduce this and report back.
To be sure I do the same as you I'll converted a clip with 150+k frames and simple load it in Hybrid through the Image Sequence dialog and then open the Vapoursynth preview. (will place the images on a normal hdd and not on an ssd since you didn't say what you did).

Since you mentioned that vsViewer is started, this is probably not an issue with Hybrid itself, but rather with Vapoursynth or your system setup.

----
Looking over at the documentation https://amusementclub.github.io/doc/plugins/imwri.html the image gets the first file number and then loads everything behind that number.
(Hybrid will then trim off the unneeded images.) So depending on your hdd speed the filter will probably at least check the file names of all files.



Cu Selur

[Selur]

What I did:

• create image sequence:
  
  ffmpeg.exe -r 1 -i "path to test file"  -r 1 e:\testing\%06d.png
  that created 159491 images.
  
• start Hybrid
  
• enable Base->Image sequence
  
• open the file open dialog
  
• press 1st frame and select the first frame
  
• press last frame and select the first frame
  
• Close 'Image sequence'-dialog by pressing 'Accept'
  
• checking the Vapoursynth Script View:
  
  # Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# source: 'E:/testing/000001.png'
# current color space: RGB24, bit depth: 8, resolution: 720x302, fps: 25, color matrix: 709, yuv luminance scale: full, scanorder: progressive
# Loading E:\testing\%06d.png using vsImageReader
clip = core.imwri.Read("E:/testing/%06d.png", firstnum=1)
clip = core.std.Trim(clip=clip, length=159491)
# Input color space is assumed to be RGB24
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_s="full")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
• Start Vapoursynth Preview
  After ~6 seconds (counted in my head), the dialog opened. (closing and opening the Vapoursynth Preview again took the same time; may be it was even a bit faster)
  

-> Can't reproduce the issue you describe here. Harddrive the images lay on, was a WD40EZRX (https://products.wdc.com/library/SpecShe...771438.pdf), which I normally use for storage.
I tested this on a Windows 11 machine, neither RAM or CPU usage were high by any means.

So my guess is that this is an issue with your system.

- core.imwri.Read is extremely slow to start processing files... once it starts processing files, then it runs normally, at expected speed.

My guess is a virus scanner or something similar is slowing down the processing.


Cu Selur

[The_Tinkerer]

Thanks very much for testing. So it's the W10 system I'm using. Awesome!

Images are being loaded from local HD, not a network drive.

Don't have any AntiVirus installed, just using Windows Defender, although I can try disabling it.

It could be some dependency but that's always a pain to troubleshoot.

Thanks again for taking the time to test it, it rules out VS and Hybrid itself as the cause... I don't currently have a second W10 PC to test with.

[The_Tinkerer]

I turned off Windows Defender and now the VS previewer works without delay.

Kind of surprised it's due to Windows Defender as that usually doesn't cause these kinds of issues. For example there's no problem with ImageReader in AviSynth with Windows Defender turned on.

Weird!

---

Adding the folder that contains the source images to Windows Defender's exclusion list fixes it.

[Selur]

I would also add Hybrids folder to it's exclusion list.