[BUG] Avisynth TemporalDergrain 2 FFT3DFilter&FFT3DGPU Crash

[Selur]

Problem seems to only occur if PostFFT is set to ff3dfilter or ff3dgpu.
I checked, Hyrid does use v2.6.6 from https://avisynth.nl/index.php/TemporalDegrain2 so it does not seem to be a bug in Hybrid itself.

with fft3dfilter, the error is:

neo_fft3d: bt must be -1 (Sharpen), 0 (Kalman), 1..5 (Wiener)

with FFT3DGPU, the error is:

Valid modes for bt are -1,0,1,2,3,4

Looking at the script, I see:

out    = (postFFT == 0) ? (postDither > 0 ? in.ConvertBits(16) : in) : \
             (postFFT == 1) ? neo_fft3d ( postDither > 0 ? in.ConvertBits(16) : in, y=LumaP?3:2, u=ChromaP?3:2, v=ChromaP?3:2, sigma=postSigma, bt=postTD, ncpu=fftThreads, bw=postBlkSz, bh=postBlkSz ) : \
             (postFFT == 11)? FFT3DFilter( postDither > 0 ? in.ConvertBits(16) : in, plane=postPlane, sigma=postSigma, bt=postTD, ncpu=fftThreads, bw=postBlkSz, bh=postBlkSz ) : \
             (postFFT == 2) ? FFT3DGPU( postDither > 0 ? in.ConvertBits(16) : in, plane=postPlane, sigma=postSigma*2/3, bt=postTD, precision=2, mode=1, bw=postBlkSz, bh=postBlkSz ) : \
             (postFFT == 3) ? neo_dfttest( postDither > 0 ? in.ConvertBits(16) : in, y=LumaP?3:2, u=ChromaP?3:2, v=ChromaP?3:2, sigma=postSigma*4, tbsize=postTD, dither=dftDither, threads=fftThreads, sbsize=postBlkSz, sosize=postBlkSz*9/12, slocation=dftsfile) : \
             (postFFT == 13)? dfttest( postDither > 0 ? in.ConvertBits(16) : in, Y=LumaP, U=ChromaP, V=ChromaP, sigma=postSigma*4, tbsize=postTD, threads=fftThreads, dither=dftDither, sbsize=postBlkSz, sosize=postBlkSz*9/12, sfile=dftsfile ) : \
             (postFFT == 4) ? DT_KNLMeansCL( postDither > 0 ? in.ConvertBits(16) : in, a=2, d=postTR, h=postSigma, Luma = LumaP, Chroma = ChromaP, device_type="GPU", device_id=devId) : \
             (postFFT == 5) ? DT_BM3D( postDither > 0 ? in.ConvertBits(16) : in, radius=postTR, sigma=postSigma, chroma=ChromaP, CUDA=cuda, device_id=devId ) : \
             (postFFT == -1)? HQDn3D( postDither > 0 ? in.ConvertBits(16) : in, 0,0,4,1, u=ChromaP?3:2, v=ChromaP?3:2) : NOP()

the problem does happen when neo_fft3d (postFFT ==1), fft3dgpu (postFFT == 2), or fft3dfilter (postFFT == 11) is used.
In all these cases, 'bt' is set to postTD which is set in line 236&267 of the script:

postTR  = (postFFT > 0) ? postTR : 0
  postTD  = postTR * 2 + 1

so for

• neo_fft3d (postFFT ==1), postTD would be 3
  
• fft3dgpu (postFFT == 2), postTD would be 5
  
• fft3dfilter (postFFT == 11), postTD would be 23
  

Lookint at what the filters support:

• fft3dfilter (postFFT == 11) supports
  
  bt - block temporal size, number of frames  (-1, 0, 1, 2 or 3, default = 3):
    0 - all previous frames (switch Kalman filter mode);
    1 - only current frame (spatial 2D Wiener filter);
    2 - previous and current frame (3D Wiener filter);
    3 - previous, current and next frame (3D Wiener filter)
    4 - two previous, current and next frame (3D Wiener filter)
    5 - two previous, current and two next frames (3D Wiener filter)
    -1 - sharpen only (2D);
  source: https://avisynth.org.ru/fft3dfilter/fft3dfilter.html
  
• fft3dgpu (postFFT == 2), supports:
  
  bt = 3
    Block temporal size, number of frames.

        -1 : sharpen only (2D)
        0 : all previous frames (switch Kalman filter mode)
        1 : only current frame (spatial 2D Wiener filter)
        2 : previous and current frame (3D Wiener filter)
        3 : previous, current and next frame (3D Wiener filter)
        4 : two previous, current and next frame (3D Wiener filter)
  source: http://avisynth.nl/index.php/FFT3DGPU
  
• neo_fft3d (postFFT ==1) supports:
  
  int  bt = 3

            Block temporal size, number of frames.

                0 : all previous frames (switch Kalman filter mode)
                1 : only current frame (spatial 2D Wiener filter)
                2 : previous and current frame (3D Wiener filter)
                3 : previous, current and next frame (3D Wiener filter)
                4 : two previous, current and next frame (3D Wiener filter)
                5 : two previous, current and two next frames (3D Wiener filter)
                -1 : sharpen only (2D)
  source: http://avisynth.nl/index.php/Neo_FFT3D
  

So 'postTD = postTR * 2 + 1' makes no sense. for these.
Replacing it with:

postTD  = (postFFT == 11 || postFFT == 1 || postFFT == 2) ? postTR : postTR * 2 + 1

and adjusting the mapping for postFFT in Hybrid might work.

=> looking into it.

Cu Selur