Possible 2 bugs in Linux & vapoursynth

[ALM]

Hi,

I use Hybrid and Vapoursynth, last versions, with Ubuntu (20.04 LTS), uptodate.
No real problem to compile Vapoursynth.
I'm trying to load a mkv with 3 video tracks and use MCTemporalDenoise on the first track, and
reencode it; others tracks are not interesting.

2 dual problems to load the file with Vapoursynth.

1. When I use "Prefer LWLibAVSource over FFMpegSource2", the generated .vpy load part is:

# Loading /global/leopard/cooking/back_Participant_10_Act_1_Trial_0.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="/global/leopard/cooking/back_Participant_10_Act_1_Trial_0.mkv", format="YUV422P8", stream_index=0, cache=0, prefer_hw=0)

prefer_hw seems coherent with Hybrid, it's the next graphic option : "Libab hardware decoding mode"
with default = 0

But IMHO, prefer_hw is an argument for LSMASHSource, not LWLibAVSource ?

The error is :
**************************************************************************************************
Python exception: LWLibavSource: Function does not take argument(s) named prefer_hw

Traceback (most recent call last):
  File "src/cython/vapoursynth.pyx", line 2231, in vapoursynth.vpy_evaluateScript
  File "src/cython/vapoursynth.pyx", line 2232, in vapoursynth.vpy_evaluateScript
  File "/global/leopard_nvme/encodingTempSynthSkript_23_58_00_8410.vpy", line 12, in <module>
    clip = core.lsmas.LWLibavSource(source="/global/leopard/cooking/back_Participant_10_Act_1_Trial_0.mkv", format="YUV422P8", stream_index=0, cache=0, prefer_hw=0)
  File "src/cython/vapoursynth.pyx", line 2042, in vapoursynth.Function.__call__
vapoursynth.Error: LWLibavSource: Function does not take argument(s) named prefer_hw
***************************************************************************************************

2. So, I'm trying to use FFMpegSource2 in place of LWLibAVSource. But my mkv has
3 video tracks. So the load part of the .vpy becomes :

# Loading source using FFMS2
clip = core.ffms2.Source(source="/global/leopard/cooking/back_Participant_10_Act_1_Trial_0.mkv",cachefile="/global/leopard_nvme/00_15_33_7110.ffindex",vtrack=0,format=vs.YUV422P8,alpha=False)

But IMHO, in core.ffms2.Source, the right name of the track is not "vtrack", but "track" ?

The error is :
**************************************************************************************************
Python exception: Source: Function does not take argument(s) named vtrack

Traceback (most recent call last):
  File "src/cython/vapoursynth.pyx", line 2231, in vapoursynth.vpy_evaluateScript
  File "src/cython/vapoursynth.pyx", line 2232, in vapoursynth.vpy_evaluateScript
  File "/global/leopard_nvme/encodingTempSynthSkript_00_15_33_7110.vpy", line 12, in <module>
    clip = core.ffms2.Source(source="/global/leopard/cooking/back_Participant_10_Act_1_Trial_0.mkv",cachefile="/global/leopard_nvme/00_15_33_7110.ffindex",vtrack=0,format=vs.YUV422P8,alpha=False)
  File "src/cython/vapoursynth.pyx", line 2042, in vapoursynth.Function.__call__
vapoursynth.Error: Source: Function does not take argument(s) named vtrack
**************************************************************************************************

WorkAround: do the job in 2 times.
First, extract the first video track in an tempo mkv without vapoursynth;
Second, use FFMpegSource2 with this new mkv with only one video track;
the argument vtrack is not used. Result is OK.

Thanks for your work & all the best,

Alain
--

[Selur]

1st please use the 'code'-tags to make your post better readable. It's a pain.
About the issues:

But IMHO, prefer_hw is an argument for LSMASHSource, not LWLibAVSource ?

I have no problem using:

# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# input color space: YUV420P8, bit depth: 8
# Loading F:\TestClips&Co\files\MPEG-4 H.264\TEST_delay-916ms.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="F:/TestClips&Co/files/MPEG-4 H.264/TEST_delay-916ms.mkv", format="YUV420P8", cache=0, fpsnum=24000, fpsden=1001, prefer_hw=1)
# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip, fpsnum=24000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting output color from: YUV420P8 to YUV420P10 for x265Model (i420)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# Output
clip.set_output()

here,... iirc prefer_hw support was added back in November 2019 by HoyWu, his github is over at: https://github.com/HolyWu/L-SMASH-Works
(was added in: https://github.com/HolyWu/L-SMASH-Works/...13931b3f29)
damn, just noticed I used the VFR-mania sources to build L-SMASH in the Linux package. :/

But IMHO, in core.ffms2.Source, the right name of the track is not "vtrack", but "track" ?

Yes, that is a bug. -> will fix

Cu Selur

[Selur]

Argh,.. how I hate all those different build systems cmake/meson/...
No clue why but I can't get mesion to work properly.

ghdl VFR-maniac/L-SMASH-Works
ghdl l-smash/l-smash

./configure --prefix="$vsprefix" --extra-cflags="$CFLAGS" || cat config.log
make -j$JOBS lib
cp liblsmash.a ..

cd ../VapourSynth
./configure --prefix="$vsprefix" \
  --extra-cflags="-I../build $CFLAGS -Wno-deprecated-declarations" \
  --extra-ldflags="-L.. $LDFLAGS" \
|| cat config.log

make -j$JOBS
cp libvslsmashsource.so.1 ../libvslsmashsource.so

cd ..
finish libvslsmashsource.so

source: https://github.com/Selur/hybrid-vapoursy...hsource.sh
works fine and I thought changing it to:

ghdl HolyWu/L-SMASH-Works
ghdl l-smash/l-smash

./configure --prefix="$vsprefix" --extra-cflags="$CFLAGS" || cat config.log
make -j$JOBS lib
cp liblsmash.a ..

cd ../VapourSynth
CFLAGS="-I../build $CFLAGS -Wno-deprecated-declarations" LDFLAGS="-L.. $LDFLAGS" meson build --prefix="$vsprefix"

make -j$JOBS
cp libvslsmashsource.so.1 ../libvslsmashsource.so

cd ..
finish libvslsmashsource.so

would make it work with HolyWus build, but it aborts with:

Run-time dependency liblsmash found: NO (tried pkgconfig and cmake)

meson.build:41:0: ERROR: Dependency "liblsmash" not found, tried pkgconfig and cmake

and I can't get my head around why. :/ (my guess it I simply don't get how to set the linker flags properly for meson,..)

-> seeing that it's not the only plugins that fails (there's also eedi3m, mvtoolssf, nnedi3cl, tcanny), I'm more and more thinking about dropping Vapoursynth support completely on Linux, since I lack the time and motivation to maintain the Vapoursynth stuff,...

Cu Selur

Ps.: sorry, for the rant, just frustrated,.. (got tons of bugs to fix and not time)

[ALM]

Hi,

A solution may be :

ghdl HolyWu/L-SMASH-Works
ghdl l-smash/l-smash

./configure --prefix="$vsprefix" --extra-cflags="$CFLAGS" || cat config.log
make -j$JOBS lib
cp liblsmash.a ..

cd ../VapourSynth

mv meson.build meson.build.ORIGINAL
sed < meson.build.ORIGINAL > meson.build \
-e "/vapoursynth_dep *=/i\
liblsmash_dep = declare_dependency(link_args : ['-L../../build', '-llsmash'],\\
                                   include_directories : ['../../build'])\n" \
-e "s/dependency('liblsmash')/liblsmash_dep/g"

if [ -z "$vsprefix" ]; then
    vsprefix="/usr/local"
fi

CFLAGS="$CFLAGS -Wno-deprecated-declarations" meson build --prefix="$vsprefix"
ninja -C build -j $JOBS

cp build/libvslsmashsource.so ../libvslsmashsource.so
cd ..
finish libvslsmashsource.so

The file meson.build uses the very strict keyword "dependency" for liblsmash.
It's useful only for official library, dpkg declared, usually dynamic.

My solution is to hack this file and replaces dependency by declare_dependency, which accepts
link and include directories. This line is included by sed before the declaration of
the vapoursynth dependencies.

I have to include and link two levels up, because ninja creates another build subdirectory.

vsprefix is empty in the shell (why ? not exported ? only for tests ?). Meson doesn't accept empty
prefix, so in this case I put it to the default /usr/local.

Vapoursynth is a fantastic toolbox; it would be a pity to lose it to Linux.

All the best,

Alain
--

[Selur]

I'll try your suggestion tomorrow.
$vsprefix isn't empty since that script isnormally called from https://github.com/Selur/hybrid-vapoursy...plugins.sh and not by itself where vsprefix is set to 'vsprefix="$HOME/opt/vapoursynth"'


Cu Selur

[ALM]

I'll try your suggestion tomorrow.
$vsprefix isn't empty since that script isnormally called from https://github.com/Selur/hybrid-vapoursy...plugins.sh and not by itself where vsprefix is set to 'vsprefix="$HOME/opt/vapoursynth"'


Cu Selur

But $vsprefix is not exported from build-plugins.sh ?
Same problem for $JOBS, defined in build-plugins.sh and not exported, and which need to be define again in build-plugins/header.sh ?

Alain
--

[Selur]

Hmm,.. it works so seems like the variable is inherited by the child script. (parent: build-plugins.sh, child: plugin-vslsmashsource.sh)
the log files show the right paths:

+ vsprefix=/home/selur/opt/vapoursynth
+ export PATH=/home/selur/opt/vapoursynth/bin:/home/selur/opt/vapoursynth/bin:/home/selur/.cargo/bin:/home/selur/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ export LD_LIBRARY_PATH=/home/selur/opt/vapoursynth/lib
+ export PYTHONUSERBASE=/home/selur/opt/vapoursynth
+ export PKG_CONFIG_PATH=/home/selur/opt/vapoursynth/lib/pkgconfig
+ pkg-config --cflags vapoursynth
+ export CFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute
+ export CXXFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute -Wno-reorder
+ export LDFLAGS=-L/home/selur/opt/vapoursynth/lib
+ ghdl HolyWu/L-SMASH-Works
+ git clone --depth 1 --recursive https://github.com/HolyWu/L-SMASH-Works build
Klone nach 'build' ...
+ cd build
+ ghdl l-smash/l-smash
+ git clone --depth 1 --recursive https://github.com/l-smash/l-smash build
Klone nach 'build' ...
+ cd build
+ ./configure --prefix=/home/selur/opt/vapoursynth --extra-cflags=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute

generating config.mak ...
...

Cu Selur

[ALM]

Hmm,.. it works so seems like the variable is inherited by the child script. (parent: build-plugins.sh, child: plugin-vslsmashsource.sh)
the log files show the right paths:

+ vsprefix=/home/selur/opt/vapoursynth
+ export PATH=/home/selur/opt/vapoursynth/bin:/home/selur/opt/vapoursynth/bin:/home/selur/.cargo/bin:/home/selur/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ export LD_LIBRARY_PATH=/home/selur/opt/vapoursynth/lib
+ export PYTHONUSERBASE=/home/selur/opt/vapoursynth
+ export PKG_CONFIG_PATH=/home/selur/opt/vapoursynth/lib/pkgconfig
+ pkg-config --cflags vapoursynth
+ export CFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute
+ export CXXFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute -Wno-reorder
+ export LDFLAGS=-L/home/selur/opt/vapoursynth/lib
+ ghdl HolyWu/L-SMASH-Works
+ git clone --depth 1 --recursive https://github.com/HolyWu/L-SMASH-Works build
Klone nach 'build' ...
+ cd build
+ ghdl l-smash/l-smash
+ git clone --depth 1 --recursive https://github.com/l-smash/l-smash build
Klone nach 'build' ...
+ cd build
+ ./configure --prefix=/home/selur/opt/vapoursynth --extra-cflags=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute

generating config.mak ...
...

Cu Selur



Well, are you sure we have the same version of build-plugins/header.sh ? My vslsmashsource.log from your original source is :

+ export LD_LIBRARY_PATH=/usr/local/lib;/usr/local/lib/vapoursynth
+ pkg-config --cflags vapoursynth
+ export CFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/usr/local/include/vapoursynth -I/usr/local/include -I/usr/include/compute
+ export CXXFLAGS=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/usr/local/include/vapoursynth -I/usr/local/include -I/usr/include/compute -Wno-reorder
+ ghdl VFR-maniac/L-SMASH-Works
+ git clone --depth 1 --recursive https://github.com/VFR-maniac/L-SMASH-Works build
Cloning into 'build'...
+ cd build
+ ghdl l-smash/l-smash
+ git clone --depth 1 --recursive https://github.com/l-smash/l-smash build
Cloning into 'build'...
+ cd build
+ ./configure --prefix= --extra-cflags=-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/usr/local/include/vapoursynth -I/usr/local/include -I/usr/include/compute

generating config.mak ...

SRCDIR = .
DESTDIR =
prefix = /usr/local

Note the prefix is empty in configure; configure takes the default /usr/local.
build-plugins/header.sh, first part of all plugin-build, is :

#!/bin/sh
set -e

JOBS=4

[... functions...]

set -x

export LD_LIBRARY_PATH="/usr/local/lib;/usr/local/lib/vapoursynth"
export CFLAGS="-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing $(pkg-config --cflags vapoursynth) -I/usr/include/compute"
export CXXFLAGS="$CFLAGS -Wno-reorder"

These 3 exports, and their order, are coherent with my log, not your ?

[Selur]

hmm,.. might be because I first build vapoursynth and installed that.
I initially had planned to look at it today some more and try your suggestion, but real live took over and I now hope to find some time tomorrow.
I attached the stuff I currently have in my folder, not totally sure how much these differ from the initial git

Cu Selur

Ps.: main problem like always is time I simply lack enough time to a. maintain Hybrid itself b. look at all the build stuff for the dependencies. :/

[ALM]

Hi,

The last lines of your version of build-plugins/header.sh file are :

set -x

vsprefix="$HOME/opt/vapoursynth"

export PATH="$vsprefix/bin:$PATH"
export LD_LIBRARY_PATH="$vsprefix/lib"
export PYTHONUSERBASE="$vsprefix"
export PKG_CONFIG_PATH="$vsprefix/lib/pkgconfig"
export CFLAGS="-pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing $(pkg-config --c
flags vapoursynth) -I/usr/include/compute"
export CXXFLAGS="$CFLAGS -Wno-reorder"
export LDFLAGS="-L$vsprefix/lib"

OK! Let the Good Times Roll!

Alain
--