Vapoursynth audio handling capabilities

[Selur]

Since Vapoursynth supports audio for quite some time so I collected what filters&o are out there and I though others might be interested in it too

External plugins

Plugin	Namespace	Functionality	Notes
VS-AudioTools	atools	Convert, Crossfade, Delay, FadeIn, FadeOut, FindPeak, Mix, Normalize, SineTone	Most comprehensive dedicated audio utility plugin found
AudioBoost	ab	Soft clipping, saturation, nonlinear amplification	WIP, VapourSynth compatibility need to be tested
AudioResample	Historical	Sample-rate and sample-format conversion	Functionality now available natively through Vapoursynth
Faveworm	fw	Visualization	Analog oscilloscope emulation, waveform rendering, trigger mode, scrolling mode, X-Y mode
BestSource	bs	Audio/video source using FFmpeg	Source plugin, sample-accurate audio access

Native VapourSynth audio functions

Function	Category	Functionality
AssumeSampleRate	Format	Assume a different sample rate
AudioGain	Gain	Linear amplification, per-channel gain
AudioLoop	Editing	Repeat audio samples
AudioMix	Mixing	Arbitrary channel mixing matrix, multiple input clips
AudioResample	Conversion	Sample-rate, sample-type, bit-depth and channel-layout conversion
AudioReverse	Editing	Reverse audio samples
AudioSplice	Editing	Concatenate audio clips
AudioTrim	Editing	Sample-accurate trimming
BlankAudio	Generation	Generate silence, sine, square, triangle or sawtooth signals
SetAudioCache	Infrastructure	Configure audio-node caching
ShuffleChannels	Channels	Extract, reorder and combine channels
SplitChannels	Channels	Split multichannel audio into mono clips

Audio-processing capabilities

Feature	Native	External
Gain adjustment	Yes	atools
Peak normalization	No	atools
Loudness normalization	No	None identified
Soft clipping	No	AudioBoost
Dynamic compression	No	No conventional compressor identified
Equalization	No	None identified
Noise reduction	No	None identified
Time stretching	No	None identified
Pitch shifting	No	None identified
Reverb / convolution	No	None identified
Fades / crossfades	No dedicated filter	atools
Delay	No dedicated DSP filter	atools
Mixing / downmixing	Yes	atools
Resampling	Yes	Native
Dithering	Yes	Native

If someone knows of a filter or feature I haven't listed, please let me know, so I can update this overview.
Thanks!

Cu Selur

[videoh]

Thank you for the highly useful post!

Maybe you could post a few sample scripts combining audio and video with audio filtering for those of us who are clueless about it?

[Selur]

I wrote an example of audio and video cutting at the same time a while ago:

# Imports import math import sys import os import vapoursynth as vs # using Vapoursynth R78 # getting Vapoursynth core core = vs.core # Limit frame cache to 48449MB core.max_cache_size = 48449 # Import scripts folder scriptPath = 'F:/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # loading plugins core.std.LoadPlugin(path="F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/vapoursynth/plugins2/libbestsource.dll") # Import scripts import validate # Source: 'g:\Test.mkv' # clip current meta; color space: YUV420P8, bit depth: 8, resolution: 1920x1080, fps: 23.976, color matrix: 709, color primaries: Unspecific, color transfer: BT.709, yuv luminance scale: limited, scanorder: progressive, full height: true (Source) # Loading 'g:\Test.mkv' using BestSource clip = core.bs.VideoSource(source="G:/Test.mkv", cachepath="J:/tmp/Test_bestSource", track=0) frame = clip.get_frame(0) # setting color matrix to 709. clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709) # setting color transfer (vs.TRANSFER_BT709), if it is not set. if validate.transferIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709) # setting color primaries info (to vs.PRIMARIES_BT709), if it is not set. if validate.primariesIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709) # setting color range to TV (limited) range. prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange' clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED}) # making sure frame rate is set to 23.976fps clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001) # making sure the detected scan type is set (detected: progressive) video = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive # Format: ac-3 # Length: 2567.840 # Bit rate: 384 # Channels: 6 # Sample rate: 48000 # Raw Audio: true # Codec: pcm # Channel Mapping: 3/2/0.1 # Dialog Normalization: -27 # Video delay: 0 # New Video delay: 0 # Audio only delay: 0 audio = core.bs.AudioSource("G:/Test.mkv") def trim_av(video: vs.VideoNode, audio: vs.AudioNode, first: int = 0, last: int = None) -> tuple: """ Trims both video and audio simultaneously based on video frame numbers. Follows VapourSynth's core.std.Trim behavior (inclusive of 'last'). """ # 1. Trim the video node v_trimmed = core.std.Trim(video, first=first, last=last) # 2. Calculate frame rate and sample rate fps = video.fps.numerator / video.fps.denominator sample_rate = audio.sample_rate # 3. Calculate corresponding audio samples # First sample starts at the beginning of the 'first' frame audio_first = math.floor(first * sample_rate / fps) if last is not None: # Last sample ends at the end of the 'last' frame audio_last = math.floor((last + 1) * sample_rate / fps) - 1 else: audio_last = None # 4. Trim the audio node a_trimmed = core.std.AudioTrim(audio, first=audio_first, last=audio_last) return v_trimmed, a_trimmed # Cut List frame numbers based on progressive input cut_list = [ (12995, 13035), (13071, 13123), (13180, 13270), (13369, 13413), (13541, 14944), (17796, 18033), (19986, 20060), (31344, 31553), (60053, 60764) ] # Do the cutting video_cuts = [] audio_cuts = [] for start, end in cut_list: v_cut, a_cut = trim_av(video, audio, first=start, last=end) video_cuts.append(v_cut) audio_cuts.append(a_cut) # combine cuts video = core.std.Splice(video_cuts) audio = core.std.AudioSplice(audio_cuts) # apply additional audio filtering # adjusting output color from YUV420P8 to YUV420P10 for NVEncModel video = core.resize.Bicubic(clip=video, format=vs.YUV420P10) # set output frame rate to 23.976fps (progressive) video = core.std.AssumeFPS(clip=video, fpsnum=24000, fpsden=1001) # output video.set_output(0) audio.set_output(1)

Cu Selur

Ps.: I'm also uploading a new dev version (will be up in ~ 1hr) which can play audio in the Vapoursynth Preview and add some visualization using Faveworm (this still needs some tuning).
(control this under Filtering->Vapoursynth->Misc->Preview)
It also has support to decode and cut audio using Vapoursynth, but those scripts are usually not inside the same script as the video.
Pss.: once can use mkv output with vspipe and feed the output to i.e. ffmpeg or play it with ffplay.