Use of audio visualization as a visual effect (2) - with “displace”¶
Watch on youtube.com- doc
https://ffmpeg.org/ffmpeg-filters.html#displace, https://ffmpeg.org/ffmpeg-filters.html#geq, https://ffmpeg.org/ffmpeg-filters.html#colorkey, https://ffmpeg.org/ffmpeg-filters.html#showcqt
`displace’ basics¶
The essence of this video is to use audio visualization as a visual effect with “displace”.
However, due to significant lack of explanation it is impossible to understand the behavior of “displace” based on the official document, so you “can not imagine this filter’s behavior absolutely without trial and error”.
So let me show you the basic behavior of “displace” before the main theme.
There are only two essences the official document should originally explain:
The pixel movement amount is determined by the difference from the intermediate value (128) of the color value.
The pixel movement amount is calculated per plane.
Calculation for each plane means that consistency of the X-axis stream, the Y-axis stream, and the target stream is important. This should be especially careful when the format is “yuv420p” (etc., those whose ratio is not 1:1:1). If the filter graph you write yourself does not result as what you want, it may work if you set it to “format=yuv444p” or “format=rgb24” as I do in the example later on.
Also note that all planes are used for movement calculation. In any non-artificial video, it is not unusual for the values of all planes to be the same. It is not very common to want to change the amount of pixel movement per plane. But it is exactly what “displace” normally does, Therefore, this often results in contrary to your intentions.
Now I will show you basic examples. Scripts are somewhat long, but since I chose the most obvious example of this filter behavior, you probably should be able to understand immediately.
00:01:25#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(X/96)*96)*255/W
'
[v]" -map '[v]' -an gradation1.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation1.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[xaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[yaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation1_xaxis_yaxis.mp4"
#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(X/96)*96)*255/W
'
[v]" -map '[v]' -an gradation1.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation1.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[yaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[xaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation1_yaxis_xaxis.mp4"
#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(Y/96)*96)*255/H
'
[v]" -map '[v]' -an gradation2.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation2.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[xaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[yaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation2_xaxis_yaxis.mp4"
#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(Y/96)*96)*255/H
'
[v]" -map '[v]' -an gradation2.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation2.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[yaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[xaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation2_yaxis_xaxis.mp4"
#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(X/96)*96)*255/W/2+abs(round(Y/96)*96)*255/H/2
'
[v]" -map '[v]' -an gradation3.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation3.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[xaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[yaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation3_xaxis_yaxis.mp4"
#! /bin/sh
ffmpeg="/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg"
dur=10
#
"${ffmpeg}" -y -filter_complex "
color=black:s=1920x1080:d=${dur},format=yuv444p
,geq='
abs(round(X/96)*96)*255/W/2+abs(round(Y/96)*96)*255/H/2
'
[v]" -map '[v]' -an gradation3.mp4
# -----------
"${ffmpeg}" -y -i "Pexels_flowers.mp4" -i "gradation3.mp4" \
-filter_complex "
[0:v]trim=0:${dur},setpts=PTS-STARTPTS,format=yuv444p,drawgrid=w=128:h=128:c=blue:t=3
,scale=1920:-1,setsar=1[target];
[1:v]scale=1920:-1,setsar=1[yaxis];
color=0x808080:s=1920x1080:d=${dur},format=yuv444p[xaxis];
[target][xaxis][yaxis]displace=edge=blank[v]
" -map '[v]' -an -shortest "out_gradation3_yaxis_xaxis.mp4"
Use of audio visualization as a visual effect with “displace”¶
Well, let’s see “using audio visualization as a visual effect with displace”.
Originally what we want to do is “move the picture in sync with the sound”. However, since there is no way to map raw audio data directly to pixel movement, it is necessary to go through the approach of converting video by audio visualization such as “showcqt” to pixel movement. “displace” can be used for this purpose.
It should be noted in this topic that, as we have seen, “displace” reacts in color value. If you do not conscious of this point, you will have unintended results for you. For example, probably you will be surprised if you give pure red. Since pure red is RGB (255, 0, 0), when converted to pixel movement this means “pixel shift for only one plane and the remaining two are left”. This may be an interesting visual effect, but in almost case it will not what you want. For this reason, it is usually better to choice colors close to grayscale.
Example 1¶
It is an example using “showwaves”. By controlling the “colors” option, it looks like this. However this is almost the same of overlaying “showwaves” results, it will not be fun for you.
00:03:25#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
"/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg" -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2,split[base1][base2];
[0:a]
volume=5.0,showwaves=split_channels=1:mode=line:s=1280x720:colors=0x101418|0xF8F4F0
,format=rgb24,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base1][vcqt]overlay[vcqt1];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][base2]displace=edge=blank,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 2¶
It is an example using “showcqt”. In ‘showcqt’ (except for background black) many white-colored colors appear, so passing the result to ‘displace’ as it is will not be a surprising result. As in the previous example, in this way it is only an effect similar to “overlay”, so it’s not interesting.
00:03:56#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
ffmpeg -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2[base];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base][vcqt]overlay,split[vcqt1][vcqt2];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][vcqt2]displace=edge=blank,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 3¶
It is an example using only specific Y of visualization by “showcqt”.
00:04:27#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
ffmpeg -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2[base];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24
,geq='p(X,363)'
,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base][vcqt]overlay,split[vcqt1][vcqt2];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][vcqt2]displace=edge=blank,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 4¶
It is an example using only the average of the specific four points of visualization by “showcqt”. “Showcqt” is a visualization based on the scale, so it’s easy to use it for effect purposes that literally shake according to “music”.
00:04:58#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
ffmpeg -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2[base];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24
,geq='p(233,363)/4+p(333,363)/4+p(500,363)/4+p(633,363)/4'
,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base][vcqt]overlay,split[vcqt1][vcqt2];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][vcqt2]displace=edge=blank,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 5¶
It is the same as before, but the value of “edge” option is set to “mirror”.
00:05:29#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
"/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg" -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2[base];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24
,geq='p(233,363)/4+p(333,363)/4+p(500,363)/4+p(633,363)/4'
,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base][vcqt]overlay,split[vcqt1][vcqt2];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][vcqt2]displace=edge=mirror,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 6¶
If it is controlled so as to be shaken around the intermediate value (128), it is a visual effect that shakes left and right (or up and down) like this.
00:06:00#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
"/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg" -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2,split[base1][base2];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24
,geq='(128-60/2)+mod(p(233,363)+p(333,363)+p(500,363)+p(633,363),60)'
,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base1][vcqt]overlay[vcqt1];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][base2]displace=edge=mirror,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"
Example 7¶
You can understand what this is doing if you understand the previous examples. Perhaps if you develop this, you will get a fairly interesting visual effect.
00:06:31#! /bin/sh
pref="`basename $0 .sh`"
bgmov="Pexels_1358988.mp4"
sound="Piano Sonata no. 11, K. 331 - III. Alla Turca (Eduardo).mp3"
#
"/c/Program Files/ffmpeg-4.1-win64-shared/bin/ffmpeg" -y \
-i "${sound}" \
-i "${bgmov}" \
-filter_complex "
color=0x808080:s=1280x720,format=rgb24,loop=-1:size=2,split[base1][base2];
[0:a]
showcqt=s=1280x720:basefreq=73.41:endfreq=1567.98
,format=rgb24
,geq='
if(lte(X,640)*lte(Y,360),p(233,363),
if(gt(X,640)*lte(Y,360),p(333,363),
if(lte(X,640)*gt(Y,360),p(500,363),
if(gt(X,640)*gt(Y,360),p(633,363),128))))'
,setsar=1,colorkey=black:similarity=0.1[vcqt];
[base1][vcqt]overlay[vcqt1];
[1:v]scale=1280:720,format=rgb24,setsar=1[bgv];
[bgv][vcqt1][base2]displace=edge=mirror,format=yuv420p[v]
" -map '[v]' -map '0:a' -shortest "${pref}.mp4"