Tom Bruce Music

Live ambient with color added

Tom Bruce Music

Live ambient with color added

by Tom Bruce Uncategorized

Last night, I did a workshop on multi-layered video for livestreaming for the gang at TE-MPO. It was basically a tutorial on producing livestreamed audiovisuals that look like this example, which is taken from a recorded livestreamed performance by Charles Shriner and I.

My notes for the presentation are below. They offer a way to create video accompaniments for live improv shows that look like the one above. You can see more work like this in my YouTube channel.

Neither the notes nor my presentation went into much detail on what JamKazam is or how it works. You can find that information on the JK site. I’ve also blogged about it a bit here.

Part 2 – how?

Components

These are the pieces and parts used in the final product. For obvious reasons — not having a person to run the video being the biggest — this is all designed to be as automated and hands-free as possible. Basically, the player only needs to intervene at the beginning and end of the show.

So here are the pieces and parts:

  • Audio, of course
    • If players are present in person, some arrangement that combines the players’ stereo feeds into a single stereo source for the streamcast.
      • That is, a mixer
    • In the case of a group performing via JamKazam,  JamKazam
    • If you’re using a multi-machine setup (a separate machine for generating graphics) the resulting audio feed will need to be split
      • One feed to the machine doing motion graphics, to give it some music to respond to.
      • One feed to whatever is doing the video and audio encoding for the live stream.
    • Note: we tend to think of audio interfaces as being things that take an analog input and digitize it, period.  But actually it’s helpful to have one that offers a variety of outputs for monitoring and other purposes.  For that reason, I really, really, really like the Zoom U-44.
  • Performer video sources (ie, cameras)
    • In the case of a group performing in person, one or more cameras. 
    • In the case of JamKazam:
      • Option A (the safe one): one camera per performer, in a layout you set in JK.
      • Option B (the unsafe one): JK offers different configuration/presentation possibilities for multiple performers and cameras. None are automated and it does not provide any real switching capabilities. Attempts to automate this with Applescript result in JK crashing.
      • In general, using JK as a switcher/multi-camera device is buggy and dangerous (it has been known to crash the whole thing)
    • What cameras do you want?
      • USB cameras are fine for the video quality you’re going to get anyway, and much easier to use with OBS. Some (now probably dated) recommendations are here.
      • The ATEM expects HDMI cameras. HDMI is better quality 
        • but you need to make sure you’ve got a “title-free” video camera. 
        • I use the Panasonic HC-V110 and HC-V180
    • Multiple cameras may or may not require additional hardware depending on what connections/interfaces your computer has
      • For solo shows, I use a couple of HDMI cameras connected directly to the switcher
      • In general, devices that will let you use USB cameras with HDMI inputs and vice versa are few and far between, and probably as expensive as just buying the right camera for the inputs you have.
  • Music-responsive graphics
    • I think G-Force (from SoundSpectrum) is the best thing there is
      • Highly detailed and responsive.
    • Any of the ProjectM versions or derivatives (like Project MilkSyphon) are also pretty good.  Others have had luck with Synesthesia and Imaginando among many, many others.
    • Both are programmable
    • Both offer the possibility of creating your own presets.
      • In the case of ProjectM-based stuff, you’re going to have to weed out some of the supplied presets; some just don’t work well when superimposed on performer image or show reels.
    • G-Force has some fairly strict licensing restrictions that you need to be aware of.
  • “Show reel”
    • My term for the videos that run in the background
    • The video segments need to be “plotless”, but offer some narrative nevertheless.
    • These are what provide the narrative and some patterns for the brain to seize on
    • I do these in black and white, for the simple reason that color performer images and motion graphics look better superimposed over something that isn’t in color itself.
  • Assorted image trash
    • Start and end cards – static graphics put on the screen at the start and end of the show. These have a purpose.
      • Start and end cards are
        • A courtesy to the audience. They say
          • “You’re in the right place”
          • “Your viewing setup is working”
        • A way to advertise other stuff
          • Recordings
          • Shows
          • Places to donate/tip/support
    • Titling with the name of the band/concert series/ your mom/ whatever
      • Could be omnipresent or you can fade it in and out
  • Video switcher (which really means “mixer”)
    • Needs to be programmable (or you need a friend to run it)
  • Streamer: a device that
    • Encodes the AV stream into something that the destination can use
    • Can be directed to a particular destination
  • Recorder (because you want a recording, don’t you?)
  • These last three functions are almost always found in one package

How, exactly?

A quick look at how all these functions can be done on the ATEM and in OBS.

WhatATEM Mini-ProOBS
Audio routingStereo feed from JK computer or performer mix boardStereo feed from performer mix board.If using JK, you have two choices. Either use Loopback (from Rogue Amoeba), or audio must be input to OBS computer using a separate interface, connected to return side of audio interface – JK messes with routing.
Multiple cameras with switchingMultiple inputs; macros
Use title-free HDMI cameras		Multiple inputs; advanced scene switcher. Use USB cameras.
Performer video via JamKazamUse as if second monitor on JamKazam machine, fed to HDMI inputIf using JK, a second monitor is needed, because OBS can only capture using display capture– the JK window is somehow not available for window capture.  Capture of second monitor MUST be done before starting up JK video and moving window to second monitor.
Music-responsive motion graphicsUse separate computer (it will also need audio feed). If you run OBS on this computer, you can also add titling.Run on OBS computer; use window capture.  For audio, see notes on audio capture above.
TitlingCan be done using media pool/superposition (that’s not how I currently do it)OBS scene(s)
“Show cards” for start and end of showATEM media pool with still imageOBS scenes with still image. Leave out of Advanced Scene Switcher rotation.
“Show reel” motion videoHDMI video player via ATEMOBS media playback scene. Beware of “loop reset” option in Advanced Scene Switcher.
Compositing/”video mixing”/superpositionMacro control of switcher fader, built into the ATEMUse opacity controls in OBS scenes.  Not adaptive at present.

Things we need to make

From the table above, we can figure out a list of things we need to make:

  • Show cards for use in either OBS or ATEM
  • Band name or other titling graphics, again for use in OBS or ATEM
  • For ATEM, a bunch of macros (one-time effort)
  • For OBS, a bunch of scenes and some work in advanced scene switcher
  • A “show reel” – the movie running behind the show (more about that later)

The first item is straightforward; they can be done in your favorite graphics software.  You may find it advantageous to use a particular canvas size to match the size of the video frame (ATEM doesn’t care about this; OBS might).

Scene transitions and superimposition in OBS are done using the Advanced Scene Switcher (for sequence and timing), and a trick in which superimposition is accomplished by adjusting the opacity of two separate video sources.   Here’s a complete list of the scenes we need to build:

  • Performer camera (might be multiple scenes if multiple cameras are used)
  • Performer camera with title superimposed (again, maybe multiple scenes if multiple cameras)
  • Music responsive motion graphics
  • Music responsive motion graphics with title superimposed
  • Show reel
  • Show reel with title superimposed
  • Superimposition of performers and motion graphics
  • Superimposition of show reel and motion graphics
  • Superimposition of show reel and performers
  • Scenes for start and end show cards

Superimposition scenes are built by using two video sources in the scene and adjusting the opacity of each. 

The scenes are then strung together in a sensible order in the Advanced Scene Switcher.  Usually you’d leave the start and end cards out of the Advanced Scene Switcher sequence and just select them manually before the show and once the show is done.

So a “sensible order” might look like this:

  • Beginning show card (manual)
  • [ Start loop in Advanced Scene Switcher]
  • Performers
  • Performers with title
  • Super performers and motion graphics
  • Motion graphics
  • Motion graphics with title
  • Super show reel and motion graphics
  • Show reel
  • Show reel with title
  • [ End loop ]
  • End show card

The timing on these can be whatever you want.  I find that 2-3 minutes is the most that you want to spend in any particular state, but it depends a bit on the material and the images you’re using. I’d recommend setting all the timings at about 2 minutes and then just spending some time with the whole thing running, observing it with more or less attention, and seeing if the timings make sense.  Maximum variety is usually achieved by making the timings of the graphics preset changes very different from those of the scene switcher, and using a variety of timings for each scene and fade in the scene switcher.  No doubt there will be some adjustments.

Scene transitions (and partial scene transitions that superimpose one scene on another) can be accomplished in ATEM using macro programming.   You can build the macros for specific types of scene transition and test them individually, and then string them together into a “show running loop”.  The difference is that you don’t need the interim “superimposed” scene; you can just halt the progress of the fader and let it sit.

Now about that show reel….

I use YouTube and the Internet Archive as my primary sources for video to use in the show reels.   I find it better to use videos that are “plotless” and that are (or can be converted to) black and white.   I use black and white because I think the superimposition of the motion graphics looks better that way – having both in color would be really confusing visually.  A lot of the early experimental-film material I use is black and white anyway.

For grabbing video from YouTube or Twitch I use a command line  tool called youtube-dl .  You can download a single video or a playlist; it also works with stuff on Twitch.  Two other good tools to know about are Lossless Cut and ffmpeg.   Lossless Cut does just what it says – it makes it easy to cut a video into chunks (or trim the beginning and the end) and export them for use in other stuff.

The ffmpeg utility is a truly amazing piece of work – a Swiss Army knife for all kinds of video manipulation.  I use it to :

  • Drop audio tracks out of video for mixing, processing, and mastering
  • Convert color video to black and white
  • Split video into chunks of a specified length
  • Transcode video from one format to another

It is very, very, very fast (so much so that it can transcode a livestream on the fly). It runs on any platform and is both well-documented and well-supported in user forums. 

I use Wondershare Filmora (an Al Baldwin recommendation) as my primary video editor.  On an M1 Mac, it is very fast and very efficient to use.

Resources and tools

OBS

You can google a million more of these.  Plus, there are books (most of them by Paul Richards).   

ATEM and ATEM macros

Livestreaming generally

Ffmpeg

Lossless Cut

Wondershare Filmora

Source material

X