The tl;dr on this:  I had a bunch of iPad gear go out from under me on Saturday night; I tried to figure out how to prevent a recurrence; there’s no sure means of prevention, but there might be some best practices.  Read on if you want the details ;).

Well, dammit all to hell.

So, Saturday night I did about 25 minutes of what was supposed to be an hour-long set with my pal Karl Fury.   That 25 minutes was broken into two parts — about 15 minutes at the beginning, and 10 at the end, with the middle 35 minutes being either inaudible or all-too-audible as the result of crackling and noise, sometimes loud enough to clip and sometimes completely inaudible, coming from my rig.   It was scary and bewildering; I’d run the rig for more than an hour of rehearsal before we began, with no sign of problems.  I spent a good chunk of yesterday trying to figure out how I could prevent a recurrence.  

Those symptoms might have had a few different causes, including USB cabling, problems with the Scarlett 18i20 that I use as the final output interface from the rig, or with the iConnect device I use to connect the iPad.  But, significantly, my sample player had not been affected. It doesn’t run from the iPad, but it does run through the Scarlett, so the chances that the interface was the problem were pretty slim. Could still have been the iConnect interface I use to connect the iPad, but that struck me as unlikely. For one thing, the iConnect has been incredibly reliable, and for another, when it *does* fail, it tends to be really spectacular,  locking up with the sort of jet-engine-level buzzing squeal that results from a sample clock losing its mind completely.   So I was inclined to look more closely at the iPad, and — because the problem had affected nearly all the instruments I was using on it — AUM in particular. 

What about buffer size?

It’s well established that snap-crackle-and-pop problems with iOS audio are usually caused by setting the sample buffer too small. People do this to cut down on latency, at the expense of greater CPU utilization, and if you take it too far, you end up with crackling, noisy audio. But in this case, I hadn’t had a problem for two hours before the show, or for the first fifteen minutes, and (even worse) it had sort of cleared itself up by the end, but not really in direct response to anything I tried to do about it.   So it seemed like the first step toward prevention in the future was to try to understand what was actually going on. How do you set buffer size, and what does it affect, and how is it affected?    I can sum up what I learned in two sentences:

1. You really do need to think through buffer size. Audio crackle is not quite independent of CPU loading, but CPU loading at the brute-force level is probably not your problem, and
2. iOS audio involves serious competition between different apps for audio resources (including the sample buffer, but also different types of CPU, memory space, and other resources) that interact in ways that are extremely complex and hard to predict. 

I can’t claim to understand either of those propositions completely, and certainly not the second one.  But at this point in my life I have enough of an eye for the kind of software platform documentation that developers consume to know when I’m looking at some kind of Wild West inside the operating system.  Those with a similar eye for such things will be entertained by this and especially this.

First of all, buffer size is believed to be “follow the leader” — the setting in the first audio app invoked determines the buffer size for everyone (that “believed to be” tells you something about how much anyone outside of Apple knows about the internals).   So, suppose you have Animoog up and running, and you’re noodling around with it, and you think, “gee, I’d like to add a reverb”, so you fire up AUM and add the reverb. At that point, the buffer size for AUM and the reverb app will be set wherever Animoog was set when you fired up AUM. And so it will be with every app you add from there on in.  Or so it is believed — even Apple’s developer documentation is unclear on this point. From reading threads like this one, it seems that Apple is pretty tight-lipped about problems with iOS audio.  My guess is it’s because their design never really contemplated much beyond people listening to Taylor Swift on earpods, or how to get a music player to duck when a phone call comes in.  The point is that there’s not much transparency, so nobody knows much about whether apps can get away with having non-conforming buffer sizes (for example) or changing the buffer, or whatever. 

In the online forums, or in any source read primarily by musicians rather than software developers, buffer size is generally portrayed as a tradeoff with latency.  Too small a buffer and you get crackle and pop because the buffer can’t be serviced fast enough.  Too big, and you’ve got latency because the rest of the audio engine is sitting around waiting for it to fill.  What’s the right size?  That depends — and like a lot of things, it depends most on being realistic about what our expectations are in different environments. Personally, I think we care a lot less about latency in live performance situations than the hot-rodders and gear salesmen among us would have us think we do (using an iPad to do tracking-type recording is a different set of needs).

Stories abound of musicians who can hear 40-sample differences in timing. I personally think those stories are mostly dick-waving. No doubt there are individuals who can do that, whether because of innate talent or training or the ability to pretend successfully, and I suggest that those individuals would be best served by not employing iPads to track vocalists.  Psychologists have pretty well established that most humans can’t perceive audio events that are separated in time by less than about 3 ms (though it varies with the nature of the events).   At a sample rate of 44100, a buffer of 1024 samples creates about 2.4 ms of latency; at 256 samples, it’s about 0.6 ms.   Others point out that latencies on that order are smaller than you’d get by standing 10 feet in front of the audio source.   And again, the perception of separate sound events depends a lot on the loudness of the events, their relative frequencies, and a bunch of human-brain stuff.  

At that level, the whole discussion reminds me of some woodworkers I know.  They’ll spend 400 hours and a lot of money tuning a table saw to the point where it will cut with .003” accuracy — accuracy that is rendered completely irrelevant by a 5% change in humidity.  I’d say that when in doubt, using the larger buffer is probably the better way to go. 

Resource competition

One way to summarize my earlier description of the iOS audio environment was as a kind of vicious competition for resources on a playing field that wasn’t really designed for professional audio performance.   How much control do we have over the size of the field and the use of resources?  Not much, except to avoid using any more than we have to.  For example, the developer documentation is quick to say that buffer size requests from applications are really just a kind of please-sir-may-I-have-some-more  to the operating system, and what the application actually gets is determined by any number of things that are going on in the system at the time of the actual request.   Those things include activity by other apps, CPU resources of different kinds, behavior of the graphics interface, and for all I know relative humidity, just like my table saw.

It seems, then, that the best way to control the use of resources is to minimize the complexity of what we’re using, introduce new apps into the overall ecology in a sensible order, and avoid introducing clutter or unused elements.  The fact is, I hadn’t done any of those things.  The iPad had not been rebooted in forever, making it a huge pile of memory leaks from faulty applications (and when it comes to memory leaks, there is no application without faults).  I’d been working on my setup in AUM for a few days, taking stuff out and putting it back, leaving empty effects slots, and so on.  And a few of the apps I use are real hogs — enough so that it’s worth asking whether i’m getting enough out of them musically to justify their huge footprint in the device.   

Given our lack of control over the environment, and the tangled competition that takes place within it, it probably makes sense to keep things simple and clean.  Here’s what I’d recommend; time will tell whether it works or not.

Rationalize your app-hosting/rig setup

I don’t know about all of you, but for me, the development of a new rig involves a lot of sketching and crossing-out and general reshuffling of stuff before I hit on the final design.  The amount of shuffling and sketching depends on whether I think I’m going to be using the design for one performance or many, but there’s always a few blind alleys and revisions and other horsing around before the thing is “finished”.  At that point, it pays to do some rationalizing.  Get rid of effects you’re not using or don’t need; delete channels that aren’t being used. Figure out whether you need all those effects or not — for example, I tend to use very similar reverbs on multiple instruments and at that point it would be far more efficient to send all those signals through one effects bus with one reverb effect on it (reverbs are particularly hard on CPU, it seems).   Do the extra steps: pare it down, simplify what you can, put channels in sensible order and label them, document everything.  We all tend to pile stuff up; the less of that, the better.  Of course, you might also ask, “Do I really need that instrument?”, but what would be the fun in that?

Incidentally, this is as good a time as any to put in a plug for just sitting down and reading through the AUM manual — it’s really well written, and told me a lot of things I hadn’t known before.  I think that iPad musicians suffer from a problem with short attention spans — and hosting software that you depend on for live performance is probably worth understanding completely. I have to confess that (for example) I had no idea that you could reorder channels — AUM’s interface is both very clean and very well thought-out, which in a touch-screen world means that a lot is hidden from you unless you know to look for it.

Start clean at set-up time, and start things up carefully.

Shut down the iPad completely, powered off.  Power off all your interfaces, laptops, and other gear and start each item up in a sensible order, iPad last. Once the iPad’s up, start your hosting software first (whether AUM or Ableton or whatever) so as to improve its chances of dictating buffer size rather than begging for it).  In the case of AUM, do NOT start instruments before starting AUM — let it invoke them in the order it wants.   

If a particular app won’t start up inside the host, that’s an indicator of potential trouble.  Borderlands used to just flat refuse to start inside AUM; the newer version fixes that.  Thor has always had trouble, but Thor won’t play nicely with anything, and I don’t use it for live performance any more as a result.  There are others that can be troublesome, and I’d take that seriously (I have my doubts about iWavestation) if something is balky about starting.

The reason to take such a structured approach is basically to make sure that any memory-leaked clutter is gone before you start, and that the right applications are laying first claim to any shared resources.  

Will any of this work?

Maybe.  It really comes under the heading of “preventive maintenance” or “prophylaxis”.  Under iOS, there doesn’t seem to be much you can do to fix things once they spiral out of control, so the best technique seems to be to try to understand enough about what it’s doing to allow you to create the most favorable environment you can.

I’ll let you know how it works out.

References

• https://forum.audiob.us/discussion/26526/aum-crackling 
• https://forum.audiob.us/discussion/29769/gen-3-ipad-pro-users-have-you-noticed-significant-cpu-spikes/
• https://forum.audiob.us/discussion/31500/tips-on-keeping-low-dsp
• https://developer.apple.com/library/archive/documentation/Audio/Conceptual/AudioSessionProgrammingGuide/OptimizingForDeviceHardware/OptimizingForDeviceHardware.html
• https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitHostingGuide_iOS/ConstructingAudioUnitApps/ConstructingAudioUnitApps.html , particularly the section called “Configuring your audio application”
• https://www.raywenderlich.com/5154-avaudioengine-tutorial-for-ios-getting-started