iBooks Author Audio Glitches - can they be fixed?

Exporting & Previewing iBooks
1

Hello wonderful Hype Forum community!

I am in the process of creating an interactive iBook for a client, but a big glitch and possibly performance issues are showing up. I'm very much hoping I can somehow fix these issues. Otherwise, I'm afraid the project may not be viable for release.

Short project description:

I am building an interactive sheet music player for practicing (sight-reading) bass players. It allows the user to selectively mute tracks from duos, trios, and quartets so that the user can practice the parts at various speeds and play the parts along with the rest of the (virtual) ensemble.

In targeting this as an iBooks release, I was very much hoping that the iBooks Author Widgets I'm building would work well across mobile, tablet and desktop platforms. Unfortunately, this does not seem to be the case. Each platform is showing some different behavior while running in iBooks.

Click here:

to download a folder that contains the iBooks Author file (that has all of the completed works so far), as well two Hype files (one duo and one quartet). Presumably, the quartet would be more taxing on each given platform than a duo or trio due to more audio being loaded/played back and more animated graphics layers, so I figured I'd include both a duo and a quartet to test both extremes.

Here are the issues I'm finding (and not finding) on each platform:

On desktop:

While running a proof in iBooks on macOS, the iBook (and all included Hype Pro widgets) runs flawlessly with no perceivable glitches.

This was tested on my 2015 MacBook Pro.

On tablet:

While running a proof within iBooks my (2nd generation 12.9 inch) iPad Pro exhibits a puzzling glitch.

The first time one of the Hype widgets is run, a rather horrible glitch appears where the audio is distorted. The audio plays back in sync as expected, but the audio is garbled and static-filled to the point of being unlistenable. However, once I 'x' out of the widget and reopen it, it plays back flawlessly. As long as I don't put the iPad to sleep (and as long as the screen doesn't go black from inactivity), I can quit and restart the app, reopen the widget that first had the glitch, or even choose another widget and playback seems to be very good and glitch free. It seems that it's the act of putting the iPad to sleep that resets things and the glitch again re-appears on whichever widget I choose to playback first.

This led me to think that it was an issue with how (and when/where) the audio was being preloaded, however, now I'm not so sure. I experimented with numerous variations on where I preloaded the sounds, but this seemed to have no effect.

In looking for potential solutions, I happened across this article from Daniel Morgan:

From reading this article, I started to think: is there perhaps a way that I might be able to harness 'Widget Events' in order to trick iBooks into thinking that the widget had been played once and then reloaded in order to bypass the glitchy first-run and then get on with correct playback? A very hacky-solution, but nonetheless I'm now at the point where I am willing to try anything to get it to work glitch-free.

On mobile:

While testing on my iPhone X running iBooks an audio glitch appears, however it seems like a different glitch than the iPad. The audio has pops and clicks occurring at regular intervals, however it's much less pronounced that the iPad. It's as if iBooks is struggling to playback the audio without pops and clicks occurring due being maxed out.

When I 'x' out of the widget and reopen it in order to reproduce the action that got rid of the glitch on the iPad, the minor pops and clicks happen again as they did on first open of the widget. In other words, re-opening the widget does not fix the clicks and pops problem like it does on the iPad. Also worth mentioning is that the pops and clicks seem much less prevalent when playing a duo and more prevalent when playing a quartet (making me think that it is indeed a processing power issue on the iPhone).

So, that's the scope of the audio glitches I'm finding on the various platforms.

I should also mention that (in an effort to determine whether iBooks running the widgets is the problem or my widgets themselves are the problem), I exported the two widget examples as an HTML5 folder and placed them on my server so that I could test the widgets directly running on a web browser on each device. Interestingly, running on a web browser, it seemed like I got no glitches whatsoever on all devices -- desktop, iPad Pro and iPhone X seem to work fine! In short, when running on the web, bypassing iBooks altogether, it seems to run great (I noticed only the slightest occasional audio pop, but it was acceptable performance-wise).

Here are links to the duo and quartet on the web to test against the same files running in iBooks:

http://raleighgreen.com/Bach2PartInvention/

http://raleighgreen.com/BachChorale/

Given all of that background, here are my specific questions for anyone who may be able to help:

  1. Is there a way to fix the startup glitch that appears on my iPad Pro? Perhaps by harnessing Apple's 'Widget Events'?
  2. Am I just pushing it too far to run in iBooks? To many audio tracks? Too many animated layers?
  3. Would it be possible (and if so, how hard would it be) to make this work as a Phone Gap App?
  4. How would a phone gap app perform compared to an iBooks eBook? Could it handle the load better as a phone gap app since it runs so much better directly on a web browser?
  5. If a phone gap app is indeed a better way to go, could I follow some of the various Hype to phonegap tutorials available online and would it work 'out of the box', or would their be extra hoops to jump through because of my audio requirements (multiple synched audio tracks being triggered by howler.js)?

In conclusion, I should mention that the notation images I'm working with are all SVG's. I have not yet tried PNG versions for the animated notation.

Also, the audio I'm working with has a sample rate of 48k with a standard 128 kbps bit rate as specified as acceptable in Apple support docs:

I would so very much appreciate anyone's input on whether or not these glitches can be fixed.

If they can't be fixed, what are my other options? Phonegap?

Thanks so much!

2

For advanced audio in a Hype project, I like to use howler.js for greater audio control.

Perhaps creating a "sound sprite" would solve the popping problems, as only one file is loaded.

I ran into a lot of trouble working on Circles with Grandma as an app because sound files may behave differently on iOS and macOS. So, a "sound sprite" might solve this, as you can specify the exact start and stop points of a sound. If iOS adds a bit of space to the end of the file, it shouldn't break the loop.

That's theoretical anyway, as I got frustrated and just dropped sound files. :laughing:

Recent testing looked more promising though. I had to figure out Sound Sprites and understand the iOS sound issue.

Are you trying to build an app? Maybe Wrapping is what you're looking for.

3

Hi Michael,

Thanks so much for your ideas and input!

I am actually already using howler.js control all of my audio.

When I was first experimenting with the early builds of this project I tried the a sound sprite approach and interestingly, back then at least, it was not as solid (latency-wise) as using howler.js to just load and trigger separate sound files.

So, I went with the latter approach.

That having been said, your suggestion about loading a single audio file got me thinking about a test that I realized might be revealing. The test, which I tried this morning, was to simply limit my preloading and playback to only one single audio file (thereby emulating the effect of loading and playing a single sound sprite). The results were indeed revealing - even with just one sound loaded and played back, I still got the playback glitch as loud as ever on the iPad Pro running iBooks!

Furthermore, I thought, why don’t also disable the scrolling SVG files (which are large and a potential resource hog) so that no animations are happening at all during playback?

I did this test as well. I disabled all preloading and playback except for a single audio track, and I also disabled the scrolling SVG’s as well (or, rather I just removed the action that triggered the scrolling in the first place). I exported the widget, loaded it in iBooks Author, and tested it on my iPad Pro - the glitch was still there as loud as ever!

I think I can now safely assume that it’s not a matter of my maxing out iBooks by playing to many audio files or being too heavy on the animations. It really seems like a glitch or a bug of some sort!

Of course, what I’d really love to do is to manipulate the audio with Hype’s native audio options (which was rock solid in every way when I was doing my initial testing). However, since Hype can only play and stop sounds (and I need the ability to pause and unpause audio), I had to go with howler.js.

Frustrating thing is, it seems like howler.js working great. I’m now thinking more than ever that it’s likely a problem with iBooks.

Again, thanks so much for the suggestion. I think at the very least, it has greatly helped to narrow down what might (or might not) be happening.

Thanks again.

4

This might be annoying for your setup since you depend so much on the time of audio syncing up, but I think what might help is staggering the start times of your audio files. Since the beginning of iBooks, I’ve seen issues with multiple audio streams playing at once. I’m pretty sure at the beginning it just didn’t work. So if you start your streams at slightly different times that might slowly ramp up the processing needs and reduce the likelihood of the warping.

5

Hi Daniel,

Thanks so much for your thoughts on this!
I’d agree that staggering the time of triggering the sounds could greatly help. I could also imagine that inserting a bit of silence at the beginning of each track in order offset the time differences for each sound would result in the effect of the audio still sounding in sync (even though the tracks are starting at different times).

However, the audio glitch was still showing up when I tested loading and playing only one audio track! I even disabled all animations to further lighten the load and the glitch was still there.

Since the glitch is there when only one track is playing, would you agree that staggering the start times would likely have little effect on fixing the glitch when they were all playing?

However, I find it so strange that the simple act of 'x’ing out of the widget and reopening it results in flawless playback. The issue only occurs the first time a widget plays after the iPad has been asleep. This is why I was thinking that their might be some way to harness ‘widget events’ as a workaround in order to trick the device into thinking the widget has been opened once and then reopened. Is there any potential solutions there that you can think of?

Thanks again Daniel!

6

Do you think you could create a Hype document that has a minimally-reproducible version? I’m having a hard time navigating your document and figuring out how you’re triggering, preloading, and playing sounds.

There’s a point when running an iOS app where the device signals a memory warning (in Simulator you can trigger this by running Shift + command + M). My guess is that some sort of memory priming is going on when you first load the document, the device runs out of available memory, and a memory warning (or some sort of audio buffer limit) is being triggered. Then when audio is played, whatever is preloaded into memory is somehow incomplete. The second load is likely re-loading all that audio content back into memory. After a ‘memory warning’, the device clears items in RAM that don’t represent the current process, so that second load (after clicking X the first time) has more RAM to work with. But long story short, this is a bug, and shouldn’t happen.

7

Hi Daniel,

I would be happy to put together a minimally-reproducible version, however I wouldn’t be able to do that until later tonight.

In the meantime, if you have the inclination, if you look at the JavaScript functions open in the top of the document, I load the sounds separately depending on whether the fast, med or slow scene is chosen from the intro popup scene.

Let’s say the user was to choose the fast scene: the function called ‘preloadFastSounds’ is called ‘on Scene Load’ on the ‘Fade to Fast Scene’ scene in order to preload the fast sounds. Then, in the ‘Fast Scene’ scene, the playButton (located in the Play Buttons group) triggers three actions On Mouse Click: Start Timeline…scrollNotation/Play, Run JavaScript…playAllFastSounds(), and Run JavaScript…markTimelineAsPlaying().

That summarizes how the sounds are preloaded and triggered to play and where they are triggered.
The slow and medium scenes are triggered the same exact way.

Sorry, I hope that’s not confusing and I definitely don’t want you to go on a wild goose chase.
Could you let me know if that clarifies things?
If not, I’d be more than happy to put together a stripped down version for you to check out later tonight.

That makes sense what you are saying about RAM memory warning issues.
If that is the case, and if it indeed is a bug, might there be a viable workaround?

Again, thanks so much.

8

Hi Daniel,

I’ve created a simplified version to share.

Below is a link to a folder contains a simplified Hype file, an exported Widget, and a simplified iBooks Author file with the widget added.

I tested this simple version and I can confirm the glitch is still happening.

Here is a link to download the folder.
Thank you so much for any ideas or input you might have!

9

So I tried something that I thought would be simple: destroying the audio object when a play is attempted:

window.sound1.once('play', function(){
  window.sound1.unload();
  window.sound1.load();
  window.sound1.play();
});

I did this for each sound (it works well on browsers) but when I load this widget alongside your widget, both widgets stop working. This may be the fault of iOS12 on my iPad, but I can’t really figure out why it isn’t working: https://cl.ly/3C1d2c3u1r04

book: https://cl.ly/1D1c2Z03360s

I did this for each song, in an attempt to shake up the memory needs of the the audio:

  window.sound1.play();
  
	window.sound1.once('play', function(){
	  window.sound1.unload();
	  window.sound1.load();
	  window.sound1.play();
	  //window.sound1.pause();
	  //window.sound1.play();
	});


  window.sound2.play();
  
  	window.sound2.once('play', function(){
	  window.sound2.unload();
	  window.sound2.load();
	  window.sound2.play();
	  //window.sound1.pause();
	  //window.sound1.play();
	});
	
  window.sound3.play();
  
  	window.sound3.once('play', function(){
	  window.sound3.unload();
	  window.sound3.load();
	  window.sound3.play();
	  //window.sound1.pause();
	  //window.sound1.play();
	});
	
  window.sound4.play();
  
  	window.sound4.once('play', function(){
	  window.sound4.unload();
	  window.sound4.load();
	  window.sound4.play();
	  //window.sound1.pause();
	  //window.sound1.play();
	});
	
  window.sound5.play();
  
  	window.sound5.once('play', function(){
	  window.sound5.unload();
	  window.sound5.load();
	  window.sound5.play();
	  //window.sound1.pause();
	  //window.sound1.play();
	});

Does this same issue occur with MP3s vs M4a files?

10

Thanks Daniel,
Sorry for the delayed reply.
I wasn’t able to get to testing until late last night, but I did get to do some testing today with interesting results.

I very much appreciate your experiment.
I tried the files you sent back to compare with my setup.

  • For the destroy-load2.ibooks file you sent, the audio did play for me on desktop iBooks, but it was pretty badly out of sync with the audio files one after another. It tried it on my iPad (running iOS 11.4) iBooks and I got no audio at all.

  • For the BachChoraleSimplifiedVersion-new.hype you sent, when previewed on desktop Chrome and desktop iBooks, it was interesting: playback starts with a very short audible glitch that happens along with the first click of the metronome. After that first glitch, it played back with flawless timing and no audio glitches. When exported as a widget and tested in an iBooks Author proof, I heard the same audio garbling on first widget opening as before. Upon restarting the widget, the behavior was exactly the same as on desktop (initial audio glitch audible on first beat of the metronome).

And to answer your question, I did try it with MP3s and the behavior seemed identical to M4a (audio glitch prevalent in the same way on iPad iBooks and no prob on desktop iBooks).

I was also able to run a bunch of other tests as well.
One of these was to update my version of howler.js (I was running v2.0.4 on all the previous tests and switched it out with v2.0.14). Curiously, I actually couldn’t get the most recent v2.0.14 to work on iPad at all!

Here are all the tests I ran and the results I got:

  • Tried with new howler.js v2.0.14 and mp3’s. This worked fine on desktop iBooks and browser and not at all on iPad (total silence).
  • Tried with new howler.js v2.0.14 and m4a’s. Also worked fine on desktop iBooks and browser and not at all on iPad (total silence, same as above).
  • Tried triggering with native Hype ‘Play Sound’ action with m4a’s. It worked on desktop iBooks, browser and iPad and the audio sounded clear, however the audio was not in sync even upon closing and restating the widget. It wasn’t drastically out of sync, but definitely out enough to be obvious (and unworkable for project that needed synced audio triggering).
  • Tried triggering with native Hype ‘Play Sound’ action with mp3’s. It worked on desktop iBooks, browser and iPad and the audio sounded clear, however the mp3 audio was strikingly less in sync than the m4a audio. This could be because the m4a audio files I was testing with were averaged to be about 500k, whereas the mp3s I was testing with averaged at just over 1mb.
  • I also tried making a new Hype doc with just a single button on only one scene to trigger sounds with native Hype actions. The audio was not at all garbled, but it was still noticeably out of sync (and especially out on the iBooks desktop version, much more so than iPad or desktop browser).
  • I tried making a new Hype doc with just a single button and only one scene with newest version of Howler.js v2.0.14 and mp3’s. Didn’t work at all - no audio whatsoever! I reverted back to using v2.0.4 and it worked again.

So, no luck on any breakthroughs with all that testing, but hopefully it might help narrow down some avenues to explore.

I think (especially due to unexpected the issues with the most recent howler.js) my next steps will be to looking into experimenting with using a different JS audio library than howler.js.

In particular, I’m curious to look more into pizzacato.js. Have you heard anything about any Hype Pro users trying is? It looks like they have the ability to create what they call a ‘group’ for playing in-sync simultaneous sounds. Looks like it may have potential.

I’m also thinking about calling the Apple iBooks support team to see if they might have any input or ideas, or at least give them a heads up on what sure looks to be a unfortunate bug.

Can you think of any other things to try?
Again, thanks so much Daniel.

11

Hi Daniel,

Just wanted to add a quick follow up:

I contacted Apple iBook Author Support and they are currently looking into the bug.
They have expedited it as high priority, so I’m hoping I will hear back soon.
Will report back with results once I get them.

1 Like
12

Hadn't heard of that library, but it appears to be using the Web Audio API using the same techniques as Howler for multiple audio streams.

Glad to hear your bug report was received well! Please let me know how I can assist. At this point you know more about the intricacies of iBooks Widget audio, but please let me know if I can be of further help.

When I try to Google techniques like 'reopening widget ibooks' I keep finding myself back on this forum -- there's not a lot of people doing this work in the clear so it's very helpful to have your notes on what is working and what isn't!

I think we agree that there should be a better option than the 'restart everything' technique listed here when restarting an out-of-sync or broken audio experience: How to refresh html widget on open - Apple Community

widget.didEnterWidgetMode = function(widgetState) {
// reload the document
}
13

Hi Daniel

Thanks so much for your thoughts on this!

For what it’s worth, I did end up trying both pizzacato.js and buzz.js. Both exhibited similar glitch issues. Curiously, with buzz.js, audio played back cleanly but unusably out of sync. Pizzacato.js acted just like howler.js - horrible audio glitch.

I heard back from Apple on Wed and they specified that the audio really should be only m4a, 48,000kHz, 256K (no mp3). The example I originally gave them was m4a, 48,000kHz, 128K. So, I made a new version with 48,000kHz, 256K audio files and tested it. In testing, the glitch was just as bad with the 256K version, so I sent Apple support a new folder with the new 256K audio widgets for them to look at. They were very thankful for the updated information and are now working on it.

In the meantime, the great news is - last night I found what looks to be a workaround! I still need to do some more testing to be 100% sure, but on both iPad and iPhone this fix totally got rid of the glitch.

It’s really simple - the first scene automatically jumps to the second scene. When the second scene loads, I tried triggering (On Scene Load) Hype’s native Play Sound… action. The Play Sound… action triggered the test audio fine with no glitches, and to my surprise, when I subsequently tried triggering the rest of the audio with holwer.js, it played back perfectly in sync and with no audio glitches whatsoever! It was the exactly the same sort of reset behavior I was getting by closing and restarting the widget.

So, I created a dummy 48,000kHz, 256K audio clip of silence 0:02 long. This way, when the widget is opened and that second scene runs, I trigger this tiny snippet of silence with Hype’s native Play Sound… action. This seems to prime the pumps and open the flood gates for the rest of the howler.js audio to play glitch free!

I’m about to apply this workaround to the rest of the widgets and do a bunch of testing to make sure it’s all good. I’m also going to let Apple support know about the workaround later today so that they can test with the updated info in mind.

In the meantime, I’m cautiously optimistic that (at least until Apple officially fixes the bug), a reliable workaround has likely been found!

1 Like
14

Very interesting workaround. Thanks for sharing this! What are you using to encode your m4a files?

15

Very interesting workaround. Thanks for sharing this! What are you using to encode your m4a files?

I bounce the m4a's directly from Logic Pro X.

16

Sorry, just to make sure I understand your question:
In the Logic Pro X bounce dialog box, encoding is set to AAC.
Hope that helps!

27%20PM
27%20PM1370×828 67.1 KB

1 Like
17

Hi Daniel,

Just wanted to let yo know that I heard back from Apple today.

Here is what they said:

"Hello Raleigh,

Thank you for your continued patience.

Our engineers have confirmed that the work around you found is functional, and we encourage you to share it on the Tumult Support Forum for Hype.

Please move forward with the work around audio files."

I also asked them if my case was closed and if they will continue to work on fixing the bug.
Here was their reply:

"Hello Raleigh,

I can confirm that the engineers are aware of the issue. However, I can not speculate as to when a fix will be issued.

And yes, the engineers have closed the case. Please let me know if you have further concerns.

So, the upshot of it looks to be - who knows when (or, I suppose, if) the bug will be fixed.
But at least the workaround is verified as workable.

Good to know!

Thanks again for all your help with this.

1 Like
18

Problem identified, case open‽ Well I guess that’s a start. At least you have a solution for now – I’m hoping that a part of their big iBooks (now known as Apple Books) redesign is some under the hood tuning for iOS 12.

19

Yes, that would indeed be great. The fact that they are rebranding as Apple Books is encouraging. Nice that they seem to be moving the platform forward.