I’m trying to play 3 synchronized audio files.
I have a play button that triggers playback for all three buttons.
When I trigger the playback in Hype’s Action inspector the audio files playback great - fully synchronized on both desktop as well as iOS. Here is the tricky thing - I need to be able to mute these tracks independently while the tracks play. This is functionality that is not built into Hype. As a result, I’ve been trying to get it to work with vanilla javascript as well as with howler.js.
I can get the tracks to play fine using javascript and with howler.js. The problem is, on iOS, the playback of the tracks loses it’s sync.
Is there a way to trigger playback of these 3 audio files with Hype’s built in Action Inspector (in order to get the audio synchronized), but somehow mute (or make the volume 0) the tracks with javascript or howler.js?
It seems the only way to affect an audio track with javascript is if it’s triggered with javascript to start with. It is true that if I trigger an audio file with the Hype UI, then I won’t be able to modify that audio file with a custom javascript function?
One of my answers was a link to the "iOS-Specific Considerations" page of the Apple Developer site re: Audio & Video. iOS does not allow volume to be set by JavaScript - that includes muting. Just below is the link:
Unfortunately Hype's built-in controls can not be accessed by JavaScript (at least that I'm aware of).
Thanks so much for your time and input on both of my questions!
It’s very much appreciated.
It’s a big help to know that iOS does not allow volume changes, or muting - however, I’m bummed that that’s the case (although I guess it makes sense that there would be this restriction so that the user can control vol from the device).
That having been said, one curious thing is that I was able to manually set the volume (to create a mute effect) using Howler.js - and this actually worked on iOS! However, no matter what I tried, the starting synchronization of the playback was way off between my three audio files. If I could somehow get them to start at the same time then I’d be all set.
In this regard, I guess I’m confused as to why synchronized audio trigging works so well on all platforms (including iOS) when using Hype’s built -in controls, while at the same time, triggering synchronized audio with javascript (vanilla or howler.js) works so poorly?
I even tried making my tracks into an audio sprite (something that howler.js accommodates) so that all three parts were playing from a single audio file. No luck - sync still way off.
Can you think of any workarounds to get muting functionality (or successfully synchronized audio playback triggered via javascript) on iOS?
Regardless, I very much appreciate your help.
How amazing would it be if Hype 4 featured more robust audio options!
Interesting to know that about Howler (setting mute) - thanks for the info.
So here goes another tidbit from the Apple Developer page I linked to earlier:
iOS will only accept one audio stream at a time... which does not explain how You managed to get all three to play, even if the syncing was "off". Another puzzler is why one just one audio file would not play. I guess they were not merged into one track but rather three separate tracks were in the format wrapper - and seen by iOS as three separate streams?
Can you think of any workarounds to get muting functionality (or successfully synchronized audio playback triggered via javascript) on iOS?
I thought You got Howler to do the muting(?). I do not know how to make the synchronized audio work on iOS via JavaScript - sorry.
However...
Below is a web page that discusses audio on iOS using the HTML5 Web Audio API. Apparently there are some work arounds of iOS audio limitations using the Web Audio API. I have not delved into this article or done any experimentation along these lines - but maybe there is something useful for You here:
Sorry I wasn’t more clear about the muting thing. To clarify, I was indeed able to do the muting (through howler,js) and I was also able to trigger the synchronized audio (with Hype’s built in controls), but I am not able to do both. I was only able to get one or the other to work, but not both at the same time (which is the functionality I need).
After thinking about it some more, I thought I had a good workaround: Dump the less important audio track (it was just a click track) and take my remaining two audio tracks and merge them into one stereo track and then programmatically pan them hard left or hard right to isolate the sounds as if they were two different tracks.
Howler.js does have panning built in, so I thought I’d be good to go! Well, I created the stereo track and added it to my Hype project. The panning with howler.js did work, but unfortunately it just took my stereo track, summed them together and then played the panned (and unintentionally merged) audio panned to the left or right speaker.
So, I’m thinking that what I actually need to make this work is to use the channelSplitterNode from the Web Audio API.
However, I’m having a really hard time visualizing how to set up creatChannelSplitter()within Hype. Do you have any idea what this function might look like to take my single audio track: src: ['${resourcesFolderName}/BachStereoTrack.m4a'] and split the channels into left and right so that I can modify the volume of the channels directly?
Again, thanks so much for your input with this - it’s a huge help - I’d so love to get this figured out!
Would You post an example of your working desktop version for examination? If there are confidential~proprietary considerations You could PM me a link or a Hype project.
This version has just a single stereo audio track that I attempted to pan left and right in order to isolate the two tracks (and get a muting effect). The panning works, but it’s summing the track and panning the whole mix, rather than isolating the tracks as intended.
For what it’s worth, I tried it on my iPhone and the panning from howler.js works well on iOS, however the iPhone speakers are badly balanced - in landscape view the left speaker (near the home button) is quite a bit louder than the right. This makes me think that even if I could figure out a way to split the stereo channels left and right, I’d still need to overcome this hardware issue and add another step of merging the channels back to a stereo clip in order to make it sound right (and not have one side way quieter than the other).
This is why I was thinking the channelSplitterNode might be an option - but I’m stuck on how to implement it.
Of course, I’d be open to considering any solutions - just thought this might be easier than dealing with trying to sync multiple tracks.
Thanks so much for giving it a look Jim - I’m really curious to see if any solutions jump out.
All the best!
I have an idea I will try. I am outlining it here in case others see flaws or improvements that could be made - as I have little design experience with iOS scripting strictures.
Split the file with the stereo channels into (2) mono audio files (isolating each unique music sequence).
Mix down the file with the stereo channels into one mono channel.
Pre-load all three mono files. (Equal to clicking “Top” ~ “Bottom” ~ “Both” buttons.)
Track the time of the currently playing audio file using JavaScript (“addEventListener”).
On demand (i.e. user button click) immediately switch to the desired audio file.
The possible (likely) “Fly-in-the-Ointment”… the latency of the file switching.
I won’t be able to experiment with this idea for at least a day. It will be cool if it works!
This sounds like a really interesting idea!
I really appreciate your experimenting with this - it’s a huge help.
I’m so curious to see if it will work as I am stumped!
I did manager to test playing two audios on my iOS earlier and using the native HTML5 Mediacontroller() worked.
Mediacontroller allower you to set the contoller for Multiple media Elements to a single control and Sync Multiple Media Elements Together
So my idea is to play both streams using the single controller and then just mute where needed.
I cannot test this one on iOS at the mo. But I think it should work. ( i hope )
Just tested and seems to work…
I added your code to my project and it works really well on iOS - the sync for the three tracks stays on point nicely and the mute works great. I was also able to hook up the metronome button to mute the click track as well. So great to finally see this all working together!
Again, thanks so much for this solution.
One thing: in testing I noticed that the sound doesn’t play on some desktop browsers (I tried Chrome, Opera and Firefox and the sound doesn’t work). However, curiously but not surprisingly, it does work fine on Safari desktop browser.
However, this doesn’t really matter since this widget is being designed for an iBook. I tested in iBooks Author on macOS and it works great - this is the important thing. That having been said, do you have any thoughts on why it’s not working on those three desktop browsers (and is there an easy fix that you can think of to make it work in case my client wants to someday put it on a web app)?
I just wanted to thank you both for your input a few days back.
As an update, I’ve had a chance to implement some of Mark’s suggestions.
At first I thought this was the solution I was looking for - it seemed to be working great in Hype reflect!
Well, as it turns out, even though it worked well in Hype reflect, it rarely works solidly when running as a widget in iBook Author.
I exported as an iBooks Author Widget and embedded it into iBooks Author. I tested it running in iBooks on my iPhone 7, my iPad Pro 2, and on desktop. It was somewhat stable running as a widget in iBooks Author on desktop, but even then it was prone to glitches (audio playing back out of sync maybe 30% of the time) when closing and opening the widget multiple times. Running on iOS, it ran in sync less than 50% of the time. So, unfortunately this solution is just not going to work out.
So, I’m wondering if there are any other ideas anyone has to try? I feel like the capability is buried in there somewhere it’s just a matter of figuring out the right approach.
Jim, I remember there was an idea you had to split the stereo file into 2 mono files.
By any chance would you have the time to revisit that idea on that chance that it might work?
Anyone else like Jonathan or Daniel have any ideas?
I managed to find myself right in the middle of the largest fire in California history - over 100 square miles (Napa, Sonoma, Solano Co.). For the past 9 days all my equipment and most of my personal belongings have been evacuated from my home. I am just now hooking things back-up. The fires are still burning - the air redolent with smoke - but things are now contained in my immediate area (in theory). Unlike many others I made it through in one piece.
Needless to say, it will be a while before I can do the experimentation I previously discussed. I will follow through when I can.
Holy cow!
So sorry to hear that you have been displaced by the fires in California. However, I’m so glad you were able to evacuate safely. Wishing you (and your neighbors) all the best in dealing with all of that!
Very much hoping those fires can be brought under control as soon as possible.
Thank You for your Best Wishes. I am trying to imagine the neural imprint of those who live with this level of destruction on a daily basis since they were born… what a toxic Hell - of which I have only skimmed the dregs.
It has now been raining for hours and I am so grateful - the Beast is Dead!
The wet ground is precious on my lips (don’t You know).
A couple of things that aren’t obvious with using audio in IBA. There is a bug that prevents audio from playing. The workaround for the bug is to take the file extensions off in your html and in the name of the actual audio file. It was a huge pain to find out the solution to this but it might help you. Your html references an audio file as “audio/bassdrum.mp4” or something to that effect. You’ll have to make it read as “audio/bassdrum” and then make sure and remove the file extension from the actual file. That did the trick for making audio work in my IBA projects.
I have no experience with Hype but thought it might be a handy solution for making complex widgets faster. I’ve done lots of synced audio in widgets with JS though. I have generally used a library called mix.js and custom skinned it and made additional functionality I needed on top of it. You can check it out at http://kevvv.in/mix/
Hope you got your stuff working but I know I had some serious pain with audio files and at one point had accepted I’d never be able to do it. Otherwise, hope this helps to recharge you.
Thanks so much for you input on all of this. I did indeed run into that audio bug.
I ended up getting around it by converting all of my audio to .m4a files. This workaround seems to work great. So in that regard, I’m all set (I used howler.js) and my widget is working really well, with no audio sync issues at all.
That having been said, I wonder which would be more lightweight and responsive to use in IBA, .mp4, or .m4a? Do you happen to know? If there is a compelling reason to go with .mp4’s (with file extensions taken off as you mentioned), then I’d consider using those instead of .m4a’s.
Again, thanks so much for the input and all the best!
