Can I apply custom javascript to audio files activated with the Hype Action Inspector?

Using JavaScript with Hype
,
#3

Hi Jim,

Thanks so much for your time and input on both of my questions!
It’s very much appreciated.

It’s a big help to know that iOS does not allow volume changes, or muting - however, I’m bummed that that’s the case (although I guess it makes sense that there would be this restriction so that the user can control vol from the device).

That having been said, one curious thing is that I was able to manually set the volume (to create a mute effect) using Howler.js - and this actually worked on iOS! However, no matter what I tried, the starting synchronization of the playback was way off between my three audio files. If I could somehow get them to start at the same time then I’d be all set.

In this regard, I guess I’m confused as to why synchronized audio trigging works so well on all platforms (including iOS) when using Hype’s built -in controls, while at the same time, triggering synchronized audio with javascript (vanilla or howler.js) works so poorly?
I even tried making my tracks into an audio sprite (something that howler.js accommodates) so that all three parts were playing from a single audio file. No luck - sync still way off.

Can you think of any workarounds to get muting functionality (or successfully synchronized audio playback triggered via javascript) on iOS?

Regardless, I very much appreciate your help.
How amazing would it be if Hype 4 featured more robust audio options!

Thx again

#4

Hi Raleigh!

Interesting to know that about Howler (setting mute) - thanks for the info.

So here goes another tidbit from the Apple Developer page I linked to earlier:
iOS will only accept one audio stream at a time… which does not explain how You managed to get all three to play, even if the syncing was “off”. Another puzzler is why one just one audio file would not play. I guess they were not merged into one track but rather three separate tracks were in the format wrapper - and seen by iOS as three separate streams?

Can you think of any workarounds to get muting functionality (or successfully synchronized audio playback triggered via javascript) on iOS?

I thought You got Howler to do the muting(?). I do not know how to make the synchronized audio work on iOS via JavaScript - sorry.

However…

Below is a web page that discusses audio on iOS using the HTML5 Web Audio API. Apparently there are some work arounds of iOS audio limitations using the Web Audio API. I have not delved into this article or done any experimentation along these lines - but maybe there is something useful for You here:

#5

Thanks Jim,

Sorry I wasn’t more clear about the muting thing. To clarify, I was indeed able to do the muting (through howler,js) and I was also able to trigger the synchronized audio (with Hype’s built in controls), but I am not able to do both. I was only able to get one or the other to work, but not both at the same time (which is the functionality I need).

After thinking about it some more, I thought I had a good workaround: Dump the less important audio track (it was just a click track) and take my remaining two audio tracks and merge them into one stereo track and then programmatically pan them hard left or hard right to isolate the sounds as if they were two different tracks.

Howler.js does have panning built in, so I thought I’d be good to go! Well, I created the stereo track and added it to my Hype project. The panning with howler.js did work, but unfortunately it just took my stereo track, summed them together and then played the panned (and unintentionally merged) audio panned to the left or right speaker.

So, I’m thinking that what I actually need to make this work is to use the channelSplitterNode from the Web Audio API.

However, I’m having a really hard time visualizing how to set up creatChannelSplitter()within Hype. Do you have any idea what this function might look like to take my single audio track: src: ['${resourcesFolderName}/BachStereoTrack.m4a'] and split the channels into left and right so that I can modify the volume of the channels directly?

Again, thanks so much for your input with this - it’s a huge help - I’d so love to get this figured out!

#6

Hi Raleigh!

Would You post an example of your working desktop version for examination? If there are confidential~proprietary considerations You could PM me a link or a Hype project.

Thank You.

#7

Hey Jim,

Sure, I’ve attached my most recent effort.

This version has just a single stereo audio track that I attempted to pan left and right in order to isolate the two tracks (and get a muting effect). The panning works, but it’s summing the track and panning the whole mix, rather than isolating the tracks as intended.

For what it’s worth, I tried it on my iPhone and the panning from howler.js works well on iOS, however the iPhone speakers are badly balanced - in landscape view the left speaker (near the home button) is quite a bit louder than the right. This makes me think that even if I could figure out a way to split the stereo channels left and right, I’d still need to overcome this hardware issue and add another step of merging the channels back to a stereo clip in order to make it sound right (and not have one side way quieter than the other).

This is why I was thinking the channelSplitterNode might be an option - but I’m stuck on how to implement it.
Of course, I’d be open to considering any solutions - just thought this might be easier than dealing with trying to sync multiple tracks.

Thanks so much for giving it a look Jim - I’m really curious to see if any solutions jump out.
All the best!

Bach-iPhoneX-pan-test.zip (780.7 KB)

#8

Hi Raleigh!

I have an idea I will try. I am outlining it here in case others see flaws or improvements that could be made - as I have little design experience with iOS scripting strictures.

  1. Split the file with the stereo channels into (2) mono audio files (isolating each unique music sequence).
  2. Mix down the file with the stereo channels into one mono channel.
  3. Pre-load all three mono files. (Equal to clicking “Top” ~ “Bottom” ~ “Both” buttons.)
  4. Track the time of the currently playing audio file using JavaScript (“addEventListener”).
  5. On demand (i.e. user button click) immediately switch to the desired audio file.

The possible (likely) “Fly-in-the-Ointment”… the latency of the file switching.

I won’t be able to experiment with this idea for at least a day. It will be cool if it works!

1 Like
#9

Hey Jim,

This sounds like a really interesting idea!
I really appreciate your experimenting with this - it’s a huge help.
I’m so curious to see if it will work as I am stumped!

Thanks again!

#10

I did manager to test playing two audios on my iOS earlier and using the native HTML5 Mediacontroller() worked.
Mediacontroller allower you to set the contoller for Multiple media Elements to a single control and Sync Multiple Media Elements Together

So my idea is to play both streams using the single controller and then just mute where needed.

I cannot test this one on iOS at the mo. But I think it should work. ( i hope )
Just tested and seems to work…

audio_mhv1.hype.zip (1.1 MB)

Update bit,

I just realised that the initail problem is not the sync, so you probably do no t need the Mediacontroller.

But in your projects you where trying to turn the sound down not mute it, the project above still shows how to mute

1 Like
#11

Hi Mark,

This is fantastic - it works!

I added your code to my project and it works really well on iOS - the sync for the three tracks stays on point nicely and the mute works great. I was also able to hook up the metronome button to mute the click track as well. So great to finally see this all working together!

Again, thanks so much for this solution.

One thing: in testing I noticed that the sound doesn’t play on some desktop browsers (I tried Chrome, Opera and Firefox and the sound doesn’t work). However, curiously but not surprisingly, it does work fine on Safari desktop browser.

However, this doesn’t really matter since this widget is being designed for an iBook. I tested in iBooks Author on macOS and it works great - this is the important thing. That having been said, do you have any thoughts on why it’s not working on those three desktop browsers (and is there an easy fix that you can think of to make it work in case my client wants to someday put it on a web app)?

Again, thanks so much!

#12

Hah,

It would seem although Apple mention the use of it in their Docs, it does not seem to be generally supported by the other browsers

#13

Hello Jim and Mark,

I just wanted to thank you both for your input a few days back.
As an update, I’ve had a chance to implement some of Mark’s suggestions.

At first I thought this was the solution I was looking for - it seemed to be working great in Hype reflect!
Well, as it turns out, even though it worked well in Hype reflect, it rarely works solidly when running as a widget in iBook Author.

I exported as an iBooks Author Widget and embedded it into iBooks Author. I tested it running in iBooks on my iPhone 7, my iPad Pro 2, and on desktop. It was somewhat stable running as a widget in iBooks Author on desktop, but even then it was prone to glitches (audio playing back out of sync maybe 30% of the time) when closing and opening the widget multiple times. Running on iOS, it ran in sync less than 50% of the time. So, unfortunately this solution is just not going to work out.

So, I’m wondering if there are any other ideas anyone has to try? I feel like the capability is buried in there somewhere it’s just a matter of figuring out the right approach.

Jim, I remember there was an idea you had to split the stereo file into 2 mono files.
By any chance would you have the time to revisit that idea on that chance that it might work?
Anyone else like Jonathan or Daniel have any ideas?

Thanks again for your time and input on this!

#14

Hi Raleigh!

I managed to find myself right in the middle of the largest fire in California history - over 100 square miles (Napa, Sonoma, Solano Co.). For the past 9 days all my equipment and most of my personal belongings have been evacuated from my home. I am just now hooking things back-up. The fires are still burning - the air redolent with smoke - but things are now contained in my immediate area (in theory). Unlike many others I made it through in one piece. :pray:

Needless to say, it will be a while before I can do the experimentation I previously discussed. I will follow through when I can.


Day 1, one of seven fires - taken from my home.

Fire_Pano_8048 8049.jpg1024×630 89.8 KB

#15

Holy cow!
So sorry to hear that you have been displaced by the fires in California. However, I’m so glad you were able to evacuate safely. Wishing you (and your neighbors) all the best in dealing with all of that!
Very much hoping those fires can be brought under control as soon as possible.

Stay safe!

1 Like
#16

I’ve been seeing images of this daily, the devastation is apocalyptic :open_mouth:

@JimScott I am very glad you are ok mate.

1 Like
#17

Glad to hear you’re out of it! Stay safe buddy.

1 Like
#18

Thank You for your Best Wishes. I am trying to imagine the neural imprint of those who live with this level of destruction on a daily basis since they were born… what a toxic Hell - of which I have only skimmed the dregs.

It has now been raining for hours and I am so grateful - the Beast is Dead!
The wet ground is precious on my lips (don’t You know).

1 Like
#19

Hey Raleigh did you ever figure this out?

A couple of things that aren’t obvious with using audio in IBA. There is a bug that prevents audio from playing. The workaround for the bug is to take the file extensions off in your html and in the name of the actual audio file. It was a huge pain to find out the solution to this but it might help you. Your html references an audio file as “audio/bassdrum.mp4” or something to that effect. You’ll have to make it read as “audio/bassdrum” and then make sure and remove the file extension from the actual file. That did the trick for making audio work in my IBA projects.

I have no experience with Hype but thought it might be a handy solution for making complex widgets faster. I’ve done lots of synced audio in widgets with JS though. I have generally used a library called mix.js and custom skinned it and made additional functionality I needed on top of it. You can check it out at http://kevvv.in/mix/

Hope you got your stuff working but I know I had some serious pain with audio files and at one point had accepted I’d never be able to do it. Otherwise, hope this helps to recharge you.

KD

1 Like
Intro to Exporting iBooks Widgets
#20

Hey KD,

Thanks so much for you input on all of this. I did indeed run into that audio bug.

I ended up getting around it by converting all of my audio to .m4a files. This workaround seems to work great. So in that regard, I’m all set (I used howler.js) and my widget is working really well, with no audio sync issues at all.

That having been said, I wonder which would be more lightweight and responsive to use in IBA, .mp4, or .m4a? Do you happen to know? If there is a compelling reason to go with .mp4’s (with file extensions taken off as you mentioned), then I’d consider using those instead of .m4a’s.

Again, thanks so much for the input and all the best!

#21

I’ve added this tip to the iBooks guide. That’s an odd workaround, but glad it does the trick for you. Was audio just not playing at all without this?

#22

Hi KD,

I had 3 quick questions about your workaround if you don’t mind:

Did you figure out this IBA audio workaround in the pre-publishing stage or after it was actually published?

Have you verified this workaround as working for published iBooks on all platforms (mac, iPad and iPhone)?

What audio file types did you use (the ones that you removed the file extensions from)?

Thanks so much!

rg

Products
Company
Social
Hype Support
Whisk Support
Store

Copyright © 2010-2020 Tumult Inc. All rights reserved.
Handcrafted in San Francisco.