Can built-in Hype events be modified with javascript?

Using JavaScript with Hype
1

Hello,

I’ve searched, but have come up with no definitive answers to this question.

I’m wondering:

If an event is triggered on an object using the Hype’s built-in controls, is it possible to then access and modify that same object with javascript?

For example, if I were to choose ‘play sound’ from the Hype’s built in Actions Inspector, is there anyway to do something like ‘window.audio.pause()’ in order to pause the sound with javascript? Or, are Hype’s built-in actions closed off from being accessible or modifiable?

Thanks!

2

If you’re looking for better audio control in your project…

3

Why do you suggest this instead of native HTML5 audio?

I think the answer is no, afaik the when sound is played through the actions it is not available to you to control with JS,

4

I know right?! I usually hate using libraries. Have you actually tried to use native HTML5 to do what Howler.js does? I did. It did not go well. :smile:

5

I have not but what I meant was can you give some details of why here since you have used it and are suggesting it.. :smile:

6

Well, the keyword was "Pause". Hype doesn't do that. Hype can "Play" a sound. Hype can "Stop" a sound (which restarts the sound from the beginning when played again). Hype cannot pause a sound. But with Howler.js, you can do that...

pause([id])

Pauses playback of sound or group, saving the seek of playback.

id: Number optional The sound ID. If none is passed, all sounds in group are paused.
stop([id])

Stops playback of sound, resetting seek to 0.

id: Number optional The sound ID. If none is passed, all sounds in group are stopped.

That's just basic functionality. Then, there's fades, loops and other features, that give much better control over audio in a project.

To even simplify the matter even further, the main question is this – Is there an Audio API in Hype? No, there isn't. There should be. But since there isn't, howler.js is a nice solution.

7

I do think it is a good option, :wink:
I was talking about Native HTML5 Audio Javascript - ( Not the Hype Actions ). I probably should have also said Web Audio API , my bad.

https://www.html5rocks.com/en/tutorials/webaudio/intro/

Before the HTML5

The Web Audio API is a high-level JavaScript API for processing and synthesizing audio in web applications. The goal of this API is to include capabilities found in modern game audio engines and some of the mixing, processing, and filtering tasks that are found in modern desktop audio production applications. What follows is a gentle introduction to using this powerful API.

Which I suspect Howlerjs is tapping into but I think Howlerjs may make it easier for the user to work with :smiley:

8

Hello Michael and Mark!

Thanks so much for all the great input here!

Just to clarify, what I’m looking to do is to build an interactive widget that is intended to run on the iBook (or hopefully even ePub) platform, and possibly the web. These will be duets (or possibly trios or more) where you can mute out individual tracks in order to play whichever parts you want as the music notation scrolls by while the song plays.

In order to do this, at the minimum, I need to be able to have three audio tracks - 1) the top audio, 2) the bottom audio 3) and the click track. I need to be able to do various things with the audio like play, stop, pause, and mute. I’ve found the really hard part here is getting all three audio files to play in sync programmatically.

I’ve tried tons of things and have not yet gotten it to work on all platforms.
Here is a rundown of what I’ve tried thus far:

  • I first tried triggering the synchronized audio with Hype’s built-in actions. It works perfectly on all platforms - macOS browsers, iOS browsers and iBooks! However, since Hype doesn’t have built-in pause or mute (and since I can’t take action on any Hype-triggered audio with javascript), this solution won’t work (arrgghh, lol).

  • I tried triggering the three audio files with howler.js and I was able to get all the functionality I needed. However (surprisingly and disappointingly) the audio playback on iOS was badly out of sync. Playback sync on macOS sync was fine, (of course), but since it was out of sync on iOS, this solution won’t work.

  • As a different approach altogether, I tried dumping the click track and making a single audio file with the duet parts panned hard left and hard right (with the hopes of achieving a muting effect when the tracks are panned to either side). I tried to use howler.js to programmatically pan the track left or right to create the mute effect. Well, I got the panning to work, but it didn’t behave as expected because Howler.js summed the left and right channels together and then just panned the entire mix to the left or right. I couldn’t figure out how to keep this from happening. Unfortunately, since the muting effect was not achieved, this was not a workable solution (plus, I’d much rather not have to dump the click track audio).

  • Then, from another post, Mark Hunte had a very thoughtful idea of triggering the three audio files with native HTML5 Mediacontroller. Using his approach, I was finally able to get audio in sync on iOS (and be able to mute, pause, etc as well)! However, three main problems cropped up: 1) even though audio playback and sync worked on iOS, the audio wouldn’t play at all on desktop browsers (except for Safari). 2) the sync worked great when previewing on Hype Reflect, but was glitchy and often badly off when previewed as an iBook using iBooks Author. 3) The audio would only work as an mp3 when triggered with Mediacontroller, and the iBooks Author docs specify that the audio needs to be in .m4a format. When triggering m4a files with the Mediacontroller, the audio was not audible at all on any platforms. I can only get audio to work as an .m4a file when triggered with the built-in Hype actions (which, again, won’t work because then I can’t mute or pause - ugh).

If only Hype had more robust audio controls (feature request for Hype 4.0!).

In the meantime, are there any other solutions that come to mind?
Since Hype does such a great job of playing the audio back in perfect sync on all platforms, I really feel this should be possible!

I’ve included my most recent Hype file effort (that uses the Mediacontroller approach) in case it would be helpful.

Thanks again for any thoughts or input - it’s all a huge help!

duetPlayerMediaController.hype.zip (1.7 MB)

9

OK Raleigh here we go…

I am using an “audio sprite” approach. Think of image sprites where all animation occurs on one sprite “sheet”. Our sheet is a single timeline audio file with (3) different components. There is no music notation overlay in my demo, just the bare bones “audio sprite” concept on display:

BachTimeTrial_JHSv1.hype.zip (1.8 MB)

The overriding concern here is getting the iOS sync, so there is just one stream. Not quite as smooth as having (3) simultaneous streams - but I think we are close.

  1. “Both” top & bottom parts playing.
  2. Just the “Top”.
  3. Just the “Bottom”.

The “Both” part starts at 0 seconds and runs to 50 seconds (your actual track length was 48 seconds).
The “Top” part starts at 50 seconds and goes to 100 seconds.
The “Bottom” runs from 100 to 150.

The idea is that 7 seconds (for instance) on “Both” = 57 seconds on "Top; and 107 seconds on “Bottom”.

So when the “Both” section is playing clicking on the “Bottom” button jumps +100 seconds to sync with the "Both"part.

While the tracks are 50 seconds each, You can change the sync time if things are a little off. So on jumping from “Both” to “Top” you could make it a 49 second differential, or 51 seconds, instead of 50, etc. etc.

The core of the syncing code is fairly basic… “bachTrack” is the (surprise!) Bach soundtrack. The code below is for switching to the “Both” section of the sound track from either the “Top” (minus 50 seconds) or “Bottom” part of the track (minus 100 seconds).

if(whatLayer == "top") {
	bachTrack.currentTime = bachTrack.currentTime - 50;
   	}
	else if(whatLayer == "bottom") {
	bachTrack.currentTime = bachTrack.currentTime - 100;
   	}
   	
   	whatLayer = "both";

There also is an unfinished section (& commented out) that is intended to be adapted so the music notation timeline keeps sync with the sound track: 
/* bachTrack.addEventListener('timeupdate', syncTimelines);
		
	function syncTimelines() {
			audioTime = bachTrack.currentTime;
			hypeDocument.goToTimeInTimelineNamed(audioTime, 'Main Timeline');
			
			if (!bachTrack.paused) {
		    hypeDocument.continueTimelineNamed('Main Timeline');
			}
			else if (bachTrack.paused) {
			hypeDocument.pauseTimelineNamed('Main Timeline');    
			}		
		} */

**Additional Note:** I made no allowance in the code for stopping the track once it hits the end of a section not at the absolute end of the track (i.e. "Bottom") - so "Both" and "Top" will start playing the next section after they complete. You can work this out by simply checking the time... if it is at the 50 or 100 second mark pause the track (or reset to the start via a load() operation or set the current time to "0").
3 Likes
10

i can play all three sounds with howler on my iPad (reflect and safari) and they are fine in sync …

are you sure load has been comleted before playing¿

11

Wow, Jim - thanks so much!

Especially with everything going on with the fires in your area this was very kind of you to take the time.
It’s much appreciated and I hope your situation is improving out there.

I tried your demo and it does work great on all platforms! This is fantastic.
This is such a wonderful and simple approach you have going on here by making the audio sprite jump to the appropriate time by adding or subtracting the appropriate amount to ‘currentTime’. Brilliant.

Since this solution does indeed work on all platforms (including iBooks on iOS!) and since it’s foolproof when it comes to the sync issue, I’m going to give this solution a shot with my project!

However, there is one issue I noticed while testing. Could I run it by you to see if you have any thoughts?
The issue seems to happen on all platforms (in a desktop browser, Hype Reflect, and as an iBooks widget).

Here are two ways to reproduce the glitch:

  1. Once the widget is loaded, click the ‘Play’ button until a bit of the music plays. Then, click the ‘Pause’ button to pause the audio. While the audio is still paused, click the ‘Bottom’ button to select that layer. Then, click the ‘Play’ button again to start the audio playing (the bottom layer starts playing). Finally, with the bottom audio continuing to play, try tapping either the ‘Top’ button or the ‘Both’ button - a glitch happens either way. Tapping the ‘Both’ button exhibits a glitch where the audio doesn’t change at all (the bottom audio just keeps playing without switching to the ‘Both’ portion of the audio sprite), and tapping the ‘Top’ button stops the audio altogether.

  2. Here is another glitch: Upon starting the widget, click either the ‘Top’ or ‘Bottom’ button to select that layer. Then, click the ‘Play’ button to start the track. The glitch here is that the full mix plays (rather than just the top or bottom layer playing as expected). However, as the full mix plays, if another layer button is pressed, it seems to snap out if it and the buttons to choose a different layer work again as expected.

Both of these glitches are reproducible every time and they both seem to have to do with issues that crop up while trying to switching layers while the audio is not playing.

Do have any thoughts (only if you truly have the time) on what might be happening and how to make it work so that the different layers can successfully be chosen whether the audio is playing or not?

Again, thanks so much for your help with this Jim, please stay safe!

12

Hans-Gerd, thanks so much for trying this.

I’m wondering - if you still have it handy, could you share your test doc so that I can try to run it on my rig? In particular I want to see if your demo works while running in iBook with either .m4a audio or .mp4 audio files. If it does, then I need to take a close look at why howler.js was acting so glitchy for me on iOS (and iBooks).

Curiously, thinking back to when I was running tests with howler, whenever I’d reload the scene and try to play the audio again it would work flawlessly - it’s was always the first play that was frequently out of sync. This makes me think perhaps it was an issue with load not being fully complete before playing? Did you put in any protection for asset loading in your example or did it just work out of the box?

Thanks again!

13

Hi Raleigh!

RE: Glitches...

I will take #2 for now...

The demo I posted above was not fully baked with logic - it was more of a proof of concept to see if the general idea would work for You at all.

The script was set-up assuming that the "Both" section of the track would initially play. There is a variable called "whatLayer" that is used for this purpose and is initialized in the "startBach()" function. Currently the "Play" button simply plays the first part of the track (i.e. "Both") - it does not initially check to see if the "Top" or "Bottom" has been selected.

The simple fix is to check which button ("Top", "Bottom", "Both") is selected on the initial play in the "startBach()" function (using the aforementioned "whatLayer" variable) and go to that section of the track.

I will look into Glitch #1 either tonight or in the next day or so (barring an Act of God :rolling_eyes:).

If You can get the howler issue fixed that of course would be the most desirable route to go - the tracks not being fully loaded would be an explanation - especially since it works for Hans.

1 Like
14

Hey Jim,

Got it - thanks very much for pointing me in the right direction with # 2).
My javascript is good enough that I can (for the most part) implement these ideas successfully, but not yet good enough for me to be able to get creative. So this is a great learning experience for me.

Once the kids are down tonight I hope to give howler a fresh shot without any animation in order to see if I missed anything that would make it work like it did for Hans.

For instance, your demo made me realize that I hadn’t thought to try a video format for the audio. So now one thing I’m curious to try is howler along with .mp4 files (.m4a files not being audible on iOS was one of the problems I was encountering - .mp4 seems to work much better and it’s one of the few formats acceptable with iBooks)!

Regardless, thanks again for your amazing input with this.

15

This version should take care of Glitch #1 & #2 (i.e. works for me)…

Demo [here]
(http://wddtrialdemos.businesscatalyst.com/BachTimeTrial_JHSv2/BachTimeTrial_JHSv2.html).

BachTimeTrial_JHSv2.hype.zip (1.8 MB)

Basically I have moved from a quickie “is this useful?” proof of concept script into something more robust. Each function now does just one thing instead of wearing several different hats.

What’s missing: elegance.
There are three scripts that in principle do the same job: “topPlaying”, “bottomPlaying”, “bothPlaying” - they should be one (functionally there will be no difference). If You can’t get howler to work for You I will cinch this part up.

Note: I have usually found that simply changing the “.m4a” suffix to “.mp4” usually works - these formats, based on my readings, are functionally the same. Your mileage may vary… :sunglasses:

2 Likes
16

Hey Jim,

Thank you so much for your efforts with this!
Your time has been so greatly appreciated.

I feel bad telling you this because you so graciously put time into getting the BachTimeTrial to work (again, thank you!). However, I’m also so excited and grateful - because with your help (and the help of a few awesome others) I have figured out how to make the audio sync work rock solid with howler.js!!

I’ve been working with howler.js today and I’m so happy to report that I have it working FLAWLESSLY with not 3, but 6 perfectly synchronized audio files. It works in all browsers and all platforms (macOS and iOS), including iBooks on all devices! Exactly what I need.

I stripped it down to the simplest demo I could think of.
Here is how I have the working version configured:

  • In the first scene I have a single button that, upon clicking, switches to the next scene.

  • When the next scene loads (triggered with an On Scene Loads Action), I call a function that loads the sounds without playing them.

  • When the play button in the middle of the new scene is clicked, a separate function is triggered that plays the sounds.

With the sounds preloaded On Scene Load, and then button-triggered with a separate function, it works like a charm!

I included the demo here for anyone else that might come along looking for a synchronized audio playback solution (and now they actually have two different approaches, counting your working version!).

In the final version, I plan to have the music examples playable in multiple speeds for sight reading practice. So, the intro screen will present buttons for .5x, .75x or 1x speeds. Clicking one of these buttons would then load the scene that contains the audio for that chosen speed. So, it’s just nice that the format of needing at least two scenes to make it work is what I need anyway.

I am also so happy that it works so well with more than just three audio files playing back in perfect sync - this is huge for scalability to trios, quartets and more.

So, again, thanks so much for your help! Now, I’m going to download and examine your solution for the BachTimeTrial so that I can check out your solution in detail and learn from it.

All of that having been said, it sounds like the best news is that you all are getting some much needed rain in your area. Hoping the fires are all squashed and under control. All the best to you and yours!

soundSyncWithHowler(4sounds).zip (2.8 MB)

2 Likes
17

Hi Raleigh!

Don't feel bad - I got to experiment with something I'd never done. I am even more enthused that You were able to get howler.js working for You - and it will be an encouragement to others to work with this library. Congratulations!

2 Likes