Control audio time On Drag?

Hello!

I have a project that features scrolling music notation in sync with an audio file (it’s a sight-reading app that allows for the easy muting of tracks for practicing musicians).

Manipulating the scrolling music notation timeline is working great - I have a rectangle element set to control the timeline ‘On Drag’. However, the audio does not change along with the timeline.

What I need is to create a javascript function that can be triggered along with the ‘On Drag’ action that allows the scrolling music notation and the audio to stay in sync. When ‘On Drag’ is triggered (and the notation is fastforwarded or rewound), then the audio should be fast-forwarded or rewound by the same amount (relatively speaking) so that the audio and the notation always stay in sync visually/aurally.

I found this excellent Hype Forum thread (where DBear’s solution gets very close to the functionality I’m looking for). However, the implementation of the javascript code he wrote is different enough from my case that I’m not sure how I would successfully take the concept and make it work on my project (I’m using howler.js).

I’ve attached my test project on the chance that someone might have any input or ideas on how to control my audio timing and playback in order to keep my scrolling notation in sync.

Many thanks and all the best!
rg

P.S. I tried to upload the file but I’m not sure it went through. Here is a dropbox link to download the test example just in case.

2 Likes

You would run this script together with the drag action you already have.

var time = hypeDocument.currentTimeInTimelineNamed('scrollNotation/Play')
window.sound1.seek(time);
window.sound2.seek(time);
window.sound3.seek(time);

Really, it’s that simple?
Many thanks DBear - I will give this a shot!

Will post back with results.
Thank you again.

Ok, so the great news is that your suggested script worked really well.

The tricky thing is, I’m not able to get it to act as desired depending on whether on not the timeline is already playing or not. What I’m hoping at this point is that I can add a condition that says that ‘if the timeline is already playing and a drag action occurs, continue playing (both audio and timeline) from that point’. However, ‘if the timeline is paused, and a drag action occurs, then keep the audio and timeline paused’ (but make sure it’s ready to go so that a click of the play button will start both the audio and the timeline from the current time/location).

I tried to do this by adding an ‘if’ statement to the function, but I couldn’t get it to work correctly (just let me know if you want me to include a link to my failed attempt).

The other issue I tried to address (unsuccessfully) is audio garble that is audible while scrubbing. For some applications this could be a nice feature, but for my purposes I’d love to ty and minimize or eliminate it.

I tried to mute the scrubbing sound in the function before seeking the time (from your script). This worked only sort of. When the volume of the audio was down or the audio was muted, it would indeed get rid of the scrubbed audio sound, but when I tried to add the audio back after the time was seeked, the audio would sound garbled on the scrub as though it had never been muted.

Here is what I was going for below:

	    var time = hypeDocument.currentTimeInTimelineNamed('scrollNotation/Play');

//when I mute the sounds without un-muting them, the scrub click goes away (which is what I'd prefer).
//however, when I un-mute the audio later in funcion (like I try to do in the 'if statement' below), the scrubbing click is still there.
window.sound1.mute(true);
window.sound2.mute(true);
window.sound3.mute(true);

window.sound1.seek(time);
window.sound2.seek(time);
window.sound3.seek(time);

if(hypeDocument.isPlayingTimelineNamed('scrollNotation/Play') == true) {
    hypeDocument.continueTimelineNamed('scrollNotation/Play', hypeDocument.kDirectionForward);
    
    window.sound1.mute(false);
	window.sound2.mute(false);
	window.sound3.mute(false);
	console.log("playing");
} else {
	hypeDocument.pauseTimelineNamed('scrollNotation/Play');
	window.sound1.mute(false);
	window.sound2.mute(false);
	window.sound3.mute(false);
	console.log("notPlaying");
}

Any thoughts on how I might:

  1. Successfully add the script you mentioned, but with conditions so that the timeline and the audio either stays playing or stays paused?

  2. Successfully mute the audio (or turn the volume down) while the drag action is scrubbing (whether the timeline/audio is currently playing or not), and then unmute the audio (or turn the audio back up) once the drag action is done?

Thanks again DBear, your input is greatly appreciated!

The reason that

hypeDocument.isPlayingTimelineNamed(...)

doesn’t work is that when you control the timeline via on Drag you are effectively hijacking the timeline and it’s not “playing” because you are controlling what it is doing.

What you can do is listen for the start and end on the drag (Hype has the ability to listen for this event built in) and then pause and play the sounds plus timeline when these events fire. Change the code I gave you above to

	var timeline = 'scrollNotation/Play';
	
	var time = hypeDocument.currentTimeInTimelineNamed(timeline)
	
	if(event['hypeGesturePhase'] === "start"){
	
		hypeDocument.pauseTimelineNamed(timeline)
		window.sound1.pause();
		window.sound2.pause();
		window.sound3.pause();
	
	}
	
	if(event['hypeGesturePhase'] === "end"){
	
		hypeDocument.continueTimelineNamed(timeline)
		window.sound1.play();
		window.sound2.play();
		window.sound3.play();
	
	}

	window.sound1.seek(time);
	window.sound2.seek(time);
	window.sound3.seek(time);

Note a few little tidy ups (always good to keep code DRY) and I’ve also added

hypeDocument.pauseTimelineNamed(timeline)

in the start event but that is not needed (explained above) but I left it in for educational purposes. :wink:

2 Likes

Hi DBear,

Thanks so much for this!
That sure does explain why hypeDocument.isPlayingTimelineNamed(...) was not working for me. Makes sense now!

I added your code and that took care of the audio scrub sound issue wonderfully!
When the timeline IS playing and I try to control the timeline On Drag, it works absolutely perfectly! (I appreciate your tidying the code as well, thanks for the refactoring example!).

However, what I’m finding is that if the timeline is NOT playing, and then I try to drag the timeline, unwanted behavior occurs: both audio playback and timeline continuation is triggered when the event['hypeGesturePhase'] === "end" is detected, whereas the timeline and the audio should both stay paused (but at the new location) and ready to play with the play button. This makes total sense that this would be happening, but I’m at a loss as to how I would make this part of it work since triggering the On Drag nullifies the playback status of the timeline!

It still seems like I’m going to need to somehow inject an if statement that determines whether or not the timeline was playing, but it seems this would need to be indicated before the drag event is even triggered.

Perhaps would it work to make a global variable that can be set to something like isPlaying or notPlaying when the timeline is manipulated (started, paused, or continued) outside of the function? That way, the global variable could be checked and reset inside the function rather than checking for hypeDocument.isPlayingTimelineNamed(...).

If this approach might work, how do I make a global variable in Hype - attach it to window.(...)?
Or, is there a better way altogether?

Overall, now that the sound issue is wonderfully taken care of, do you have any thoughts on how to check the status of the timeline (and get around the On Drag hijacking issue)?

Again, thanks for your input with this DBear.
Excited to get this working and also learning a ton from you as my Javascript (especially when applied to Hype) is still in the developmental stages. :slight_smile:

I would (as you say) create a global with

window["isPlaying"] = true; // or
window.isPlaying = true;

and probably set this when the user clicks the play button … so “play” sets the variable to true and “pause” sets it to false. And, then you can check against that with a conditional statement.

So, logic is:

  1. user clicks “play” button and the timeline + audio starts and the variable is set to true.
  2. user then drags the handle and the drag function kicks in with the added logic that …
    if the timeline is playing (isPlaying = true) then it will function as it is but if the timeline is not playing (!isPlaying) then the condition will leave the state as “pause”.
    if(isPlaying){
        hypeDocument.continueTimelineNamed(timeline)
        window.sound1.play();
        window.sound2.play();
        window.sound3.play();
    }
    

You would want to probably set the variable in more than one function based on your interactions with said “buttons”.

Notice there is no “else” above as the state on drag “pauses” everything anyway so it will just stay that way. But, the button states may need adjusting. I’ll leave it with you. :wink:

1 Like

Wonderful!
Thanks so much DBear - I will give this a shot!

1 Like

Hi DBear,

Thanks to your pointing me in the right direction, I now have it working perfectly!
I’m now very happy to have this new feature working (and even happier to have learned a ton about integrating javascript into Hype).
Hope you have a great week.
All the best and thanks again!

1 Like

You’re welcome!

1 Like

Hi Raleigh, I came across this thread looking for a scrubbing & scrolling component for Howler, I was wondering if you would be so kind as to share your working code so that I don’t have to reinvent the wheel? :slight_smile:

Hi Jason,

Sure, I’d be happy to share my code. Also got your email - many thanks! I’ll try to send you an email tonight. :slight_smile: