Stop or continue Hype project audio on iOS lock screen in Xcode

@MarkHunte I assumed it self-explanatory due to my ignorance when stating that all my audio is triggered and played from within the HTML folder of the Hype project wrapped inside Xcode. I assume it’s thus HTML Audio? I’m an eager learner, so feel free to correct me.

Lot to read… :smile:

Yes it could be or you used some of my control examples to control the app.

I have not played with Audio for a little while so would need to refresh my Memory.

But do you have background mode turned on in the Xcode project in capabilities.

If so what do you get if you turn it off.

@MarkHunte I have Background Mode turned off yet STILL get all the functions of background audio. That’s one of the amazing parts of why I thought I’d share this; I am not telling the app to play audio on the home screen, in another app, or on a locked screen, yet it DOES do that. Just like a radio app.

I will try and dig out some of my old projects and see.

But you should be able to use some of my applicationDidEnterBackground examples. But again I would need to refresh my memory…

Thanks, @MarkHunte. I can send you the Xcode project so you can just dig around. It’s quite bizarre to see small audio loops being treated like a “radio station” on your iPhone…

I can get it to stop the sound (pause ) using the enters background but it still displays the Control centre.

That’s about as far as I got with killing the entire app using UIApplicationExistsOnSuspend; The audio still sat in the control center. A zombie, so to speak…

So my thoughts are : this does suck.

I have had mixed results with audio/ wkwebview and Xcode before when it comes to the command controls etc.
And I wanted background sound ( playing a stream radio ) I also had issues with resume after call. ( still do )

So I have come to the conclusion that playing a sound like how you want is best done via something like the AVAudioPlayer (iOS)


I just did an experiment that when I tap the play button a message is posted to the iOS app.

The message body is the current **string ** for the audio file.

var audioPlayer_ = document.getElementById('audioPlayer')
	
window.audiosrc = audioPlayer_.querySelector('source').src;
 window.webkit.messageHandlers.helloWorld.postMessage(window.audiosrc);

I pick this message up in the app and have it play the file via AVAudioPlayer .
This means I probably can get more control over the media command centre display etc.

In the testing. (So far)

The sound plays
Exiting the app sound stops
Going to lock screen the sound fades out,stops and the media commands fade out and disappear


Some code snippets.

In the app delegate.

import AVFoundation
 import MediaPlayer

UIApplication.shared.beginReceivingRemoteControlEvents()
            do {
                try AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryPlayback)
                print("Playback OK")
                try AVAudioSession.sharedInstance().setActive(true)
                print("Session is Active")
            } catch {
                print(error)
            }
        
        let commandCenter = MPRemoteCommandCenter.shared()
        commandCenter.playCommand.isEnabled = false
        
        commandCenter.togglePlayPauseCommand.isEnabled = false
        
        commandCenter.stopCommand.isEnabled = false
        
        
        //commandCenter.pauseCommand.isEnabled = false
        commandCenter.pauseCommand.addTarget(self, action: #selector(donothing))
        
        commandCenter.seekForwardCommand.isEnabled = false
        commandCenter.seekBackwardCommand.isEnabled = false
        commandCenter.nextTrackCommand.isEnabled = false
        commandCenter.previousTrackCommand.isEnabled = false

wkwebview file

class WKWebViewController: UIViewController, WKNavigationDelegate  ,WKScriptMessageHandler {
   var audioPlayer = AVAudioPlayer()
   var anURL :URL? = nil

... 

...

    private func enterBackground(notification:Notification) -> Void {
                print("call 1")
                
              webView!.evaluateJavaScript("HYPE.documents['souncheck'].functions().playStopAudioAnimation()", completionHandler:nil )
                audioPlayer.stop()
            }

   func userContentController(_ userContentController: WKUserContentController, didReceive message: WKScriptMessage) {
        
        
        
        
        //-- Posted messages  from Hype page
     if message.name == "prepplay"{
            
        
          anURL  = URL.init(string: message.body as! String)!
        
        do {
             audioPlayer =   try  AVAudioPlayer.init(contentsOf: anURL!)
            audioPlayer.prepareToPlay()
            audioPlayer.play()
        } catch {
            
            // catch error
            
        }
       }
        
        
        
    }

override func viewDidLoad() {
        
        super.viewDidLoad()
      
        //-- configure the WKWebView
        
        //-- init config and controller
        let wconfiguration = WKWebViewConfiguration()
        let wcontroller = WKUserContentController()
        
        //-- We must add the webkit scripts Posted messages we expect to get from the Hype Page to the controller
        wcontroller.add(self, name: "prepplay")

bottom of viewDidload

      NotificationCenter.default.addObserver(forName:Notification.Name(rawValue:"UIApplicationWillResignActiveNotification"),
                                                   object:nil, queue:nil,
                                                   using:enterBackground)

Note this code is not checked for bad syntax in relation to swift options unwrap/nil etc…

Hype project used with corresponding code
souncheck.hype.zip (167.4 KB)

2 Likes

I don’t know a solution, but I’d try giving the page visibility api a try for a time to pause/stop the audio.

I will have a look at that. but initially this looks good for another way to pause the audio.
But I suspect since the audio is still registered on a html audio the controls would still show up.

Here's the code I added to Circles With Grandma to solve that problem...

// Look at me when I'm talking to you!
function visible() {
 if (document.hidden) {
  speechSynthesis.cancel();
 }
}

document.addEventListener("visibilitychange", visible);

In that example, I'm cancelling the synthesized speech, but that area could be used to run all sorts of code... such as pausing the audio, pausing the game and/or changing the animation.

Howler.js might also be helpful, as it has advanced sound controls.

Yes you do. It was that.

Heh, neat. Someone's actually using the app. Is it working out for you?

2 Likes

Thanks @MarkHunte! Very thorough and concise.

I am, however, starting to doubt my decision to use wrapped html5 for this project because your workaround illustrates perfectly that Hype’s core product is not intended for what I am trying to use it right now. That’s a good thing: Nothing worse than a product that tries to be everything for everyone.

Thanks, @Photics. Where exactly are you adding this code? For every audio event or is your example solely using audio for speech synthesis?

Yes, I love your Wrapping App.
Even though Xcode has improved tons since I last touched it 10 years ago it just doesn't work right with my head: I am fluent in 7 languages but you can bring me to tears with a single line of code. I'm just not that guy...

I think they can work together well just some situations are not as straight forward as we would like or expect.

I personally feel the sound control issue in the lock screen is more limited by Apple.
I have had to do the opposite of what you want as in needing the controls to show up and with my own info.

I ran into some issues piping through hype but this was more how iOS coped with streaming sound. ( Sound from a radio station not a file ) The iOS control swift code will be ignored when it is not going through the iOS AVPlayer/ Session )

So I then went down the route of implementing the stream in iOS/Swift Audio tools only and have the app control the animations of the hype scenes using the events etc..

Most of that worked but the big issue was again the control centre.
A simple iOS audio setup for streaming is not simple at all. I did manage to get it working but things like resume after call just do not work as they should. Apple even say that this behaviour is not guaranteed.
So all my sound being run by iOS was just as bothersome as being run by Hype but in slightly different ways.


@Photics code would normally only need to run once. So I would add it on scene load ( it goes in a Hype function )

2 Likes

So just got a moment to run @Photics code in a iOS app.

It works in so far as pausing the sound when the App is no longer visible. i.e lock screen.

And this may be a very good way of updating the interface/animations when hidden or visible in an iOS app
But as I suspected, iOS still detects that the sound is/has been coming from html and pops up the control commands when going into lock screen.


So I wondered if removing the source or the audio node would stop iOS seeing the audio on lock screen.
Nope seems that the WKWebView cache the data.

I even tested in Desktop Safari. by removing the Audio node. Toggled tabs to trigger the visibilitychange.
In the DOM I could see the Audio node had gone but the play button still played the audio…

1 Like

Great findings, @MarkHunte. One of the reasons I brought up this issue was that I fear Apple rejecting apps with persistent audio despite not declaring so in Capabilities / Backround Mode.

Let me make this experiment and submit my app later this week as-is and see if they reject it and make explicit reference to the html audio sticking around in ugly and annoying ways or if they’re happy with a sloppy user experience (insert your own “Apple in 2018-joke” here).

1 Like

Cool, let me know what they say…

Also a Tip.

When you load your app directly to your iPhone ( actual device by cable ) from Xcode ,

You can then go to Safari Developer Menu on the Desktop, select your phone and the iOS app’s web page to bring up it’s web inspectors. Very, very useful to say the least.

Also in cases like going to lock screen like this the Console errors may not update visibly with any messages that got generated during the transition. But the console error counter badge at the top may. So just click on that and it will the display the error.

That's right. I only load it once, as it's an event listener. It's loaded when the title screen is loaded, via the custom "home" function.

10%20PM

Here's the breakdown of the code...

document.addEventListener("visibilitychange", visible);

"document" means the whole HTML document. This section lets you be specific, by adding event listeners to specific elements in the document, but in this example it's an event listener for the whole document.

"addEventListener" ... this is a JavaScript thing. You're saying... "Hey, listen!"

"visibilitychange" ...this is the event you're listening for. It's a predetermined thing in JavaScript. There are many different types. Fortunately, there's one for determining if the page is visible.

"visible" ...This is the name of the function that is run whenever there is a change in visibility. It's a name I gave to the function. I could have called it Potato or Cauliflower, it doesn't matter. The point is that you'll need a function to call when there's a change in visibility.

Here's a line-by-line breakdown of the function...

// Look at me when I'm talking to you!

The double-slash means this is a comment. It's a note to yourself, or anyone else working with the code, as it can be a descriptive message as to what this part of the code does. I'm notorious for leaving goofy comments in my code, even use of emoji. Happy World Emoji Day. :crazy_face:

function visible() {

Here's how a function is created. Notice how it's called "visible". This is the function that the event listener is going to run whenever the "visibilitychange" event occurs.

 if (document.hidden) {

Here I'm using the read-only document property to see if the page is actually hidden. Document: hidden property - Web APIs | MDN ...but it could be reversed... if(!document.hidden) ...the exclamation point means "not". So if the document is not hidden, then do something. Since I only wanted to turn the speech off when the document is hidden, I didn't need to add other code.

  speechSynthesis.cancel();

That's how I stop the voice sound effects. If I had background music, or other sound effects, then I could add additional code here. Like if I was using Howler.js... maybe I could fade out the sound. That's theoretical though. I have had significant trouble getting sound to work properly in my apps.

Ha ha great. Maybe leave a review to let other people know. :man_shrugging:t2:
The more popular the app, the more likely it gets updated.

The new App Store Connect app update lets me see all the reviews for the app, from all the countries its available. So I've been looking to see if anyone has been reviewing it.

1 Like

Oooh. I love that. Sure beats my current way of sending each binary to Testflight, then being puzzled after a long wait for Apple's servers to make it available...

Thanks @Photics for the blow-by-blow breakdown.
So I could use Mozilla's (experimental) cancelAndHoldAtTime() function (link) to stop all sounds and in an ideal world even pick up where we left off upon change to visible...

Though, all that goodness still won't eliminate the control center displaying the last played audio file the way @MarkHunte managed to correct by letting the AVAudioPlayer (iOS) handle audio.

1 Like

Apple just approved my iOS app despite the violation of background audio declaration.

I hope this helps anyone using Hype to create iOS apps without wanting to start coding around Mozilla’s and iOSs weird tango of HTML audio handling.

@MarkHunte and @Photics: Thanks for the generous help and insights.
Yes, @Photics, I left a gushing review in the Mac App Store for your app “Wrapping”.

Now I’ll go and improve the app’s user experience by using @MarkHunte’s code above. Apple doesn’t require it nor will most users ever be put off by the persistent audio but you may as well do things right…

3 Likes