Stop or continue Hype project audio on iOS lock screen in Xcode

backpalm · July 16, 2018, 11:45am

First time poster, long time lurker/learner here. I love this forum and its community. The wealth of sharing, knowledge, and civility is exemplary.

I discovered a few audio goodies after wrapping my Hype project in an Xcode webView for adhoc iOS distribution, using @Photics wrapping app. Some of the insights might be new and of help to the inquisitive mind.

A wrapped Hype project on iOS will:

Continue audio when the app goes into background (such as switching to another app or home screen).
Continue audio when switching to lock screen (“switching off” the phone).
- Play audio without prior touch event (on scene/layout load for example). This last one isn’t new to this Forum but so refreshing to experience live after experimenting solely in mobile Safari.

These 3 audio behaviors are already well worth the hassle of wrapping your project in Xcode if your project is mobile-mainly and/or you have a sound based app in mind.
From what I understand “Audio on lock screen” is also one of the most searched for audio related topics in Xcode centric posts elsewhere.

There IS a downside to this.
Apple will bemoan that behavior if you intend to submit your app to the App Store because you’re not using the official Background Mode in Xcode’s Capabilities tab. You’re not officially declaring that your app’s audio will persist (such as a music player or VOIP app).

Additionally your audio file will be represented by a confusing and buggy looking audio player in the tray on your locked screen (see image).

It becomes critical if you DON’T want audio to persist after the app goes off-focus. For example, in a game with sound effects. The last audio file played in your Hype project (say a “boing” or a “whoosh”) will persist on a user’s lock screen. This turns into a nightmare (or hilarity) with looped sound effects (for a walk-cycle for instance).

I’d like to kick off this thread for people interested in keeping audio persistent LEGALLY in iOS and folks wanting their audio to gracefully exit upon the app moving to the background (such as me).

Apple’s documentation about sticking to the rules and programmatically handling audio in those cases is here:

Apple Documentation on Background State

Apple Documentation on Persistent Audio

Apple Documentation on Activating and Deactivating Audio Sessions

As for me: I’m trying to stop all audio if the app is not in the foreground. I tried to simply “kill” the app if not in foreground by using UIApplicationExistsOnSuspend “NO” in info.plist but while that method stops the sounds it also resets Hype’s timelines and the last played audio file STILL sits in the device’s buffer (see screenshot).

I KNOW it has to be somewhere in the AppDelegate.swift inside func applicationDidEnterBackground but I don’t know how to declare which player to stop.

@MarkHunte has some brilliant insights in related posts regarding callbacks between Xcode projects and Hype 3 projects: Get Control of & feedback from Webpage Or iOS/Mac App using the (newish) WKWebView

Could that be close? I’m not trying to control one specific audio file. Just all audio used in the Hype 3 project embedded in Xcode.

Can anyone help? Thanks!

MarkHunte · July 16, 2018, 12:07pm

You have not said how your audio is actually playing. I.e via the AVAudioSession or through HTML Audio.

backpalm · July 16, 2018, 12:11pm

@MarkHunte I assumed it self-explanatory due to my ignorance when stating that all my audio is triggered and played from within the HTML folder of the Hype project wrapped inside Xcode. I assume it’s thus HTML Audio? I’m an eager learner, so feel free to correct me.

MarkHunte · July 16, 2018, 12:15pm

Lot to read…

Yes it could be or you used some of my control examples to control the app.

I have not played with Audio for a little while so would need to refresh my Memory.

But do you have background mode turned on in the Xcode project in capabilities.

If so what do you get if you turn it off.

backpalm · July 16, 2018, 12:21pm

@MarkHunte I have Background Mode turned off yet STILL get all the functions of background audio. That’s one of the amazing parts of why I thought I’d share this; I am not telling the app to play audio on the home screen, in another app, or on a locked screen, yet it DOES do that. Just like a radio app.

MarkHunte · July 16, 2018, 12:32pm

I will try and dig out some of my old projects and see.

But you should be able to use some of my applicationDidEnterBackground examples. But again I would need to refresh my memory…

backpalm · July 16, 2018, 12:40pm

Thanks, @MarkHunte. I can send you the Xcode project so you can just dig around. It’s quite bizarre to see small audio loops being treated like a “radio station” on your iPhone…

MarkHunte · July 16, 2018, 1:50pm

I can get it to stop the sound (pause ) using the enters background but it still displays the Control centre.

backpalm · July 16, 2018, 2:09pm

That’s about as far as I got with killing the entire app using UIApplicationExistsOnSuspend; The audio still sat in the control center. A zombie, so to speak…

MarkHunte · July 16, 2018, 4:22pm

So my thoughts are : this does suck.

I have had mixed results with audio/ wkwebview and Xcode before when it comes to the command controls etc.
And I wanted background sound ( playing a stream radio ) I also had issues with resume after call. ( still do )

So I have come to the conclusion that playing a sound like how you want is best done via something like the AVAudioPlayer (iOS)

I just did an experiment that when I tap the play button a message is posted to the iOS app.

The message body is the current **string ** for the audio file.

var audioPlayer_ = document.getElementById('audioPlayer')
	
window.audiosrc = audioPlayer_.querySelector('source').src;
 window.webkit.messageHandlers.helloWorld.postMessage(window.audiosrc);

I pick this message up in the app and have it play the file via AVAudioPlayer .
This means I probably can get more control over the media command centre display etc.

In the testing. (So far)

The sound plays
Exiting the app sound stops
Going to lock screen the sound fades out,stops and the media commands fade out and disappear

Some code snippets.

In the app delegate.

import AVFoundation
 import MediaPlayer

UIApplication.shared.beginReceivingRemoteControlEvents()
            do {
                try AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryPlayback)
                print("Playback OK")
                try AVAudioSession.sharedInstance().setActive(true)
                print("Session is Active")
            } catch {
                print(error)
            }
        
        let commandCenter = MPRemoteCommandCenter.shared()
        commandCenter.playCommand.isEnabled = false
        
        commandCenter.togglePlayPauseCommand.isEnabled = false
        
        commandCenter.stopCommand.isEnabled = false
        
        
        //commandCenter.pauseCommand.isEnabled = false
        commandCenter.pauseCommand.addTarget(self, action: #selector(donothing))
        
        commandCenter.seekForwardCommand.isEnabled = false
        commandCenter.seekBackwardCommand.isEnabled = false
        commandCenter.nextTrackCommand.isEnabled = false
        commandCenter.previousTrackCommand.isEnabled = false

wkwebview file

class WKWebViewController: UIViewController, WKNavigationDelegate  ,WKScriptMessageHandler {
   var audioPlayer = AVAudioPlayer()
   var anURL :URL? = nil

...

–

...

    private func enterBackground(notification:Notification) -> Void {
                print("call 1")
                
              webView!.evaluateJavaScript("HYPE.documents['souncheck'].functions().playStopAudioAnimation()", completionHandler:nil )
                audioPlayer.stop()
            }

   func userContentController(_ userContentController: WKUserContentController, didReceive message: WKScriptMessage) {
        
        
        
        
        //-- Posted messages  from Hype page
     if message.name == "prepplay"{
            
        
          anURL  = URL.init(string: message.body as! String)!
        
        do {
             audioPlayer =   try  AVAudioPlayer.init(contentsOf: anURL!)
            audioPlayer.prepareToPlay()
            audioPlayer.play()
        } catch {
            
            // catch error
            
        }
       }
        
        
        
    }

override func viewDidLoad() {
        
        super.viewDidLoad()
      
        //-- configure the WKWebView
        
        //-- init config and controller
        let wconfiguration = WKWebViewConfiguration()
        let wcontroller = WKUserContentController()
        
        //-- We must add the webkit scripts Posted messages we expect to get from the Hype Page to the controller
        wcontroller.add(self, name: "prepplay")

bottom of viewDidload

      NotificationCenter.default.addObserver(forName:Notification.Name(rawValue:"UIApplicationWillResignActiveNotification"),
                                                   object:nil, queue:nil,
                                                   using:enterBackground)

Note this code is not checked for bad syntax in relation to swift options unwrap/nil etc…

Hype project used with corresponding code
souncheck.hype.zip (167.4 KB)

jonathan · July 16, 2018, 4:30pm

I don’t know a solution, but I’d try giving the page visibility api a try for a time to pause/stop the audio.

MarkHunte · July 16, 2018, 5:06pm

I will have a look at that. but initially this looks good for another way to pause the audio.
But I suspect since the audio is still registered on a html audio the controls would still show up.

Photics · July 16, 2018, 9:33pm

Here's the code I added to Circles With Grandma to solve that problem...

// Look at me when I'm talking to you!
function visible() {
 if (document.hidden) {
  speechSynthesis.cancel();
 }
}

document.addEventListener("visibilitychange", visible);

In that example, I'm cancelling the synthesized speech, but that area could be used to run all sorts of code... such as pausing the audio, pausing the game and/or changing the animation.

Howler.js might also be helpful, as it has advanced sound controls.

Yes you do. It was that.

Heh, neat. Someone's actually using the app. Is it working out for you?

backpalm · July 17, 2018, 1:55pm

Thanks @MarkHunte! Very thorough and concise.

I am, however, starting to doubt my decision to use wrapped html5 for this project because your workaround illustrates perfectly that Hype’s core product is not intended for what I am trying to use it right now. That’s a good thing: Nothing worse than a product that tries to be everything for everyone.

backpalm · July 17, 2018, 2:03pm

Thanks, @Photics. Where exactly are you adding this code? For every audio event or is your example solely using audio for speech synthesis?

Yes, I love your Wrapping App.
Even though Xcode has improved tons since I last touched it 10 years ago it just doesn't work right with my head: I am fluent in 7 languages but you can bring me to tears with a single line of code. I'm just not that guy...

MarkHunte · July 17, 2018, 2:31pm

I think they can work together well just some situations are not as straight forward as we would like or expect.

I personally feel the sound control issue in the lock screen is more limited by Apple.
I have had to do the opposite of what you want as in needing the controls to show up and with my own info.

I ran into some issues piping through hype but this was more how iOS coped with streaming sound. ( Sound from a radio station not a file ) The iOS control swift code will be ignored when it is not going through the iOS AVPlayer/ Session )

So I then went down the route of implementing the stream in iOS/Swift Audio tools only and have the app control the animations of the hype scenes using the events etc..

Most of that worked but the big issue was again the control centre.
A simple iOS audio setup for streaming is not simple at all. I did manage to get it working but things like resume after call just do not work as they should. Apple even say that this behaviour is not guaranteed.
So all my sound being run by iOS was just as bothersome as being run by Hype but in slightly different ways.

@Photics code would normally only need to run once. So I would add it on scene load ( it goes in a Hype function )

MarkHunte · July 17, 2018, 5:18pm

So just got a moment to run @Photics code in a iOS app.

It works in so far as pausing the sound when the App is no longer visible. i.e lock screen.

And this may be a very good way of updating the interface/animations when hidden or visible in an iOS app
But as I suspected, iOS still detects that the sound is/has been coming from html and pops up the control commands when going into lock screen.

So I wondered if removing the source or the audio node would stop iOS seeing the audio on lock screen.
Nope seems that the WKWebView cache the data.

I even tested in Desktop Safari. by removing the Audio node. Toggled tabs to trigger the visibilitychange.
In the DOM I could see the Audio node had gone but the play button still played the audio…

backpalm · July 17, 2018, 5:45pm

Great findings, @MarkHunte. One of the reasons I brought up this issue was that I fear Apple rejecting apps with persistent audio despite not declaring so in Capabilities / Backround Mode.

Let me make this experiment and submit my app later this week as-is and see if they reject it and make explicit reference to the html audio sticking around in ugly and annoying ways or if they’re happy with a sloppy user experience (insert your own “Apple in 2018-joke” here).

MarkHunte · July 17, 2018, 5:53pm

Cool, let me know what they say…

Also a Tip.

When you load your app directly to your iPhone ( actual device by cable ) from Xcode ,

You can then go to Safari Developer Menu on the Desktop, select your phone and the iOS app’s web page to bring up it’s web inspectors. Very, very useful to say the least.

Also in cases like going to lock screen like this the Console errors may not update visibly with any messages that got generated during the transition. But the console error counter badge at the top may. So just click on that and it will the display the error.

Photics · July 17, 2018, 10:19pm

That's right. I only load it once, as it's an event listener. It's loaded when the title screen is loaded, via the custom "home" function.

10%20PM

Here's the breakdown of the code...

document.addEventListener("visibilitychange", visible);

"document" means the whole HTML document. This section lets you be specific, by adding event listeners to specific elements in the document, but in this example it's an event listener for the whole document.

"addEventListener" ... this is a JavaScript thing. You're saying... "Hey, listen!"

"visibilitychange" ...this is the event you're listening for. It's a predetermined thing in JavaScript. There are many different types. Fortunately, there's one for determining if the page is visible.

"visible" ...This is the name of the function that is run whenever there is a change in visibility. It's a name I gave to the function. I could have called it Potato or Cauliflower, it doesn't matter. The point is that you'll need a function to call when there's a change in visibility.

Here's a line-by-line breakdown of the function...

// Look at me when I'm talking to you!

The double-slash means this is a comment. It's a note to yourself, or anyone else working with the code, as it can be a descriptive message as to what this part of the code does. I'm notorious for leaving goofy comments in my code, even use of emoji. Happy World Emoji Day.

function visible() {

Here's how a function is created. Notice how it's called "visible". This is the function that the event listener is going to run whenever the "visibilitychange" event occurs.

 if (document.hidden) {

Here I'm using the read-only document property to see if the page is actually hidden. Document: hidden property - Web APIs | MDN ...but it could be reversed... if(!document.hidden) ...the exclamation point means "not". So if the document is not hidden, then do something. Since I only wanted to turn the speech off when the document is hidden, I didn't need to add other code.

  speechSynthesis.cancel();

That's how I stop the voice sound effects. If I had background music, or other sound effects, then I could add additional code here. Like if I was using Howler.js... maybe I could fade out the sound. That's theoretical though. I have had significant trouble getting sound to work properly in my apps.

Ha ha great. Maybe leave a review to let other people know.
The more popular the app, the more likely it gets updated.

The new App Store Connect app update lets me see all the reviews for the app, from all the countries its available. So I've been looking to see if anyone has been reviewing it.