Google giveth, and Google taketh away. Every new Android release introduces a lot of new APIs that either improve the functionality of existing apps or open up new categories of apps. On the other hand, Google also removes access to or restricts the capabilities of existing APIs to improve the security of the Android platform. The Android Q release is no different, and we’ve already talked a lot about its changes to shared storage, clipboard access, notifications, and other APIs. At this year’s Google I/O developer conference, Google announced new APIs in Android Q that are especially interesting for users and mobile game developers: the AudioPlaybackCapture and Thermal APIs.
Recording The Audio From Other Apps With AudioPlaybackCapture in Android Q
Mobile gaming is a huge industry with no signs of slowing down anytime soon, especially in markets like the U.S., China, and India. With more people spending time on their smartphones for gaming and social media, it’s no surprise that people want to share their gaming experiences online.
Android has supported taking screenshots since basically the very beginning, but capturing videos of the screen has only been supported since Android 5.0 Lollipop with the introduction of the MediaProjection API. Sadly, this API does not allow third-party apps to record the internal audio output in other apps, forcing third-party screen recorders to record external audio output from the device’s microphone(s). OEMs like Samsung and Huaweiprovide system apps to record the internal audio output during a screen recording, but the only way for owners of other devices to properly record the video and audio output from their devices is to buy a third-party accessory. Fortunately, Google is finally introducing a way in Android Q for third-party apps to record the audio from other apps.
The AudioPlaybackCapture API is described by Google as the analog of screen capture, but for audio recording. Although the only app so far to use the API is Google’s Live Caption, an accessibility tool that adds captions to any video being played on the device, the API’s documentation states that its primary purpose is to allow streaming apps to capture the audio being played by games. Think mobile gamers streaming their sessions onto services like Twitch or YouTube. The API is designed to not affect the audio latency of the app whose audio is being captured, making it perfect for gameplay recording so the audio and video don’t get desynced.
Unlike in the early days of Android, Google’s approach to new APIs is more cautious. AudioPlaybackCapture can only be used if both the app whose audio is being captured and the app doing the capturing meet specific requirements. Here’s a summary of those requirements:
- For an app’s audio to be captured by a third-party app, the following requirements must be met:
- The app must have allowAudioPlaybackCapture either set to true or unset in their Manifest. (It defaults to true for apps targeting Android Q, but can be manually set for apps targeting Android Pie.)
- The app producing audio must have its audio usage of type USAGE_MEDIA, USAGE_GAME, or USAGE_UNKNOWN.
- The audio capture policy (setAllowedCapturePolicy) must be set to ALLOW_CAPTURE_BY_ALL. This policy can be set at runtime, but the playback must be restarted for the policy change to take effect. If ALLOW_CAPTURE_BY_SYSTEM is set, then only system apps can record the audio (but are restricted in that they can’t save the audio, pass it to a third-party app, or record at > 16kHz 16-bit mono quality.)
- For an app to record audio from another app, the following requirements must be met:
- The app must bring up the MediaProjectionManager.createScreenCaptureIntent prompt and the user must accept it.
- The app must hold the RECORD_AUDIO permission.
- The app must be in the same user profile as the app whose audio will be recorded.
I would imagine that most games will allow their audio to be captured by apps using this new API, but it may take some time for games to get updated to support it. On the other hand, don’t expect any music or video streaming apps to support this API for third-parties because of issues with licensing. However, the limitations that Google has imposed if an app sets its audio capture policy to ALLOW_CAPTURE_BY_SYSTEM means that developers of music and video streaming apps should be okay with enabling audio capture by the system to improve accessibility.
To read more about the AudioPlaybackCapture API in Android Q, check out this page.
Monitoring Throttling With Thermal API in Android Q
Thermal throttling is almost entirely managed by the system with no input from, or even notification to, the apps that are most affected by throttling. If the device’s resources are too strained, most devices will throttle the CPU and GPU speeds to the detriment of your game’s performance. There’s nothing you, as a game developer, can really do about it except warn the user before they change any settings about the potential impact their changes may have on performance.
With the new Thermal API in Android Q, however, apps can receive callbacks for different stages of performance throttling – giving developers the chance to help reduce the strain on the device by ramping down on activities that require significant CPU, GPU, or modem usage. For example, a game could dynamically reduce the resolution until the device is no longer GPU throttling, or a Maps app could disable enhanced features like augmented reality navigation to reduce the load on the CPU.
To use this new API, apps register a listener in PowerManager (addThermalStatusListener) and the system sends the app one of the following thermal status codes:
- THERMAL_STATUS_NONE: “Not under throttling.”
- THERMAL_STATUS_LIGHT: “Light throttling where UX is not impacted.”
- THERMAL_STATUS_MODERATE: “Moderate throttling where UX is not largely impacted.”
- THERMAL_STATUS_SEVERE: “Severe throttling where UX is largely impacted.”
- THERMAL_STATUS_CRITICAL: “Platform has done everything to reduce power.”
- THERMAL_STATUS_EMERGENCY: “Key components in platform are shutting down due to thermal condition. Device functionalities will be limited.”
- THERMAL_STATUS_SHUTDOWN: “Need shutdown immediately.
Support for the Thermal API obviously requires Android Q, but another requirement is the addition of a new HAL. Google says the Pixel devices on Android Q (meaning the Pixel, Pixel XL, Pixel 2, Pixel 2 XL, Pixel 3, Pixel 3 XL, Pixel 3a, and Pixel 3a XL) support the Thermal API, but Google is working on adding support for other devices.