Add System audio loopback on Mac using Screen Capture Kit API

[jkarthic]

Mac's CoreAudio API doesn't support capture of system audio. But the newly introduced ScreenCaptureKit API support capturing of system audio. This will require the user to provide screen sharing permissions though.

This is the initial version that supports only mono capture and blocking APIs.
But it can be easily extended to support other features as well.

[fightingforyou46]

I have successfully built and replaced libportaudio.2.dylib into my directory
sounddevice/portaudio/ and renamed it to libportaudio.dylib. After checking, there was no loopback device added to the audio device list

[jkarthic]

I have successfully built and replaced libportaudio.2.dylib into my directory sounddevice/portaudio/ and renamed it to libportaudio.dylib. After checking, there was no loopback device added to the audio device list

@fightingforyou46
I tested with the following standard commands and it lists the loopback device for me.

git clone https://github.com/get-wrecked/portaudio/
cd portaudio
git checkout mac_screen_capture_kit
mkdir build
cd build
cmake .. -DPA_BUILD_TESTS=ON
cmake --build . --config Release
test/patest_enumerate_default_latency

System Audio [Loopback] is the name of this new device.

Could you please try the above steps just to rule out any issues with your test or build setup.
Also this Screen Capture Kit API will need MacOS 13.0 or above.

[fightingforyou46]

I saw it like this:

|_. Bad Default Latency? |_. Device Number |_. I/O Channels |_. Device Name |_. Host API |_. Default High Input Latency |_. Default Low Input Latency |_. Default High Output Latency |_. Default Low Output latency |
|   |   0 | 2/0 |  == Built-in Line Input == | Core Audio |   0.0141 |   0.0039 |   0.1000 |   0.0100 |
|   |   1 | 2/0 |  == Built-in Digital Input == | Core Audio |   0.0130 |   0.0037 |   0.1000 |   0.0100 |
|   |   2 | 0/2 |  == Built-in Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
|   |   3 | 0/2 |  == Built-in Line Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
|   |   4 | 0/2 |  == Built-in Digital Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
| X |   5 | 1/0 |  == System Audio [Loopback] == | Mac ScreenCaptureKit |   0.0000 |   0.0000 |   0.0000 |   0.0000 |

and I used it , I got this error: Error opening InputStream: Can't write to a callback stream [PaErrorCode -9976]

[jkarthic]

I got this error: Error opening InputStream: Can't write to a callback stream [PaErrorCode -9976]

So, I see the device is being listed correctly.
Which app are you using to test audio capture? Could you please provide a sample app code, with which you are seeing this error. Just to clarify, this is a output stream(similar to a Mic stream) and not a input stream. You will have to read from it, and not write into it(as your error suggests). Also please note that this doesn't yet support callback(as mentioned in the PR description). You will have to use the blocking API a.k.a., Pa_ReadStream.

[fightingforyou46]

I use it with python sounddevice. I just replace libportaudio.dylib in sounddevice/portaudio_binary/ then use it with my code

[RossBencina]

Thanks for your contribution. This seems to be at the proof-of-concept stage. Are you planning on fleshing this out and maintaining it? For now I'm marking this as draft. We're not going to merge this until it is more complete. The following issues are apparent:

1. The CI is failing
2. No PortAudio callback API support
3. No stereo capture

The first two definitely need to be addressed. The last would need to be justified.

[philburk]

A callback API should be possible. Screen Capture Kit API doc says:

When a stream captures a new audio or video sample buffer, it calls the stream output’s stream:didOutputSampleBuffer:ofType: method, passing it the captured data and an indicator of its type.

[RossBencina]

@jkarthic could you please rebase this on the latest master, since we've fixed the issue that was causing Mac CI to fail.

Also, awaiting your response to our other comments. Thank you.

[jkarthic]

@RossBencina
Sorry for the delayed reply. And thanks for reviewing this PR. Actually I am not looking to maintain this feature actively, as the changes I have done so far seems to suffice my usecase so far. I just wanted to ensure that the work I had done so far benefits the very nice PortAudio community as well.
My goal was to just push a rudimentary version of Mac loopback capture working so that the community can take it forward from there, based on the contributor's requirements and priorities.
Nevertheless I have rebased and added the callback API support as it seems to be the minimum requirement for adding a device API in PortAudio. But I am not planning to add the stereo support, as it is not part of my current requirements.

[jkarthic]

Also all the CI tests seems to passing for the dummy PR I created in my repo. get-wrecked#1

[RossBencina]

Thanks for clarifying and for your contribution. Right now we don't think it is ready to merge. We don't want to discard your good work, so we will keep this PR around as a draft.

Right now we're not going to merge this because:

1. It is unfinished (e.g. no stereo, possibly other things) and there is no current prospect of it being bought up to standard.
2. No one has yet stepped up to maintain it and I'm not sure there is anyone who will.

If anyone would like to step up and complete and maintain this we would be happy for this new host API to reach our master branch when it's ready.

[DonaldTrump-2020]

@jkarthic I have tested the newest commit of you. here is the resuilt:

[100%] Built target paloopback
|_. Bad Default Latency? |_. Device Number |_. I/O Channels |_. Device Name |_. Host API |_. Default High Input Latency |_. Default Low Input Latency |_. Default High Output Latency |_. Default Low Output latency |
|   |   0 | 2/0 |  == Built-in Line Input == | Core Audio |   0.0141 |   0.0039 |   0.1000 |   0.0100 |
|   |   1 | 2/0 |  == Built-in Digital Input == | Core Audio |   0.0130 |   0.0037 |   0.1000 |   0.0100 |
|   |   2 | 0/2 |  == Built-in Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
|   |   3 | 0/2 |  == Built-in Line Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
|   |   4 | 0/2 |  == Built-in Digital Output == | Core Audio |   0.1000 |   0.0100 |   0.0141 |   0.0039 |
| X |   5 | 1/0 |  == System Audio [Loopback] == | Mac ScreenCaptureKit |   0.0000 |   0.0000 |   0.0000 |   0.0000 |

[arkadijs]

Curious, as if macOS could indicate that only audio is captured and not the screen?