MediaMuxer has been driving me mad for two days & nights now :-(
The situation: I receive a H264 encoded 1280x720 video stream via UDP.
The h264 stream contains NALU 1 - slice and NALU 5 - keyframe (5 is always preceded by NALU 7 - SPS and NALU 8 - PPS).
This stream appears to be stable 30fps with at least one NALU 5 keyframe per second. Bitrate is variable but less than 4Mbps.
MediaCodec sucessfully decodes the stream and renders it in a surface view so that part works well.
But now I need to save the H.264 into a local mpg file.
I set up a MediaMuxer with all MediaFormat information that I have, and feed it with the sample data from the stream.
Each sample contains one frame (NALU 1 or 5), and the first data sent to MediaMuxer is a keyframe (NALU 5). The presentation time is calculated based on framenumber and framerate.
All involved methods are called from the same thread.
But the mpg file is never created. As you can see in the output below the data in the ByteBuffer does start with NALU headers followed by varying size of data. MediaMuxer seems to "see" frames in the data as it counts the frames.
So what is wrong here?
Minimum API is 21, and I have tested with a Samsung Galaxy S4 running stock Android 5 and a couple of devices running Lineageos Oreo and Nougat.
Here is the code to setup the MediaMuxer:
void setupMuxer(File f) throws IOException {
if (DEBUG) Log.d(TAG, "Setup Muxer: " + f.getAbsolutePath() +" can write: " + f.canWrite());
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, decoderWidth, decoderHeight);
format.setInteger(MediaFormat.KEY_BIT_RATE, 4000000);
format.setInteger(MediaFormat.KEY_FRAME_RATE, 29);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
format.setByteBuffer("csd-0", ByteBuffer.wrap(sps)); // sps and pps have been retrieved from the stream's NAL 7/8
format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
muxer = new MediaMuxer(f.getPath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
videoTrack = muxer.addTrack(format);
muxer.start();
}
This method is called for each (complete) NALU 1 and NALU 5:
void muxFrame(ByteBuffer buf, int frame) {
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
bufferInfo.offset = buf.arrayOffset();
bufferInfo.size = buf.position() - bufferInfo.offset;
bufferInfo.flags = (buf.get(4) & 0x1f) == 5 ? MediaCodec.BUFFER_FLAG_KEY_FRAME : 0;
bufferInfo.presentationTimeUs = computePresentationTime(frame);
if (DEBUG)
Log.d(TAG, "muxFrame frame: " + frame + " size: " + bufferInfo.size + " NAL: " + (buf.get(4) & 0x1f) + " Flags: " + bufferInfo.flags + " PTS: " + bufferInfo.presentationTimeUs + " content: " + BitByteUtil.toByteString(buf.array(), buf.arrayOffset(), 8));
try {
muxer.writeSampleData(videoTrack, buf, bufferInfo);
} catch (Exception e) {
Log.w(TAG, "muxer failed", e);
} finally {
}
}
private static long computePresentationTime(int frameIndex) {
return 42 + frameIndex * 1000000 / FRAME_RATE;
}
Here is my output if MediaMuxer is stopped after it has consumed 100 NALUs.
05.651 D/VideoDecoderView: Setup Muxer: /storage/emulated/0/Pictures/test.mpg can write: true
05.656 I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3317 bytes
06.263 D/VideoDecoderView: muxFrame frame: 2 size: 7257 NAL: 5 Flags: 1 PTS: 66708 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015
06.264 I/MPEG4Writer: setStartTimestampUs: 66708
06.264 I/MPEG4Writer: Earliest track starting time: 66708
06.308 D/VideoDecoderView: muxFrame frame: 3 size: 8998 NAL: 1 Flags: 0 PTS: 100042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020
06.342 D/VideoDecoderView: muxFrame frame: 4 size: 13664 NAL: 1 Flags: 0 PTS: 133375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020
06.375 D/VideoDecoderView: muxFrame frame: 5 size: 13674 NAL: 1 Flags: 0 PTS: 166708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020
06.409 D/VideoDecoderView: muxFrame frame: 6 size: 13772 NAL: 1 Flags: 0 PTS: 200042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020
06.483 D/VideoDecoderView: muxFrame frame: 7 size: 13707 NAL: 1 Flags: 0 PTS: 233375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:162 7:020
06.520 D/VideoDecoderView: muxFrame frame: 8 size: 13778 NAL: 1 Flags: 0 PTS: 266708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:194 7:020
06.555 D/VideoDecoderView: muxFrame frame: 9 size: 13743 NAL: 1 Flags: 0 PTS: 300042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:226 7:020
06.575 D/VideoDecoderView: muxFrame frame: 10 size: 7338 NAL: 5 Flags: 1 PTS: 333375 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015
06.593 D/VideoDecoderView: muxFrame frame: 11 size: 9059 NAL: 1 Flags: 0 PTS: 366708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020
06.618 D/VideoDecoderView: muxFrame frame: 12 size: 13587 NAL: 1 Flags: 0 PTS: 400042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020
06.644 D/VideoDecoderView: muxFrame frame: 13 size: 13650 NAL: 1 Flags: 0 PTS: 433375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020
06.671 D/VideoDecoderView: muxFrame frame: 14 size: 13797 NAL: 1 Flags: 0 PTS: 466708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020
.... [snip]
09.620 D/VideoDecoderView: muxFrame frame: 97 size: 7212 NAL: 5 Flags: 1 PTS: 3233375 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015
09.661 D/VideoDecoderView: muxFrame frame: 98 size: 8814 NAL: 1 Flags: 0 PTS: 3266708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020
09.692 D/VideoDecoderView: muxFrame frame: 99 size: 13566 NAL: 1 Flags: 0 PTS: 3300042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020
09.737 D/VideoDecoderView: muxFrame frame: 100 size: 13733 NAL: 1 Flags: 0 PTS: 3333375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020
09.771 D/VideoDecoderView: muxFrame frame: 101 size: 13771 NAL: 1 Flags: 0 PTS: 3366708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020
09.775 D/MPEG4Writer: Video track stopping. Stop source
09.775 I/MPEG4Writer: Received total/0-length (100/1) buffers and encoded 100 frames. - Video
09.775 D/MPEG4Writer: Video track source stopping
09.775 D/MPEG4Writer: Video track source stopped
09.775 D/MPEG4Writer: Video track stopped. Stop source
09.775 D/MPEG4Writer: Stopping writer thread
09.776 D/MPEG4Writer: 0 chunks are written in the last batch
09.779 D/MPEG4Writer: Writer thread stopped
09.780 I/MPEG4Writer: Ajust the moov start time from 66708 us -> 66708 us
09.780 D/MPEG4Writer: Video track stopping. Stop source
#greeble31: You are right. The first log entry clearly states "Pictures" and not "Videos".
I spent hours looking at this problem without noticing a simple cut&paste mistake in my preferences keys. How stupid is that!!?!
Note to myself: Coding two days & nights in a row is not heroic but just plain stupid.
Related
I am using the google stun server in both android and iOS. For data communication using the firebase. I am able to connect android to android video call and iOS to iOS video call. But while connect the android to iOS and iOS to android it show the remote video as black screen
iOS code
let configuration = RTCConfiguration()
configuration.iceServers = [
RTCIceServer(urlStrings: [
"stun:stun1.l.google.com:19302", "stun:stun2.l.google.com:19302"
])
]
private let factory: RTCPeerConnectionFactory = {
let videoEncoderFactory = RTCSimluatorVideoEncoderFactory()
let videoDecoderFactory = RTCSimulatorVideoDecoderFactory()
print(videoEncoderFactory)
videoEncoderFactory.preferredCodec = RTCVideoCodecInfo(name: kRTCVideoCodecVp8Name)
return RTCPeerConnectionFactory(encoderFactory: videoEncoderFactory, decoderFactory: videoDecoderFactory)
}()
videoSource!.adaptOutputFormat(toWidth: 1280, height: 720, fps: 30)
videoCapturer = RTCCameraVideoCapturer(delegate: videoSource!)
localVideoTrack = factory.videoTrack(with: videoSource!, trackId: "local_track")
localAudioTrack = factory.audioTrack(with: audioSource, trackId: "local_track" + "_audio")
localAudioTrack.isEnabled = true
// audioSource.volume = 1
super.init()
let constraints = RTCMediaConstraints(mandatoryConstraints: nil,
optionalConstraints: ["DtlsSrtpKeyAgreement": kRTCMediaConstraintsValueTrue])
peerConnection = factory.peerConnection(with: configuration, constraints: constraints, delegate: self)
peerConnection?.add(localVideoTrack, streamIds: ["local_track"])
peerConnection?.add(localAudioTrack,streamIds: ["local_track"])
remoteVideoTrack = peerConnection?.transceivers.first { $0.mediaType == .video }?.receiver.track as? RTCVideoTrack
Android code
private val iceServer = listOf(
PeerConnection.IceServer.builder("stun:stun.l.google.com:19302")
.createIceServer(),
PeerConnection.IceServer.builder("stun:stun2.l.google.com:19302")
.createIceServer()
)
private fun initPeerConnectionFactory(context: Application) {
val options = PeerConnectionFactory.InitializationOptions.builder(context)
.setEnableInternalTracer(true)
.setFieldTrials("WebRTC-H264HighProfile/Enabled/")
.createInitializationOptions()
PeerConnectionFactory.initialize(options)
}
private fun buildPeerConnectionFactory(): PeerConnectionFactory {
return PeerConnectionFactory
.builder()
.setVideoDecoderFactory(DefaultVideoDecoderFactory(rootEglBase.eglBaseContext))
.setVideoEncoderFactory(DefaultVideoEncoderFactory(rootEglBase.eglBaseContext, true, true))
.setOptions(PeerConnectionFactory.Options().apply {
disableEncryption = false
disableNetworkMonitor = true
})
.createPeerConnectionFactory()
}
fun startLocalVideoCapture(localVideoOutput: SurfaceViewRenderer) {
val surfaceTextureHelper = SurfaceTextureHelper.create(Thread.currentThread().name, rootEglBase.eglBaseContext)
(videoCapturer as VideoCapturer).initialize(surfaceTextureHelper, localVideoOutput.context, localVideoSource.capturerObserver)
videoCapturer.startCapture(1280, 720, 60)
localAudioTrack = peerConnectionFactory.createAudioTrack(LOCAL_TRACK_ID + "_audio", audioSource);
localVideoTrack = peerConnectionFactory.createVideoTrack(LOCAL_TRACK_ID, localVideoSource)
localVideoTrack?.addSink(localVideoOutput)
val localStream = peerConnectionFactory.createLocalMediaStream(LOCAL_STREAM_ID)
localStream.addTrack(localVideoTrack)
localStream.addTrack(localAudioTrack)
peerConnection?.addStream(localStream)
}
I need help for connecting android and ios without black screen in remote video
Android:
v=0o=- 6563448285196734461 2 IN IP4 127.0.0.1s=-t=0 0a=group:BUNDLE audio videoa=msid-semantic: WMSm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 100 101 127 124 125c=IN IP4 0.0.0.0a=rtcp:9 IN IP4 0.0.0.0a=ice-ufrag:I69Ta=ice-pwd:RY4fL2nYEJVHJ1vDc7fsws0qa=ice-options:trickle renominationa=fingerprint:sha-256 91:33:FC:D0:9C:BD:FE:A8:F4:7D:8F:6E:1F:71:05:4F:5B:F3:A0:DA:86:14:8A:CA:29:F0:45:AB:88:02:EE:50a=setup:activea=mid:audioa=extmap:14 urn:ietf:params:rtp-hdrext:toffseta=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=extmap:13 urn:3gpp:video-orientationa=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delaya=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-typea=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timinga=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-spacea=recvonlya=rtcp-muxa=rtcp-rsizea=rtpmap:96 VP8/90000a=rtcp-fb:96 goog-remba=rtcp-fb:96 transport-cca=rtcp-fb:96 ccm fira=rtcp-fb:96 nacka=rtcp-fb:96 nack plia=rtpmap:97 rtx/90000a=fmtp:97 apt=96a=rtpmap:98 H264/90000a=rtcp-fb:98 goog-remba=rtcp-fb:98 transport-cca=rtcp-fb:98 ccm fira=rtcp-fb:98 nacka=rtcp-fb:98 nack plia=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640c1fa=rtpmap:100 H264/90000a=rtcp-fb:100 goog-remba=rtcp-fb:100 transport-cca=rtcp-fb:100 ccm fira=rtcp-fb:100 nacka=rtcp-fb:100 nack plia=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01fa=rtpmap:101 rtx/90000a=fmtp:101 apt=100a=rtpmap:127 red/90000a=rtpmap:124 rtx/90000a=fmtp:124 apt=127a=rtpmap:125 ulpfec/90000m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126c=IN IP4 0.0.0.0a=rtcp:9 IN IP4 0.0.0.0a=ice-ufrag:I69Ta=ice-pwd:RY4fL2nYEJVHJ1vDc7fsws0qa=ice-options:trickle renominationa=fingerprint:sha-256 91:33:FC:D0:9C:BD:FE:A8:F4:7D:8F:6E:1F:71:05:4F:5B:F3:A0:DA:86:14:8A:CA:29:F0:45:AB:88:02:EE:50a=setup:activea=mid:videoa=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-levela=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01a=recvonlya=rtcp-muxa=rtpmap:111 opus/48000/2a=rtcp-fb:111 transport-cca=fmtp:111 minptime=10;useinbandfec=1a=rtpmap:103 ISAC/16000a=rtpmap:104 ISAC/32000a=rtpmap:9 G722/8000a=rtpmap:102 ILBC/8000a=rtpmap:0 PCMU/8000a=rtpmap:8 PCMA/8000a=rtpmap:106 CN/32000a=rtpmap:105 CN/16000a=rtpmap:13 CN/8000a=rtpmap:110 telephone-event/48000a=rtpmap:112 telephone-event/32000a=rtpmap:113 telephone-event/16000a=rtpmap:126 telephone-event/8000
iOS
v=0o=- 8886095848359872555 2 IN IP4 127.0.0.1s=-t=0 0a=group:BUNDLE audio videoa=msid-semantic: WMS local_trackm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125c=IN IP4 0.0.0.0a=rtcp:9 IN IP4 0.0.0.0a=ice-ufrag:1BBja=ice-pwd:uaFw/ANPPPKHdZmf+qr31phba=ice-options:trickle renominationa=fingerprint:sha-256 1B:D0:0D:16:73:CD:FF:B9:F5:C3:8E:86:3D:83:DF:5D:3D:40:86:B1:67:D5:D3:9D:A4:92:AA:0F:C6:09:AE:ECa=setup:actpassa=mid:audioa=extmap:14 urn:ietf:params:rtp-hdrext:toffseta=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=extmap:13 urn:3gpp:video-orientationa=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delaya=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-typea=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timinga=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-spacea=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mida=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-ida=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-ida=sendrecva=msid:local_track local_tracka=rtcp-muxa=rtcp-rsizea=rtpmap:96 VP8/90000a=rtcp-fb:96 goog-remba=rtcp-fb:96 transport-cca=rtcp-fb:96 ccm fira=rtcp-fb:96 nacka=rtcp-fb:96 nack plia=rtpmap:97 rtx/90000a=fmtp:97 apt=96a=rtpmap:98 H264/90000a=rtcp-fb:98 goog-remba=rtcp-fb:98 transport-cca=rtcp-fb:98 ccm fira=rtcp-fb:98 nacka=rtcp-fb:98 nack plia=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640c1fa=rtpmap:99 rtx/90000a=fmtp:99 apt=98a=rtpmap:100 H264/90000a=rtcp-fb:100 goog-remba=rtcp-fb:100 transport-cca=rtcp-fb:100 ccm fira=rtcp-fb:100 nacka=rtcp-fb:100 nack plia=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01fa=rtpmap:101 rtx/90000a=fmtp:101 apt=100a=rtpmap:127 red/90000a=rtpmap:124 rtx/90000a=fmtp:124 apt=127a=rtpmap:125 ulpfec/90000a=ssrc-group:FID 1956454411 1435441277a=ssrc:1956454411 cname:BnVPai9gO/G5EEuLa=ssrc:1956454411 msid:local_track local_tracka=ssrc:1956454411 mslabel:local_tracka=ssrc:1956454411 label:local_tracka=ssrc:1435441277 cname:BnVPai9gO/G5EEuLa=ssrc:1435441277 msid:local_track local_tracka=ssrc:1435441277 mslabel:local_tracka=ssrc:1435441277 label:local_trackm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126c=IN IP4 0.0.0.0a=rtcp:9 IN IP4 0.0.0.0a=ice-ufrag:1BBja=ice-pwd:uaFw/ANPPPKHdZmf+qr31phba=ice-options:trickle renominationa=fingerprint:sha-256 1B:D0:0D:16:73:CD:FF:B9:F5:C3:8E:86:3D:83:DF:5D:3D:40:86:B1:67:D5:D3:9D:A4:92:AA:0F:C6:09:AE:ECa=setup:actpassa=mid:videoa=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-levela=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mida=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-ida=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-ida=sendrecva=msid:local_track local_track_audioa=rtcp-muxa=rtpmap:111 opus/48000/2a=rtcp-fb:111 transport-cca=fmtp:111 minptime=10;useinbandfec=1a=rtpmap:103 ISAC/16000a=rtpmap:104 ISAC/32000a=rtpmap:9 G722/8000a=rtpmap:102 ILBC/8000a=rtpmap:0 PCMU/8000a=rtpmap:8 PCMA/8000a=rtpmap:106 CN/32000a=rtpmap:105 CN/16000a=rtpmap:13 CN/8000a=rtpmap:110 telephone-event/48000a=rtpmap:112 telephone-event/32000a=rtpmap:113 telephone-event/16000a=rtpmap:126 telephone-event/8000a=ssrc:2967951894 cname:BnVPai9gO/G5EEuLa=ssrc:2967951894 msid:local_track local_track_audioa=ssrc:2967951894 mslabel:local_tracka=ssrc:2967951894 label:local_track_audio
#Test
fun test():Unit= runBlocking(){
var count =0
List(1000){
GlobalScope.launch() {
println("${Thread.currentThread().name} $count")
count++
}
}.joinAll()
println(count)
}
I except it output less than 1000,but it always output 1000.I tried many times,even change count 10000,it alwaya output 1000 or 10000.When i read the log,it seems to be a thread synchronization problem, but the end result is correct again.Log like this
DefaultDispatcher-worker-7 #coroutine#8 0
DefaultDispatcher-worker-9 #coroutine#10 0
DefaultDispatcher-worker-6 #coroutine#7 0
DefaultDispatcher-worker-4 #coroutine#5 0
DefaultDispatcher-worker-2 #coroutine#3 0
DefaultDispatcher-worker-1 #coroutine#2 0
DefaultDispatcher-worker-8 #coroutine#9 0
DefaultDispatcher-worker-11 #coroutine#12 6
DefaultDispatcher-worker-5 #coroutine#6 0
DefaultDispatcher-worker-3 #coroutine#4 0
DefaultDispatcher-worker-10 #coroutine#11 4
DefaultDispatcher-worker-12 #coroutine#13 10
DefaultDispatcher-worker-8 #coroutine#14 12
DefaultDispatcher-worker-10 #coroutine#15 12
at the end like this
DefaultDispatcher-worker-8 #coroutine#908 899
DefaultDispatcher-worker-9 #coroutine#982 984
DefaultDispatcher-worker-9 #coroutine#998 985
DefaultDispatcher-worker-9 #coroutine#999 986
DefaultDispatcher-worker-9 #coroutine#1000 987
DefaultDispatcher-worker-9 #coroutine#1001 988
DefaultDispatcher-worker-8 #coroutine#997 989
DefaultDispatcher-worker-10 #coroutine#996 990
DefaultDispatcher-worker-1 #coroutine#995 991
DefaultDispatcher-worker-7 #coroutine#994 992
DefaultDispatcher-worker-4 #coroutine#993 993
DefaultDispatcher-worker-5 #coroutine#992 994
DefaultDispatcher-worker-2 #coroutine#991 995
DefaultDispatcher-worker-11 #coroutine#990 996
DefaultDispatcher-worker-6 #coroutine#909 899
DefaultDispatcher-worker-12 #coroutine#907 899
DefaultDispatcher-worker-3 #coroutine#906 899
1000
Can someone explain why this is
I'm maintaining an Android video replay app through Android Studio. All aspects of the application work fine across multiple devices (Samsung S8, S10, S20 etc) except for 64bit devices where the video decode fails. I'm looking for advice and/or an explaination as to what's causing the problem.
Below is a snippet of the code that is causing me grief. Below are the resultant logs. Basically the application works fine when run using jniLibs/armeabi however fails when run using jnilibs/arm64-v8a. The decoder threads keep failing with IllogicalArgumentException.
Code:
private class PlayerThread3 extends Thread {
private MediaCodec decoder;
private Surface surface;
public PlayerThread3(Surface surface) {
this.surface = surface;
}
#Override
public void run() {
Log.d("PlayerThread3->Run", "Entry surface=" + surface.toString());
MediaFormat format = MediaFormat.createVideoFormat("video/avc", mSurfaceWidth, mSurfaceHeight);
try {
Log.d("PlayerThread3->Run", "MediaFormat=" + format.toString());
decoder = MediaCodec.createByCodecName("OMX.google.h264.decoder");
Log.d("PlayerThread3->Run", "MediaCodec=" + decoder.toString());
if (surface.isValid()) {
Log.d("PlayerThread3->Run", "surface.isValid=" + surface.toString());
decoder.configure(format, surface, null, 0);
} else {
Log.i("debug","decoder31");
Message msg = Message.obtain();
msg.what = 3;
msg.obj = "Failed to start live video(surface)";
handler.sendMessage(msg);
return;
}
} catch (IllegalArgumentException e) {
Log.d("PlayerThread3->Run", "IA surface=" + surface.toString());
e.printStackTrace();
Log.i("debug", "decoder32");
Message msg = Message.obtain();
msg.what = 3;
msg.obj = "Failed to start live video(arg)";
handler.sendMessage(msg);
return;
} catch (IOException e) {
e.printStackTrace();
Log.i("debug", "decoder33");
Message msg = Message.obtain();
msg.what = 3;
msg.obj = "Failed to start live video(IO)";
handler.sendMessage(msg);
return;
}
try {
decoder.start();
inputBuffers3 = decoder.getInputBuffers();
Message msg = Message.obtain();
msg.what = 3;
handler.sendMessage(msg);
while (preferences_isAlways.getBoolean("indexDecode3" + index, false)) {
if (decoder_array3 != null) {
offerDecoder(decoder_array3, decoder_array3.length);
decoder_array3 = null;
}
}
decoder.stop();
decoder.release();
} catch (IllegalStateException e) {
e.printStackTrace();
Log.i("debug", "decoder34");
Message msg = Message.obtain();
msg.what = 3;
msg.obj = "Failed to start live video(IS)";
handler.sendMessage(msg);
}
}
Log:
I/frameRate: 2
I/whichPartOfFrame: 0
I/s_frame_serial: 40
I/frame_packet_body: 1952
D/PlayerThread3->Run: Entry surface=Surface(name=null)/#0x68f7d48
D/PlayerThread3->Run: MediaFormat={mime=video/avc, width=536, height=437}
D/CCodec: allocate(c2.android.avc.decoder)
I/CCodec: Created component [c2.android.avc.decoder]
D/CCodecConfig: read media type: video/avc
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
extent() != 1 for single value type: output.subscribed-indices.values
D/ReflectedParamUpdater: extent() != 1 for single value type: input.buffers.allocator-ids.values
extent() != 1 for single value type: output.buffers.allocator-ids.values
extent() != 1 for single value type: algo.buffers.allocator-ids.values
extent() != 1 for single value type: output.buffers.pool-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.pool-ids.values
D/ReflectedParamUpdater: ignored struct field coded.color-format.locations
D/CCodecConfig: ignoring local param raw.size (0xd2001800) as it is already supported
ignoring local param raw.color (0xd2001809) as it is already supported
D/ReflectedParamUpdater: ignored struct field raw.hdr-static-info.mastering
I/CCodecConfig: query failed after returning 12 values (BAD_INDEX)
D/CCodecConfig: c2 config is Dict {
c2::u32 coded.pl.level = 20496
c2::u32 coded.pl.profile = 20481
c2::u32 coded.vui.color.matrix = 0
c2::u32 coded.vui.color.primaries = 0
c2::u32 coded.vui.color.range = 2
c2::u32 coded.vui.color.transfer = 0
c2::u32 default.color.matrix = 0
c2::u32 default.color.primaries = 0
c2::u32 default.color.range = 0
c2::u32 default.color.transfer = 0
c2::u32 input.buffers.max-size.value = 57600
c2::u32 input.delay.value = 0
string input.media-type.value = "video/avc"
c2::u32 output.delay.value = 8
string output.media-type.value = "video/raw"
c2::u32 raw.color.matrix = 0
c2::u32 raw.color.primaries = 0
c2::u32 raw.color.range = 2
c2::u32 raw.color.transfer = 0
c2::u32 raw.max-size.height = 240
c2::u32 raw.max-size.width = 320
c2::u32 raw.pixel-format.value = 35
c2::i32 raw.rotation.flip = 0
c2::i32 raw.rotation.value = 0
c2::u32 raw.sar.height = 1
c2::u32 raw.sar.width = 1
c2::u32 raw.size.height = 240
c2::u32 raw.size.width = 320
c2::u32 ra
W/ColorUtils: expected specified color aspects (2:0:0:0)
D/PlayerThread3->Run: MediaCodec=android.media.MediaCodec#da3c906
surface.isValid=Surface(name=null)/#0x68f7d48
D/SurfaceUtils: connecting to surface 0x7645de8010, reason connectToSurface
I/MediaCodec: [c2.android.avc.decoder] setting surface generation to 27948035
D/SurfaceUtils: disconnecting from surface 0x7645de8010, reason connectToSurface(reconnect)
connecting to surface 0x7645de8010, reason connectToSurface(reconnect)
D/CCodecConfig: no c2 equivalents for native-window
config failed => CORRUPTED
Bad parameter value
D/CCodecConfig: c2 config is Dict {
c2::u32 coded.pl.level = 20496
c2::u32 coded.pl.profile = 20481
c2::u32 coded.vui.color.matrix = 0
c2::u32 coded.vui.color.primaries = 0
c2::u32 coded.vui.color.range = 2
c2::u32 coded.vui.color.transfer = 0
c2::u32 default.color.matrix = 0
c2::u32 default.color.primaries = 0
c2::u32 default.color.range = 0
c2::u32 default.color.transfer = 0
c2::u32 input.buffers.max-size.value = 57600
c2::u32 input.delay.value = 0
string input.media-type.value = "video/avc"
c2::u32 output.delay.value = 8
string output.media-type.value = "video/raw"
c2::u32 raw.color.matrix = 0
c2::u32 raw.color.primaries = 0
c2::u32 raw.color.range = 2
c2::u32 raw.color.transfer = 0
c2::u32 raw.max-size.height = 240
c2::u32 raw.max-size.width = 320
c2::u32 raw.pixel-format.value = 35
c2::i32 raw.rotation.flip = 0
c2::i32 raw.rotation.value = 0
c2::u32 raw.sar.height = 1
c2::u32 raw.sar.width = 1
c2::u32 raw.size.height = 240
c2::u32 raw.size.width = 536
c2::u32 ra
W/CCodec: failed to configure c2 params
E/MediaCodec: Codec reported err 0xffffffea, actionCode 0, while in state 3
D/SurfaceUtils: disconnecting from surface 0x7645de8010, reason disconnectFromSurface
E/MediaCodec: configure failed with err 0xffffffea, resetting...
I/MediaCodec: Codec shutdown complete
D/CCodec: allocate(c2.android.avc.decoder)
I/CCodec: Created component [c2.android.avc.decoder]
D/CCodecConfig: read media type: video/avc
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
extent() != 1 for single value type: output.subscribed-indices.values
extent() != 1 for single value type: input.buffers.allocator-ids.values
extent() != 1 for single value type: output.buffers.allocator-ids.values
extent() != 1 for single value type: algo.buffers.allocator-ids.values
extent() != 1 for single value type: output.buffers.pool-ids.values
extent() != 1 for single value type: algo.buffers.pool-ids.values
D/ReflectedParamUpdater: ignored struct field coded.color-format.locations
I/frameRate: 2
I/whichPartOfFrame: 0
I/s_frame_serial: 41
I/frame_packet_body: 46
D/CCodecConfig: ignoring local param raw.size (0xd2001800) as it is already supported
ignoring local param raw.color (0xd2001809) as it is already supported
D/ReflectedParamUpdater: ignored struct field raw.hdr-static-info.mastering
I/CCodecConfig: query failed after returning 12 values (BAD_INDEX)
D/CCodecConfig: c2 config is Dict {
c2::u32 coded.pl.level = 20496
c2::u32 coded.pl.profile = 20481
c2::u32 coded.vui.color.matrix = 0
c2::u32 coded.vui.color.primaries = 0
c2::u32 coded.vui.color.range = 2
c2::u32 coded.vui.color.transfer = 0
c2::u32 default.color.matrix = 0
c2::u32 default.color.primaries = 0
c2::u32 default.color.range = 0
c2::u32 default.color.transfer = 0
c2::u32 input.buffers.max-size.value = 57600
c2::u32 input.delay.value = 0
string input.media-type.value = "video/avc"
c2::u32 output.delay.value = 8
string output.media-type.value = "video/raw"
c2::u32 raw.color.matrix = 0
c2::u32 raw.color.primaries = 0
c2::u32 raw.color.range = 2
c2::u32 raw.color.transfer = 0
c2::u32 raw.max-size.height = 240
c2::u32 raw.max-size.width = 320
c2::u32 raw.pixel-format.value = 35
c2::i32 raw.rotation.flip = 0
c2::i32 raw.rotation.value = 0
c2::u32 raw.sar.height = 1
c2::u32 raw.sar.width = 1
c2::u32 raw.size.height = 240
c2::u32 raw.size.width = 320
c2::u32 ra
W/ColorUtils: expected specified color aspects (2:0:0:0)
D/PlayerThread3->Run: IA surface=Surface(name=null)/#0x68f7d48
W/System.err: java.lang.IllegalArgumentException
at android.media.MediaCodec.native_configure(Native Method)
W/System.err: at android.media.MediaCodec.configure(MediaCodec.java:2023)
at android.media.MediaCodec.configure(MediaCodec.java:1951)
at au.com.FreedomVMS.iFreedomVMSpro.live.ShowFragment$PlayerThread3.run(ShowFragment.java:5789)
I/debug: decoder32
I/.iFreedomVMSpr: Compiler allocated 15MB to compile void au.com.FreedomVMS.iFreedomVMSpro.live.ShowFragment.handleSocket1(int, byte, java.lang.String, int, java.lang.String, java.lang.String)
I/frameRate: 2
I/whichPartOfFrame: 1
I/s_frame_serial: 42
I/frame_packet_body: 10240
Simples, problem solved. Surface dimensions, width and height, have to be multiples of 16 under Android 10. The earlier versions of Android tested against don't appear to be worried by this constraint.
Doesn't add reverb effect to mediaplayer.
Added permission to the manifest. Checked if the device supports effects.
Check device:
D/Constraints: Loudness Enhancer, type: fe3199be-aed0-413f-87bb-11260eb63cf1
D/Constraints: Multichannel Downmix To Stereo, type: 381e49cc-a858-4aa2-87f6-e8388e7601b2
D/Constraints: Visualizer, type: e46b26a0-dddd-11db-8afd-0002a5d5c51b
D/Constraints: Insert Preset Reverb, type: 47382d60-ddd8-11db-bf3a-0002a5d5c51b
D/Constraints: Auxiliary Preset Reverb, type: 47382d60-ddd8-11db-bf3a-0002a5d5c51b
D/Constraints: Insert Environmental Reverb, type: c2e5d5f0-94bd-4763-9cac-4e234d06839e
D/Constraints: Auxiliary Environmental Reverb, type: c2e5d5f0-94bd-4763-9cac-4e234d06839e
D/Constraints: Volume, type: 09e8ede0-ddde-11db-b4f6-0002a5d5c51b
D/Constraints: Equalizer, type: 0bed4300-ddd6-11db-8f34-0002a5d5c51b
D/Constraints: Virtualizer, type: 37cc2c00-dddd-11db-8577-0002a5d5c51b
D/Constraints: Dynamic Bass Boost, type: 0634f220-ddd4-11db-a0fc-0002a5d5c51b
private var player: MediaPlayer ?= null
player = MediaPlayer.create(context, R.raw.r2);
val mReverb = PresetReverb(0, 0)
mReverb.preset = PresetReverb.PRESET_LARGEROOM
mReverb.enabled = true
player?.attachAuxEffect(mReverb.id)
player?.setAuxEffectSendLevel(1.0f)
player?.start()
There are no error messages in Logcat
I'm developing an android app with the libav and I'm trying decode a 3gp with code below:
#define simbiLog(...) __android_log_print(ANDROID_LOG_DEBUG, "simbiose", __VA_ARGS__)
...
AVCodec *codec;
AVCodecContext *c = NULL;
int len;
FILE *infile, *outfile;
uint8_t inbuf[AUDIO_INBUF_SIZE + FF_INPUT_BUFFER_PADDING_SIZE];
AVPacket avpkt;
AVFrame *decoded_frame = NULL;
simbiLog("inbuf size: %d", sizeof(inbuf) / sizeof(inbuf[0]));
av_register_all();
av_init_packet(&avpkt);
codec = avcodec_find_decoder(AV_CODEC_ID_AMR_NB);
if (!codec) {
simbiLog("codec not found");
return ERROR;
}
c = avcodec_alloc_context3(codec);
if (!c) {
simbiLog("Could not allocate audio codec context");
return ERROR;
}
int open = avcodec_open2(c, codec, NULL);
if (open < 0) {
simbiLog("could not open codec %d", open);
return ERROR;
}
infile = fopen(inputPath, "rb");
if (!infile) {
simbiLog("could not open %s", inputPath);
return ERROR;
}
outfile = fopen(outputPath, "wb");
if (!outfile) {
simbiLog("could not open %s", outputPath);
return ERROR;
}
avpkt.data = inbuf;
avpkt.size = fread(inbuf, 1, AUDIO_INBUF_SIZE, infile);
int iterations = 0;
while (avpkt.size > 0) {
simbiLog("iteration %d", (++iterations));
simbiLog("avpkt.size %d avpkt.data %X", avpkt.size, avpkt.data);
int got_frame = 0;
if (!decoded_frame) {
if (!(decoded_frame = avcodec_alloc_frame())) {
simbiLog("out of memory");
return ERROR;
}
} else {
avcodec_get_frame_defaults(decoded_frame);
}
//below the error, but it isn't occur on first time, only in 4th loop interation
len = avcodec_decode_audio4(c, decoded_frame, &got_frame, &avpkt);
if (len < 0) {
simbiLog("Error while decoding error %d frame %d duration %d", len, got_frame, avpkt.duration);
return ERROR;
} else {
simbiLog("Decoding length %d frame %d duration %d", len, got_frame, avpkt.duration);
}
if (got_frame) {
int data_size = av_samples_get_buffer_size(NULL, c->channels, decoded_frame->nb_samples, c->sample_fmt, 1);
size_t* fwrite_size = fwrite(decoded_frame->data[0], 1, data_size, outfile);
simbiLog("fwrite returned %d", fwrite_size);
}
avpkt.size -= len;
avpkt.data += len;
if (avpkt.size < AUDIO_REFILL_THRESH) {
memmove(inbuf, avpkt.data, avpkt.size);
avpkt.data = inbuf;
len = fread(avpkt.data + avpkt.size, 1, AUDIO_INBUF_SIZE - avpkt.size, infile);
if (len > 0)
avpkt.size += len;
simbiLog("fread returned %d", len);
}
}
fclose(outfile);
fclose(infile);
avcodec_close(c);
av_free(c);
av_free(decoded_frame);
but I'm getting the follow log and error:
inbuf size: 20488
iteration 1
avpkt.size 3305 avpkt.data BEEED40C
Decoding length 13 frame 1 duration 0
fwrite returned 640
fread returned 0
iteration 2
avpkt.size 3292 avpkt.data BEEED40C
Decoding length 13 frame 1 duration 0
fwrite returned 640
fread returned 0
iteration 3
avpkt.size 3279 avpkt.data BEEED40C
Decoding length 14 frame 1 duration 0
fwrite returned 640
fread returned 0
iteration 4
avpkt.size 3265 avpkt.data BEEED40C
Error while decoding error -1052488119 frame 0 duration 0
the audio file I'm trying decode:
$ avprobe blue.3gp
avprobe version 0.8.6-6:0.8.6-1ubuntu2, Copyright (c) 2007-2013 the Libav developers
built on Mar 30 2013 22:23:21 with gcc 4.7.2
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'blue.3gp':
Metadata:
major_brand : 3gp4
minor_version : 0
compatible_brands: isom3gp4
creation_time : 2013-09-19 18:53:38
Duration: 00:00:01.52, start: 0.000000, bitrate: 17 kb/s
Stream #0.0(eng): Audio: amrnb, 8000 Hz, 1 channels, flt, 12 kb/s
Metadata:
creation_time : 2013-09-19 18:53:38
thanks a lot!
EDITED
I read on ffmper documentation about the method avcodec_decode_audio4 the follow:
#warning The input buffer, avpkt->data must be FF_INPUT_BUFFER_PADDING_SIZE larger than the actual read bytes because some optimized bitstream readers read 32 or 64 bits at once and could read over the end.
#note You might have to align the input buffer. The alignment requirements depend on the CPU and the decoder.
and I see here a solution using posix_memalign, to android i founded a similar method called memalign, so i did the change:
removed:
uint8_t inbuf[AUDIO_INBUF_SIZE + FF_INPUT_BUFFER_PADDING_SIZE];
inserted:
int inbufSize = sizeof(uint8_t) * (AUDIO_INBUF_SIZE + FF_INPUT_BUFFER_PADDING_SIZE);
uint8_t *inbuf = memalign(FF_INPUT_BUFFER_PADDING_SIZE, inbufSize);
simbiLog("inbuf size: %d", inbufSize);
for (; inbufSize >= 0; inbufSize--)
simbiLog("inbuf position: %d index: %p", inbufSize, &inbuf[inbufSize]);
I'm getting the correct memory sequence position, but the error not changed.
A piece of outpout:
inbuf position: 37 index: 0x4e43d745
inbuf position: 36 index: 0x4e43d744
inbuf position: 35 index: 0x4e43d743
inbuf position: 34 index: 0x4e43d742
inbuf position: 33 index: 0x4e43d741
inbuf position: 32 index: 0x4e43d740
inbuf position: 31 index: 0x4e43d73f
inbuf position: 30 index: 0x4e43d73e
inbuf position: 29 index: 0x4e43d73d
inbuf position: 28 index: 0x4e43d73c
inbuf position: 27 index: 0x4e43d73b
inbuf position: 26 index: 0x4e43d73a
inbuf position: 25 index: 0x4e43d739
inbuf position: 24 index: 0x4e43d738
inbuf position: 23 index: 0x4e43d737
inbuf position: 22 index: 0x4e43d736
inbuf position: 21 index: 0x4e43d735
inbuf position: 20 index: 0x4e43d734
inbuf position: 19 index: 0x4e43d733
You are trying to decode without demuxing.