Send Android h264 capture over a rtp stream - android

I'm writing a rtp video streamer for android that reads h264 coded data from an Android local socket and packetize it. The thing is that I did it but I keep getting black frames in the client side (Voip).
The communication goes like this: Android -> Asterisk -> Jitsi (Osx) (and reverse)
There are a few things that I haven't understood yet:
1) Android's mediarecorder gives me a raw h264 stream, How can I know when a NAL starts / ends based on that stream? It doesn't have any 0x000001 pattern but it does have a 0x0000 (which I'm assuming is a separator)
EDIT:
Adding more information. These are 2 (first, second) different reads of the input buffer (in order). If I got it right the first 4 bytes should be used to get the NALU length and the 5th one (index 4) is the NALU header.
I'll copy here the byte's values for future usage:
1) 0 0 12 114 101 -72 4 25 -15 -1 -121 -53 .....
length = (114&0xFF) + (12&0xFF)*256 + (0&0xFF)
length -> 3186
forbidden = 101 & 0x80
forbidden -> 0
nri = 101 & 0x60
nri -> 96
nal_unit_type = 101 & 0x1F
nal_unit_type -> 5
2) 0 0 1 -93 97 -32 32 103 -14 93 -1 ....
length = (-93&0xFF) + (1&0xFF)*256 + (0&0xFF)
length -> 419
forbidden = 97 & 0x80
forbidden -> 0
nri = 97 & 0x60
nri -> 96
nal_unit_type = 97 & 0x1F
nal_unit_type -> 1
is this correct?
2) How can I get the NALu timestamp and its length from that stream?
3) For some reason the packets are being marked (Even when I unset the marker). (In case you check the pcap file) [FIXED: I wasn't using the same SSCR for every packet]
Here's a pcap capture of the stream coming from asterisk (wich comes from Android). The Android device is an Asus Transform Prime with Android ICS.
I'm sending the packetization-mode (1) and profile-level-id (42801e) in the sdp, I've also tried sending the sprops (sps: Z0KAHpWgUHxA, psp: aM48gA==) parameter but nothing changed.
Cheers.

There is no "android h264 stream format". Packetization should follow RFC6184 if RTP is used as the transport protocol. The start code is not present of the RTP payload format. What is contained in an RTP packet, depends entirely on the packetization mode which specifies if NAL units may be aggregated, fragmented, etc. Read the RFC section on packetization modes for more info. Packetization mode is usually communicated via SDP.
Timestamp is part of RTP header. Length of a NAL unit again depends on packetization mode.
What do you mean by marked? Do you mean that the RTP marker bit is set? If so, this again must follow the rules laid out in the RFC.

Related

IOException: write failed: EINVAL (Invalid argument) on UVC FileOutputStream in Kotlin

I'm trying to write Android Camera stream frames to the UVC Buffer using FileOutputStream. For context: the UVC Driver is working on the device and it has a custom built kernel.
I get 24 frames per second using imageAnalyzer:
imageAnalyzer = ImageAnalysis.Builder()
.setTargetAspectRatio(screenAspectRatio)
.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
...
imageAnalysis.setAnalyzer(cameraExecutor) { image ->
val buffer = image.planes[0].buffer
val data = buffer.toByteArray()
...
}
Then based on UVC Specifications I build the header of the frame:
val header = ByteBuffer.allocate(26)
val frameSize = image.width * image.height * ImageFormat.getBitsPerPixel(image.format) / 8
val EOH = 0x01
val ERR = 0x00
val STI = 0x01
val REST = 0x00
val SRC = 0x00
val PTS = (System.currentTimeMillis() - referenceTime) * 10000
val endOfFrame = 0x01
val FID = (frameId).toByte()
Add all of the above to the header
header.putInt(frameSize)
header.putShort(image.width.toShort())
header.putShort(image.height.toShort())
header.put(image.format.toByte())
header.put(((EOH shl 7) or (ERR shl 6) or (STI shl 5) or (REST shl 4) or SRC).toByte())
header.putLong(PTS)
header.put(endOfFrame.toByte())
header.put(FID)
Open the FileOutputStream and try to write the header and the image:
val uvcFileOutputStream = FileOutputStream("/dev/video3", true)
uvcFileOutputStream.write(header.toByteArray() + data)
uvcFileOutputStream.close()
Tried to tweak the header/payload but I'm still getting the same error:
java.io.IOException: write failed: EINVAL (Invalid argument)
at libcore.io.IoBridge.write(IoBridge.java:654)
at java.io.FileOutputStream.write(FileOutputStream.java:401)
at java.io.FileOutputStream.write(FileOutputStream.java:379)
What could I be doing wrong? is the header format wrong?
I don't know the answer directly, but I was curious to look and have some findings. I focused on the Kotlin part, as I don't know about UVC and because I suspect the problem to be there.
Huge assumption
Since there's no link to the specification I just found this source:
https://www.usb.org/document-library/video-class-v15-document-set
within the ZIP I looked at USB_Video_Payload_Frame_Based_1.5.pdf
Page 9, Section 2.1 Payload Header
I'm basing all my findings on this, so if I got this wrong, everything else is. It could still lead to a solution though if you validated the same things.
Finding 1: HLE is wrong
HLE is the length of the header, not the image data. You're putting the whole image size there (all the RGB byte data). Table 2-1 describes PTS and SCR bits control whether PTS and SCR are present. This means that if they're 0 in BFH then the header is shorter. This is why HLE is either 2, 6, 12.
Confirmation source + the fact that the field is 1 byte long (each row of Table 2-1 is 1 byte/8 bits) which means the header can be only up to 255 bytes long.
Finding 2: all the header is misaligned
Since your putting HLE with putInt, you're writing 4 bytes, from this point on, everything is messed up in the header, the flags depend on image size, etc.
Finding 3: SCR and PTS flag inconsistencies
Assuming I was wrong about 1 and 2. You're still setting the SRC and PTS bit to 0, but pushing a long (8 bytes).
Finding 4: wrong source
Actually, something is really off at this point, so I looked at your referenced GitHub ticket and found a better example of what your code represents:
Sadly, I was unable to match up what your header structure is, so I'm going to assume that you are implementing something very similar to what I was looking at, because all PDFs had pretty much the same header table.
Finding 5: HLE is wrong or missing
Assuming you need to start with the image size, the HLE is still wrong because it's the image format's type, not in connection with SCR and PTS flags.
Finding 6: BFH is missing fields
If you're following one of these specs, the BFH is always one byte with 8 bits. This is confirmed by how the shls are putting it together in your code and the descriptions of each flag (flag = true|false / 1/0).
Finding 7: PTS is wrong
Multiplying something that is millisecond precise by 10000 looks strange. The doc says "at most 450 microseconds", if you're trying to convert between ms and us, I think the multiplier would be just 1000. Either way it is only an int (4 bytes) large, definitely not a long.
Finding 8: coding assistant?
I have a feeling after reading all this, that Copilot, ChatGPT or other generator wrote your original code. This sound confirmed by you looking for a reputable source.
Finding 9: reputable source example
If I were you I would try to find a working example of this in GitHub, using keyword search like this: https://github.com/search?q=hle+pts+sti+eoh+fid+scr+bfh&type=code the languages don't really matter since these are binary file/stream formats, so regardless of language they should be produced and read the same way.
Finding 10: bit order
Have a look at big endian / little endian byte order. If you look at Table 2-1 in the PDF I linked you can see which bit should map to which byte. You can specify the order you need easily on the buffer BEFORE writing to it, by the looks of the PDF it is header.order(ByteOrder.LITTLE_ENDIAN). I think conventionally 0 is the lowest bit and 31 is the highest. I can't cite a source on this, I seem to remember from uni. Bit 0 should be the 2^0 component (1) and bit 7 is the 2^7 (128). Reversing it would make things much harder to compute and comprehend. So PTS [7:0] means that byte is the lowest 8 bits of the 32 bit PTS number.
If you link to your specification source, I can revise what I wrote, but likely will find very similar guesses.

BLE Heart Rate Senser Value Interpretation

I have an Android App where I get Heart Rate Measurements from a Polar H10 Device.
I'm totally lost on how to interpret the heart rate. Various links to the bluetooth.com site are resulting in 404 errors unfortunately.
The characteristics value is i.e.
[16, 59, 83, 4]
From what I understood the second byte (59) is the heart rate in BPM. But this does not seem to be decimal as the value goes up to 127 and then goes on -127, -126, -125, ... It is not hex either.
I tried (in kotlin)
characteristic.value[1].toUInt()
characteristic.value[1].toInt()
characteristic.value[1].toShort()
characteristic.value[1].toULong()
characteristic.value[1].toDouble()
All values freak out as soon as the -127 appears.
Do I have to convert the 59 to binary (59=111011) and see it in there? Please give me some insight.
### Edit (12th April 2021) ###
What I do to get those values is a BluetoothDevice.connectGatt().
Then hold the GATT.
In order to get heart rate values I look for
Service 0x180d and its
characteristic 0x2a37 and its only
descriptor 0x2902.
Then I enable notifications by setting 0x01 on the descriptor. I then get ongoing events in the GattClientCallback.onCharacteristicChanged() callback. I will add a screenshot below with all data.
From what I understood the response should be 6 bytes long instead of 4, right? What am I doing wrong?
On the picture you see the characteristic on the very top. It is linked to the service 180d and the characteristic holds the value with 4 bytes on the bottom.
See Heart Rate Value in BLE for the links to the documents. As in that answer, here's the decode:
Byte 0 - Flags: 16 (0001 0000)
Bits are numbered from LSB (0) to MSB (7).
Bit 0 - Heart Rate Value Format: 0 => UINT8 beats per minute
Bit 1-2 - Sensor Contact Status: 00 => Not supported or detected
Bit 3 - Energy Expended Status: 0 => No Present
Bit 4 - RR-Interval: 1 => One or more values are present
So the first byte is a heart rate in UInt8 format, and the next two bytes are an RR interval.
To read this in Kotlin:
characteristic.getIntValue(FORMAT_UINT8, 1)
This return a heart rate of 56 bpm.
And ignore the other two bytes unless you want the RR.
It seems I found a way by retrieving the value as follows
val hearRateDecimal = characteristic.getIntValue(BluetoothGattCharacteristic.FORMAT_UINT8, 1)
2 things are important
first - the format of UINT8 (although I don't know when to use UINT8 and when UINT16. Actually I thought I need to use UINT16 as the first byte is actually 16 (see the question above)
second - the offset parameter 1
What I now get is an Integer even beyond 127 -> 127, 128, 129, 130, ...

How to write two types of information into nfc tag and make reader read them at the same time?

I am working on a Mobile Programming project. The Mobile program aims to write uri information like http://example.com as uri payload and wifi credentials so that when you tap nfc tag you can connect to wifi and open up uri in the browser.
I have tested to write two information into one nfc tag but it does not seem to provide two operations at the same time.
If the project is successful on tap onto nfc tag the mobile has to popup wifi connection and open uri in the browser.
If anyone has the clue i will be so pleased to follow!
NDEF message is composed of NDEF Records. In this case, the message should consist of 4 records namely:
URI Record Type representing http://example.com
Recommended Action Record
Title Record
Wifi Simple Configuration Record defined by Wi-Fi Alliance Mime Type being application/vnd.wfa.wsc (Android Code that writes such record)
NDEF message consisting of the first 3 records is as follows. Please update it to contain the 4th record yourself.
----------
Offset Content Length Explanation
0 0xD1 1 NDEF header. TNF = 0x01(Well Known Type). SR=1, MB=1, ME=1
1 0x02 1 Record name length (2 bytes)
2 0x30 1 Length of the Smart Poster data (48 bytes)
3 “Sp” 2 The record name
5 0x81 1 NDEF header. TNF = 0x01, SR=0, MB=1, ME=0
6 0x01 1 Record name length (1 byte)
7 0x00, 0x00, 0x00, 0x0C 4 The length of the URI payload (12 bytes) (long format)
11 “U” 1 Record type: “U”
12 0x03 1 Abbreviation: “http://”
13 “example.com” 11 The URI itself.
24 0x11 1 NDEF record header (SR=1, TNF=0x01)
25 0x03 1 The length of the record name
26 0x01 1 The length of the “act” payload.
27 “act” 3 Record type: “act”
30 0x00 1 Action = Launch browser
31 0x11 1 NDEF record header (SR=1, TNF=0x01)
32 0x01 1 Length of the record name
33 0x12 1 Length of the record payload (18 bytes)
34 “T” 1 Record type: “T” (=Text)
35 0x05 1 Status byte for the Text (UTF-8, five-byte code)
36 “en-US” 5 ISO Language code: US-English
41 “Hello, world” 12 The text: “Hello world”, encoded in UTF-8.
----------
To answer your question of whether its possible to get the Android reader to handle these multiple records we should look at the NFC Tag Dispatch system
This system creates an intent with
- action = NfcAdapter.ACTION_NDEF_DISCOVERED
- MimeType = mimetype of the first record
- Intent's Extra NfcAdapter.EXTRA_NDEF_MESSAGES contains complete Ndef message
Therefore, Android populates mimetype of first record only.
Its left to the application or service that picks up this intent to process all the NDEF records as needed.

How to get smaller buffer size in multi-channel audio application with Oboe

I'm using Oboe 1.2 in an audio android application. When I call getFramesPerBurst(), which gives the endpoint buffer size, I get expected results (240 frames) if the number of output channels is set to 2. However when I set 4 output channels, the value returned by getFramesPerBurst() is around 960 (!). Is that normal ? Is that a limitation of the hardware (I tested on 4 different devices though, with different os version) ? A limitation of Oboe ? I notice also that this value is different than the value given by the property PROPERTY_OUTPUT_FRAMES_PER_BUFFER of AudioManager from the AudioService.
oboe::AudioStreamBuilder builder;
if (!oboe::AudioStreamBuilder::isAAudioRecommended()){
builder.setAudioApi(oboe::AudioApi::OpenSLES);
}
builder.setSharingMode(oboe::SharingMode::Exclusive);
builder.setFormat(oboe::AudioFormat::Float);
builder.setChannelCount(4);
builder.setCallback(&_oboeCallback);
builder.setPerformanceMode(oboe::PerformanceMode::LowLatency);
oboe::Result result = builder.openStream(&_stream);
if (result == oboe::Result::OK) {
int framePerBurst = _stream->getFramesPerBurst(); // gives value around 960 for 4 channels, 240 for 2 channels
_stream->setBufferSizeInFrames(2*framePerBurst);
}
Unless you are connecting to an audio device which actually has 4 independent channels (e.g. a USB audio interface or DJ controller like this one) then your 4 channel stream will need to be mixed into an N channel stream where N is the number of channels in your audio device. This could be 2 (stereo) for headphones or 1 (mono) for a built-in speaker.
The mixer introduces latency and larger buffer sizes. This is the difference in buffer sizes you see when you request a channel count of 2 vs 4.
For the lowest latency always leave the channel count unspecified when creating the stream, then do any channel count conversion inside your own app. There's an example of this here.

Polar Wearlink Bluetooth packet

i am looking at the code of a project called MyTracks:
http://code.google.com/r/jrgert-polar-bluetooth/source/browse/MyTracks/src/com/google/android/apps/mytracks/services/sensors/PolarMessageParser.java?r=ebc01faf49550bc9801633ff38bb3b8ddd6f5698
Now I am having problems with the method isValid(byte[] buffer). I don´t understand what exactly is he checking here. We want to know if the first byte in the array is the header containing 0xFE. I don´t quite understand the following lines :
boolean goodHdr = ((buffer[0] & 0xFF) == 0xFE);
boolean goodChk = ((buffer[2] & 0xFF) == (0xFF - (buffer[1] & 0xFF)));
return goodHdr && goodChk;
any ideas?
Ewoks is correct, refer to this blog post:
http://ww.telent.net/2012/5/3/listening_to_a_polar_bluetooth_hrm_in_linux
"Digging into src/com/google/android/apps/mytracks/services/sensors/PolarMessageParser.java we find a helpful comment revealing that, notwithstanding Polar's ridiculous stance on giving out development info (they don't, is the summary) the Wearlink packet format is actually quite simple.
Polar Bluetooth Wearlink packet example
Hdr - Len - Chk - Seq - Status - HeartRate - RRInterval_16-bits
FE - 08 - F7 - 06 - F1 - 48 - 03 64
where
Hdr always = 254 (0xFE),
Chk = 255 - Len
Seq range 0 to 15
Status = Upper nibble may be battery voltage
bit 0 is Beat Detection flag."
&0xff simply converts signed byte to unsigned int for doing the comparison
First line is checking is received buffer are starting with 0xFE as it should be with this Polar Wearable.
Second line is checking if length byte is correct as well because it's value by specification is 255-value writen is size byte..
This together is super simple verification that messages are correct (more complicated implementation would include CRC or other verification methods). cheers

Categories

Resources