Develop Reference

Stream h264 to javafx possibly using javacv/ffmpeg

I'm really stuck on getting a video stream to play on a java fx project.
-- Short version:
I'm streaming h264/avcc flavor video from an android phone to a desktop computer. However javafx doesn't have an easy solution for displaying stream. I'm attempting to use javacv / ffmpeg in an attempt to make this work. However I am getting errors from ffmpeg.
1) Is there a better way to display streaming video on javafx?
2) Do you have a sample project or good tutorial for javacv ffmpegframegrabber?
3) I think I may be missing some small detail in mycode but Im not sure what i would be.
-- Longer Version:
1) On the android end Im getting video using mediarecorder. In order to get the sps/pps info I record and save a small movie to the device and then parse the sps and pps data.
2) Next, on the android, I split up the nalus to meet MTU req and send them over a udp connection to my desktop
3)On my desktop I reassmble the nalus( or trash them if they loose data) and feed those to an input stream that I gave to the framegreabber constructor.
-- The Code and Logs:
The errors are long and numerous depending on the flavor I feed it. Here are two separate examples which are usually repeated at great length
[h264 # 0000020225907a40] non-existing PPS 0 referenced
[h264 # 0000020225907a40] decode_slice_header error
[h264 # 0000020225907a40] no frame!
[h264 # 00000163d8637a40] illegal aspect ratio
[h264 # 00000163d8637a40] pps_id 3412 out of range
[AVBSFContext # 00000163e28a0e00] Invalid NAL unit 0, skipping.
!! One big caveat that I am aware off is that I have not implemented timestamps
which I created on the android device when feeding ffmpeg. I think it should still show distorted images without this though
Because I have spent all day guessing and trying I have several "flavors" of data I have shoved through. I am only showing the first section of each nal which I believe if correct would at least show a garbage image as long as my sps and pps are right
sps: 67 80 80 1E E9 01 68 22 FD C0 36 85 09 A8
pps: 68 06 06 E2
Below is annex B style.
These were each prefixed with either 00 00 01 and 00 00 00 01
Debug transfer 65 B8 40 0B E5 B8 7B 80 5B 85
Debug transfer 41 E2 20 7A 74 34 3B D6 BE FA
Debug transfer 41 E4 40 2F 01 E0 0C 06 EE 91
Debug transfer 41 E6 60 3E A1 20 5A 02 3C 6D
Debug transfer 41 E8 80 13 B0 B9 82 C3 03 F4
Debug transfer 41 EC C0 1B A3 0C 28 F1 B0 C8
Debug transfer 41 EE E0 1F CE 07 30 EE 05 06
Debug transfer 41 F1 00 08 ED 80 9C 20 09 73
Debug transfer 41 F3 20 09 E9 00 86 60 21 C3
VideoDecoderaddPacket type: 24
Debug transfer 67 80 80 1E E9 01 68 22 FD C0
Debug transfer 68 06 06 E2
Debug transfer 65 B8 20 00 9F 80 78 00 12 8A
Debug transfer 41 E2 20 09 F0 1E 40 7B 0C E0
Debug transfer 41 E4 40 09 F0 29 30 D6 00 AE
Debug transfer 41 E6 60 09 F1 48 31 80 99 40
[h264 # 000001c771617a40] non-existing PPS 0 referenced
Here I tried Avcc style. You can see the first line is the combination of the sps pps followed by idr and then repeated non idr
Debug transfer 18 00 0E 67 80 80 1E E9 01 68
Debug transfer 00 02 4A 8F 65 B8 20 00 9F C5
Debug transfer 00 02 2F DA 41 E2 20 09 E8 0F
Debug transfer 00 02 2C 34 41 E4 40 09 F4 20
Debug transfer 00 02 4D 92 41 E6 60 09 FC 2B
Debug transfer 00 02 47 02 41 E8 80 09 F0 72
Debug transfer 00 02 52 50 41 EA A0 09 EC 0F
Debug transfer 00 02 58 8A 41 EC C0 09 FC 6F
Debug transfer 00 02 55 F9 41 EE E0 09 FC 6E
Debug transfer 00 02 4D 79 41 F1 00 09 F0 3E
Debug transfer 00 02 4D B6 41 F3 20 09 E8 64
The following class is where I try to get javacv/ffmpeg to show the video. I dont think its an ideal solution and am researching canvasfram as a replacement to the image view.
public class ImageDecoder {
private final static String TAG = "ImageDecoder ";
private ImageDecoder(){
}
public static void streamImageToImageView(
final ImageView view,
final InputStream inputStream,
final String format,
final int frameRate,
final int bitrate,
final String preset,
final int numBuffers
)
{
System.out.println("Image Decoder Starting...");
try( final FrameGrabber grabber = new
FFmpegFrameGrabber(inputStream))
{
final Java2DFrameConverter converter = new Java2DFrameConverter();
grabber.setFrameNumber(frameRate);
grabber.setFormat(format);
grabber.setVideoBitrate(bitrate);
grabber.setVideoOption("preset", preset);
grabber.setNumBuffers(numBuffers);
System.out.println("Image Decoder waiting on grabber.start...");
grabber.start(); //---- this call is blocking the loop
System.out.println("Image Decoder Looping---------------------------
-------- hit stop");
while(!Thread.interrupted()){
//System.out.println("Image Decoder Looping");
final Frame frame = grabber.grab();
if (frame != null){
final BufferedImage bufferedImage =
converter.convert(frame);
if (bufferedImage != null){
Platform.runLater(() ->
view.setImage(SwingFXUtils.toFXImage(bufferedImage, null)));
}else{
System.out.println("no buf im");
}
}else{
System.out.println("no fr");
Thread.currentThread().interrupt();
}
}
}catch (Exception e){
System.out.print(TAG + e);
}
}
}
Any help is greatly appreciated.

So I had two problems.
The first was that my sps pps parsing method had a mistake. Notice the 2nd and 3rd bytes are same
The second was I accidentally over sized a buffer and created large 0x00 padded areas which emulated start codes.
This was a big project for me and I want to help other. Please visit my website where I wrote a lengthy multi-part discussion about streaming h264

Android 5.0 BluetoothGattServer.notifyCharacteristicChanged how to find MTU?

I'm trying to send BLE Notification from android (GAP:central, GATT:server) to peer device (GAP: peripheral, GATT: client).
The problem is that on android 5.0 there is command: BluetoothGatt.requestMtu(int mtu).
But I do not know the way how to find out if peer device support requested MTU and what is actually negotiated MTU.
The required function:
BluetoothGattCallback.onMtuChanged(BluetoothGatt gatt, int mtu, int status)
was added only in API level 22 (Android L 5.1).
My problem is that I do not know how many bytes in packet I can send.
I write a test code to send bigger packet than 20B and it seems android sends just first 20B of data and never tell me that it discarded rest of the data!!! That is terrible behavior. Either I'm missing something, or Android 5.0 is useless for bigger packets than 20 bytes :(
I wrote test code and logs prove that all is sent and returns true:
BluetoothGattCharacteristic mCharacVal;
BluetoothGattServer mGattServer;
...
//log: I/vbeOryNotify: bleNotify snd:msg body; id:27; len:60 :I should have known those alien maggots booby-trapped this s
ret = mCharacVal.setValue(toSnd);
Log.i("vbeOry","bleNotify: setValue "+AppCommon.ByteArrayToHexStr(toSnd)+" ret:"+ret);
//log: I/vbeOry: bleNotify: setValue 03 1B 00 00 00 3C 49 20 73 68 6F 75 6C 64 20 68 61 76 65 20 6B 6E 6F 77 6E 20 74 68 6F 73 65 20 61 6C 69 65 6E 20 6D 61 67 67 6F 74 73 20 62 6F 6F 62 79 2D 74 72 61 70 70 65 64 20 74 68 69 73 20 73
//log: ret:true
byte[] dataRdBck = mCharacVal.getValue();
Log.i("vbeOry","bleNotify: getValue "+AppCommon.ByteArrayToHexStr(dataRdBck));
//log: I/vbeOry: bleNotify: getValue 03 1B 00 00 00 3C 49 20 73 68 6F 75 6C 64 20 68 61 76 65 20 6B 6E 6F 77 6E 20 74 68 6F 73 65 20 61 6C 69 65 6E 20 6D 61 67 67 6F 74 73 20 62 6F 6F 62 79 2D 74 72 61 70 70 65 64 20 74 68 69 73 20 73
ret = mGattServer.notifyCharacteristicChanged(device, mCharacVal, false);
Log.i("vbeOry","bleNotify: notifyCharacteristicChanged "+AppCommon.ByteArrayToHexStr(toSnd)+" ret:"+ret);
//log: I/vbeOry: bleNotify: notifyCharacteristicChanged 03 1B 00 00 00 3C 49 20 73 68 6F 75 6C 64 20 68 61 76 65 20 6B 6E 6F 77 6E 20 74 68 6F 73 65 20 61 6C 69 65 6E 20 6D 61 67 67 6F 74 73 20 62 6F 6F 62 79 2D 74 72 61 70 70 65 64 20 74 68 69 73 20 73
//log: ret:true
Even from call back I do not get any error:
private BluetoothGattServerCallback mGattServerCallback = new BluetoothGattServerCallback() {
public void onNotificationSent(BluetoothDevice device, int status){
Log.i("vbeGattServ", "onNotificationSent status: "+status);
//log: I/vbeGattServ: onNotificationSent status: 0
but then I look with my BLE analyzer and I see that only first 20 B of notification data is send:
So my question is either how to find out negotiated MTU or at least how to find out that not all data was sent? (Constrain is Android 5.0).
Perhaps it has no solution for Android 5.0 :( And I have to stick with 20B even when both device could support higher. Only work around would be to implement mechanism for returning MTU from peer device as was suggested here on stack overflow.

Your issue is not Android but Bluetooth / the app running on your central.
Increasing the useful data size from 20B is only possible with BLE v4.2 and newer. I suppose your radio is running with BLE v4.0 or v4.1 or there is an issue setting up your MTU.
Either way this should not affect reading data. The central device can read a characteristic with an offset. Assuming you have 60B, you would have 3 requests from the central:
Read, offset=0
Read, offset=20
Read, offset=40
TL;DR: Your GATT central simply handles reads wrong.

WhatsApp video as Gif sharing on Android programatically

How can i convert a mp4 video file to a WhatsApp gif file (it's simple showed as gif inside app UI but internally is a specific mp4 format) to be use in android share intent, being recognized as this type of media by whatsapp chat app???
I search a lot but i can't find any information from WhatsApp docs (they don't have this kind of doc anyway) or any dev with the same problem as i.
WHAT I HAVE:
I have discovered that at beginning of whatsapp "gif" mp4 files is present a loop value if you read they on hex editor, all files have this. Remove this value make whatsapp receive as regular video (not sharing as gif).
How can i add this value using ffmpeg encoding? (editing my mp4 files manually with this value corrupt the files, maybe i have to fix some mp4 header index that i don't know yet...)
FIRST 80 BYTES in hexadecimal (from beginning to start of "moov" atom from mp4 structure):
00 00 00 1C 66 74 79 70 6D 70 34 32 00 00 00 01 6D 70 34 31 6D 70 34 32 69 73 6F 6D 00 00 00 18 62 65 61 6D 01 00 00 00 01 00 00 00 00 00 00 00 05 00 00 00 00 00 00 0C 6C 6F 6F 70 00 00 00 00 00 00 00 08 77 69 64 65 00 00 04 9F 6D 6F 6F 76
A short mp4 file generated by WhatsApp that internally (at app) was showed as a Gif (with different UI):
https://www.dropbox.com/s/kpynmx1bg3z76lz/VID-20171024-WA0009.mp4?dl=0

"...The problem is that I can't edit another MP4 file to add this atom
without corrupt the file.
Test this small_VC1edit.mp4 in WhatsApp. If it does what you want then read on...
To make a playable MP4 :
Using original small.mp4 as an editing example (download file and open with a hex editor).
1) In a blank byte array, add the first 72 bytes of the shown WhatsApp-style MP4 header.
00 00 00 1C 66 74 79 70 6D 70 34 32 00 00 00 01 6D 70 34 31 6D 70 34 32 69 73 6F 6D 00 00 00 18 62 65 61 6D 01 00 00 00 01 00 00 00 00 00 00 00 05 00 00 00 00 00 00 0C 6C 6F 6F 70 00 00 00 00 00 00 00 08 77 69 64 65
You've shown 80 bytes but the last 8 bytes are not needed yet (also four of those eight byte's values must be different for your output file).
2) Calculate the deltas.
Note the (new) WhatsApp header is 72 bytes (before moov atom).
Note the (original) Small.mp4 has 160 bytes of header (before moov atom).
So just use this logic (a or b):
a) If WhatsApp header is bigger than input MP4 :
delta = ( WhatsApp_header - input_MP4_header)
b) If input MP4 header is bigger than WhatsApp :
delta = ( input_MP4_header - WhatsApp_header )
So for the input small.mp4 which has 160 header bytes (is followed next by 4 bytes of moov's SIZE (as 00 00 0D 83) and then follows another 4 bytes now of moov's NAME (as 6D 6F 6F 76 or utf-8 text of "moov").
We can say : 160 MP4 bytes - 72 WhatsApp bytes = Delta of 88.
If you delete these original 160 bytes and replace them with the shorter 72 WhatsApp bytes, they'll be 88 less bytes which must now be accounted for in the another section of MOOV data. That section is the STCO atom.
3) Update the STCO atom with new offsets:
In small.mp4 the STCO atom begins at offset 1579 (as 73 74 63 6F). The previous 4 bytes (offsets: 1575 to 1578) are stco's SIZE bytes (as 00 00 00 B8) which is decimal value 184. This total SIZE of bytes length includes accounting for those 4 SIZE bytes too.
Skip 12 bytes from the starting byte 73 of the stco's NAME bytes 73 74 63... so skip these:
73 74 63 6F 00 00 00 00 00 00 00 2A
Now you reach point to sequentially update every 32-bit integer (4 bytes) of offsets with the new delta value. But how many offsets to update?
atomEditTotal = ( (stco_SIZE - 16) / 4); //gives 42 //PS: Minus by 16 is to trim off non-offset bytes.
So there are 42 entries to edit. Our Delta as 88 so for each integer we read value, minus it by 88, then write it back new value at same place, repeat another 41 times (using While loop with an if condition to break; the loop).
For testing, given a corrupt file, if you edit the first entry it should be enough to show frame 1 of video (if non-audio file).
PS: After editing the STCO offsets of small.mp4, just delete its starting 160 bytes and join/concat those remaining MP4 bytes to the back/end of the 72 byte WhatsApp header. Save array as new file and test.

Error in TLS handshake when using mutual authentication

While testing my app (https://play.google.com/apps/testing/com.degoo.android) I've found that on some devices the TLS handshake between the app and our servers fail if the https request requires a client certificate (i.e. mutual authentication). The same code works on Windows and on OS X, so I know that it's not caused by an incorrect cert or that I have forgotten to include the cert into the SSLContext. I have detected it on both some Android 4.4 devices and on some 5.0 devices. Unfortunately I haven't found any common denominator which causes it to fail (other than Android). However, on devices were it fails it fails 100% of the time.
I've analyzed the network traffic to see more precisely when the error occurs. The following thing works on all devices:
The connection is established and the client successfully validates the server's certs.
The client sends it's HTTP request.
The server detects that the request is for protected area and sends a certificate_request.
The client decides which client certificate to send (we can see that it selects the correct certificate) and sends it.
After the client has responded with its certificate it sends two more TLS records and then the handshake fails.
Here's an example of how what these two TLS records looks like on a device were the handshake SUCCEEDS:
14 03 01 00 20 9c 07 49 78 9f ba 09 03 41 6b 66 ad 46 e2 75 94 f7 cf 18 bd 11 cf 35 a2 eb 5e b8 a8 4c 2a 1d c5
16 03 01 00 30 20 0e 13 d7 48 b9 6e b2 1b 96 6f 10 56 67 81 63 d9 d8 c7 73 23 95 3b f9 da f9 ce f4 f8 d1 7e 1b a4 12 92 4c 4f 54 a5 f8 49 75 d5 46 f4 2d 29 97
Here's an example of how what these two TLS records looks like on a device were the handshake FAILS:
14 03 01 00 20 a9 86 c2 fd 03 0a f8 08 fa f8 9e eb b7 97 07 56 6f 27 c0 d6 8f 95 be 77 c1 44 84 e9 e1 56 6f 6a
16 03 01 00 30 ff 36 76 e5 47 87 84 71 1c ce c9 08 41 45 fc 09 c6 ef 08 e6 21 ff 45 3a 10 ae 8d 5a 99 5f ca c5 ac bd bf 7e ca 69 32 4d 1f 01 c6 30 83 8e 06 cb
From the first byte of the records I can see that both clients send a change cipher spec record and then a handshake record. A funny thing about the failing devices is that byte with index 5 of the second record has value ff. When I look in TLS specification I don't see any handshake record type with that value. The device that succeeds has that byte set to 20, which means "finished".
Any idea on what's going on? What could cause this?

ISO 7816-4 APDU command Select File and Write Record

I am trying to communicate between PN532 and HCE on Android with ISO 7816-4 command, I am successfully select the AID (a DF), but when I select the EF under that DF (that EF does not exist, so I assume that Select command will create that EF), and then write the records to that EF but it display like this:
inList passive target
write: 4A 1 0
read: 4B 1 1 0 4 60 4 8 23 5A 4D 5 75 80 70 2
write: 40 1 0 A4 4 0 7 F0 1 2 3 4 5 6 0
read: 41 0 48 65 6C 6C 6F 20 44 65 73 6B 74 6F 70 21
Successfully hehe
48 65 6C 6C 6F 20 44 65 73 6B 74 6F 70 21 Hello Desktop!
write: 40 1 0 A4 2 C 1 1 0
read: 41 0 48 65 6C 6C 6F 20 44 65 73 6B 74 6F 70 21
Not enough space
write: 40 1 0 D2 0 0 7 42 41 4F 47 49 41 40 0
read: 41 0 4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 30
Not enough space
write: 40 1 0 D2 0 2 4 44 4F 41 4E 0
read: 41 0 4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 31
Not enough space
write: 40 1 0 B2 0 0 7 42 41 4F 47 49
read: 41 0 4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 32
Not enough space
write: 40 1 0 B2 0 2 4 44 4F 41 4E 0
read: 41 0 4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 33
Not enough space
I don't know what I am doing wrong here?
On Android, the log is :
04-15 09:36:54.024: D/HostEmulationManager(929): notifyHostEmulationData
04-15 09:36:54.024: W/System.err(17710): [B#41ed5970
04-15 09:36:54.024: I/HCEDEMO(17710): Received: ???????BAOGI
04-15 09:36:54.024: D/HostEmulationManager(929): Sending data
04-15 09:36:54.164: D/BrcmNfcJni(929): RoutingManager::stackCallback: event=0x17
04-15 09:36:54.164: D/BrcmNfcJni(929): RoutingManager::stackCallback: NFA_CE_DATA_EVT; h=0x302; data len=10
04-15 09:36:54.164: D/HostEmulationManager(929): notifyHostEmulationData
04-15 09:36:54.164: W/System.err(17710): [B#41ed5e20
04-15 09:36:54.164: I/HCEDEMO(17710): Received: ?????DOAN??
04-15 09:36:54.174: D/HostEmulationManager(929): Sending data
04-15 09:36:54.885: D/BrcmNfcJni(929): RoutingManager::stackCallback: event=0x19
04-15 09:36:54.885: D/HostEmulationManager(929): notifyHostEmulationDeactivated
04-15 09:36:54.885: I/HCEDEMO(17710): Deactivated: 0
04-15 09:36:54.885: D/HostEmulationManager(929): Unbinding from service ComponentInfo{de.grundid.hcedemo/de.grundid.hcedemo.MyHostApduService}
04-15 09:36:54.895: E/BrcmNfcNfa(929): UICC[0x0] is not activated
It displays that it can receive some data, but it misses some elements I want to transmit, but, from PN532, when I use read records, it does not display these data?

The commands that your Android HCE emulated smartcard application understands and processes are completely up to you (as long as they are formatted as valid ISO 7816-4 APDUs).
In your case, your Android HCE service obviously processes the SELECT (by DF name) APDU,
00 A4 04 00 07 F0010203040506 00
and gives this as a response:
48 65 6C 6C 6F 20 44 65 73 6B 74 6F 70 21 ("Hello Desktop!" when interpreted as ASCII)
(Note that this response is not a valid response APDU according to ISO 7816-4 as it lacks a status word.)
The next command you send is an invalid SELECT (by EF) command:
00 A4 02 0C 01 01 00
For that command, Lc should be 2 and the EF identifier should consist of two bytes if following ISO 7816-4. In response to that, your Android HCE service again sends
48 65 6C 6C 6F 20 44 65 73 6B 74 6F 70 21 ("Hello Desktop!" when interpreted as ASCII)
(Note that this response is not a valid response APDU according to ISO 7816-4 as it lacks a status word.)
So I would guess, that your Android HCE service performs a check like this:
public byte[] processCommandApdu(byte[] apdu, Bundle extras) {
if (apdu[1] == (byte)0xA4) {
return "Hello Desktop!".getBytes("US-ASCII");
}
}
The next command that you send is a malformed WRITE RECORD command that tries to write "BAOGIA#" in the first record of the cuirrently selected file (malformed, because a WRITE RECORD command normally does not have an Le field):
00 D2 00 00 07 42 41 4F 47 49 41 40 00
As a response your Android HCE service sends:
4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 30 ("Message from android: 0" when interpreted as ASCII)
(Note that this response is again not a valid response APDU according to ISO 7816-4 as it lacks a status word.)
You then repeat the WRITE RECORD command with a different record payload and after that you send two malformed READ RECORD commands:
00 D2 00 02 04 44 4F 41 4E 00
00 B2 00 00 07 42 41 4F 47 49
00 B2 00 02 04 44 4F 41 4E 00
As a response your Android HCE service sends:
4D 65 73 73 61 67 65 20 66 72 6F 6D 20 61 6E 64 72 6F 69 64 3A 20 xx ("Message from android: X" when interpreted as ASCII)
Where xx seems to be an ASCII digit X that is incremented for each received command.
So I would guess, that your Android HCE service looks like this:
private int mCommandCounter = 0;
public byte[] processCommandApdu(byte[] apdu, Bundle extras) {
String response;
if (apdu[1] == (byte)0xA4) {
response = "Hello Desktop!";
} else {
response = "Message from android: " + Integer.toString(mCommandCounter);
++mCommandCounter;
}
return response.getBytes("US-ASCII");
}
So, to summarize this, your Android HCE service will understand and process only those commands that you (or whoever develops it) implement. So it is up to you what commands you can send to the HCE device. There is no file system behind it. ISO 7816-4 only suggests a file system layout for smartcard applications.