Develop Reference

Does RTMP support the Display Orientation SEI Message in h264 streams?

I'm streaming video h264 video and AAC audio over RTMP on Android using the native MediaCodec APIs. Video and audio look great, however while the video is shot in potrait mode, playback on the web or with VLC is always in landscape.
Having read through the h264 spec, I see that this sort of extra metadata can be specified in Supplemental Enhancement Information (SEI), and I've gone about adding it to the raw h264 bit stream. My SEI NAL unit for this follows this rudimentary format, I plan to optimize later:
val displayOrientationSEI = {
val prefix = byteArrayOf(0, 0, 0, 1)
val nalHeader = byteArrayOf(6) // forbidden_zero_bit:0; nal_ref_idc:0; nal_unit_type:6
val display = byteArrayOf(47 /* Display orientation type*/, 3 /*payload size*/)
val displayOrientationCancelFlag = "0" // u(1); Rotation information follows
val horFlip = "1" // hor_flip; u(1); Flip horizontally
val verFlip = "1" // ver_flip; u(1); Flip vertically
val anticlockwiseRotation = "0100000000000000" // u(16); value / 2^16 -> 90 degrees
val displayOrientationRepetitionPeriod = "010" // ue(v); Persistent till next video sequence
val displayOrientationExtensionFlag = "0" // u(1); No other value is permitted by the spec atm
val byteAlignment = "1"
val bitString = displayOrientationCancelFlag +
horFlip +
verFlip +
anticlockwiseRotation +
displayOrientationRepetitionPeriod +
displayOrientationExtensionFlag +
byteAlignment
prefix + nalHeader + display + BigInteger(bitString, 2).toByteArray()
}()
Using Jcodec's SEI class, I can see that my SEI message is parsed properly. I write out these packets to the RTMP stream using an Android JNI wrapper for LibRtmp.
Despite this, ffprobe does not show the orientation metadata, and the video when played remains in landscape.
At this point I think I'm missing a very small detail about how FLV headers work when the raw h264 units are written out by LibRtmp. I have tried appending this displayOrientationSEI NAL unit:
To the initial SPS and PPS configuration only.
To each raw h264 NAL units straight from the encoder.
To both.
What am I doing wrong? Going through the source of some RTMP libraries, like rtmp-rtsp-stream-client-java, it seems the SEI message is dropped when creating FLV tags.
Help is much, much appreciated.

Does RTMP support the Display Orientation SEI Message in h264 streams?
RTMP is unaware of the very concept. from RTMPs perspective, the SEI is just a series of bytes it copys. It never looks at them, it never parses them.
The thing that needs to support it, is the h.264 decoder (which RTMP is also unaware of) and the player software. If it is not working for you, you must check the player, or the validity of the encoded SEI, not the transport.

HEVC Encoder (convert from JPG to HEIC)

I have some trouble when converting from JPG to HEIC format (using HEVC Encoding) on Android
I'm using Nokia HEIF's library (https://github.com/nokiatech/heif/) for parsing data which I need to put some data includes:
Decoder config data (as a byte array) following the name called by Nokia, I think that data can be profile data, header data, ... that store the VPS/PPS, Exif, metadata ...
Image data following the name called by Nokia -> This may is a bitstream of HEVC Encoding.
So now, I got the image data bitstream (2) based FFMPEG tool by command as bellow:
ffmpeg -i oktest.png -crf 12 -preset medium -pix_fmt yuv420p -f hevc.bin
This is a referring link from Nokia that they guide how to do feed data and encode to HEIF format.
https://github.com/nokiatech/heif/issues/44
But, I didn't understand how I can get the decoder config data from the FFMPEG tool because I'm developing on Android platforms instead. I mean some definitions are determined different from the mind so I can't find any solutions for this. I already request a ticket to Nokia support, but no response yet.
val heif = HEIF()
val decoderConfig = context.assets.open(bitStreamDecoderConfig).readBytes()
val decoderConfig = decoderBuffer?.array() ?: ByteArray(0)
val imageData = FileInputStream(bitStreamData).readBytes()
val imageItem =
HEVCImageItem(heif, Size(imageWidth, imageHeight), decoderConfig, imageData)
heif.primaryImage = imageItem
heif.majorBrand = HEIF.BRAND_MIF1
heif.addCompatibleBrand(HEIF.BRAND_HEIC)
heif.save("$targetOutputFolder/$fileNameNoneExtension.${destinationType()}")

There is a page at http://jpgtoheif.com/ which suggest use ffmpeg itself instead the nokia tool.
there is another solution libheif with
a windows build is at https://github.com/pphh77/libheif-Windowsbinary/releases

Android MediaCodec : Decoding is success but encoded file is not playable

I am trying to write a video compression application that will run on Jellybean version of Android. Till now I could decode the given input to video/raw format and it is playable with mplayer. My problem is that when I Encode this video/raw item into video/avc format with width = 320, height = 240, bitRate = (480*1024), frameRate = 20, iFrameInterval = 7 and colorFormat = YUV420Planar, the output file is not playable by just double clicking on it. Can anyone suggest me a way to play it using any player.? Or can you tell me if I had made any mistake in giving the above parameters like bitrate, framerate etc..
Thanks in advance.! :)

Android - Include native StageFright features in my own project

I am currently developing an application that needs to record audio, encode it as AAC, stream it, and do the same in reverse - receiving stream, decoding AAC and playing audio.
I successfully recorded AAC (wrapped in a MP4 container) using the MediaRecorder, and successfully up-streamed audio using the AudioRecord class. But, I need to be able to encode the audio as I stream it, but none of these classes seem to help me do that.
I researched a bit, and found that most people that have this problem end up using a native library like ffmpeg.
But I was wondering, since Android already includes StageFright, that has native code that can do encoding and decoding (for example, AAC encoding and AAC decoding), is there a way to use this native code on my application? How can I do that?
It would be great if I only needed to implement some JNI classes with their native code. Plus, since it is an Android library, it would be no licensing problems whatever (correct me if I'm wrong).

yes, you can use libstagefright, it's very powerful.
Since stagefright is not exposed to NDK, so you will have to do extra work.
There are two ways:
(1) build your project using android full source tree. This way takes a few days to setup, once ready, it's very easy, and you can take full advantage of stagefright.
(2) you can just copy include file to your project, it's inside this folder:
android-4.0.4_r1.1/frameworks/base/include/media/stagefright
then you will have export the library function by dynamically loading libstagefright.so, and you can link with your jni project.
To encode/decode using statgefright, it's very straightforward, a few hundred of lines can will do.
I used stagefright to capture screenshots to create a video, which will be available in our Android VNC server, to be released soon.
the following is a snippet, I think it's better than using ffmpeg to encode a movie. You can add audio source as well.
class ImageSource : public MediaSource {
ImageSource(int width, int height, int colorFormat)
: mWidth(width),
mHeight(height),
mColorFormat(colorFormat)
{
}
virtual status_t read(
MediaBuffer **buffer, const MediaSource::ReadOptions *options) {
// here you can fill the buffer with your pixels
}
...
};
int width = 720;
int height = 480;
sp<MediaSource> img_source = new ImageSource(width, height, colorFormat);
sp<MetaData> enc_meta = new MetaData;
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_H263);
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_MPEG4);
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
enc_meta->setInt32(kKeyWidth, width);
enc_meta->setInt32(kKeyHeight, height);
enc_meta->setInt32(kKeySampleRate, kFramerate);
enc_meta->setInt32(kKeyBitRate, kVideoBitRate);
enc_meta->setInt32(kKeyStride, width);
enc_meta->setInt32(kKeySliceHeight, height);
enc_meta->setInt32(kKeyIFramesInterval, kIFramesIntervalSec);
enc_meta->setInt32(kKeyColorFormat, colorFormat);
sp<MediaSource> encoder =
OMXCodec::Create(
client.interface(), enc_meta, true, image_source);
sp<MPEG4Writer> writer = new MPEG4Writer("/sdcard/screenshot.mp4");
writer->addSource(encoder);
// you can add an audio source here if you want to encode audio as well
//
//sp<MediaSource> audioEncoder =
// OMXCodec::Create(client.interface(), encMetaAudio, true, audioSource);
//writer->addSource(audioEncoder);
writer->setMaxFileDuration(kDurationUs);
CHECK_EQ(OK, writer->start());
while (!writer->reachedEOS()) {
fprintf(stderr, ".");
usleep(100000);
}
err = writer->stop();

How to calculate FrameLength without using AudioSystem?

I'm porting Java project to Android.
As you know, the Android does not have javax.sound package.
I need to calculate frameLength.
My sound file size is 283KB. And frame size is 4 and frame rate is 44100 and sample size in bits is 16.
The frame length was 69632 when I used just pure Java.
Do you know any equation to get this?
Thank you.

For the raw PCM data, basically, you have 4 bytes per sample.
((283 KB) * (1024 bytes per KB)) / (4 bytes per frame) ==> 72448
But a wav can include a lot besides the raw PCM, e.g., song title or artist info.
Here's some more info about wav format. You might have to load the file as raw bytes to parse the header, but the header has the frame size in a predictable location.
Maybe someone else with Android experience has already concocted a method.
Maybe Google should treat Java as an intact entity and properly license and implement it.