Develop Reference

How to record screen and take screenshots, using Android API?

Background
Android got a new API on Kitkat and Lollipop, to video capture the screen. You can do it either via the ADB tool or via code (starting from Lollipop).
Ever since the new API was out, many apps came to that use this feature, allowing to record the screen, and Microsoft even made its own Google-Now-On-tap competitor app.
Using ADB, you can use:
adb shell screenrecord /sdcard/video.mp4
You can even do it from within Android Studio itself.
The problem
I can't find any tutorial or explanation about how to do it using the API, meaning in code.
What I've found
The only place I've found is the documentations (here, under "Screen capturing and sharing"), telling me this:
Android 5.0 lets you add screen capturing and screen sharing
capabilities to your app with the new android.media.projection APIs.
This functionality is useful, for example, if you want to enable
screen sharing in a video conferencing app.
The new createVirtualDisplay() method allows your app to capture the
contents of the main screen (the default display) into a Surface
object, which your app can then send across the network. The API only
allows capturing non-secure screen content, and not system audio. To
begin screen capturing, your app must first request the user’s
permission by launching a screen capture dialog using an Intent
obtained through the createScreenCaptureIntent() method.
For an example of how to use the new APIs, see the MediaProjectionDemo
class in the sample project.
Thing is, I can't find any "MediaProjectionDemo" sample. Instead, I've found "Screen Capture" sample, but I don't understand how it works, as when I've run it, all I've seen is a blinking screen and I don't think it saves the video to a file. The sample seems very buggy.
The questions
How do I perform those actions using the new API:
start recording, optionally including audio (mic/speaker/both).
stop recording
take a screenshot instead of video.
Also, how do I customize it (resolution, requested fps, colors, time...)?

First step and the one which Ken White rightly suggested & which you may have already covered is the Example Code provided officially.
I have used their API earlier. I agree screenshot is pretty straight forward. But, screen recording is also under similar lines.
I will answer your questions in 3 sections and will wrap it up with a link. :)
1. Start Video Recording
private void startScreenRecord(final Intent intent) {
if (DEBUG) Log.v(TAG, "startScreenRecord:sMuxer=" + sMuxer);
synchronized(sSync) {
if (sMuxer == null) {
final int resultCode = intent.getIntExtra(EXTRA_RESULT_CODE, 0);
// get MediaProjection
final MediaProjection projection = mMediaProjectionManager.getMediaProjection(resultCode, intent);
if (projection != null) {
final DisplayMetrics metrics = getResources().getDisplayMetrics();
final int density = metrics.densityDpi;
if (DEBUG) Log.v(TAG, "startRecording:");
try {
sMuxer = new MediaMuxerWrapper(".mp4"); // if you record audio only, ".m4a" is also OK.
if (true) {
// for screen capturing
new MediaScreenEncoder(sMuxer, mMediaEncoderListener,
projection, metrics.widthPixels, metrics.heightPixels, density);
}
if (true) {
// for audio capturing
new MediaAudioEncoder(sMuxer, mMediaEncoderListener);
}
sMuxer.prepare();
sMuxer.startRecording();
} catch (final IOException e) {
Log.e(TAG, "startScreenRecord:", e);
}
}
}
}
}
2. Stop Video Recording
private void stopScreenRecord() {
if (DEBUG) Log.v(TAG, "stopScreenRecord:sMuxer=" + sMuxer);
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.stopRecording();
sMuxer = null;
// you should not wait here
}
}
}
2.5. Pause and Resume Video Recording
private void pauseScreenRecord() {
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.pauseRecording();
}
}
}
private void resumeScreenRecord() {
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.resumeRecording();
}
}
}
Hope the code helps. Here is the original link to the code that I referred to and from which this implementation(Video recording) is also derived from.
3. Take screenshot Instead of Video
I think by default its easy to capture the image in bitmap format. You can still go ahead with MediaProjectionDemo example to capture screenshot.
[EDIT] : Code encrypt for screenshot
a. To create virtual display depending on device width / height
mImageReader = ImageReader.newInstance(mWidth, mHeight, PixelFormat.RGBA_8888, 2);
mVirtualDisplay = sMediaProjection.createVirtualDisplay(SCREENCAP_NAME, mWidth, mHeight, mDensity, VIRTUAL_DISPLAY_FLAGS, mImageReader.getSurface(), null, mHandler);
mImageReader.setOnImageAvailableListener(new ImageAvailableListener(), mHandler);
b. Then start the Screen Capture based on an intent or action-
startActivityForResult(mProjectionManager.createScreenCaptureIntent(), REQUEST_CODE);
Stop Media projection-
sMediaProjection.stop();
c. Then convert to image-
//Process the media capture
image = mImageReader.acquireLatestImage();
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * mWidth;
//Create bitmap
bitmap = Bitmap.createBitmap(mWidth + rowPadding / pixelStride, mHeight, Bitmap.Config.ARGB_8888);
bitmap.copyPixelsFromBuffer(buffer);
//Write Bitmap to file in some path on the phone
fos = new FileOutputStream(STORE_DIRECTORY + "/myscreen_" + IMAGES_PRODUCED + ".png");
bitmap.compress(CompressFormat.PNG, 100, fos);
fos.close();
There are several implementations (full code) of Media Projection API available.
Some other links that can help you in your development-
Video Recording with MediaProjectionManager - website
android-ScreenCapture - github as per android developer's observations :)
screenrecorder - github
Capture and Record Android Screen using MediaProjection APIs - website
Hope it helps :) Happy coding and screen recording!
PS: Can you please tell me the Microsoft app you are talking about? I have not used it. Would like to try it :)

example of AlwaysOnHotwordDetector in Android

Can someone provide an example of how to use the new AlwaysOnHotwordDetector class in Android?
I'd like to build an app, that when the app is running in the background, can detect a hotword like "next", or "back", or "pause".

Unless I have a huge blind spot, I don't think third-party applications can make use of this API. Its strange that AlwaysOnHotwordDetector (and related classes VoiceInteractionService etc.) have been granted public access.
If you are building a privileged app, look through these test projects from AOSP:
Voice Interaction - Basic AlwaysOnHotwordDetector usage/implementation
http://androidxref.com/5.0.0_r2/xref/frameworks/base/tests/VoiceInteraction/
Voice Enrollment - http://androidxref.com/5.0.0_r2/xref/frameworks/base/tests/VoiceEnrollment/
While trying to make this work, I came upon this:
AlwaysOnHotwordDetector's constructor:
/**
* #param text The keyphrase text to get the detector for.
* #param locale The java locale for the detector.
* #param callback A non-null Callback for receiving the recognition events.
* #param voiceInteractionService The current voice interaction service.
* #param modelManagementService A service that allows management of sound models.
*
* #hide
*/
public AlwaysOnHotwordDetector(String text, Locale locale, Callback callback,
KeyphraseEnrollmentInfo keyphraseEnrollmentInfo,
IVoiceInteractionService voiceInteractionService,
IVoiceInteractionManagerService modelManagementService) {
mText = text;
mLocale = locale;
mKeyphraseEnrollmentInfo = keyphraseEnrollmentInfo;
mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);
mExternalCallback = callback;
mHandler = new MyHandler();
mInternalCallback = new SoundTriggerListener(mHandler);
mVoiceInteractionService = voiceInteractionService;
mModelManagementService = modelManagementService;
new RefreshAvailabiltyTask().execute();
}
The statement of interest here is:
mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);
What does KeyphraseEnrollmentInfo#getKeyphraseMetadata(String, Locale) do?
/**
* Gets the {#link KeyphraseMetadata} for the given keyphrase and locale, null if any metadata
* isn't available for the given combination.
*
* #param keyphrase The keyphrase that the user needs to be enrolled to.
* #param locale The locale for which the enrollment needs to be performed.
* This is a Java locale, for example "en_US".
* #return The metadata, if the enrollment client supports the given keyphrase
* and locale, null otherwise.
*/
public KeyphraseMetadata getKeyphraseMetadata(String keyphrase, Locale locale) {
if (mKeyphrases == null || mKeyphrases.length == 0) {
Slog.w(TAG, "Enrollment application doesn't support keyphrases");
return null;
}
for (KeyphraseMetadata keyphraseMetadata : mKeyphrases) {
// Check if the given keyphrase is supported in the locale provided by
// the enrollment application.
if (keyphraseMetadata.supportsPhrase(keyphrase)
&& keyphraseMetadata.supportsLocale(locale)) {
return keyphraseMetadata;
}
}
Slog.w(TAG, "Enrollment application doesn't support the given keyphrase/locale");
return null;
}
At this point, my example project kept telling me: Enrollment application doesn't support keyphrases. So, digging a bit further - to support keyphrases, we need to provide additional meta-data:
// Taken from class KeyphraseEnrollmentInfo
/**
* Name under which a Hotword enrollment component publishes information about itself.
* This meta-data should reference an XML resource containing a
* <code><{#link
* android.R.styleable#VoiceEnrollmentApplication
* voice-enrollment-application}></code> tag.
*/
private static final String VOICE_KEYPHRASE_META_DATA = "android.voice_enrollment";
Additionally, we will need the android.permission.MANAGE_VOICE_KEYPHRASES permission. This is where the example project gets stuck:
<!-- Must be required by hotword enrollment application,
to ensure that only the system can interact with it.
#hide <p>Not for use by third-party applications.</p> -->
<permission android:name="android.permission.MANAGE_VOICE_KEYPHRASES"
android:label="#string/permlab_manageVoiceKeyphrases"
android:description="#string/permdesc_manageVoiceKeyphrases"
android:protectionLevel="signature|system" />
The permission required to support hotword detection is not available to third-party applications. I still can't figure out why package android.service.voice package has public access. Perhaps I am missing something here.

What is MINI_THUMB_MAGIC and how to use it?

Background
I've noticed a weird column for MediaStore.Images.ImageColumns called "MINI_THUMB_MAGIC" .
the documentation says just that :
The mini thumb id.
Type: INTEGER
Constant Value: "mini_thumb_magic"
The question
my guess is that this field is related to MediaStore.Images.Thumbnails .
Is it correct ? if not, what is this and how do you use it?
if it is correct , i have other questions related to it:
Is it a mini sized image of the original one? does it use the same aspect ratio or does it do center-cropping on it?
how come the size of "MICRO" is square (96 x 96) and the size of "MINI" is a non-square rectangle ( 512 x 384 ) ?
How do you use it? My guess is that it's done by using "THUMB_DATA", which is a blob, so you use it like this, but then what is the purpose of using "getThumbnail" if you already have this field?
does it get a rotated thumbnail in case the orientation value is not 0 ? meaning that if I wish to show it, I won't need to rotate the image?
Is it possible to do a query of the images together with their thumbnails? maybe using inner join?
Is it available for all Android devices and versions?
Why is it even called "magic" ? Is it because it's also available for videos (and for some reason doesn't exist for music, as it could be the album's cover photo, for example) ?

Check this file: https://github.com/android/platform_packages_providers_mediaprovider/blob/master/src/com/android/providers/media/MediaThumbRequest.java in the Android source code. This value is some magic number which allows to determine if the thumbnail is still valid. I didn't investigate that file further, but it should be no bit issue to dive deeper.
To your questions:
No, no mini-sized image
Well, I guess it's a definition by Google who want to have a square thumbnail for some lists, where only very small previews should be visible and where many items should fit on the screen and there's another thumbnail format where the images are bigger...
I don't know that, but according to Google's doc, one (THUMB_DATA) is only some raw byte array of the thumbnail (dunno in which format) and the other one (getThumbnail) retrieves a full-fledged bitmap object...
don't know
don't know
I guess so, as it's part of AOSP source code.
The word "magic" is often used for some kind of identifier. There are "magic packets" who can wake up a computer from sleep or shutdown over the network, there are magic numbers on hard disks, where some sectors (e.g. the MBR) has the hexadecimal values AA 55 on its last two byte positions, there are also magic numbers in image files which help software packages determine the image type (e.g. GIF files begin with GIF89a or GIF87a (ASCII), JPEG files begin with FF D8 hexadecimal) and there are many, many more examples. So, magic numbers are a very common term here :-)

According to the source code at the following URL, the Magic Number is the Id of the original image * a constant. That value is then used to check for a long int. If the int isn't as expected, it's considered out of sync with the image media.
http://grepcode.com/file/repo1.maven.org/maven2/org.robolectric/android-all/4.4_r1-robolectric-0/android/media/MiniThumbFile.java#MiniThumbFile.getMagic%28long%29
// Get the magic number for the specified id in the mini-thumb file.
// Returns 0 if the magic is not available.
public synchronized long getMagic(long id) {
// check the mini thumb file for the right data. Right is
// defined as having the right magic number at the offset
// reserved for this "id".
RandomAccessFile r = miniThumbDataFile();
if (r != null) {
long pos = id * BYTES_PER_MINTHUMB;
FileLock lock = null;
try {
mBuffer.clear();
mBuffer.limit(1 + 8);
lock = mChannel.lock(pos, 1 + 8, true);
// check that we can read the following 9 bytes
// (1 for the "status" and 8 for the long)
if (mChannel.read(mBuffer, pos) == 9) {
mBuffer.position(0);
if (mBuffer.get() == 1) {
return mBuffer.getLong();
}
}
} catch (IOException ex) {
Log.v(TAG, "Got exception checking file magic: ", ex);
} catch (RuntimeException ex) {
// Other NIO related exception like disk full, read only channel..etc
Log.e(TAG, "Got exception when reading magic, id = " + id +
", disk full or mount read-only? " + ex.getClass());
} finally {
try {
if (lock != null) lock.release();
}
catch (IOException ex) {
// ignore it.
}
}
}
return 0;
}
I got the runtime exception when trying to get the original Id of a thumbnail by looking up the thumbnail's path. (BTW, the disk isn't full and it's not read-only.)

It's a bit strange parameter...
While exploring the Gallery source code,
noticed that the value is being read from the cursor, but then is Never used:
#Override
protected BaseImage loadImageFromCursor(Cursor cursor) {
long id = cursor.getLong(INDEX_ID);
String dataPath = cursor.getString(INDEX_DATA_PATH);
long dateTaken = cursor.getLong(INDEX_DATE_TAKEN);
if (dateTaken == 0) {
dateTaken = cursor.getLong(INDEX_DATE_MODIFIED) * 1000;
}
// here they read it ====>>
long miniThumbMagic = cursor.getLong(INDEX_MINI_THUMB_MAGIC);
int orientation = cursor.getInt(INDEX_ORIENTATION);
String title = cursor.getString(INDEX_TITLE);
String mimeType = cursor.getString(INDEX_MIME_TYPE);
if (title == null || title.length() == 0) {
title = dataPath;
}
// and not use at all ==>>>
return new Image(this, mContentResolver, id, cursor.getPosition(),
contentUri(id), dataPath, mimeType, dateTaken, title,
orientation);
}
Maybe it was used on the previous APIs.
ref: https://android.googlesource.com/platform/packages/apps/Gallery/+/android-8.0.0_r12/src/com/android/camera/gallery/ImageList.java?autodive=0%2F%2F.
and videos list:
https://android.googlesource.com/platform/packages/apps/Gallery/+/android-8.0.0_r12/src/com/android/camera/gallery/VideoList.java?autodive=0%2F%2F

Get private photos using flickrj-android

I am using the open source library: https://code.google.com/p/flickrj-android/ and there is an example how do I get photos from flickr. Main problem is that I get only public photos. How can I manage getting private streams/photos?
Did anyone managed to get private streams?

With Flickrj-android you'd want to use this method:
Flickr flickr = new Flickr(API_KEY,SHARED_SECRET,new REST());
Set<String> extras = new HashSet();
// A set of extra info we want Flickr to give back. Go to the API page to see the other size options available.
extras.add("url_o");
extras.add("original_format");
//A request for a list of the photos in a set. The first zero is the privacy filter,
// the second is the Pages, and the third is the Per-Page (see the Flickr API)
PhotoList<Photo> photoList = flickr.getPhotosetsInterface().getPhotos(PHOTOSET_ID, extras, 0, 0, 0);
//We'll use the direct URL to the original size of the photo in order to download it. Remember: you want to make as few requests from flickr as possible!
for(Photo photo : photoList){
//You can also get other sizes. Just ask for the info in the first request.
URL url = new URL(photo.getOriginalSize().getSource());
InputStream is = url.openStream();
OutputStream os = new FileOutputStream(PATH_OF_FOLDER + photo.getTitle() + "." + photo.getOriginalFormat());
byte[] b = new byte[2048];
int length;
while ((length = is.read(b)) != -1) {
os.write(b, 0, length);
}
is.close();
os.close();
}
Use this method for a single-photo inputstream.
InputStream inputStream = flickr.getPhotosInterface().getImageAsStream(flickr.getPhotosInterface().getPhoto(PHOTO_ID), Size.ORIGINAL);

I'm not very familiar with Java and that framework but will try to help.
I found next method name in that framework:
public class PeopleInterface {
public static final String METHOD_GET_PHOTOS = "flickr.people.getPhotos";
/**
* Returns photos from the given user's photostream. Only photos visible the
* calling user will be returned. this method must be authenticated.
*
* #param userId
* #param extras
* #param perpage
* #param page
* #return
* #throws IOException
* #throws FlickrException
* #throws JSONException
*/
public PhotoList getPhotos(String userId, Set<String> extras, int perPage,
int page)
From Flick API docs I found next:
flickr.people.getPhotos Return photos from the given user's photostream. Only photos visible to the calling user will be returned.
This method must be authenticated; to return public photos for a user,
use flickr.people.getPublicPhotos.
So, it mean that you must be authenticated with 'read' permissions to get your private pohotos (your account).
You can also get private photos of some users only in case if you are contact/friend of that user.

Android Twitter getting profile image size issue

I'm trying to get the profile images of my followers for use within a ListView as thumbnails.
These thumbnails are around 125x125, but the standard twitter4j call of User.getProfileImageURL(); returns much smaller size of 48x48 and is also recommended to not be used as the image source.
I've tried creating a ProfileImage object and supplying it as a parameter, User.getProfileImageURL(profile image object.Original),
But this code takes some time to simply retrieve the url, which when loading a list of thumbnails, is inefficient.
Any suggestions on how to go about this?

Edit
Twitter API v1 has been disabled, so my old answer is no longer valid. Refer to API v1.1, which I believe requires authentication.
If you know the screen name, the twitter api allows for you to fetch the profile image at 4 different resolutions;
https://api.twitter.com/1/users/profile_image?screen_name=Krylez&size=mini
https://api.twitter.com/1/users/profile_image?screen_name=Krylez&size=normal
https://api.twitter.com/1/users/profile_image?screen_name=Krylez&size=bigger
https://api.twitter.com/1/users/profile_image?screen_name=Krylez&size=original
The "bigger" image is 73x73, which is going to interpolate in your 125x125 container. If you're not okay with this, you can try to fetch the "original" photo, but this photo could be very large (slow) and it's not necessarily a square.
Whatever method you choose, make sure you're not fetching and/or decoding Bitmaps on the UI thread. The Android API documentation has excellent guidelines for the correct way to do this.
Also we can make use of the Twitter4j using:
mTwitter.getUserProfileImage();

From the official doc:
You can obtain a user’s most recent profile image from GET users/show. Within the user object, you’ll find the profile_image_url
and profile_image_url_https fields. These fields will contain the
resized “normal” variant of the user’s uploaded image. This “normal”
variant is typically 48x48px.
By modifying the URL, you can retrieve other variant sizings such as
“bigger”, “mini”, and “original”.
Following the code:
TwitterApiClient twitterApiClient = TwitterCore.getInstance().getApiClient();
twitterApiClient.getAccountService().verifyCredentials(false, false, new Callback<User>() {
#Override
public void success(Result<User> userResult) {
String name = userResult.data.name;
String email = userResult.data.email;
// _normal (48x48px) | _bigger (73x73px) | _mini (24x24px)
String photoUrlNormalSize = userResult.data.profileImageUrl;
String photoUrlBiggerSize = userResult.data.profileImageUrl.replace("_normal", "_bigger");
String photoUrlMiniSize = userResult.data.profileImageUrl.replace("_normal", "_mini");
String photoUrlOriginalSize = userResult.data.profileImageUrl.replace("_normal", "");
}
#Override
public void failure(TwitterException exc) {
Log.d("TwitterKit", "Verify Credentials Failure", exc);
}
});
For further information refer to Twitter API Documentation | Profile Images and Banners
To create a custom size pic, for example 90x90 you can use the createScaledBitmap() method.
private final int PROFILE_PIC_SIZE = 90;
Bitmap originalPic = null;
Bitmap resizedPic = null;
try {
InputStream in = new java.net.URL(photoUrlOriginalSize).openStream();
originalPic = BitmapFactory.decodeStream(in);
resizedPic = Bitmap.createScaledBitmap(originalPic, PROFILE_PIC_SIZE, PROFILE_PIC_SIZE, false);
} catch (Exception exc) {
Log.e("Error", exc.getMessage());
exc.printStackTrace();
}

You can use getOriginalProfileImageURL() for example. This is as large as it gets.
Smaller ones are getBiggerProfileImageURL() and getProfileImageURL().
These are the urls you retrieve:
http://pbs.twimg.com/profile_images/NUMBER/c62p-cAD_normal.jpeg
http://pbs.twimg.com/profile_images/NUMBER/c62p-cAD_bigger.jpeg
http://pbs.twimg.com/profile_images/NUMBER/c62p-cAD.jpeg