Develop Reference

Is there a way to delay a video feed from a cell phone camera in Unity?

I am making a project where you are supposed to be able to change the delay with which the feed from the cell phone camera is shown, so that people can see how their brains handle the delay/latency. I have managed to show the camera feed on a canvas that follows the camera around and fills the whole view of the Google Cardboard, but I am wondering how I could delay this video feed. Perhaps by using an image array of some sort?
I have tried searching for sollutions online, but I have come up short of an answer. I have tried a texture2D array, but the performance was really bad (I tried a modified version of this).
private bool camAvailable;
private WebCamTexture backCam;
private Texture defaultBackground;
public RawImage background;
public AspectRatioFitter fit;
// Start is called before the first frame update
void Start()
{
defaultBackground = background.texture;
WebCamDevice[] devices = WebCamTexture.devices;
if (devices.Length == 0 )
{
Debug.Log("No camera detected");
camAvailable = false;
return;
}
for (int i = 0; i < devices.Length; i++)
{
if (!devices[i].isFrontFacing)
{
backCam = new WebCamTexture(devices[i].name, Screen.width, Screen.height); // Used to find the correct camera
}
}
if (backCam == null)
{
Debug.Log("Unable to find back camera");
return;
}
backCam.Play();
background.texture = backCam;
camAvailable = true;
} //Tell me if this is not enough code, I don't really have a lot of experience in Unity, so I am unsure of how much is required for a minimal reproducible example
Should I use some sort of frame buffer or image/texture array for delaying the video? (Start "recording", wait a specified amount of time, start playing the "video" on the screen)
Thanks in advance!

Scan QRcode with inverted colors using Vision API

After struggling a few hours on making my app detect this QRCode:
I realized that the problem was the in the QRCode appearance. After inverting the colors, the detection was working perfectly..
Is there a way to make Vision API detect the first QRCode? I tried to enable all symbologies but it did not work. I guess it is possible because the app QR Code Reader detects it.

I improved googles example app "barcode-reader" to detect both inverted colored barcodes and regular ones.
here is a link to googles example app:
https://github.com/googlesamples/android-vision/tree/master/visionSamples/barcode-reader
I did so by editing "CameraSource" class,
package: "com.google.android.gms.samples.vision.barcodereader.ui.camera".
I added a parameter: private boolean isInverted = false;
and changed function void setNextFrame(byte[] data, Camera camera):
void setNextFrame(byte[] data, Camera camera) {
synchronized (mLock) {
if (mPendingFrameData != null) {
camera.addCallbackBuffer(mPendingFrameData.array());
mPendingFrameData = null;
}
if (!mBytesToByteBuffer.containsKey(data)) {
Log.d(TAG,
"Skipping frame. Could not find ByteBuffer associated with the image " +
"data from the camera.");
return;
}
mPendingTimeMillis = SystemClock.elapsedRealtime() - mStartTimeMillis;
mPendingFrameId++;
if (!isInverted){
for (int y = 0; y < data.length; y++) {
data[y] = (byte) ~data[y];
}
isInverted = true;
} else {
isInverted = false;
}
mPendingFrameData = mBytesToByteBuffer.get(data);
// Notify the processor thread if it is waiting on the next frame (see below).
mLock.notifyAll();
}
}

I think this is still an open issue, please see link for details. One workaround for this as stated by a developer:
Right, the barcode API generally doesn't support color-inverted codes. There's no parameter or option to control this at the moment. Though some APIs support them, I don't believe it's a common feature.
For a workaround, you could preprocess the colors in the bitmap before passing them to the barcode API (perhaps inverting colors on alternate frames).
Hope this helps.

Performance: Android JCIFS file reading is slow

First of all, I saw an existing question (JCIFS: file retrieval is too slow to be usable), but it was for Java, not Android, and none of the suggested answers worked.
I created a default project for Android SDK 25 (7.1.1) in Android Studio 2.3, linked the library with compile 'jcifs:jcifs:1.3.17', and typed the following simple test code. The result is below the code.
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
jcifs.Config.setProperty("jcifs.util.loglevel", "3");
//jcifs.Config.setProperty("jcifs.smb.client.dfs.disabled", "false");
//jcifs.Config.setProperty("jcifs.resolveOrder", "DNS");
try
{
NtlmPasswordAuthentication auth = new NtlmPasswordAuthentication("", ID, PASSWORD);
final SmbFile smb = new SmbFile("smb://192.168.XX.XX/Share/FileName", auth);
Thread t = new Thread(new Runnable()
{
#Override
public void run()
{
Log.d(TAG, "Test Start");
for(int i = 1000; i<10000; i+=1000)
measure(i);
Log.d(TAG, "Test End");
}
private void measure(int bufferSize)
{
Log.d(TAG, "=====Buffer: " + bufferSize + "============");
try
{
byte[] buffer = new byte[bufferSize];
int read = 0;
InputStream str = smb.getInputStream();
long start = System.nanoTime();
while(read < 1000000)
read += str.read(buffer);
long end = System.nanoTime();
str.close();
float time = (float) ((end - start) / 1000000000d);
float speed = (float) read / 1048576 / time;
Log.d(TAG, "Time:" + time + ", size =" + read);
Log.d(TAG, "Speed = " + speed + "MB/s");
}
catch(IOException exc)
{
exc.printStackTrace();
}
}
});
t.start();
}
catch(Exception exc)
{
Log.d(TAG, exc.toString());
}
}
Result
Test Start
=====Buffer: 1000============
Time:2.210785, size =1000000
Speed = 0.43137363MB/s
=====Buffer: 2000============
Time:1.4158936, size =1000000
Speed = 0.6735495MB/s
=====Buffer: 3000============
Time:1.0556641, size =1002000
Speed = 0.9051948MB/s
=====Buffer: 4000============
Time:0.7543335, size =1000000
Speed = 1.2642609MB/s
=====Buffer: 5000============
Time:3.6557617, size =1000000
Speed = 0.26086885MB/s
=====Buffer: 6000============
Time:3.292389, size =1002000
Speed = 0.2902396MB/s
=====Buffer: 7000============
Time:2.9179688, size =1001000
Speed = 0.32715496MB/s
=====Buffer: 8000============
Time:2.462616, size =1000000
Speed = 0.38726068MB/s
=====Buffer: 9000============
Time:3.9379272, size =1008000
Speed = 0.24411413MB/s
Test End
Read speed is about 0.2MB/s ~ 1.2MB/s. The device is connected to a 150Mbps Wi-Fi, so, theoretically it can achieve above 10MB/s. The SMB server is not slow either. When I copied the file to a laptop, the read speed was about 30MB/s.
Why is this so slow? What should I check? Why is the read speed about 5 times higher (1.2MB/s) if the buffer size is 4000?
By the way, I have tested copying the same file with other commercial apps. File Commander, Asus File Manager showed similary low speed, ES File Explorer showed about 2MB/s, and Solid Explorer showed about 5MB/s. Since I am pretty sure that all of them use JCIFS (albeit perhaps slightly different versions of it), there must be a way to achieve at least 5MB/s as Solid Explorer does.

After using WireShark (network analysis tool) on the Windows computer, I have found that no matter which buffer size I set, the read SMB command always gives 4286 bytes to the Windows computer. It seems that SmbFileInputStream.java is using the max buffer size from the server.
But when I saw the packets from Soild Explorer, it was 32768 bytes. So, I decompiled Solid Explorer's APK (it was of course obfuscated), and saw the SmbFileInputStream.java file inside of it (that file belongs to JCIFS). It seems that the developers of Solid Explorer has modified that file, and set a bigger readSize. So, I tried a similar thing. And then I achieved 5MB/s for the same code above.
Since JCIFS comes with LGPL, the fact that Solid Explorer is using a modified JCIFS without disclosing the source code is a violation of JCIFS' licence. But, oh well, it seems a lot of Android app developers ignores licence of the libraries they use anyway. They do not even properly credit the open-source libraries they used.

Did you try with: jcifs.Config.setProperty("jcifs.smb.client.dfs.disabled", "true");
In my case (though Java) it was helpful for slow connection. By default it is false
Only problem for me is that I'm not sure will that property "break" something else which is working fine..

There is a patch for large buffer size reading:
https://github.com/kohsuke/jcifs/issues/11
https://github.com/kohsuke/jcifs/tree/master/patches
https://jcifs.samba.org/src/patches/LargeReadWrite.patch:
From inside the README:
This patch adds two SMBs that supposedly improves read and write
performance considerably. Unfortunately it's not crystal clear that
all implementation properly support the commands. Note that in
addition to this patch an '& 0xFFFF' needs to be added in
SmbTransport.java:doRecv:~437 to appear as:
int size = Encdec.dec_uint16be( BUF, 2 ) & 0xFFFF;
although this change has been made in 1.2.7.
Not sure if this works with Android, but the solution could be similar.

How to record screen and take screenshots, using Android API?

Background
Android got a new API on Kitkat and Lollipop, to video capture the screen. You can do it either via the ADB tool or via code (starting from Lollipop).
Ever since the new API was out, many apps came to that use this feature, allowing to record the screen, and Microsoft even made its own Google-Now-On-tap competitor app.
Using ADB, you can use:
adb shell screenrecord /sdcard/video.mp4
You can even do it from within Android Studio itself.
The problem
I can't find any tutorial or explanation about how to do it using the API, meaning in code.
What I've found
The only place I've found is the documentations (here, under "Screen capturing and sharing"), telling me this:
Android 5.0 lets you add screen capturing and screen sharing
capabilities to your app with the new android.media.projection APIs.
This functionality is useful, for example, if you want to enable
screen sharing in a video conferencing app.
The new createVirtualDisplay() method allows your app to capture the
contents of the main screen (the default display) into a Surface
object, which your app can then send across the network. The API only
allows capturing non-secure screen content, and not system audio. To
begin screen capturing, your app must first request the user’s
permission by launching a screen capture dialog using an Intent
obtained through the createScreenCaptureIntent() method.
For an example of how to use the new APIs, see the MediaProjectionDemo
class in the sample project.
Thing is, I can't find any "MediaProjectionDemo" sample. Instead, I've found "Screen Capture" sample, but I don't understand how it works, as when I've run it, all I've seen is a blinking screen and I don't think it saves the video to a file. The sample seems very buggy.
The questions
How do I perform those actions using the new API:
start recording, optionally including audio (mic/speaker/both).
stop recording
take a screenshot instead of video.
Also, how do I customize it (resolution, requested fps, colors, time...)?

First step and the one which Ken White rightly suggested & which you may have already covered is the Example Code provided officially.
I have used their API earlier. I agree screenshot is pretty straight forward. But, screen recording is also under similar lines.
I will answer your questions in 3 sections and will wrap it up with a link. :)
1. Start Video Recording
private void startScreenRecord(final Intent intent) {
if (DEBUG) Log.v(TAG, "startScreenRecord:sMuxer=" + sMuxer);
synchronized(sSync) {
if (sMuxer == null) {
final int resultCode = intent.getIntExtra(EXTRA_RESULT_CODE, 0);
// get MediaProjection
final MediaProjection projection = mMediaProjectionManager.getMediaProjection(resultCode, intent);
if (projection != null) {
final DisplayMetrics metrics = getResources().getDisplayMetrics();
final int density = metrics.densityDpi;
if (DEBUG) Log.v(TAG, "startRecording:");
try {
sMuxer = new MediaMuxerWrapper(".mp4"); // if you record audio only, ".m4a" is also OK.
if (true) {
// for screen capturing
new MediaScreenEncoder(sMuxer, mMediaEncoderListener,
projection, metrics.widthPixels, metrics.heightPixels, density);
}
if (true) {
// for audio capturing
new MediaAudioEncoder(sMuxer, mMediaEncoderListener);
}
sMuxer.prepare();
sMuxer.startRecording();
} catch (final IOException e) {
Log.e(TAG, "startScreenRecord:", e);
}
}
}
}
}
2. Stop Video Recording
private void stopScreenRecord() {
if (DEBUG) Log.v(TAG, "stopScreenRecord:sMuxer=" + sMuxer);
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.stopRecording();
sMuxer = null;
// you should not wait here
}
}
}
2.5. Pause and Resume Video Recording
private void pauseScreenRecord() {
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.pauseRecording();
}
}
}
private void resumeScreenRecord() {
synchronized(sSync) {
if (sMuxer != null) {
sMuxer.resumeRecording();
}
}
}
Hope the code helps. Here is the original link to the code that I referred to and from which this implementation(Video recording) is also derived from.
3. Take screenshot Instead of Video
I think by default its easy to capture the image in bitmap format. You can still go ahead with MediaProjectionDemo example to capture screenshot.
[EDIT] : Code encrypt for screenshot
a. To create virtual display depending on device width / height
mImageReader = ImageReader.newInstance(mWidth, mHeight, PixelFormat.RGBA_8888, 2);
mVirtualDisplay = sMediaProjection.createVirtualDisplay(SCREENCAP_NAME, mWidth, mHeight, mDensity, VIRTUAL_DISPLAY_FLAGS, mImageReader.getSurface(), null, mHandler);
mImageReader.setOnImageAvailableListener(new ImageAvailableListener(), mHandler);
b. Then start the Screen Capture based on an intent or action-
startActivityForResult(mProjectionManager.createScreenCaptureIntent(), REQUEST_CODE);
Stop Media projection-
sMediaProjection.stop();
c. Then convert to image-
//Process the media capture
image = mImageReader.acquireLatestImage();
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * mWidth;
//Create bitmap
bitmap = Bitmap.createBitmap(mWidth + rowPadding / pixelStride, mHeight, Bitmap.Config.ARGB_8888);
bitmap.copyPixelsFromBuffer(buffer);
//Write Bitmap to file in some path on the phone
fos = new FileOutputStream(STORE_DIRECTORY + "/myscreen_" + IMAGES_PRODUCED + ".png");
bitmap.compress(CompressFormat.PNG, 100, fos);
fos.close();
There are several implementations (full code) of Media Projection API available.
Some other links that can help you in your development-
Video Recording with MediaProjectionManager - website
android-ScreenCapture - github as per android developer's observations :)
screenrecorder - github
Capture and Record Android Screen using MediaProjection APIs - website
Hope it helps :) Happy coding and screen recording!
PS: Can you please tell me the Microsoft app you are talking about? I have not used it. Would like to try it :)

What is MINI_THUMB_MAGIC and how to use it?

Background
I've noticed a weird column for MediaStore.Images.ImageColumns called "MINI_THUMB_MAGIC" .
the documentation says just that :
The mini thumb id.
Type: INTEGER
Constant Value: "mini_thumb_magic"
The question
my guess is that this field is related to MediaStore.Images.Thumbnails .
Is it correct ? if not, what is this and how do you use it?
if it is correct , i have other questions related to it:
Is it a mini sized image of the original one? does it use the same aspect ratio or does it do center-cropping on it?
how come the size of "MICRO" is square (96 x 96) and the size of "MINI" is a non-square rectangle ( 512 x 384 ) ?
How do you use it? My guess is that it's done by using "THUMB_DATA", which is a blob, so you use it like this, but then what is the purpose of using "getThumbnail" if you already have this field?
does it get a rotated thumbnail in case the orientation value is not 0 ? meaning that if I wish to show it, I won't need to rotate the image?
Is it possible to do a query of the images together with their thumbnails? maybe using inner join?
Is it available for all Android devices and versions?
Why is it even called "magic" ? Is it because it's also available for videos (and for some reason doesn't exist for music, as it could be the album's cover photo, for example) ?

Check this file: https://github.com/android/platform_packages_providers_mediaprovider/blob/master/src/com/android/providers/media/MediaThumbRequest.java in the Android source code. This value is some magic number which allows to determine if the thumbnail is still valid. I didn't investigate that file further, but it should be no bit issue to dive deeper.
To your questions:
No, no mini-sized image
Well, I guess it's a definition by Google who want to have a square thumbnail for some lists, where only very small previews should be visible and where many items should fit on the screen and there's another thumbnail format where the images are bigger...
I don't know that, but according to Google's doc, one (THUMB_DATA) is only some raw byte array of the thumbnail (dunno in which format) and the other one (getThumbnail) retrieves a full-fledged bitmap object...
don't know
don't know
I guess so, as it's part of AOSP source code.
The word "magic" is often used for some kind of identifier. There are "magic packets" who can wake up a computer from sleep or shutdown over the network, there are magic numbers on hard disks, where some sectors (e.g. the MBR) has the hexadecimal values AA 55 on its last two byte positions, there are also magic numbers in image files which help software packages determine the image type (e.g. GIF files begin with GIF89a or GIF87a (ASCII), JPEG files begin with FF D8 hexadecimal) and there are many, many more examples. So, magic numbers are a very common term here :-)

According to the source code at the following URL, the Magic Number is the Id of the original image * a constant. That value is then used to check for a long int. If the int isn't as expected, it's considered out of sync with the image media.
http://grepcode.com/file/repo1.maven.org/maven2/org.robolectric/android-all/4.4_r1-robolectric-0/android/media/MiniThumbFile.java#MiniThumbFile.getMagic%28long%29
// Get the magic number for the specified id in the mini-thumb file.
// Returns 0 if the magic is not available.
public synchronized long getMagic(long id) {
// check the mini thumb file for the right data. Right is
// defined as having the right magic number at the offset
// reserved for this "id".
RandomAccessFile r = miniThumbDataFile();
if (r != null) {
long pos = id * BYTES_PER_MINTHUMB;
FileLock lock = null;
try {
mBuffer.clear();
mBuffer.limit(1 + 8);
lock = mChannel.lock(pos, 1 + 8, true);
// check that we can read the following 9 bytes
// (1 for the "status" and 8 for the long)
if (mChannel.read(mBuffer, pos) == 9) {
mBuffer.position(0);
if (mBuffer.get() == 1) {
return mBuffer.getLong();
}
}
} catch (IOException ex) {
Log.v(TAG, "Got exception checking file magic: ", ex);
} catch (RuntimeException ex) {
// Other NIO related exception like disk full, read only channel..etc
Log.e(TAG, "Got exception when reading magic, id = " + id +
", disk full or mount read-only? " + ex.getClass());
} finally {
try {
if (lock != null) lock.release();
}
catch (IOException ex) {
// ignore it.
}
}
}
return 0;
}
I got the runtime exception when trying to get the original Id of a thumbnail by looking up the thumbnail's path. (BTW, the disk isn't full and it's not read-only.)

It's a bit strange parameter...
While exploring the Gallery source code,
noticed that the value is being read from the cursor, but then is Never used:
#Override
protected BaseImage loadImageFromCursor(Cursor cursor) {
long id = cursor.getLong(INDEX_ID);
String dataPath = cursor.getString(INDEX_DATA_PATH);
long dateTaken = cursor.getLong(INDEX_DATE_TAKEN);
if (dateTaken == 0) {
dateTaken = cursor.getLong(INDEX_DATE_MODIFIED) * 1000;
}
// here they read it ====>>
long miniThumbMagic = cursor.getLong(INDEX_MINI_THUMB_MAGIC);
int orientation = cursor.getInt(INDEX_ORIENTATION);
String title = cursor.getString(INDEX_TITLE);
String mimeType = cursor.getString(INDEX_MIME_TYPE);
if (title == null || title.length() == 0) {
title = dataPath;
}
// and not use at all ==>>>
return new Image(this, mContentResolver, id, cursor.getPosition(),
contentUri(id), dataPath, mimeType, dateTaken, title,
orientation);
}
Maybe it was used on the previous APIs.
ref: https://android.googlesource.com/platform/packages/apps/Gallery/+/android-8.0.0_r12/src/com/android/camera/gallery/ImageList.java?autodive=0%2F%2F.
and videos list:
https://android.googlesource.com/platform/packages/apps/Gallery/+/android-8.0.0_r12/src/com/android/camera/gallery/VideoList.java?autodive=0%2F%2F