Develop Reference

Why is Android file creation suddenly failing when it worked on Android 11 previously?

In the last few days, my Android app is suddenly failing to download files from a web server to store in the app. This is the same for all users I have contacted. It was previously working in Android 11, so it's something that has only just changed. It's a (free) niche app for UK glider pilots to process NOTAMS, and has relatively large number of users who I don't want to let down.
The published app uses getExternalFilesDir(null) to return the directory in which to store the downloaded files, with android:requestLegacyExternalStorage set to "true" in the manifest.
I changed getExternalFilesDir(null) to getFilesDir() in Android Studio since that's what I understand should now be used for internal app data files. This returns /data/user/0/(my package name)/files. I'm running the Pixel 2 API 30 emulator for debugging, and the File Explorer shows that /data/data/(my package name)/files directory has been created. Everything I've read on here says that this is what is supposed to happen and it should all work. However no file was created when I attempted the download.
I changed android:requestLegacyExternalStorage to "false", and this time a file was created as expected. However it was empty and the download thread was giving an exception "unexpected end of stream on com.android.okhttp.Address#89599f3f".
This is the relevant code in my DownloadFile class which runs as a separate thread (comments removed for compactness):
public class DownloadFile implements Runnable
{
private String mUrlString;
private String mFileName;
private CountDownLatch mLatch;
public DownloadFile(String urlString, String fileName, CountDownLatch latch)
{
mUrlString = urlString;
mFileName = fileName;
mLatch = latch;
}
public void run()
{
HttpURLConnection urlConnection = null;
// Note for StackOverflow: following is a public static variable defined in the main activity
Spine.mDownloadStatus = false;
try
{
URL url = new URL(mUrlString);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
urlConnection.setUseCaches(false);
urlConnection.connect();
File file = new File(Spine.dataDir, mFileName);
FileOutputStream fileOutput = new FileOutputStream(file);
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int bufferLength = 0; // used to store a temporary size of the buffer
while ((bufferLength = inputStream.read(buffer)) > 0)
{
fileOutput.write(buffer, 0, bufferLength);
}
fileOutput.close();
Spine.mDownloadStatus = true;
}
// catch some possible errors...
catch (IOException e)
{
Spine.mErrorString = e.getMessage();
}
if (urlConnection != null)
urlConnection.disconnect();
// Signal completion
mLatch.countDown();
}
}
I now believe the problem lies with the URL connection, rather than the changes to local file storage access which is what I first thought. Incidentally, if I enter the full URL into my web browser the complete text file is displayed OK, so it's not a problem with the server.

The problem has been narrowed down to changes to the functionality of the website that hosts the data files to be downloaded. It's been made https secure and they are currently working on further changes.
I temporarily moved the hosting to my own website in Android Studio and everything worked so it's down to those website changes and nothing to do with my code (at least it may need changing later to support the upgrade to the main hosting site).
Thanks to all for responding.

Unable to track upload progress with HttpURLConnection as well as OkHttp

For some reason HttpURLConnection appears to be buffering the upload data no matter what I try. I can show the progress percentage of the data, but it is clear that the progress advances way too fast while the data is not flowing at that high rate.
The receiving server is not in the intranet, but hosted somewhere. The edge router is throttling the upload bandwidth to 2mbit in order to simulate a slow network, and in the bandwidth graph of the router I can see the data rate graph for the development device. The WiFi AP also allows me to see a graph of the data rate, and it looks just like the one of the edge router, so no device in the intranet is buffering the data. It is definitely the development device (Nexus 5X)
The following is the code that is being used:
HttpURLConnection hucConnection = (HttpURLConnection) url.openConnection();
//hucConnection.setUseCaches(false); // does not solve the issue
//hucConnection.setDefaultUseCaches(false); // does not solve the issue
//hucConnection.setAllowUserInteraction(true); // does not solve the issue
hucConnection.setConnectTimeout(6 * 1000);
hucConnection.setReadTimeout(30 * 1000);
hucConnection.setRequestProperty("content-type", "application/json; charset=UTF-8");
hucConnection.setRequestMethod("POST");
hucConnection.setDoInput(true);
hucConnection.setDoOutput(true);
// Data to transfer
byte[] bData = joTransfer.toString().getBytes("UTF-8");
int iDataLength = bData.length;
//hucConnection.setRequestProperty("content-transfer-encoding", "binary"); // does not solve the issue
// use compression
hucConnection.setRequestProperty("content-encoding", "deflate");
ByteArrayOutputStream stream = new ByteArrayOutputStream();
Deflater deflater = new Deflater(Deflater.DEFAULT_COMPRESSION);
DeflaterOutputStream zip = new DeflaterOutputStream(stream, deflater);
zip.write(bData);
zip.close();
deflater.end();
byte[] bZippedData = stream.toByteArray();
Integer iZippedDataLength = bZippedData.length;
int iChunk = 1000;
hucConnection.setChunkedStreamingMode(iChunk);
//hucConnection.setFixedLengthStreamingMode(iZippedDataLength); // does not solve the issue
hucConnection.connect();
OutputStream osOutputStream = hucConnection.getOutputStream();
// FROM HERE ---->>>
int iUploadedLength;
for (iUploadedLength = 0; iUploadedLength < iZippedDataLength - iChunk; iUploadedLength += iChunk) {
LogWrapper.e(TAG, "l -> f:" + iUploadedLength + " t:" + (iUploadedLength+iChunk));
osOutputStream.write(Arrays.copyOfRange(bZippedData, iUploadedLength , iUploadedLength+iChunk));
osOutputStream.flush();
}
LogWrapper.e(TAG, "r -> f:" + iUploadedLength + " t:" + iZippedDataLength);
osOutputStream.write(Arrays.copyOfRange(bZippedData, iUploadedLength, iZippedDataLength));
osOutputStream.flush();
osOutputStream.close();
// <<<---- TO HERE ---- XXXXXXXXX max 1 second XXXXXXXXX
// FROM HERE ---->>>
int iResponseCode = hucConnection.getResponseCode();
// <<<---- TO HERE ---- XXXXXXXXX about 10 seconds XXXXXXXXX
if (iResponseCode != HttpURLConnection.HTTP_OK) {
...
I expected the calls to osOutputStream.flush(); to force the HttpURLConnection to send the data to the server, but for some reason that isn't happening.
It appears to get buffered somewhere, because after the osOutputStream.close(); and before the hucConnection.getResponseCode(); the data is getting uploaded to the server.
All the transfers (upload and download) are working properly, no data is damaged.
Is there a way to fix this, or an alternative to using HttpURLConnection? I've read that the Socket class does not have this problem, but I'm not sure if it handles redirects and stuff like that properly. I don't need to use cookies or some other stuff.
The aprox. 10 seconds it takes for hucConnection.getResponseCode(); to finish is when about 3MB are uploaded (3MB*8b/B = 24Mb, 24Mb/2Mb/s = 12s), the data that is downloaded is getting sent after that call. The progress of the downloaded data is precise.
Is it possible that a 3rd party library is altering HttpURLConnection's behavior and doing some proxying? Like Firebase or something? I already disabled Crashlytics, but I think that Firebase also does some kind of stats gathering (response time). I think I had some strange issues about 1-2 months ago in another app, where I was getting a Proxy error issue in the domain name resolution, as if something inside of Android was proxying network traffic.
I'm about to give OkHttp a try, one of their recipies has a Post Streaming example (https://github.com/square/okhttp/wiki/Recipes)
Update: I implemented it using okhttp3, following the above mentioned recipie. I have the exact same problem there.
This is on Android 8.1
The server is an nginx instance.
I also ran the app on a Genymotion emulator instance, same OS, and it looks like it's better there, yet the problem still seems to be present, a bit. While radical throttling on the edge router has no effect on the Nexus 5X, it does have an effect on the emulator. But nonetheless, even the emulator upload tracking precision leaves much to be desired.
Would it make sense to use a WebSocket connection for that? That would be my last resort.

The logic is for downloading used in AsyncTask, but I think, that it should be the same (just a switching input>output and so on)
InputStream inputStream = null;
try {
try {
OutputStream outputStream = new FileOutputStream(documentFile, false);
try {
inputStream = httpConn.getInputStream();
byte[] buffer = new byte[4 * 1024]; // or other buffer size
long downloaded = 0;
long target = dataLength;
int readed;
long updateSize = target / 10;
long updateHelp = 0;
while ((readed = inputStream.read(buffer)) != -1) {
downloaded += readed;
updateHelp += readed;
if (updateHelp >= updateSize) {
updateHelp = 0;
publishProgress(downloaded, target);
}
outputStream.write(buffer, 0, readed);
if (isCancelled()) {
return false;
}
}
outputStream.flush();
outputStream.close();
return true;
} catch (Exception e) {
return false;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (inputStream != null) {
inputStream.close();
}
}
} catch (Exception e) {
e.printStackTrace();
}

How to cache a video in background in android?

I am building an android application where a user can view some listed video. Those videos are categories into some channel. Once a channel is selected by user I want to cache all the video related to that channel in my cache memory so can play the video when there is no internet also.
Can anyone have more understanding about video cache without playing please help me in understanding how I can achieve this task.
Right now I am able to cache video If it's played using some library.

I have find the following working solution for caching video in background (single/multiple) using below lib, no need of player/video_view.use AsyncTaskRunner
Videocaching Lib
Add following in line in your gradle file
compile 'com.danikula:videocache:2.7.0'
Since we just need to kick start the prefetching, no need to do anything in while loop.
Or we can use ByteArrayOutputStream to write down the data to disk.
URL url = null;
try {
url = new URL(cachingUrl(cachingUrl));
InputStream inputStream = url.openStream();
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int length = 0;
while ((length = inputStream.read(buffer)) != -1) {
//nothing to do
}
} catch (IOException e) {
e.printStackTrace();
}
Important code from lib. to do
Create static instance in application class using following code
private HttpProxyCacheServer proxy;
public static HttpProxyCacheServer getProxy(Context context) {
Applications app = (Applications) context.getApplicationContext();
return app.proxy == null ? (app.proxy = app.newProxy()) : app.proxy;
}
private HttpProxyCacheServer newProxy() {
//return new HttpProxyCacheServer(this);
return new HttpProxyCacheServer.Builder(this)
.cacheDirectory(CacheUtils.getVideoCacheDir(this))
.maxCacheFilesCount(40)
.maxCacheSize(1024 * 1024 * 1024)
.build();
}
Write following code in your activity to pass url
public String cachingUrl(String urlPath) {
return Applications.getProxy(this).getProxyUrl(urlPath, true);
}

Is it a good way to call AsyncTask in infinity loop to pull data from server?

Well in app I'm trying to pull the data from sever for every 4 sec,and update the app.
I'm using handler,in that I'm calling AsynTask to fetch the data from server for every 4 sec.
Just I'm worried about the instance created for AsynTask every 4'sec causes any problem ?
This is what I'm doing.
private static final int DELAY = 1000 * 4;
final Handler printHandler = new Handler();
private boolean keepLooping = true;
printHandler.postDelayed(printStuff, DELAY);
Runnable printStuff = new Runnable(){
#Override
public void run(){
// call AsynTask to perform network operation on separate thread
new DownloadMainScore().execute("http://server/root/score.php");
if(keepLooping)
printHandler.postDelayed(this, DELAY);
}
};

On your choice of concurrency tool:
You are right that this is not so good. AsyncTasks are designed to be useful helpers when designing occasional asynchronous calls that then need to update a UI. As such, in old (< 1.6) versions of Android the maximum thread pool size was 10!
It would be better to go straight to the very robust Java out of which AsyncTask is built. Given you want to do this repeatedly, try a ScheduledExecutorService. I see they've even made a nice example for you.
Or, given that you seem to be getting a score down, best might be to maintain a persistent connection over a protocol like XMPP, for which there are many Java server and clients.
Finally, you might like to look at gcm.
On design issues in general
I see you want to print a score frequently. Once every four seconds in fact. But what's the point is the score hasn't changed? Furthermore, what if you've got a slow internet connection, and eight seconds later the one for four seconds ago hasn't finished? Right now you will set off yet another download request, even though the other one when it comes back will be up to date!
The solution is to decouple the download mechanism and the UI update mechanism. One way to do it is to have your scheduled download on a single threaded executor- not something you can control in an AsyncTask, which when finishes causes the UI to update and show the score.
Wishing you the best of luck!
Code sketch
Don't have environment set up right now, but in a very rough code sketch (check syntax), using a scheduled executor would look like:
In class:
private final ScheduledExecutorService downloadScheduler = Executors.newSingleThreadScheduledExecutor(1);
Then elsewhere, wherever you start doing this
final Runnable scoreHttpRunnable = new Runnable() {
#Override public void run() {
...
//do Http Syncronously here- I guess whatever is in the doInBackground(...) part of that Async task you wrote!
...
final int newScoreResult = ... (do whatever you do here)
...
runOnUiThread(new Runnable() { #Override public void run() { yourView.updateHoweverYouLike(newScoreResult); } })
...
};
downloadScheduler.scheduleAtFixedRate(scoreHttpRunnable, 0, 4, TimeUnit.SECONDS);
Going one of the other two routes is really too much to post in a single answer to a question. That'd be a another SO question if there isn't already one.

Be sure that next call send to asyc class only after once its done for that make a variable(IsLoadRunning) and make it true in on preExecute() and false in onPOstExecute and add a condition if(!IsLoadRunning){new DownloadMainScore().execute();}

As official documentation states
AsyncTasks should ideally be used for short operations (a few seconds at the most.)
Services can serve better in you case. Have a look at the accepted answer here

#Override
protected String doInBackground(String... params) {
Log.d(TAG, "type - " + params[0] + ", url = " + params[1] + ", name = " + params[2]);
downloadFile(params[1], params[2]);
return null;
}
here is download method
URL url = new URI(Url.replace(" ", "%20")).toURL();
URLConnection connection = url.openConnection();
connection.setConnectTimeout(1000);
int fileLength = connection.getContentLength();
mSavePath = CommonUtilities.getFileSavePath(mContext, fileName, fileLength);
Log.d(TAG, "*** saveFilePath - " + mSavePath);
InputStream inputStream = connection.getInputStream();
if (inputStream != null) {
File file = new File(mSavePath);
BufferedOutputStream bufferOutputStream = new BufferedOutputStream(new FileOutputStream(file));
byte byteArray[] = new byte[1024];
int len = 0;
long total = 0;
while ((len = inputStream.read(byteArray)) != -1) {
bufferOutputStream.write(byteArray, 0, len);
total += len;
}
bufferOutputStream.flush();
bufferOutputStream.close();
inputStream.close();
} else {
Log.d(TAG, "*** inputStream is null");
}

Android: Too many open files error

I have the following operation which runs every 3 seconds.
Basically it downloads a file from a server and save it into a local file every 3 seconds.
The following code does the job for a while.
public class DownloadTask extends AsyncTask<String, Void, String>{
#Override
protected String doInBackground(String... params) {
downloadCommandFile( eventUrl);
return null;
}
}
private void downloadCommandFile(String dlUrl){
int count;
try {
URL url = new URL( dlUrl );
NetUtils.trustAllHosts();
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.connect();
int fileSize = con.getContentLength();
Log.d(TAG, "Download file size = " + fileSize );
InputStream is = url.openStream();
String dir = Environment.getExternalStorageDirectory() + Utils.DL_DIRECTORY;
File file = new File( dir );
if( !file.exists() ){
file.mkdir();
}
FileOutputStream fos = new FileOutputStream(file + Utils.DL_FILE);
byte data[] = new byte[1024];
long total = 0;
while( (count = is.read(data)) != -1 ){
total += count;
fos.write(data, 0, count);
}
is.close();
fos.close();
con.disconnect(); // close connection
} catch (Exception e) {
Log.e(TAG, "DOWNLOAD ERROR = " + e.toString() );
}
}
Everything works fine, but if I leave it running for 5 to 10 minutes I get the following error.
06-04 19:40:40.872: E/NativeCrypto(6320): AppData::create pipe(2)
failed: Too many open files 06-04 19:40:40.892: E/NativeCrypto(6320):
AppData::create pipe(2) failed: Too many open files 06-04
19:40:40.892: E/EventService(6320): DOWNLOAD ERROR =
javax.net.ssl.SSLException: Unable to create application data
I have been doing some researches for the last 2 days.
There are suggestions that they are many connections open, like this one https://stackoverflow.com/a/13990490/1503155 but still I can not figure out what's the problem.
Any ideas what may cause the problem?
Thanks in advance.

I think you get this error because you have too many files open at the same times, meaning that you have too many async tasks running in the same time (each async task opens a file), which makes sense if you say that you run a new one every 3 seconds.
You should try to limit the number of async task running in the same time using a thread pool executor.

Try using OkHttp Instead.
Your issue isn't with too many threads, although that's what's causing your issue to surface.
As #stdout mentioned in the comments, AsyncTask already runs in a threadpool that is common and shared amongst all AsyncTasks, unless you specify otherwise. The issue here is that the file descriptors are not being closed properly and in time.
The issue is that your file descriptors are not being closed fast enough.
I struggled with this for hours/days/weeks, doing everything you should like setting small read/connect timeouts and using a finally block to close out connections, input streams, output streams, etc. But we never found a working solution. It seemed like HttpsUrlConnection was flawed in some way.
So we tried OkHttp as a drop-in replacement for HttpsUrlConnection and voila! It worked out of the box.
So, if you're struggling with this and are having a really hard time fixing it, I suggest you try using OkHttp as well.
Here are the basics:
Once you get the Maven dependency added, you can do something like the following to download a file:
OkHttpClient okHttpClient = new OkHttpClient.Builder().build();
OutputStream output = null;
try {
Request request = new Request.Builder().url( download_url ).build();
Response response = okHttpClient.newCall( request ).execute();
if ( !response.isSuccessful() ) {
throw new FileNotFoundException();
}
output = new FileOutputStream( output_path );
output.write( response.body().bytes() );
}
finally {
// Ensure streams are closed, even if there's an exception.
if ( output != null ) output.flush();
if ( output != null ) output.close();
}
Switching to OkHttp instantly fixed our leaked file descriptor issue so it's worth trying if you're stuck, even at the expense of adding another library dependency.

I had to download several hunders of files at a time and meet the error.
You may check open descriptors with the following command:
adb shell ps
Find your application PID in the list and use another command:
adb shell run-as YOUR_PACKAGE_NAME ls -l /proc/YOUR_PID/fd
I see about 150 open descriptors on a usual launch. And there are 700+ when files are downloading. Their number decreases only after some minutes, looks like Android frees them in the backgound, not when you can close on a stream.
The only working solution was use a custom ThreadPool to limit the concurrency. Here is the Kotlin code:
private val downloadDispatcher = Executors.newFixedThreadPool(8).asCoroutineDispatcher()
private suspend fun downloadFile(sourceUrl: String, destPath: String, progressBlock: suspend (Long) -> Unit) = withContext(downloadDispatcher) {
val url = URL(sourceUrl)
url.openConnection().apply { connect() }
url.openStream().use { input ->
FileOutputStream(File(destPath)).use { output ->
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
var bytesRead = input.read(buffer)
var bytesCopied = 0L
while (bytesRead >= 0) {
if (!coroutineContext.isActive) break
output.write(buffer, 0, bytesRead)
bytesCopied += bytesRead
progressBlock(bytesCopied)
bytesRead = input.read(buffer)
}
}
}
}