As Android is very inconsistent between different major Versions regarding File access, I feel a bit lost.
I try to describe the problem as easy as possible:
My Company uses a commercial native DRM to protect other native library's we provide. We have a Licensing App, which invoked some Voodoo to end up with Licensing files in say /sdcard/companyname/LicenseContainer/. Other protected Apps looked at this directory in native code, checking if the user has a valid License.
The Android 10 update however, invalidated this workflow completely as it only provides scoped storage access. We could do a workaround using Storage Manager to grant access, which is unfortunately also deprecated now.
So my Question is now:
How can one App save files to a location on /sdcard/FOLDER which are
not deleted on App deletion
Accessible in native code by other apps
I'm a bit overwhelmed with all the possible solutions (SAF, FileProvider, etc), which invoke often that one app grants permissions to the other. But the files should be accessible without an installed first app who put it there.
I know there must be a solution, as recent FileManagers (i.e. Files by Google) get access to the whole /sdcard/ directory.
Whats the easiest, future-proof route to go here without invoking "hacks" like android:requestLegacyExternalStorage="true"
You may ask the user to give you access to any file or directory, including the root of internal storage or external SD card. You can make this access permanent for your app, be able to read/write files anywhere with the Scoped Storage API afterwards, until the app is uninstalled or reset.
Then, if you need to read or write a file in native C/C++ code, you may get Linux file descriptor (int number) of the file and pass it to native code to use with fdopen() call for example.
Here is a Java code snippet to get a file descriptor form a single file Uri (which in string form is like content://...)
ParcelFileDescriptor parcelFileDescriptor =
getContentResolver().openFileDescriptor(uri, "r"); // gets FileNotFoundException here, if file we used to have was deleted
int fd = parcelFileDescriptor.getFd(); // detachFd() if we want to close in native code
If you have source code for your native libraries, or can call them with C FILE* - it will work fine. The only problem is when you don't have the source code and they expect a file path/name. * UPDATE *: it is still possible to use the path/file name strings to pass to C/C++ functions that expect a file name. Simply instead of the "real path/file name", create a name to symbolic link like this:
// fd is file descriptor obtained in Java code above
char fileName[32];
sprintf(fileName, "/proc/self/fd/%d", fd);
// The above fileName can be passed to C/C++ functions that expect a file name.
// They can read or write to it, depending on permissions to fd given in Java,
// but I guess C/C++ code can not create a new file. Someone correct me,
// if I'm mistaken here.
However, at this time I'm not sure that when you create a file in a directory beyond the app "sandbox" in that way, if the system will delete this file too after uninstall... Would need to write a quick test on Android 10 to find out, and then we still won't know if Google won't change this behavior in future.
If you want to save files in shared storage (where it can be accessed by users & other apps) you need to use
for Media Files (Images, Videos, Audio, Downloads) use MediaStore
for Documents and Other Files use Storage Access Framework (this is simply a system file picker)
For instance you can use the following snippet to save a pdf file using Storage Access Framework
const val CREATE_FILE = 1
private fun createFile(pickerInitialUri: Uri) {
val intent = Intent(Intent.ACTION_CREATE_DOCUMENT).apply {
addCategory(Intent.CATEGORY_OPENABLE)
type = "application/pdf"
putExtra(Intent.EXTRA_TITLE, "invoice.pdf")
// Optionally, specify a URI for the directory that should be opened in
// the system file picker before your app creates the document.
putExtra(DocumentsContract.EXTRA_INITIAL_URI, pickerInitialUri)
}
startActivityForResult(intent, CREATE_FILE)
}
After the user has picked a directory we still need to handle the result Uri in on onActivityResult method.
override fun onActivityResult(
requestCode: Int, resultCode: Int, resultData: Intent?) {
if (requestCode == CREATE_FILE && resultCode == Activity.RESULT_OK) {
// The result data contains a URI for directory that
// the user selected.
resultData?.data?.also { uri ->
// save your data using the `uri` path
}
}
}
You can read about this in more detail on the following blogpost
https://androidexplained.github.io/android/android11/scoped-storage/2020/09/29/file-saving-android-11.html
Related
I'm trying to read a given directory recursively and find all folders and files. The directory and subdirectory is found, but no files.
The storage structure on my virtual device:
The code used to "walk through the given directory"
import android.util.Log
import java.io.File
// path = "/storage/1B10-1D17/ReadTest/"
class FileHelper {
fun walkTest(path: String){
File(path).walkTopDown().forEach {
// println(it)
Log.e("walkTest extension", it.extension)
if(it.isDirectory){
Log.e("walkTest", "Directory: ${it.name}")
}else{
Log.e("walkTest", "File: ${it.name}")
}
}
}
}
This will output the following:
E/walkTest extension:
E/walkTest: Directory: ReadTest
E/walkTest extension:
E/walkTest: Directory: SubFolder
In my android manifest file I have the following line and the permission is granted:
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
What is the flaw in my code, how can I find the three pdf files stored in these directories?
Update 1/2
As can be seen in the screenshot below, the media access is granted:
If I run the following command (as suggested by dan1st), I get the following output, which shows that pdf file in place:
> adb shell ls /storage/1B10-1D17/ReadTest/
File_1.pdf
SubFolder
You generally have no access to arbitrary files on removable storage. This is not changed by READ_EXTERNAL_STORAGE.
Your proposed solution is to use MANAGE_EXTERNAL_STORAGE. Technically, this works, and it is a fine solution for personal-use apps. If your intention is to distribute this app, you will need to file paperwork with Google (and perhaps other app stores) justifying your use of this permission. Stores that disagree with your justification may elect to not distribute your app.
Alternatively, you can use the Storage Access Framework. Use ACTION_OPEN_DOCUMENT_TREE / ActivityResultContracts.OpenDocumentTree and let the user decide where on the user's device (or in the user's cloud storage) the user wants your app to access the user's content. You can then use DocumentFile to walk the document tree to find sub-trees and documents.
I'll have an Android App currently compiled against the API29 which I'm working on an update for API31.
The App is using an huge ultralite database which is located in the directory
/sdcard/Documents/myApp/db
Because I now have to apply scoped storage', I'll use the App-Directory on the external sdcard by using context.getExternalFilesDir(null).
I'll have to fulfill an important requirement, that the User needs to download this big database by PC only once
and store this db- file in a dedicated directory on the Android Device.
This will be done from a person (in a staging role) that prepares dozens of devices for dozens of users in a staging process,
so not alle devices needs to download the DB for themselfes.
When the App is starting it will have a look on this staging-directory and would then move the DB-file to the internal Working storage of the APP.
So my question is:
How can I get the permission for myApp to grant myApp access to a dedicated directory on the SDCard e.g. /sdcard/Documents/myApp/db in a way
that the user can provide there a db for myAPP, which myApp will move to internal storage in case of no DB, is present?
It would be also important to just set this permissions once on a device and having then the app confirmed to have access to this directory as long as installed on the device.
(/sdcard/Documents/myApp/db -> app-internal-dir)
Constraints:
It's no option to use via API29 in the AndroidManifest android:requestLegacyExternalStorage="true". Reason:
I anyway need to provide up a mechanism, where the user can copy a db file to the SD-Card which the app will use in case of no DB is present.
The app should then move (not copy) this huge file into its app directory.
I would also like to have a dedicated directory on the sdcard, where the application can move the current DB into it. (Do the opposite: app-internal-dir -> /sdcard/Documents/myApp/db-export)
It is no option to use the MANAGE_EXTERNAL_STORAGE permission, due this app is not a classified App that would get the permission from Google to do that (like a filebrowser or a antivirus App)
Can anybody give me some recommendation which approach to use?
The whole Logic is currently used by File Objects and Paths to working directories with the java.io.File-API
Thanks in advance!
EDITED:
I'm able to open the folder and query the DocumentFiles when it returns from the picker:
Here is my code that starts the picker:
// Choose a directory using the system's file picker.
val intent = Intent(Intent.ACTION_OPEN_DOCUMENT_TREE).apply {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
// Optionally, specify a URI for the directory that should be opened in
// the system file picker when it loads.
val pathAfterPrimaryIndicator = bootstrapViewModel.getDbMigrationDirectoryName()?.substringAfter("/0/")
val uri =DocumentsContract.buildDocumentUri("com.android.externalstorage.documents", "primary:$pathAfterPrimaryIndicator")
putExtra(DocumentsContract.EXTRA_INITIAL_URI, uri)
}
}
startActivityForResult(intent, DOCUMENT_TREE_ACTIVITY_REQUEST)
Here is some code that handles the return:
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (resultCode == RESULT_OK && requestCode == DOCUMENT_TREE_ACTIVITY_REQUEST) {
val treeUri: Uri? = data!!.data
val pickedDir = DocumentFile.fromTreeUri(this, treeUri!!)
pickedDir?.listFiles()?.forEach {
// test if its a Db file and move it to the internal storage
if (it.isFile) {
// QUESTION: --> How can I MOVE those files (some files are huge) found here to the app-internal directory?
}
}
The question is now still open how to MOVE the directory content into the app internal directory. Because some of the files are huge so copy them is not really an Option. How can this be done?
Android 11+:
If the file comes from a PC then the officer can directly place the file in getExternalFilesDir(null) and you are done.
If the officer places the file in a public directory of external storage then your app should use ACTION_OPEN_DOCUMENT to let the user choose the file.
Since Android Q came out there are some privacy changes in the permissions about read from external storage. I have a chat application that the user can choose a photo from Downloads etc... and send it. So i need to have access to that files. The way i did this is by using contentProvider.
context.contentResolver.openInputStream(uri)?.use { inputStream ->
while (true) {
numBytesRead = inputStream.read(buffer)
// .. do stuff
}
}
The uri that is available at that time is -> file:///storage/emulated/0/Download/myFile.pdf
However i get a FileNotFoundException but the file trully exists.
I have set all the permissions in the manifest and granted them on the launch of the app. From Android <= 9 it works properly.
So what do i have to do...?
I have a chat application that the user can choose a photo from Downloads etc... and send it
That will not be possible on Android 10 and higher. You need to switch to something else. For example, you could use ACTION_OPEN_DOCUMENT to allow the user to choose content from anywhere the user wants. Then, use the resulting Uri to upload it to your chat server, akin to how you are using the Uri in your code snippet. Or, better yet, don't read it all into memory — use something like this OkHttp InputStreamRequestBody implementation.
For Android 10, you can add android:requestLegacyExternalStorage="true" to your <application> element in the manifest. This opts you into the legacy storage model, and your existing external storage code will work. That will not work on Android R and higher, though, so this is only a short-term fix.
Background
Up until Android Q, if we wanted to get information about an APK file, we could use WRITE_EXTERNAL_STORAGE and READ_EXTERNAL_STORAGE to get access to the storage, and then use PackageManager.getPackageArchiveInfo function on the file-path.
Similar cases exist, such as using ZipFile class on a compressed file, and probably countless framework APIs and third party libraries.
The problem
Google announced a huge amount of restrictions recently on Android Q.
One of them is called Scoped Storage, which ruins storage permission when it comes to accessing all files the device has. It lets you either handle media files, or use the very restricted Storage-Access-Framework (SAF) which can't allow apps to reach and use files using File API and file-paths.
When Android Q Beta 2 was published, it broke a lot of apps because of it, including of Google. The reason was that it was turned on by default, affecting all apps, whether they target Android Q or not.
The reason is that many apps, SDKs and Android framework itself - all use File API quite often. On many cases, they also don't support InputStream or SAF-related solutions. An example for this is exactly the APK parsing example I wrote about (PackageManager.getPackageArchiveInfo).
On Q beta 3, however, things changed a bit, so that app that target Q will have the scoped storage, and there is a flag to disable it and still use the normal storage permissions and File API as usual. Sadly the flag is only temporary (read here), so it's delaying the inevitable .
What I've tried
I've tried and found the next things:
Using the storage permission indeed didn't let me read any file that's not media file (I wanted to find APK files). It's as if the files don't exist.
Using SAF, I could find the APK file, and with some workaround to find its real path (link here), I've noticed that File API can tell me that indeed the file exist, but it couldn't get its size, and the framework failed to use its path using getPackageArchiveInfo . Wrote about this here
I tried to make a symlink to the file (link here), and then read from the symlink. It didn't help.
For the case of parsing APK files, I tried to search for alternative solutions. I've found 2 github repositories that handle the APK using a File class (here and here), and one that uses InputStream instead ( here). Sadly the one that uses InputStream is very old, missing various features (such as getting the app's name and icon) and isn't going to be updated anytime soon. Besides, having a library requires maintenance to keep up with future versions of Android, otherwise it might have issues in the future, or even crash.
The questions
Generally, is there a way to still use File API when using SAF ? I'm not talking about root solutions or just copying the file to somewhere else. I'm talking about a more solid solution.
For the case of APK parsing, is there a way to overcome this issue that the framework only provides file-path as a parameter? Any workaround or a way to use InputStream perhaps?
How to handle SAF when I can only handle File or file-path? It is possible, even if you can send only a Java File object, or path string to a library function which you cannot modify:
First, obtain a Uri to a file you need to handle (in String form it would be like "content://..."), then:
try {
ParcelFileDescriptor parcelFileDescriptor =
getContentResolver().openFileDescriptor(uri, "r"); // may get FileNotFoundException here
// Obtain file descriptor:
int fd = parcelFileDescriptor.getFd(); // or detachFd() if we want to close file in native code
String linkFileName = "/proc/self/fd/" + fd;
// Call library function with path/file string:
someFunc(/*file name*/ linkFileName);
// or with File parameter
otherFunc(new File(linkFileName));
// Finally, if you did not call detachFd() to obtain the file descriptor, call:
parcelFileDescriptor.close();
// Otherwise your library function should close file/stream...
} catch (FileNotFoundException fnf) {
fnf.printStackTrace(); // or whatever
}
Posting another answer just to have more room and let me insert code snipes. Given the file descriptor as explained in my previous answer, I tried using net.dongliu:apk-parser package mentioned by #androiddeveloper in the original question, as follows (Lt.d is my shorthand to using Log.d(SOME_TAG, string...)):
String linkFileName = "/proc/self/fd/" + fd;
try (ApkFile apkFile = new ApkFile(new File(linkFileName))) {
ApkMeta apkMeta = apkFile.getApkMeta();
Lt.d("ApkFile Label: ", apkMeta.getLabel());
Lt.d("ApkFile pkg name: ", apkMeta.getPackageName());
Lt.d("ApkFile version code: ", apkMeta.getVersionCode());
String iconStr = apkMeta.getIcon();
Lt.d("ApkFile icon str: ", iconStr);
for (UseFeature feature : apkMeta.getUsesFeatures()) {
Lt.d(feature.getName());
}
}
catch (Exception ex) {
Lt.e("Exception in ApkFile code: ", ex);
ex.printStackTrace();
}
}
It gives me the correct app label, for the icon it gives me only a string to the resource directory (like "res/drawable-mdpi-v4/fex.png"), so again raw ZIP reading functions would have to be applied to read the actual icon bits. Specifically I was testing ES File Explorer Pro APK (bought this product and saved APK for my own backup, got the following output:
I/StorageTest: ApkFile Label: ES File Explorer Pro
I/StorageTest: ApkFile pkg name: com.estrongs.android.pop.pro
I/StorageTest: ApkFile version code: 1010
I/StorageTest: ApkFile icon str: res/drawable-mdpi-v4/fex.png
I/StorageTest: android.hardware.bluetooth
I/StorageTest: android.hardware.touchscreen
I/StorageTest: android.hardware.wifi
I/StorageTest: android.software.leanback
I/StorageTest: android.hardware.screen.portrait
My application will be used offline and I plan daily pull/push synchronizations via USB cable. My users are in a very primitive situation: no wifi, and no cell phone towers. My question is not about synchronization, but rather just getting access to the data so that I can synchronize.
I connect the cable, select USB for file transfer, and I can see Internal Shared Storage. But I cannot find my SqlLite database anywhere. I have tried using these paths for the database:
Environment.GetFolderPath(System.Environment.SpecialFolder.ApplicationData);
Environment.GetFolderPath(Environment.SpecialFolder.Personal);
Android.App.Application.Context.FilesDir.AbsolutePath;
The app works fine with any of those paths, the data is stored and retrieved, but I cannot see the database from my PC.
I have also tried this but it blows up:
Android.OS.Environment.GetExternalStoragePublicDirectory(Android.OS.Environment.DirectoryDownloads).AbsolutePath;
I have tried debugging with Xamarin Live but then I get this error:
"You need to call SQLitePCL.raw.SetProvider();"
I have tried adding console.writeline and Log.Error to add in some diagnostics but I can't find any log files in Internal Shared Storage.
I have WRITE_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE permissions in the manifest; even tho I think that is not necessary.
If I could store files in Internal Shared Storage then I could put the database there and access it to synchronize. And I could create a simple text logging facility to write a text log to the same place.
I have rarely asked for help in 40 years but I've been at this for days. Thanks!
To get files onto Internal Shared Storage accessible via USB took 3 steps:
1: Get runtime permissions
ActivityCompat.RequestPermissions(activity, new String[] { Manifest.Permission.WriteExternalStorage, Manifest.Permission.ReadExternalStorage }, 1);
2) use this path:
string extPath = Android.OS.Environment.ExternalStorageDirectory.AbsolutePath;
extPath = Path.Combine(extPath, "RtTrace.txt");
File.AppendAllText(extPath, "new content" + System.Environment.NewLine);
3) Media scan the resulting file to make it visible:
MediaScannerConnection.ScanFile(Android.App.Application.Context, new String[] { extPath }, null, null);
The path on Environment.SpecialFolder.Personal refers to a private area where only the app (and the OS itself when you clear data from the app, for example) have access. I don't know about ApplicationData.
You can easily copy your app file (the protected one) to a public folder, like Downloads or create a new folder MyAppDirectory at the public storage space, that will allow access from other devices. Then, you can clear local data that you don't need anymore (after the sync process).
To create a new public folder on Android:
var folder = System.IO.Path.Combine(Android.OS.Environment.ExternalStorageDirectory.AbsolutePath, "MyAppDirectory");
System.IO.Directory.CreateDirectory(folder);
You'll get this:
Then, copy the file:
var newFile = Path.Combine(folder, "MySharedFile"); // The database, xml, json, text or any file you want to share
var sourceFullName = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Personal),"MyOriginalFile");
System.IO.File.Copy(sourceFullName, newFile, true /* overwrite */);
I hope it helps.
SQLite db is stored in application private memory and can't be accessed over USB, try exporting contents of your DB to an external file in your internal memory or External memory to be able to access it via a USB chord.