Develop Reference

How can I use Android scoped storage to store large app specific files?

I have an old drawing app on Android, which stores drawings (.PNG files) out on a specific folder on external storage. With Scoped Storage in Android 11, I need to find a way to save files, but I can't seem to find a solution that meets my needs. My app has its own gallery UI, like many drawing apps, so that I can control the selection/view UX.
I've tried:
MediaStore: I inserted files with MediaStore. The issue I ran into here is that I couldn't figure out a way to query just the files that my app created. Querying mediastore always returns a bunch of stuff I don't want.
Storage Access Framework. My intuition with this framework, is that I would probably spend a lot of time and potentially regret going down that road. I don't like the idea of not controlling the file selector (gallery). I also don't like relying on intents for a critical part of my app.
MANAGE_EXTERNAL_FILES. Based on the criteria defined by Google, my app wouldn't qualify to use this, though it would be great (since I could keep my old code).
Use internal file storage. I coded this up, and it's really clean, but I hesitate to roll this out, because uninstalling would mean that users lose their documents. Note that i have a Share intent, so users can "export" files one at a time.
What I want:
To write files where they aren't removed when the user uninstalls my app.
To write my files somewhere that the user can back them up one way or another.
Simplicity. I don't want to confuse existing users by changing things drastically.
Am I missing something regarding my options here? Can Mediastore effectively partition my files so that I can show them in the gallery? Does internal storage seem like the best option for me?

Just create your files in the old fashioned way in your apps sub folder in public Documents directory.
No special permissions needed.

Android DB in app-specific folder vs external folder

I have an Android app that writes data to a db. Initially, when we started, we stored the db in an app-specific folder (something like /storage/org.domain.app/db/db.sync and so on). Things worked with no issues. However, the issue was that this DB would get wiped out everytime we reinstall the app.
We therefore moved the DB to a different folder outside the app. (Now the path is something like /storage/data/db/db.sync). Now, if we reinstall the app, we still have the old DB entries. However, now reading and writing to the DB is a lot lot slower.
I am not sure if this slowness is because the amount of data has shot up. Since the data is now persistent across multiple reinstalls.
I suspect this could also be because the DB is now in a folder external to the app. To rule this out, I setup the DB in an app specific folder and intend to copy the entire DB from some previous tests (that contains lot more data - the amount of data which caused the DB access to slow down). However, I am unable to do so.
Some questions:
1. I am not able to actually access the folder where the DB is setup. The folder is created in Context.MODE_PRIVATE mode. Therefore, only the app itself can access it. I tried changing it to MODE.WORLD_READABLE. However, this is deprecated and throws an exception while using it. Is there a way to make this folder easily editable?
I also tried accessing the DB from Device file explorer in Android studio. I opened the existing DB (db.sync file) and just copy-pasted the large DB I already have. However, it looks like this method does not allow us to copy or even display more than 400 KB of information. The DB I have is around 60-70 MB. Hence this approach did not work either.
Is there a way I can edit the db.sync file?
Does DB being in a folder external to the app make it slower? Esp when there are too many entries. In my case, certain search operations happen via a local cache and happens in a giffy. However, we also need to log some stuff to the DB and that takes forever. I figured this out by adding timer logs in the code.
Anything else, I could do to identify if the DB path is causing the issue?
Thanks,
Anand

How to Manage increasing size of sqlite in Mobile App?

My app are sometime needed syncing with web servers and pull the data in mobile sqlite database for offline usages, so database size is keep growing exponentially.
I want to know how the professional app like whatsapp,hike,evernote etc manage their offline sqlite database.
Please suggest me the steps to solve this problem.
PS: I am asking about offline database (i.e growing in the size after syncing) management do not confuse with database syncing with web servers.

I do not know how large is your data size is. However, I think it should not be a problem storing reasonably large data into the internal memory of an application. The internal memory is shared among all applications and hence it can grow until the storage getting filled.
In my opinion, the main problem here is the query time if you do not have the proper indexing to your database tables. Otherwise, keeping the databases in your internal storage is completely fine and I think you do not have to be worried about the amount of data which can be stored in the internal storage of an application as the newer Android devices provide better storage capability.
Hence, if your database is really big, which does not fit into the internal memory, you might consider having the data only which is being used frequently and delete otherwise. This highly depends on the use case of your application.
In one of the applications that I developed, I stored some large databases in the external memory and copied them into the internal memory whenever it was necessary. Copying the database from external storage into internal storage took some time (few seconds) though. However, once the database got copied I could run queries efficiently.
Let me know if you need any help or clarification for some points. I hope that helps you.

For max size databases. AFAIK You don't want to loose what's on the device and force a reload.
Ensure you don't drop the database with each new release of your app when a simple alter table add column will work.
What you do archive and remove from the device give the user a way to load it in the background.

There might be some Apps / databases where you can find a documentation, but probably this case is limited and an exception.
So to know exactly what's going on you need to create some snapshots of the databases. You can start with that of one app only, or do it directly with several, but without analyzing you won't get a reliable statement.
The reasons might be even different for each app as databases and app-features differ naturally too.
Faster growth in size than amount of incoming content might be related to cache-tables or indexing for searches, but perhaps there exist other reasons too. Without verification and some important basic-info about it, it's impossible to tell you a detailed reason.
It's possible that table-names of a database give already some hints, but if tablenames or even fields just use meaningless strings, then you've to analyze the data inside including the changes between snapshots.

The following link will help in understanding what exactly Whatsapp is using,
https://www.quora.com/How-is-the-Whatsapp-database-structured
Not really sure if you have to keep all the data all the time stored on the device, but if you have a choice you can always use cloud services (like FCM, AWS) to store or backup most of the data. If you need to keep all the data on the device, then perhaps one way is to use Caching mechanisms in your app.
For Example - Using LRU (Least Recently Used) to cache/store the data that you need on the device, while storing the rest on the cloud, and deleting whats unneeded from the device. If needed you can always retrieve the data on demand (i.e. if the user tries to pull to refresh or on a different action) and delete it whenever its not being used.

Do I need to store paths to my files or can I simply note their storage type?

I did some testing and, with an Android emulator, outputted a few pathways:
Internal Storage context.getFilesDir() has path /data/data/package_name/files
Private External Storage with context.getExternalFilesDir(null) has path /storage/emulated/0/Android/data/packagename/files/
Public External Storage with Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES) has path /storage/emulated/0/Pictures/
In my app, I've defined an object that has a timestamp string and an array for images (more specifically, paths to those images), and this information is stored in a sqlite database.
But is this a bad idea? Because if someone wanted to move this app to another device, wouldn't the paths change? I'm a little confused as to how files can be moved around, how different devices impact this situation, etc. I am confused as to what the correct approach is for storing image references in the sqlite table. Do you store absolute paths like above? Or do you store a number representing whether the pic is internal/private external/public external and pull that way regardless of device/file structure?
I'm just trying to make sure I don't publish an app that lacks some critical feature simply because I misunderstood something. Like if I start using an app that can take pictures/store them internally, it's all on internal storage, all on this phone. But if I want to start saving pics to an SD card or something else, I have to change where I store the pics and how I reference them. And what if I upgrade my phone? I'd have to somehow carry/transfer all that stuff to a new infrastructure, etc. Or the "cloud", even.
In other words I am just trying to understand all the possible use cases, here, in terms of how Android app data is stored and transferred.

Your questions are a little broad. I will try to answer them in turns:
But is this a bad idea?
Not in itself. The issue here is how are you managing changes and the Android alterations that may occur in the future (as well as the versions you are currently supporting)
Because if someone wanted to move this app to another device, wouldn't the paths change?
Relative paths would never change. They were implemented (I am guessing around API 2~6) to ensure that hard disk management would not be necessary in apps. By getting the Enviroment.getExternalFilesDir(stuffs..) you would garantee a path to the same file, regardless of readl disk allocation.
how files can be moved around, how different devices impact this situation, etc.
Files from Desktops (PC, Mac, Linux, Ubuntu, etc) follow a contracted with Android devices, that simulate the same disk system. This way, a PC can save a file (png, mp3, custom, etc) in a "root folder" that becomes the same root for Enviroment.getExternalFilesDir(stuffs..) BUT data private to your application is never accessible (on non-rooted devices), to the outside system.
Do you store absolute paths like above?
You can. But I believe this goes against the system best-practices. Saving a relative path, then building a File object, in regards to that path is the correct approach. Do note that the File class in Android expects you to know a bit about the file you are refferencing.... either its extension, name regex, physical location, relational location, etc... you need to know at least one of those.
Finally, what I think you asked:
Can the user replicate the same content in different devices?
For this, you must ensure that the user is "know" for instance, using the Play Store account, you can then have a repository, and have devices syncronize with that repository. Or even add a "export" function, that copies files, and imports them somewhere else.

Is there any way to use a pre-existing database from Xamarin without copying it from Assets?

I have a rather large SQLite database (~20 mb) I need to access from my Android Xamarin-Forms app.
Everything online I've read says you can read the database by copying it to the filesystem first. For example, this question. However, won't that mean the large database is duplicated, wasting users' precious space (and nearly doubling the footprint of my app)?
There must be a way to read the SQLite database directly from the assets, or use some other method to bundle the database with my app that won't waste so much space. But how can I do this?

You don't want to use it from assets, even if you could, because assets is a compressed read only file, part of your installation. You can't write updates into it, which kills 90% of database use. And its inefficient for reading as its zipped up. So you really do need to copy it. If you're worried about disk space, consider downloading it from the web rather than keeping it in your apk.