Slurp a disk image of the data partition on an Android device - android

I hope it won't sound like I'm up to no good -- I'm actually motivated by this problem -- but I'm trying to slurp down a raw disk image of the data partition of an Android device. Ie, not just the files stored there but all the orphaned file fragments and everything.
If I can achieve that then I can write some code to sift through that raw data and do some data forensics.
(I hope the recovery part of this question is sufficiently programming-related. I don't think I'm going to get an answer anywhere else and I intend to offer a large bounty for any answers here.)

Can't be done on "normal" devices via application: http://developer.android.com/guide/topics/security/security.html
Possibly it can be done on rooted devices, but this will depend on particular ROM. I guess you'd need a dd command.

It is unfortunate that the data is on the built-in storage. Does your phone automatically provide USB Mass storage mode access to the hardwired storage when you plug it into a computer? If so, you might be in luck still.
I'd suggest using a Linux machine and make sure that udevd(8) won't try to mount the drive when you plug it in -- you want to mount it read-only, which won't be the default. (Depending upon your distribution, it might have some other mechanism to automatically mount filesystems on USB mass storage drives when they are attached. If so, find them and turn them off.) Maybe service udev stop or /etc/init.d/udev stop are sufficient to stop udevd(8) from pretending it knows better.
Check dmesg(1) output to find the device name that the hardware takes when it is plugged in. I'll pretend it reports /dev/sde.
dd if=/dev/sde of=~/android_backup bs=4096
chmod 400 ~/android_backup
The if specifies the input file, the of specifies the output file, and bs asks for a blocksize of 4096. (There's nothing magical about 4096 -- it is the size of a memory page on many platforms, so blocks of this size can sometimes be moved around more efficiently than smaller blocks.)
The chmod(1) command makes it more difficult to modify the backup by removing write permission.
This makes a copy of the entire "drive" -- all the filesystems. This might be more difficult for programs to handle, so grab the specific partition your data is on while it is mounted -- it'll never be easier than now:
dd if=/dev/sde1 of=~/android_partition bs=4096
chmod 400 ~/android_partition
(If there are multiple, maybe grab them all. Disk space is cheap.)
Once you've got the image, you need to figure out how to salvage the data from it. It might be there, it might not be there, but there are some immensely helpful tools available that can help.
I've used Autopsy (part of The Sleuth Kit) before to recover deleted images from FAT-based camera storage. Granted, FAT is an easier problem, but they claim to support many filesystems, and it would be my first choice.
I haven't used Scalpel but it looks promising. It uses binary magic numbers to identify and slurps files out. It claims to be suitable for forensics use, but I don't know happily it will recover data from deleted files. (Since it claims it can handle raw partitions, I've got a good feeling about it.)
I've fiddled around with debugfs(8) before, but never used it when the stakes mattered. It looked amazing -- but also looked like it requires the user to have more advanced knowledge of ext* filesystems to do anything really useful. Read the manpage and guess if you think it's worth trying.
And don't forget you're not alone -- companies are available that can help recover data. (I've selected this specific company because I know Erik from this company and know him to be careful, thoughtful, and good at what he does. There are more companies than just this one but it'd be nice to find one that does more than just run Autopsy on images. You can do that.)

Related

Cloning data partition to other Android devices

We have several devices running Android in an industrial environment.
We think about completely setting up one device and then copy the whole /data directory to the other devices. The expected benefit is to configure things like wireless settings and install the required apps and updates only once on one device and somehow bring this onto the other devices.
Is this a good idea?
I am not sure if there is device specific information stored in data such as a device ID or something else that we would mess up with our idea.
Besides this what about 'uid's that are responsible to represent app permissions? Do you expect problems if our apps are not installed onto a device but simply come by a copied /data partition?
Is something specific established on a device after the first boot when /data is created and filled, that should not be replaced with something else?
Would it be better to copy only selected directories or even files? This is much more effort but ok if there is no other way...
I would appreciate any input about what can go wrong and what we should consider.

A Program for Transfering a folder from a Mac to an Android Phone

I've been trying to write a small program to copy a folder on my mac (10.10.3), containing a set of songs, into the music folder on my Galaxy S3 (GS3) when it is connected via USB. Clearly I could just use Android File Transfer (AFT), and copy the files in (manually), however this wouldn't teach me anything. My goal is thus to automate this act. Python seems like a good choice for this project, as it seems like mostly scripting (in fact a Bash script may suffice).
This should be simple, using a bash script like cp ~/../music_on_mac /.../music_target_on_android
However, the file structure of the GS3 doesn't show up in the finder (like you would expect from an ordinary USB drive). I can only see the file structure via Android File Transfer. From what I've read, this is expected behavior (I suppose due to formatting differences?). Thus I've been unable to find the target directory /.../music_target_on_android
My best guess (getting a bit out of my depth here), is that I need to copy the music folder, and then pipe it to AFT, and have AFT place it in/on top of the target Music folder on my GS3. Is this correct? If so, could anyone offer suggestions on implementing this? If not, alternative approaches or suggestions would also be appreciated.
I'd also note that I considered using ./adb push <local> <remote> to try to copy the files directly, however this pushed back that the device was read only. I'm also not familiar enough to find the proper directory on the GS3 (the one containing the target for the "Music" folder) using ./adb shell. There's also the added downside that in order to use a solution involving the ADT, one must have the ADT (which most people don't). Moreover, I want to keep it simple.
Research Update:
I've found that my phone will not show up in Finder because Google has disabled USB mass storage (aka mounting the phone as a disk) in favor of the Media Transfer Protocol (MTP). AFT is just an MTP client, needed because unlike Windows and Linux, OSX does not support MTP.
http://www.howtogeek.com/192732/android-usb-connections-explained-mtp-ptp-and-usb-mass-storage/
Based on tips, I've been able to implement an alternative based on creating an FTP server on my phone. Connecting to this gives me file system access and write privileges from my terminal, which is half the battle, however its quite slow. Thus I'd still like to find a way to automate instructions to Android File Transfer.

"Insufficient Storage Available" even there is lot of free space in device memory

The total space of my app is 10 MB, and after installation it will take less than 20 MB. In Galaxy Note I, while updating my app, it's saying "Insufficient Storage Available", where there is 214 MB of free space in device memory (internal). It happens even trying to download a new app.
I searched long for the solution, and a perfect reason for the cause of this problem, but I can't find it. How do I fix this problem?
This is the result of the 'adb shell df' in my another device which has the same problem. It has 35 MB free space:
/dev: 115788K total, 0K used, 115788K available (block size 4096)
/mnt/asec: 115788K total, 0K used, 115788K available (block size 4096)
/system: 179840K total, 168376K used, 11464K available (block size 4096)
/data: 201856K total, 168524K used, 33332K available (block size 4096)
/cache: 108544K total, 1284K used, 107260K available (block size 4096)
/cdrom: 8960K total, 8632K used, 328K available (block size 4096)
/tmp: 2048K total, 28K used, 2020K available (block size 4096)
/pds: 1536K total, 1320K used, 216K available (block size 4096)
/mnt/sdcard: 1928992K total, 1014496K used, 914496K available (block size 32768)
/mnt/secure/asec: Permission denied
Here's a very simple solution that works on my Samsung Galaxy S II and Note 1; I have no idea about other models:
Open the phone app and switch to keypad.
Dial *#9900#
On the screen that appears, click on the button labelled "Delete dumpstate/logcat".
I've restored about one GB of system space this way.
At first I tried Berislav Lopac's answer, but I got Connection problem or invalid MMI code. when I tried to dial *#9900#. I was using CyanogenMod on the phone, and I believe phones with custom ROMs don't use the stock dialer, so they lack the SysDump functionality.
Basically, Delete dumpstate/logcat in SysDump clears out the log files in /data/log. But you can also do this manually without SysDump. (This is assuming your phone has been rooted, which will be the case if your phone is running CyanogenMod or any other non-stock ROM.)
Make sure Superuser and Terminal Emulator apps are installed. (They come with most custom ROMs.)
Run Terminal Emulator
Type in su, hit return.
This will bring up a Superuser prompt. Grant access. (You will have to wait three seconds before you can click "Allow".)
Change current directory by typing in cd /data/log, followed by return.
MAKE SURE you are in the data/log directory by typing in pwd, followed by return. It should print out the present working directory you are in: /data/log. It is very important to make sure you are in the right directory as the next step removes all files in whatever working directory you presently are in.
Remove all the files in the directory by typing in rm *, followed by return.
Close the terminal window or app, or type in exit to leave the su session.
I deleted roughly 1,500 1 MB files like this and fixed my "Insufficient Storage Available" problem.
As with the other posters, I own a Galaxy S II, so it seems to be a problem with that model.
If anyone knows of the permanent solution to stop the log files building up, please let me know.
NOTE: Some file managers will falsely list /data/log to be empty as they are running unprivileged and hence lack the permissions to view the files inside.
The memory may be in reserve by the OS to be used for running what you normally run (kind of like a swap file). You may be able to squeeze in another app or two by
Trying to install them right after a restart, or
By force closing some apps that are running (but that second option may not be a good idea -- see the first link),
But the only very good fix might be to
Repartition your SD card so that apps can be installed directly to it (see the second link).
Take a look at forum post It was bound to happen: low memory warning!.
The important part is:
The OS knows how much memory it needs to run the apps you already have. This is a perfect example.
Now you may be able to "fool" the OS by force closing some apps that
are sitting in RAM. This will increase your "bucket" of memory which
may let you install an app, but remember if you do these types of
things you will only cause issues down the road.. lagg, error
messages, etc. (because you are fooling the OS in thinking you have
given it additional memory which in fact you did.. you only force
closed).
Another good explanation of what is happening is in forum post Low Internal Memory.
The important part is:
The reason why your internal space is filling up is 3-fold. First,
when an app is "moved" to the SD card, it isn't completely moved. Only
portions of it actually go. Second, the Dalvik cache of the app is
still stored on the internal memory (which takes up a substantial
amount of space). Three, the data for apps and all your system
settings are stored in the internal memory (yes, some apps use the
SD card for portions of their data, but every app has data stored on
the internal memory).
And the thread includes suggestions on what partitioning you can do to your SD card to allow 'moar apps'!
The package manager (“installer”) has a design problem: it can’t distinguish between a bunch of possible errors and regularly comes up with the “insufficient storage” excuse.
The first steps are done: identify it’s an install problem (1.) and not related to storage shortage (2.)
It happens on the console (pm install file.apk), with Google Play, other markets and manual GUI-install (for example, “clicking” on a downloaded APK file); it is not a download issue, ...
Packages end up entirely on the /data partition -or- mostly on the SD card (and a little on /data). – Both places show enough space as indicated by the original poster (33 MB and >900 MB respectively) for the <20 MB package. –And– the /data partition has more than 10% free (33 MB is more than 10% of 200 MB).
Surprisingly most answers don’t take this into account...
In reality, the /data partition needs a cleanup from residues from previous installs.
Identify the common name of the problematic package (for example, com.abc.def)
Uninstall the package (for example, pm uninstall com.abc.def)
Check what’s left of it in data (for example, find /data -name 'com.abc.def*')
Delete that stuff
The installer chokes on those, returning with the wrong reason. – The interesting part is: if the package gets installed on the SD card (forced or by other means) some (all?) leftovers on /data don’t hurt... which leads to the false belief that it is indeed a space problem (more space on the SD card...)!
The Stack Overflow question where I got half of this from is Solution to INSTALL_FAILED_INSUFFICIENT_STORAGE error on Android.
The first thing to do is to check the details of the error message. For this you could use the LogCat App.
For me the problem was an error like
Cannot rename native library directory /data/app-lib/vmdl-... to /data/app-lib/com.xyz
The solution was to activate the common sense function in my brain and look for the com.xyz folder in the app-lib folder with ES-Explorer. I recognized that this folder was already there. So removing it solved the renaming problem and the apps can now install properly.
The same problem was coming for my phone and this resolved the problem:
Go to Application Manager/ Apps from Settings.
Select Google Play Services.
Click Uninstall Updates button to the right of the Force Stop button.
Once the updates are uninstalled, you should see Disable button
which means you are done.
You will see lots of free space available now.
I also had this issue while installating an app after I had uninstalled that. I resolved downloading Lucky Patcher and then click on menu - troubleshooting - remove fixes and backups (insufficient storage available). Please notice you need your device to be rooted.
I have an un-rooted Nexus 4 (which has only internal storage, no SD card) and was getting this error with larger apps updating. Smaller apps would update fine.
I discovered that it was because I have recently signed up to the Play Music All Access service and had pinned several albums.
These are downloaded to the hidden /data partition and it was this that had run out of space (I assume)
I unpinned a couple of albums and now have no problems installing apps.
1. Restart the phone and then re-install the application!
I was also getting the same problem Insufficient Storage Available on my device, but I restarted my device, and it worked fine!
PS.:
2. Install application on external storage
For this, set Storage Location with the following command
adb shell pm set-Install-Location 2 // 2 for external storage ([SD card][1])
adb shell pm set-Install-Location 1 // 2 for internal storage
adb shell pm set-Install-Location 0 // for auto
I had this problem even with plenty of internal memory and SD memory. This solution is only for apps that won't update, or have previously been installed on the phone and won't install.
It appears that in some cases there are directories left over from a previous install and the new app cannot remove or overwrite these.
The first thing to do is try uninstalling the app first and try again. In my case this worked for a couple of apps.
For the next step you need root access on your phone:
With a file manager go to /data/app-lib and find the directory (or directories) associated with the app. For example for kindle it is com.amazon.kindle. Delete these. Also go to /data/data and do the same.
Then goto play store and re-install the app. This worked for all apps in my case.
I had the same problem, and it was solved by using App Cache Cleaner.
(HT: acejavelin#Android Forums)
I tried several of the suggested solutions, but none of them worked for me. After some research I stumbled upon a hint to move some apps from /data/app to /system/app. That freed up enough space to install new apps and update existing ones.
I can recommend the free utility SystemCleanup for moving the apps.
This is the easiest thing to do. Go to settings
look for storage or memory touch it and look for cached data. touch it
and clear your data from there. SIMPLE!!!
Does the app necessarily have to be installed in internal storage? If you are not running any service, you could try installing it on the external storage. This can be done by adding the following code in your manifest:
manifest
xmlns:android="http://schemas.android.com/apk/res/android"
android:installLocation="preferExternal".....
This usually works on Android 2.2 and higher in most of the cases. Be sure that your app will work properly if it is installed on the external storage. You'll get a good idea on what kind of apps can be installed on external storage in App Install Location.
When it comes to areal device, the behavior of devices seem different to a different group of devices.
Some of the strange collection of the opinion I heard form different people is:
Restart your device after unplugging
Remove some apps from device and free at-least 100 MB
Try to install your app from the command line, ./adb install ~Application_path
Move your application to SD card storage or make it default in SD card in the Android manifest file, android:installLocation="preferExternal"
You got a lot of memory acquiring stuff in the Raw folder which installs a copy in phone memory while instating an APK file and the device doesn't have enough memory to load them
Root your device and install some good ROM which help to letting the device know about its remaining memory.
I hope one of them is relevant to you! ;)
Most of the space you have available is reserved by the OS. The best and easy fix is to move your apps to external storage. This will free up a lot of space for you.
Some apps need to reboot to completely install. Android just says it has insufficient memory for some reason - it should say it needs reboot to complete the installation. Try it - it will install completely automatically when you reboot.
I resolved this issue for myself. Though, the internal and SD memory was showing a lot of free space. It was an issue with phone memory, which was almost full.
Hence, I moved many of my apps from the phone memory to internal iemory, to free up the phone memory: Settings -> Storage -> Apps (under the internal storage section) -> Internal tab
Here are the ones which are not checked and that are occupying the space on the phone memory.
Click on the Apps (one by one)
Click on the button: 'Move to Internal Storage'.
Once you free up a considerable amount of space on the phone memory this way, the error should not come.
After uninstalling a few apps I'm able to install the new one...
I think OS calculates the total memory required to run all apps. If it doesn't fit then it says "in sufficient memory".
I had the same issue on Galaxy S4 (i9505) on stock ROM (4.2.2 ME2). I had free space like this: 473 MB on /data, 344 MB on /system, 2 GB on /cache. I was getting the free spate error on any download from Play Store (small app, 2.5 MB), I checked LogCat, it said "Cancel download of ABC because insufficient free space".
Then I freed up some space on /data, 600 MB free, and now it's working fine, apps download and install ;). So it seems like this ROM needs a little more free space to work OK...
Clearing the Google Play cache memory will also help you... Go to the app information page of Google Play and clear it.
I did not find a free solution that worked, but I found a solution: I used the non-free version of Titanium backup, clicked on the context button and chose to check the memory occupied by apps. Find the download app, and you will see that it has a certain amount of space allocated to its cache. Clear data is the option that you want.
I got the same error message in case the package name was too long (>128 chars). Just using a shorter name fixed the issue.
I had more than 2 GB internal space and yet I was not able to install / update applications either from Google Play or manually.
Whatever may be the reason, wiping the cache partition solved my purpose.
Steps:
Recovery -> Wipe cache partition -> Reboot system now
If you have root, delete all of the folders on the path:
/data/app-lib/
And then restart your device.
I had this issue many times, and this fix worked for me each time. It even has an XDA thread.
I write all folders, because if there is a problem with one app, there is a good chance you have this issue with other apps too. Plus, it's annoying to find just the folders of the problematic app/s .
Go to Settings, Apps, All and uninstall Google Play Store.
This will replace by the old version and then you can download without the "Insufficient Storage ERROR"
It works for me
I kept having this problem, and I cleaned up the Dalvik cache using Titanium Backup. You'll need to have your phone rooted. As soon as I did that I was able to update Swiftkey and Beautiful Widgets.

Watch for changes in external USB storage for Honeycomb or later Android versions

I've got a DSLR camera and Samsung Galaxy Tab running Android Honeycomb. DSLR connected to a tablet using USB-cable (via USB kit enabling host functionality on a tablet). I'd like to being notified when user takes a photo using this external camera, in order to download this image to the tablet or do something else with it like showing Toast notification containing meta-information taken from the image.
As far as I get all of the existing tools (like FileObserver using underlying inotify mechanism, MediaContentProvider etc) allowing to watch for changes, demand a specific file or a filesystem path to be watched. This was good enough till we had a block layer protocol support in 2.x and earlier Android versions - when you connected device it'd been mounted somewhere on the device's filesystem and you was able to use this mountpoint as a watch point for those tools.
Since Honeycomb Google has changed the way of accessing external USB devices to Media Transfer Protocol with PTP as a subset of this. Now when I connect external USB device to an Android device I won't see any mountpoints for it (I'm using adb shell and subsequent mount command for getting them). Moreover, MTP implementation uses storage ids which apparently act as a higher level of abstraction and are just plain integer values. I was hoping there is a way to somehow translate these storage ids to the real paths/mountpoint/whatever but apparently there does not appear to be.
Thinking about Android MediaScanner which is already running on my device I guessed it could manage this issue with a special Intent broadcasted when there're changes in media files accessible from the device, so I started looking for already existing and suitable Intents for being notified, but no luck - I found only ACTION_MEDIA_MOUNTED and ACTION_MEDIA_REMOVED which are broadcasted only when device is connected and disconnected respectively. That means MediaScanner can't notice any changes on the device until you remount it (I've double checked it using stock Gallery app - it doesn't see any newly created images on the camera until you unplug and then plug it into the Android device again).
Trying to get this mount path for external sdcard, I used Environment.getExternalStorageDirectory() API call but it yields emulated Galaxy's sdcard path
which is /mnt/sdcard, not the camera's one. So it doesn't work for me either.
I managed to work out this issue only having launched periodic Timer event with AsyncTask acting as a TimerTask. This task does initialize usb connection, open device,
scan the whole device memory, getting only the last taken photo and then close device descriptor and usb connection.
It doesn't look like the best and efficient way of doing that taking into account it has to do all of these actions every time which could be pretty often, say each 5 or 10 seconds. It definitely quickly drains battery out and produces unnecessary system I/O for only taking last taken photo and comparing it with the previous last taken photo (in 99% it'd the same image), but I haven't found any better working solution for doing this. It'd much better off to have an observer mechanism with event-based notifications.
So my question is there more efficient way of being notified about changes in external USB storage for Honeycomb or later Android versions rather than one described above?
If you would like a more efficient way the camera would have to send out some sort of signal over usb that it has taken a photo. I guess it is not doing that.
Therefore you will have to check manually by doing the way your are discribing:
mount storage --> check for changes --> do your thing with your detected changes.
I dont know what you used to read "the MTP way" but here an example application:
https://github.com/ynakanishi/Honeycomb-MTP-sample
To not scan the entire storage every time you could save the result of read out file names for example every time you check and compare it to find the new ones. Usually the naming of the file also starts with the same number on a camera. So if you start a session with an empty sd card you know already the file name the photo will have. lets say img0001.jpg. So you just need to write a function to grab that file until it succeeds. if you want the next one img0002.jpg you can write a task/service/function to grab that file until successful, and so on.
If you want to save on battery you could implement an additional battery/power source inbetween for powering the usb port.
Instead of an Async task or timerTask you could try a ScheduledExecutorService and see if it uses less power.
Hope that gave you some new thoughts

Android: How to move Dalvik cache BACK to internal memory?

I've used A2SD script to move the dalvik cache to a special partition on my SD card in order to use it as an extended internal memory. However I did it under a beta release of ICS on my HTC Desire, which does not officially support a2sd. Now I have several usability issues and I would like to move everything back to the internal memory.
Is that possible through a terminal emulator (I have SU credentials on a rooted phone)?
I've tried APP2SDGUI from the market hoping that it has reverse option, but it says it cannot start for some reason.
I know I am messing with stuff that's not supported or even tested, but if anyone knows any script / command I would like to try them out.
P.S.: Here's a link to the script I've used
Try hard reset (pressing power button and upper volume key simultaneously at the time of booting the mobile) and chose factory reset.I hope this should solve your problem.

Categories

Resources