Is there anyone using git in such a fashion?
I would like to distribute some multimedia content from a server to some Android remote devices. I would like them sending back a log file with device usage statistics (provided by an android app I will write).
The server could be anything but I would prefer a linux box.
I thought that since git handle and sych only differences between files, It would be a nice tool for this purpose and I would have content revision history as a bonus.
I need some piece of advice on how the repositories architecture could be organized: does It have to be a star topology or something different?
The remote end of the sistem don't need any interactivity, in other words the remote git repository could pull and push whatever It needs to, autonomously and automatically.
UPDATE: I've found here on SO the author of git internals (I'm downloading It right now), Scott Chacon talking about the architecture I would like to implement.
UPDATE 2: OK I read the chapter about "Non-SCM uses of Git" and here is what the author says about a Peer to Peer CDN:
You have to get new content [...]
consist of any combination of xml
files, images, animations, text and
sound. You need to build a content
distribution framework that will
easily and efficiently transfer all
the necessary content to the machines
on your network. You need to
constantly determine what content each
machine has and what it needs to have
and transfer the difference as
efficiently as possible.[...]
It turns out that Git is an
excellent solution to this problem.
I don't find anything about mentioning little portions of the book inside it, so I hope that I'm not violating any copyright. In any case I will delete It if someone complain.
I would suggest against using git for such purpases. For starters, Git will use extra phone storage for the revision history, and it will send entire files (not deltas) anyway because multimedia content is binary and diffing does not work on it. Just implement a method to list server-side multimedia with last-modification dates and another method to download updated files (I would suggest HTTP as it is the simplest). On the server side, you can of course use git internally for versioning the multimedia files, but I'd rather not expose the git interface.
The git protocol tries to send patches instead of whole files, but the git storage engine always stores whole files, and always keeps old versions of the files. git is probably not the tool for the job if you aren't trying to keep file history.
rsync is a mature file distribution system that can work over ssh or its own protocol (the same as git), can make binary patches, and doesn't necessarily keep change history. Probably start looking there to see if you can get that work.
So in a previous job, we used Git for exactly this and the reason was that our media assets were not often changing, so no matter what we used it was likely we would have to send the whole file anyways - thus, the issues with binary deltifying, though also an issue with other content distribution tools, was not important.
The main advantage to rsync (and presumably unison, though I've never used it) is that you can build the content trees in the index and store the trees in Git under a branch per client rather than having to have everything on disk to run rsync on. If you have several variations on content, it's pretty cool to be able to record unique trees of content needed by each client - of which you could have thousands of combinations - and have a simple pull command fetch only what's needed and update it on the client. That was the reason we choose Git instead of rsync to do that. If every client needs exactly the same set of data, perhaps rsync would be easier, however the other nice thing about Git is that you get a history of the content on each client - when and how it changed for every single client.
We also used it to record log files - since they are generally pretty uniform and text based, they delta excellently and transfer very efficiently - we were very happy with using that to record and transfer back upstream our log data.
Related
This might be a trivial question but I dont know the answer to it. Looking for android gurus.
My task is to compile android kernel and run on a pandaboard. I cannot just take the uImages there are drivers which I need to modify..
Now one way is to download every thing from AOSP by repo init... and downloading 3-4 Gb of git from day 1 of android git init.
Is there a way to just get the source files for ICS. It is easy for linux just go to git.kernel.org and get the snapshot of the tag like 3.0 , is there a same git snapshot available in android ?
Any help is welcome....
TLDR: Not likely. Don't bother and simply repo sync to the branch you need.
Longer explanation:
If somebody has created snapshot of your particular tree, then you might be able to do that.
However, I don't think anybody will bother to do so, because having this git-less snapshot is pretty much useless from developer's standpoint, and space/traffic savings over repo sync are miniscule.
Note that git is extremely efficient at packing historic metadata in object store (provided it was packed by git gc recently). It is relatively rare situation when compressed git metadata takes more space than simple checkout of the top of the tree. Since repo sync only downloads compressed metadata, this process is very efficient.
So speed this up even further and avoid handling small objects, Android projects creates single-file compressed bundles for most repositories included in standard manifest, and modern repo script understands how to get these bundles first for efficiency, and then get small deltas afterwards. This allows to perform first repo sync much faster than otherwise. Basically, you are only limited by the speed of your internet connection.
I use Eclipse to develop in java for Android. I have installed Mercurial to control the source files with other programmers.
But I don't like how mercurial works, without options to lock the files when someone are modifing it.
I know mercurial works in this way, but we prefer the lock "approach".
Is there any alternative to mercurial with lock files?
Is there any way to config Eclipse to avoid write on read-only files? If I have a read-only source file, with Eclipse I can open it and write in it.
Thanks.
Quote from Mercurial: The Definitive Guide:
For instance, a distributed revision control system cannot, by its
nature, offer a file locking facility. There is thus no built-in
mechanism to prevent two people from making conflicting changes to a
binary file. If you have a team where several people may be editing
binary files frequently, it may not be a good idea to use Mercurial—or
any other distributed revision control system—to manage those files.
However, right underneath the paragraph, there is a reference to an extension you can use, named "lock/unlock extension". A quick search located a page that describes the extension, but I didn't use it yet. The linked page also hints at the existence of another, older extension of the same name.
As a final note, personally, I agree with the quote from the book. If you need file or directory locking in your workflow, trying to use a distributed version control system is only going to create problems for you. You will probably be better off using a more traditional version control system.
I've been trying to use SVN Kit to gain access to an SVN repository from an Android app.
Our project is supposed to check for changes to files and download them if they have changed, but still with the ability to rollback to previous versions if nessecary.
I set up a regular Java project, and had no problem using SVN Kit, I logged into the SVN server and retrived a list of the files stored there.
But when I tryed to do this from an Android project all hell broke loose. The VM run out of heap space, just trying to build it and Eclipse went down in sreaming flames mith comments like 'GC overhead limit exceeded'.
I get the impression that this is due to the diffrent type of virtural machine that Android uses, and the SVNKit jars are compiled for a diffrent type of VM.
This Guy claims to have got it working, with what looks like an older version of Android.
Now I have suggested that we use some kind of ftp server approach, uploading new versions of the files, perhaps with seperate files containing versioning information, but I have to explore this path before I write it off as a dead end, or at least suggest that to my boss!
anyu help or suggestions would be greatly appricated.
My answer may not be directly related to SVNKit but would actually try to address the original problem.
As I understand your use-case, you want to download files if they have changed and are trying to use SVN to do this with its update command. I would rather create a simple web-service which should return me the list of files with their checksums or md5Hash. Now I am assuming that only the relevant files are present on the server. Upon receiving the hashes, you can compare them with the hashes on the device and decide to update based on any difference. This approach will work for rollback too as you need to just replace the file on the server with old file and this would be treated as a new one.
As far as heap overflow and out of memory is concerned, they are mostly because of the limited heap space around 10MB.
I created a SVN client based on SVNKit it is available for free here : https://play.google.com/store/apps/details?id=com.valleytg.oasvnlite.android
I am not sure if that will work for your purpose or you have to build your own. If you are looking to build your own, svnkit will not work directly on the android platform. Some of the libraries used by SVNkit are not available on the Android platform.
I'm working on a small project for fun, which involves Android and a web server.
I want some suggestion on the protocol/model to use that is best suited for the following scenario.
Server side
I have setup the following 5 components:
A config file rule.csv which contains 6 columns and about 20 rows. Each row in that file a rule.
A "switch" file RUN.on or RUN.off
A Java program that runs according to those rules, if RUN.on exists
A cronjob that runs the Java program every 5min.
A PHP page. control.php?run=on will rename RUN.off to RUN.on
control.php?run=off will set RUN.off
On Android
An app that submits to control.php to change the running status.
Goal
Now I want to add a feature to that android app, such that I can view and add/remove the rule in rule.csv. That requires a Android <--> Server <--> File communication. However, getting the whole file using PHP, and transfer it back after editing does not seem to be a good way in this case.
Since this is a project just for fun, getting it to work is not my priority; I'd like to look for a good model that handles this kind of task and I'm willing to learn other languages if necessary. Any advise?
I'm guessing the rule file is kinda big or you would just transfer it.
Adding a new rule seems pretty simple - enter the data on the Android and send the rule to the server. But how do you want the Android user to pick the rule to remove? Can you just transfer the names of all of the rules, let the user choose the rule, and then just send the rule name back?
I am doing a forensic course and as a requirement I have been asked to develop a forensic investigation tool (windows based) for Google's Android OS. The requirement is such that given an image file, the tool should be able to display the databases that the applications are using, call history, messages and etc..
I have little experience in Java but I have no experience in Android development. The research so far has given me nothing on how to go about this. If anyone could point me in the right direction I would much appreciate it.
Thanks in advance.
Step 1 would be mounting the filesystem. Since Android is Linux based, there's a huge array of filesystems available, and individual vendors may or may not decide to write their own filesystems, just for the fun of it. On Windows, your options include ext2fsd or ext2read, among other possibilities.
Once you've got the filesystem mounted, then you get to deal with the per-application data storage. I'd wager a fair amount of applications use SQLite3, because it is an amazing tool. But you'll have to figure out, for each type of data you want to read, where it is stored and in what format. (The standard file(1) tool on Linux systems can come in handy, it knows heuristics that are surprisingly good at showing what type of file you might be dealing with.)
If you have the .apk of an application, a tool such as dex2jar, used in combinaison with something like jd-gui, can get you the JAVA source-code of the application (which can help, if not obfuscated).
After that, an .apk is basically a zip-file -- which means opening it with an unzip-ing application will allow you to get the images and resources it uses.
Then, databases used by Android applications tend to be SQLite, on which you can do SQL queries, using an SQLite client.