Develop Reference

Git unable to commit or rollback deleted file

I have two files in my android repo, :ls and :ls.pub.
When I clone the repo they are always missing and unstaged as deleted
If I commit the change, I receive "Changes not staged for commit: modified: :ls modified: :ls.pub"
if I stage them I receive "pathspec ':ls.pub' did not match any files"
If I do a git checkout I get "error: unable to create file :ls: File exists"
If I rollback I get "Error:pathspec ':ls.pub' did not match any file(s) known to git"
How can I remove or ignore them ?

AIMIN PAN's advice (to avoid "odd" characters like : in path names) is good. Git does allow : in path names, but : is a special first character in pathspecs, so it must be quoted or avoided.
The simplest method of doing that is usually to add ./ in front:
$ echo foo >':ls'
$ git add :ls
fatal: pathspec ':ls' did not match any files
$ git add ./:ls
$ git status --short
A :ls
An alternative is to use double colon or :(literal), e.g.:
$ git rm :ls
fatal: pathspec ':ls' did not match any files
$ git rm ':(literal):ls'
error: the following file has changes staged in the index:
:ls
(use --cached to keep the file, or -f to force removal)
$ git rm :::ls
error: the following file has changes staged in the index:
:ls
(use --cached to keep the file, or -f to force removal)
Note that a colon in a path name will trip up Windows systems. Again, Git can handle it (with care), but not every OS can.

Advise: stop using special characters in your file names.
Well I also want to know if colon ":" is supported character in git file name or not, but really don't have time to dig it out... On my Windows it is not allowed in file name.
If there is a strong reason to use : in file name, you should be prepared for numerous problems.

Clone manifest file with JGIT

I'm working in android project with git. In Linux, the command init project is:
repo init -u $URL -b $BRANCH -m $MANIFEST_FILE
Then i can get the MANIFEST_FILE in ./.repo folder (it includes the source path to build android project).
My code snippet in java application using JGIT is:
try {
Git repo = Git.cloneRepository()
.setURI(REMOTE_URL_SSH)
.setDirectory(localRepo)
.setBranchesToClone(Arrays.asList("refs/remotes/origin/build"))
//.setNoCheckout(true)
.call();
} catch (GitAPIException ex) {
System.out.println("clone fail: " + ex);
}
It will download all the manifest file in branch "refs/remotes/origin/build" to my local repo (in fact, this branch has more than 100 xml files. It burns out my memory space, ~ 2GB). I want to get only the xml file that I want like using command in linux above. How can it possible in java application ?
Thank in advance

As you expecting from command line to pull required files, you can follow these commands
Clone Repo:
repo init -u $URL -b $BRANCH -m $MANIFEST_FILE
To update only required file:
git fetch
git checkout origin/$BRANCH -- path/to/file

How do you update a mirrored repo workspace?

I created a mirrored repo workspace like this:
repo init -u $url -b $branch -m $manifest --mirror
repo sync
I would like to update the mirror to use a different manifest or an updated manifest and then re-sync to pull down any new or missing changes. If I try to run the same init command, it complains:
fatal: --mirror is only supported when initializing a new workspace.
Either delete the .repo folder in this workspace, or initialize in another location.
I suppose that I could manually fetch within .repo/manifests and then update the symlink, .repo/manifest.xml. I'd like to use the repo tool instead, though, if possible.
Or I could remove the entire .repo directory and then run the init command again. I'm avoiding this because unfortunately my manifest file is inside a 12GB repository. So, re-cloning is time consuming.

I ended up abandoning this idea. Instead, I create a throw-away git repo locally containing the manifest. Then I init from that. The shell functions below will create the temporary manifest repo, create/update the mirror, and then create a working repo for building.
function parameters
mirror
/path/to/mirror
url
/path/to/temporary/manifests_repo
branch
branch name to use.
This is not strictly necessary, but it makes it easy to label your build directories. I use the hash from our main repo corresponding to the current manifest.xml. Then, I can easily see which revision a build directory is based upon:
$ repo info Manifest
...
Manifest merge branch: refs/heads/BRANCH_NAME
manifest
relative/path/to/manifest_file from the current directory
build_dir
/path/to/new_build_directory
The reason for creating the mirror is to save time fetching as this code was designed for an automated build server. Notice that the build directory does a shallow clone from the mirror and even gets its copy of Google's repo tool from a local bundle allowing for offline building. The sync step uses options to only fetch the changesets required by the manifest and excludes other branches and commits.
function pushd_quiet() {
pushd "$#" >/dev/null
}
function popd_quiet() {
popd "$#" >/dev/null
}
function build_android_os_update_mirror() {
local mirror="$1"
local url="$2"
local branch="$3"
local manifest="$4"
# Create a temporary manifest repo
rm -rf "${url}"
mkdir -p "${url}"
cp -rf "${manifest}" "${url}"
pushd_quiet "${url}"
{
git init
git add .
git commit -m $branch
git branch -m $branch
popd_quiet
}
# Update mirror
mkdir -p $mirror
pushd_quiet $mirror
{
rm -rf .repo
repo init -u $url -b $branch -m $(basename $manifest) --mirror \
--repo-url ${BUILD_HOME}/tools/git-repo-clone.bundle
repo sync -c --optimized-fetch -j 16
popd_quiet
}
}
function build_android_os_create_build_tree() {
local build_dir="$1"
local mirror="$2"
local url="$3"
local branch="$4"
local manifest="$5"
rm -rf $build_dir
mkdir -p $build_dir
pushd_quiet $build_dir
{
repo init -u $url -b $branch -m $(basename $manifest) --reference=$mirror --depth=1 \
--repo-url ${BUILD_HOME}/tools/git-repo-clone.bundle
repo sync -d -c --optimized-fetch -j 16
popd_quiet
}
}

repo sync results in "fatal: Not a git repository: 'working_dir/.repo/projects/external/ipsec-tools.git'"

I am trying to download the Android source code and am following the procedure described in https://source.android.com/source/downloading.html, which consists of installing the repo script and then running
$ repo init -u https://android.googlesource.com/platform/manifest
$ repo sync
to get the main branch. I have also tried using the -f -j1 flags to go through errors and use only one core, as was suggested somewhere online. At this point, I think I am very close to getting the entire thing downloaded but it will invariably stop towards the end when working on ipsec-tools. It always gives the following output:
Fetching project platform/external/ipsec-tools
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1 100 1 0 0 3 0 --:--:-- --:--:-- --:--:-- 3
100 1690k 100 1690k 0 0 2307k 0 --:--:-- --:--:-- --:--:-- 2307k
fatal: Not a git repository: '/home/sebastian/WORKING_DIR/.repo/projects/external/ipsec-tools.git'
fatal: Not a git repository: '/home/sebastian/WORKING_DIR/.repo/projects/external/ipsec-tools.git'
fatal: Not a git repository: '/home/sebastian/WORKING_DIR/.repo/projects/external/ipsec-tools.git'
Traceback (most recent call last):
File "/home/sebastian/WORKING_DIR/.repo/repo/main.py", line 500, in <module>
_Main(sys.argv[1:])
File "/home/sebastian/WORKING_DIR/.repo/repo/main.py", line 476, in _Main
result = repo._Run(argv) or 0
File "/home/sebastian/WORKING_DIR/.repo/repo/main.py", line 155, in _Run
result = cmd.Execute(copts, cargs)
File "/home/sebastian/WORKING_DIR/.repo/repo/subcmds/sync.py", line 635, in Execute
fetched = self._Fetch(to_fetch, opt)
File "/home/sebastian/WORKING_DIR/.repo/repo/subcmds/sync.py", line 339, in _Fetch
self._FetchProjectList(**kwargs)
File "/home/sebastian/WORKING_DIR/.repo/repo/subcmds/sync.py", line 234, in _FetchProjectList
success = self._FetchHelper(opt, project, *args, **kwargs)
File "/home/sebastian/WORKING_DIR/.repo/repo/subcmds/sync.py", line 275, in _FetchHelper
no_tags=opt.no_tags, archive=self.manifest.IsArchive)
File "/home/sebastian/WORKING_DIR/.repo/repo/project.py", line 1110, in Sync_NetworkHalf
no_tags=no_tags)):
File "/home/sebastian/WORKING_DIR/.repo/repo/project.py", line 1845, in _RemoteFetch
self.bare_git.pack_refs('--all', '--prune')
File "/home/sebastian/WORKING_DIR/.repo/repo/project.py", line 2483, in runner
p.stderr))
error.GitError: platform/external/ipsec-tools pack-refs: fatal: Not a git repository: '/home/sebastian/WORKING_DIR/.repo/projects/external/ipsec-tools.git'
Since it says it is not a git repository, I just went on a hunch and ran git init on the directory it indicated, which didn't work. I also tried deleting the entire ipsec-tools.git directory and running repo sync again so that it would start from scratch but it always gives the same error.
Any ideas?

Its not necessary to start sync from scratch. Android is a huge repository to download from scratch and in interest of saving time and internet one can remove specific git repository (generally I have seen only 1 git directory being corrupt) and re-run the sync.
repo stores projects in
.repo/projects
but these are only soft links to actual git data, which is stored in
.repo/project-objects
follow the links in 'projects' and remove both the repositories and run sync again.
It worked for me.
Sample for platform/frameworks/support:
rm -fr .repo/projects/frameworks/support.git/ .repo/project-objects/platform/frameworks/support.git/
Sync only this project:
repo sync -j4 platform/frameworks/support
[I wanted to add this as comment, but its not allowing me - so added as answer]

I had a similar problem, where the repo command would exit with the same error.
I made a temporary folder, did a
repo init
then did
repo sync -j4 [PROJECT NAME]
where PROJECT NAME was the git which was getting corrupted. I then copied the .git directory from .repo/projects folder to the main directories' .repo/projects and again ran repo sync to finish it.

Shorter answer: Remove the offending repositories/data:
rm -Rf platform/external/ipsec-tools # from current path
rm -Rf .repo/projects/platform/external/ipsec-tools.git # soft links
rm -Rf .repo/project-objects/platform/external/ipsec-tools.git # actual git data
Better sync only the offending one:
repo sync platform/external/ipsec-tools
That worked for me for lineageos.

How to download google source code for android

As you know, there is a list of several hundred projects in https://android.googlesource.com/. I'd like to download them all in windows machine. According to Google's document,
To install, initialize, and configure Repo:
$ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo
To clone the entire platform, install repo, and run:
mkdir mydroid
cd mydroid
repo init -u https://android.googlesource.com/platform/manifest
repo sync
In my machine, however, I cannot "repo init" in Git Bash because it says it does not have python. I have python installed but git bash does not recognize it. Note that I set the python directory to the system path too. If anybody can give a tip, I would appreciate it. Thanks
UPDATE: I believe it's problem with new version of Git Bash for Windows. System path is not applied to Git Bash at all - I could easily test if system path worked with command prompt. Anyway, I tried this instead and it actually ran with error of course.
/c/python27/python.exe ../bin/repo init -u https://android.googlesource.com/platform/manifest
The error message is
$ /c/python27/python.exe ../bin/repo init -u https://android.googlesource.com/platform/manifest
Traceback (most recent call last):
File "../bin/repo", line 91, in <module>
import readline
ImportError: No module named readline
OK. I passed this error by installing pyreadline in windows:
easy_install pyreadline
If you got an error, you must install setuptools from
http://pypi.python.org/pypi/setuptools#files
And finally ran the command again to get this:
$ repo init -u https://android.googlesource.com/platform/manifest
fatal: unable to start d:\mywork\dev\GoogleAndroid\working_dir\.repo\repo/main.py
fatal: [Errno 8] Exec format error

With one click, download the latest code as .tar.gz file, from here
https://android.googlesource.com/platform/frameworks/base/+archive/master.tar.gz, the android could be found under core folder
Edit
Alternative here:
http://grepcode.com/project/repository.grepcode.com/java/ext/com.google.android/android/
Just select the version then a download options within.

If you consider, as an example, this other program "sympy" which also needs git bash and python, it is only a matter to add python to your PATH prior to launching the git bash session.
Install Python from:
http://python.org/download/
by downloading the "Python 2.7 Windows installer" (or Python 2.6 or 2.5) and running it.
Add python directory to your system environment path variable
(My Computer -> Advanced -> Environment Variables -> Path -> Edit).
Note that the repo script itself must be in the path, as mentioned in the Version Control page of android:
Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow.
Repo is not meant to replace Git, only to make it easier to work with Git in the context of Android.
The repo command is an executable Python script that you can put anywhere in your path.

This answer explains how to fix this error:
fatal: unable to start c:\path\.repo\repo/main.py
fatal: [Errno 8] Exec format error
Summary: I finally used the python packaged by Cygwin.
Details: Below is the full story.
The tip from the repo bug tracking is to add '/c/app/Python27/python ':
line 136 in v1.20
REPO_MAIN = '/c/app/Python27/python ' + S_repo + '/main.py'
line 735 in v1.20 (beginning of function main)
wrapper_path = '/c/app/Python27/python ' + os.path.abspath(__file__)
But we get the error TypeError: coercing to Unicode: need string or buffer, NoneType found
Therefore I reverted these changes above and performed the other changes below (on version 1.20):
line 136, replaced single slash by double back-slash:
REPO_MAIN = S_repo + '\\main.py'
line 766, added python absolute path as first element of me:
me = ['C:\\app\\Python27\\python.exe', repo_main,
'--repo-dir=%s' % rel_repo_dir,
'--wrapper-version=%s' % ver_str,
'--wrapper-path=%s' % wrapper_path,
'--']
line 776, replaced os.execv(repo_main, me) by
os.execv('C:\\app\\Python27\\python.exe', me)
However we get still an error:
$ Traceback (most recent call last):
File "c:\path\.repo\repo\main.py", line 39, in <module>
from subcmds.version import Version
File "c:\path\.repo\repo\subcmds\__init__.py", line 36, in <module>
['%s' % name])
File "c:\path\.repo\repo\subcmds\forall.py", line 17, in <module>
import fcntl
ImportError: No module named fcntl
The Python v2.7 fcntl documentation says fcntl is available for platform Unix only.
I finally reverted again all changes in repo script and installed Cygwin including its python and git packages: it succeeded as a charm.
But, as the symlinks simulated by Cygwin are not recognized by the MSysGit, we have to use the Cygwin git. And GUIs on top of git are not fully compliant with Cygwin git...
(see also my other post)
Edit:
Cygwin can use native NTFS symlinks (just set CYGWIN=winsymlinks:native and be Admin). Therefore MSysGit can be used and any other GUI based on it :-)