Develop Reference

How to list all possible packages / modules in AOSP via command line?

In AOSP, I often use the following to setup my environment and build target and build the entire image:
$ cd ~/aosp
$ source build/envsetup.sh
$ lunch sdk_phone_x86_64
$ m
I also like to use the following to build a specific package/module instead of the entire image:
$ m <my package>
or
$ cd <path_to_package>
$ mm
Is there a way to simply list all possible packages/modules in all of AOSP? Similar to how running lunch displays all possible targets, I'm interested in seeing a list of all possible packages/modules (e.g. defined in Android.bp and Android.mk).
Follow up question: given a package/module name, is there a way to get the directory or path to the file that defines it?

build/make/envsetup.sh has define the shell function:
- allmod: List all modules.
- pathmod: Get the directory containing a module.

Script that will transfer photos from phone camera using adb

Story
I take photos and record videos with my phone camera and keep all of them on my internal storage/sdcard. I periodically back them up on my PC, so I keep these camera photos on PC storage in sync with phone storage.
For years, I've been backing up my phone camera photos to my PC in the following way:
Plug in phone into PC and allow access to phone data
Browse phone storage → DCIM → Camera
Wait several minutes for the system to load a list of ALL photos
Copy only several latest photos which haven't been backed up yet
I figured that waiting several minutes for all photos to load is an unnecessary drag so I downloaded adb platform tools. I've added the folder bin to my Path environment variable (i.e. %USERPROFILE%\Tools\adb-platform-tools_r28.0.3) so that I can seamlessly use adb and not write its full path each time.
The script
I wrote the following script for Git Bash for Windows. It is also compatible with Unix if you change the $userprofile variable. Essentially, the script pulls camera photos between two dates from phone storage to PC.
# Attach device and start deamon process
adb devices
# Initialize needed variables
userprofile=$(echo "$USERPROFILE" | tr "\\" "/") # Windows adjustments
srcFolder="//storage/06CB-C9CE/DCIM/Camera" # Remote folder
dstFolder="$userprofile/Desktop/CameraPhotos" # Local folder
lsFile="$dstFolder/camera-ls.txt"
filenameRegex="2019061[5-9]_.*" # Date from 20190615 to 20190619
# Create dst folder if it doesn't exist
mkdir -p "$dstFolder"
# 1. List contents from src folder
# 2. Filter out file names matching regex
# 3. Write these file names line by line into a ls file
adb shell ls "$srcFolder" | grep -E "$filenameRegex" > "$lsFile"
# Pull files listed in ls file from src to dst folder
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder" # adb: error: ...
done < "$lsFile"
# Clean up
rm "$lsFile"
# Inform the user
echo "Done pulling files to $dstFolder"
The problem
When I run the script (bash adb-pull-camera-photos.sh), everything runs smoothly except for the adb pull command in the while-loop. It gives the following error:
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg
I am not sure why the output is broken. Sometimes when I resize the Git Bash window some of the text goes haywire. This is the actual error text:
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg': No such file or directory
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg': No such file or directory
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg': No such file or directory
I am sure that these files exist in the specified directory on the phone. When I manually execute the failing command in bash, it succeeds with the following output:
$ adb pull "//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg" "C:/Users/User/Desktop/CameraPhotos/"
//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg: 1 file pulled. 15.4 MB/s (1854453 bytes in 0.115s)
The question
I can't figure out what's wrong with the script. I thought the Windows system might be causing a commotion, because I don't see the reason why the same code works when entered manually, but doesn't work when run in a script. How do I fix this error?
Additional info
Note that I had to use // in the beginning of an absolute path on Windows because Git Bash would interpret / as its own root directory (C:\Program Files\Git).
I've echoed all variables inside the script and got all the correct paths that otherwise work via manual method.
camera-ls.txt file contents
20190618_124656.jpg
20190618_204522.jpg
20190619_225739.jpg
Additional questions
Is it possible to navigate to external sdcard without using its name? I had to use /storage/06CB-C9CE/ because /sdcard/ navigates to internal storage.
Why does tr "\\" "/" give me this error: tr: warning: an unescaped backslash at end of string is not portable?

Windows batch script
Here's a .bat script that can be run by Windows Command Prompt or Windows PowerShell. No Git Bash required.
:: Start deamon of the device attached
adb devices
:: Pull camera files starting from date
set srcFolder=/storage/06CB-C9CE/DCIM/Camera
set dstFolder=%USERPROFILE%\Desktop\CameraPhotos
set lsFile=%USERPROFILE%\Desktop\CameraPhotos\camera-ls.txt
set dateRegex=2019061[5-9]_.*
mkdir %dstFolder%
adb shell ls %srcFolder% | adb shell grep %dateRegex% > %lsFile%
for /F "tokens=*" %%A in (%lsFile%) do adb pull %srcFolder%/%%A %dstFolder%
del %lsFile%
echo Done pulling files to %dstFolder%
Just edit the srcFolder to point to your phone camera folder,
plug a pattern into the dateRegex for matching the date interval and
save it as a file with .bat extension, i.e: adb-pull-camera-photos.bat.
Double-click the file and it will pull filtered photos into CameraPhotos folder on Desktop.
Keep in mind that you still need have adb for Windows on your PC.

The problem was with Windows line delimiters.
Easy fix
Just add the IFS=$'\r\n' above the loop so that the read command knows the actual line delimiter.
IFS=$'\r\n'
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder"
done < "$lsFile"
Explanation
I tried plugging the whole while-loop into the console and it failed with the same error:
$ bash adb-pull-camera-photos.sh
List of devices attached
9889db343047534336 device
tr: warning: an unescaped backslash at end of string is not portable
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg
Done pulling files to C:/Users/User/Desktop/CameraPhotos
This time I started investigating why the output was broken. I remembered that windows uses \r\n as newline, which means Carriage Return + Line Feed, (CR+LF), so some text must have been overwritten.
It was because of broken values stored inside the $filename variable.
This is the loop from the script:
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder"
done < "$lsFile"
Since each iteration of the while-loop reads a line from $lsFile in the following form:
exampleFilename.jpg\r\n
It misinterprets the newline symbols as part of the file name, so adb pull tries to read files with these whitespaces in their names, but fails and it additionally writes a broken output.

Adb Photo Sync
This might not be the answer but might be useful for others looking for android photo/files backup solution.
I use this script on my Windows with git bash. This can be easily used for Linux. A common issue with a long backup process is that it might get interrupted and you might have to restart the entire copy process from start.
This script saves you from this trouble. You can restart the script or interrupt in between but it will resume copy operation from the point it left.
Just change the rfolder => android folder, lfolder => local folder
#!/bin/sh
rfolder=sdcard/DCIM/Camera
lfolder=/f/mylocal/s8-backup/Camera
adb shell ls "$rfolder" > android.files
ls -1 "$lfolder" > local.files
rm -f update.files
touch update.files
while IFS= read -r q; do
# Remove non-printable characters (are not visible on console)
l=$(echo ${q} | sed 's/[^[:print:]]//')
# Populate files to update
if ! grep -q "$l" local.files; then
echo "$l" >> update.files
fi
done < android.files
script_dir=$(pwd)
cd $lfolder
while IFS= read -r q; do
# Remove non-printable characters (are not visible on console)
l=$(echo ${q} | sed 's/[^[:print:]]//')
echo "Get file: $l"
adb pull "$rfolder/$l"
done < "${script_dir}"/update.files

After upgrading Gradle to v2.8 Travis doesn't pass Findbugs

I just updated my gradle from version 2.2 to latest one 2.8. I didn't have any issue with findbugs on version 2.2.
I'm working on an Android project that contains two modules. In order to use find bugs in both modules I have following configuration on main build.gradle file of root directory.
configure(allprojects) {
apply plugin: 'findbugs'
task findbugs(type: FindBugs) {
ignoreFailures = false
effort = "max"
classes = fileTree('build/intermediates/classes/')
source = fileTree('src/main/java')
classpath = files()
excludeFilter = file("exclude.xml")
reportLevel = "high"
reports {
xml.enabled = false
html.enabled = true
}
}
}
When I run ./gradlew findbugs on my local machine everything is fine and build is successful however when I push my PR to Github and Trivis tries to build then I get error:
:findbugs UP-TO-DATE
:passenger-app:findbugs
:passenger-sdk:findbugs FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':passenger-sdk:findbugs'.
> FindBugs rule violations were found. See the report at: file:///home/travis/build/project-name/passenger-android/passenger-sdk/build/reports/findbugs/findbugs.html
I'm really confused why I have no problem on my local machine while Travis shows error! I tried to print out the contents findbugs.html on Travis but I got permission denied :(
I'm using java 1.8 while Travis is using 1.7. Does problem relates to this? Thanks
Update:
In order to print the contents of findbugs.html on Trivis I created a print_findbugs.sh and this is its contents.
#!/usr/bin/env bash
echo '**********************'
echo '*** Print Findbugs ***'
echo '**********************'
echo file:///home/travis/build/company/passenger-android/passenger-sdk/build/reports/findbugs/findbugs.html
Then I set sudo true in my .travis.yml file. What I have in this file.
sudo: true
language: android
android:
components:
- build-tools-23.0.1
- android-23
- extra-android-support
- extra-google-google_play_services
- extra-google-m2repository
- extra-android-m2repository
env:
global:
// some settings are there
before_cache:
# Delete the gradle lock file which forces creation of a new build cache
- rm ~/.gradle/caches/modules-2/modules-2.lock
cache:
directories:
- ~/.gradle
before_script:
# Overwrite the keystore if it is a pull request
- ./before_script.sh
script:
# Override Travis default script to not run connectedCheck until this bug is fixed:
# https://code.google.com/p/android/issues/detail?id=59592
- ./gradlew clean build findbugs -PdisablePreDex
- ./print_findbugs.sh
before_deploy:
# Clean up the output folder
# Link up the new builds into individual html files
- ./before_deploy.sh
after_deploy:
# Upload to...
and finally my travis prints:
:findbugs UP-TO-DATE
:passenger-app:findbugs
:passenger-sdk:findbugs
FindBugs rule violations were found. See the report at: file:///home/travis/build/company/passenger-android/passenger-sdk/build/reports/findbugs/findbugs.html
BUILD SUCCESSFUL
Total time: 8 mins 9.966 secs
The command "./gradlew clean build findbugs -PdisablePreDex" exited with 0.
$ ./print_findbugs.sh
/home/travis/build.sh: line 41: ./print_findbugs.sh: Permission denied
The command "./print_findbugs.sh" exited with 126.
before_cache
$ rm ~/.gradle/caches/modules-2/modules-2.lock
cache.2
Done. Your build exited with 1.

I don't use it and I need more information about the permission denied.
Html reports
In the past, I printed html reports using Travis-ci, point 5 here. I downloaded lynx using apt-get (it's not possible now using container-infrastructure and sudo: false) and converted and printed the reports.
before_script:
# - echo 'LOGCAT'
# Check logcat debug output: http://developer.android.com/tools/help/logcat.html
# Check debugging log: http://developer.android.com/tools/debugging/debugging-log.html
# Comment the lines belows to debug output and redirect it to a file. Custom tags for your app.
- adb -e logcat *:W | tee logcat.log > /dev/null 2>&1 &
after_failure:
# - echo 'FAILURE'
# Check apt configuration: http://docs.travis-ci.com/user/ci-environment/#apt-configuration
# Comment out the lines below to show log about tests with app name customized on exports section.
- sudo apt-get install -qq lynx
- export MOD_NAME=yourappmodulename
- export LOG_DIR=${TRAVIS_BUILD_DIR}/${MOD_NAME}/build/outputs/reports/androidTests/connected/
- lynx --dump ${LOG_DIR}com.android.builder.testing.ConnectedDevice.html > myConnectedDevice.log
- lynx --dump ${LOG_DIR}com.android.builder.testing.html > myTesting.log
- for file in *.log; do echo "$file"; echo "====================="; cat "$file"; done || true
Xml reports
Can be used in both, legacy and container based infrastructure.
I read you can enable xml reports like this:
reports {
xml.enabled = true
html.enabled = true
}
You can easily print xml reports on Travis-ci using cat like here:
- cat ${TRAVIS_BUILD_DIR}/ui/espresso/*/app/build/outputs/androidTest-results/connected/* # logs
Adding * will include all the subfolders at this point.
You need to first find locally the folder for the xml reports, in my case is not the same folder than html, and add to your travis.yml file something like this:
after_failure:
- cat /home/travis/build/*/passenger-android/passenger-sdk/build/reports/findbugs/*
This doesn't solve your issue but perhaps helps to find reason.
Update:
I suggest you to try to use cat and the xml version without the script first.
A good explanation about the permission issue here and how to solve it making the file executable:
before_script:
- chmod +x yourscript
Update 2:
A better approach to fix the permission denied issue explained here.
Use this and commit the changes:
git update-index --chmod=+x yourscript

In my case, ignoreFailures was true, but nevertheless the findbugs task was failing after upgrading gradle because build.gradle was depending on findbugs:annotations:3.0.0 instead of findbugs:findbugs-annotations:3.0.1. (Note: annotations vs. findbugs-annotations, which is a new artifact.

Finally I was able to get everything passed.
I first used following command rather than simple $ ./gradlew findbugs.
./gradlew clean aGD findbugs -PdisablePreDex
aGD is abbreviation of my task something like assembleDebug.
I could see a lot of complains by gradle. Probably Travis was showing this (afterDebug) but since it prints html page in console is not human readable. So I added filters into Findbug's exclude file (ref) in order to pass Findbugs check.
I was successful! Once I pushed my code, Travis failed again.
But since I printed out Findbug's output on Travis and there was only an issue I could find it in my code and Fix it. After that Travis didn't failed and passed my PR.
So the problem is still exist unfortunately. I can get Findbugs passed on my local project while Travis sometimes finds more issues. I'm suspicious that Travis is able to load recent versions of Findbugs but mine isn't able, due to cache things. But yes, this is my assumption!

Jenkins - Warnings Plugin & IntelliJ IDEA Parser

I am having issues opening the files that contains IntelliJ IDEA Inspection warnings using Jenkins and Warnings Plugin.
The path to my files looks weird - file://$PROJECT_DIR$/app/src/main..../Foo.java in the .xml generated files by ./inspect.sh
When I click on the file I get the following error:
`java.io.IOException: Failed to copy file:/$PROJECT_DIR$/app/src/main/java`/
These are some screenshots of my files:
I am using the ./inspect.sh from Android Studio, not from actual IntelliJ. Could that be the problem?
Thanks in advance.

In the end I have added an extra build step where I clean the .xml files that are output by ./inspect.sh.
I have also added a line of code that removes all the inspects related to the build (generated) directory. Hope this helps anyone who was stuck with this issue.
Below you have the tweaked inspect.sh file and the two separate build steps:
inspect.sh
#!/bin/sh
#
# ------------------------------------------------------
# Android Studio offline inspection script.
# ------------------------------------------------------
#
export DEFAULT_PROJECT_PATH="$(pwd)"
cd ..
exec "MacOS/studio" inspect "$#"
Build Step 1
cd ${WORKSPACE}/inspectionsresults
rm *.xml
cd "${ANDROID_STUDIO_HOME_BIN}"
./inspect.sh ${WORKSPACE} ${WORKSPACE}/inspections_profile.xml ${WORKSPACE}/inspectionsresults -v2 -d ${WORKSPACE}/app
Build Step 2
cd ${WORKSPACE}/inspectionsresults
sed -i .bak "s,file://\\\$PROJECT_DIR\\\$,${WORKSPACE},g" *.xml
sed -i .bak "s,file:///,,g" *.xml
/usr/local/bin/xml ed -L -d "/problems/problem[contains(file,'generated')]" *.xml
rm *.bak
I have used xmlstarlet to remove the "generated" problems.
brew install xmlstarlet

Building webrtc library

I am trying to build webrtc android demo application. I was following README applied by the project. First thing was mentioned about gclient but when i tried to run it "no external or internal command". Then i got this link according to the this link i cloned the depot_tool repository but when at the step when i try to run gclient i got this
So after hours of trying i am not able to run it. if any one has experience with this WebRTC library please provide some help. Any guidance? Any Hlep will be appreciated

I heard few things were changed from svn to GIT. We tried building webRTC to windows7 using visual studio. See if you can use any parts on this.
Get depot_tools software and extract to a location (set this path in environment variable).
cd to depot_tools directory and hit 'gclient' (this should take sometime)
Install python 2.7 and put it path.
create a folder of your choice for webRTC and open command prompt there.
Now hit, gclient config http://webrtc.googlecode.com/svn/trunk
then, gclient sync.
Then set GYP_GENERATORS=msvs environment variable (use ninja if you want that).
Also set msvs_version=2012 (depending on your visual studio version)
now run, gclient runhooks --force
This should generate solutions files for your folder. fireup the libjingle/webrtc.sln files and start building it. I'm not sure if it builds as is. But we had hicupps with that.
See this link on WebRTC native development if you haven't already. From my personal experience building this tricky job initially.
Also note that links might be deprecated as they moved to git, don't know if they are still maintaining it.

How to compile WEBRTC On ubuntu
download script and run:
https://cs.chromium.org/chromium/src/build/install-build-deps.sh
./build/install-build-deps.sh --no-chromeos-fonts
Now download depot tools
$ git clone --depth 1 https://chromium.googlesource.com/chromium/tools/depot_tools.git
Put it in Path
$ export PATH=`pwd`/depot_tools:"$PATH"
Define your settings:
export GYP_DEFINES="target_arch=x64 host_arch=x64 build_with_chromium=0 use_openssl=0 use_gtk=0 use_x11=0 include_examples=0 include_tests=1 fastbuild=1 remove_webcore_debug_symbols=1 include_pulse_audio=0 include_internal_video_render=0 clang=1 "
make file to download the source
mkdir webrtc-checkout
cd webrtc-checkout
I chose branch 50 you can change it
gclient config https://chromium.googlesource.com/external/webrtc.git#branch-heads/50 --name=src
gclient sync --force --with_branch_heads
gclient sync --force --with_branch_heads --nohooks
cd src
git checkout branch-heads/50
gclient runhooks
ninja -C ./out/Release
Collect your LIB & Include file to use it in your project
mkdir ../lib/
find ./ -name "*.o" -and -not -name do_not_use -and -not -name protoc -and -not -name genperf -exec ar crs ../lib/libwebrtc.a {} +
mkdir ../include
find ./ -name *.h -exec cp --parents '{}' ../include ';'