All the git projects for the AOSP are cloned by the repo tool, which reads this xml: https://android.googlesource.com/platform/manifest/+/refs/heads/master/default.xml.
AOSP guide says the in order to build, we should run source build/envsetup.sh on the folder where repo cloned all repositories. So let's look at the platform/build on the default.xml from the manifest repository. We get
<project path="build/make" name="platform/build" groups="pdk" >
<copyfile src="core/root.mk" dest="Makefile" />
<linkfile src="CleanSpec.mk" dest="build/CleanSpec.mk" />
<linkfile src="buildspec.mk.default" dest="build/buildspec.mk.default" />
<linkfile src="core" dest="build/core" />
<linkfile src="envsetup.sh" dest="build/envsetup.sh" />
<linkfile src="target" dest="build/target" />
<linkfile src="tools" dest="build/tools" />
</project>
We confirm where envsetup.sh is located., it is in platform/build. It defines the function m which according to the AOSP guide, builds the entire AOSP project:
function _trigger_build()
(
local -r bc="$1"; shift
if T="$(gettop)"; then
_wrap_build "$T/build/soong/soong_ui.bash" --build-mode --${bc} --dir="$(pwd)" "$#"
else
echo "Couldn't locate the top of the tree. Try setting TOP."
fi
)
function m()
(
_trigger_build "all-modules" "$#"
)
Ok, so looks like build/soong/soong_ui.bash is the place called when we run the m function, so this script should build everything.
Here's soong_ui.bash. It sources source ${TOP}/build/soong/scripts/microfactory.bash and then calls soong_build_go soong_ui android/soong/cmd/soong_ui
Here's microfactory.bash, where we find function soong_build_go
soong_build_go
{
BUILDDIR=$(getoutdir) \
SRCDIR=${TOP} \
BLUEPRINTDIR=${TOP}/build/blueprint \
EXTRA_ARGS="-pkg-path android/soong=${TOP}/build/soong -pkg-path github.com/golang/protobuf=${TOP}/external/golang-protobuf" \
build_go $#
}
We find build_go in microfactory.bash from build/blueprint:
Looks like all of this is for building the microfactory.go project. I think it has something to do with the soong build system.
I'm now lost. After building microfactory.go, what happens? Where does actual Android code gets built?
microfactory.sh says build_go does this: Bootstrap microfactory from source if necessary and use it to build the requested binary. The requested binary is android/soong/cmd/soong_ui
I'm trying to find android/soong/cmd/soong_ui but I don't know what/where it is, but I'd guess is the soong build system, not the AOSP project yet.
UPDATE:
on soong_ui.bash, I noticed it end with
cd ${TOP}
exec "$(getoutdir)/soong_ui" "$#"
Remember that this is called form envsetup.sh. Well, ${TOP}, I guess, is the place where repo clones everything. Looks like it's trying to execute soong_ui with the arguments from envsetup.sh which are --build-mode --${bc} --dir="$(pwd)" "$#", where this $# is "all-modules" "$#", I guess.
I assume song_ui is the soong executable. It should look for Android.bp on the ${TOP}, but I don't think there is one on the place where repo cloned everything.
You have already found out a lot, and you are right with the link from m to soong_ui.bash and then starting microfactory.
From my reading of the code, the purpose of soong_build_go is to build the package android/soong/cmd/soong_ui, with the binary name soong_ui. Like Yong said in the other answer, this creates the binary soong_ui under the directory $(getoutdir), and the source for that binary is located at build/soong/cmd/soong_ui/main.go.
As for your updated question about an Android.bp file, it is symlinked from build/soong/root.bp when repo sync is run, but as you can see, the file is empty.
Instead, in m it tells Soong to build all_modules, which eventually runs another tool called kati. From the description in https://github.com/google/kati, Kati processes GNU makefiles and turns them into Ninja build files.
At this point we can (mostly) assume regular Make semantics, even though the underlying build system is actually Kati and Ninja and Soong etc. Since the working directory is $TOP, the Makefile at the root directory, which is symlinked from build/make/core/root.mk is used. Which includes main.mk which then includes build/make/core/Makefile. Inside that makefile you can see how the different .img files are built (e.g. system.img).
Let's take make systemimage as an example:
The call sequence is:
prebuilts/build-tools/linux-x86/bin/makeparallel --ninja build/soong/soong_ui.bash --make-mode "systemimage".
And $(getoutdir)/soong_ui is build by "build_go soong_ui android/soong/cmd/soong_ui"
build/soong/cmd/soong_ui/main.go#main()
soong/ui/build/build.go#Build()
Related
In the Android project that I am working on, we are using GitLab CI/CD to automatically build and upload in Diawi the .apk file. In current settings build application's name is static because it needs to be known and sent to the Diawi framework as a curl request. The script looks like this:
- ./gradlew assembleRelease && cp app/build/outputs/apk/release/app-release.apk app-release.apk && curl -v --http1.1 https://upload.diawi.com/ -F token=$DIAWI_TOKEN -F file=#app-release.apk -F find_by_udid=0 -F callback_emails="mymail#company.com"
But this causes some troubles during the manual testing because the .apk files with the same name can be easily mistaken or overwritten.
My idea is to add some metadata in the .apk file's name to be unique and to avoid such errors. Do you have any ideas about how this could be done?
First thing you can do is create version bump functionality. You can do it via version bumping tool and git sync.
For version bumping you can use our open source tool - https://github.com/relizaio/versioning . Now, assuming that you sync via a file called apk_version, you first initialize this file with the current version like:
docker run --rm relizaio/versioning -s YY.0M.Patch > apk_version
Then in your CI context, you can bump it with something like
docker run --rm relizaio/versioning -s YY.0M.Patch -v $(cat apk_version) -a bump > apk_version
Now, in GitLab CI you need to have a block that resolves this. Note, that you also need to commit apk_version file at the end of the bump.
For inspiration - see this question How to grant permission for semantic-release to push code to master and also our sample CD project on GitLab - https://gitlab.com/taleodor/sample-helm-cd/-/blob/master/.gitlab-ci.yml
Also my article here may be helpful (note it's a bit outdated at this point): https://worklifenotes.com/2020/02/27/automatic-version-increments-with-reliza-hub-2-strategies/
Finally, for other things that you can add to version GitLab has a list of pre-defined env variables that you can use in whole or partially as modifiers: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
I'd like to know if some of you worked out how to generate signed APKs and IPA with multiple configurations (ie : beta for hockeyapp, and production for stores) in a single command.
I'm exploring all the possibilities there, it looks like there are lots of ways to do this.
I'd also like to be able to pass a variable like ENVIRONMENT to set Javascript constants such as an API url or turn on/off debugs.
Here's what I'm thinking right now :
Using https://www.npmjs.com/package/react-native-config to solve the former problem.
On android :
I'm thinking about adding a type in buildTypes in gradle. So far, I couldn't get it to work, I'm not very experienced with native configurations.
I would then make a bash script to create the offline bundle with the chosen env (staging / dev / prod) and use gradle's assembleRelease / assembleBeta. Do you think that's doable ?
On iOS, it looks a bit more complicated :
It looks like it's hard to change the project's configuration in CLI when building. So I was thinking that I should duplicate the project for each environement : project-dev.xcodeproj, project-prod.xcodeproj... you got the point.
Once again, I would make the bundle, then cp it inside the given project. A nice touch would be to trigger the xcode compilation in CLI too, I don't know if this is hard to set up.
What do you think about this ? Maybe some of you are already using custom scripts to do that ?
The icing on the cake would be to use HockeyApp's 'puck' cli tool to upload it, but that should be quite easy to set up once the application has been build for both iOS and Android.
This is the process I use for command line build with parameters. My system builds a release version .ips file and then copies and resigns that file with a development provisioning profile that I can put on my development devices to test exactly what is being sent to the customer or to the app store. Not all of this will be useful to you, but hopefully some will.
First, I have a variables file that sets the global parameters that I'm going to use for the build:
Scheme="(The scheme I am going to build inside my project)"
WindowsSavePath="(path to my source archive directory on a shared computer)"
InstallSavePath="(path to my .ipa archive directory on a shared computer"
Customer="(relative path inside archive directories)"
Fleet="(Continued relative path)"
PathToProject="(path to the folder on my Mac with the xcode project file)"
ProjectName="(project name).xcodeproj"
PlistPath="$Scheme-Info.plist"
appScheme="$Scheme"
AppName="(The name that I want to give my .ipa file)"
Version="(The version number for this build)"
AppendedFileName="-QA" //I use this for QA and Production distinction
exportPath="$InstallSavePath/$Fleet/$Customer/$Version$AppendedFileName/"
CompanyAppIdentifierPrefix="(my generic provisioning profile identifier)"
ArchiveLocation="$WindowsSavePath/$Fleet/$Customer/$Version$AppendedFileName"
SourceCodeDestination="(my compiled absolute path to the archive directories)"
This is all contained in a SetVariables.sh file. Now we get to start building the project. In another .sh file first I call the SetVariables file, then I start compilation:
#Update the version number in the plist file for the project
/usr/libexec/PListBuddy -c "Set :CFBundleShortVersionString $Version" "$PathToProject/$PlistPath"
/usr/libexec/PListBuddy -c "Set :CFBundleVersion $BuildVersion" "$PathToProject/$PlistPath"
#Now build and archive the project, the create the .ipa file for either submittal or giving to customers
mkdir -p "$exportPath"
mkdir -p "${ArchiveLocation}/dSYM"
xcodebuild -project "$PathToProject/$ProjectName" -scheme "$Scheme" DSTROOT="$exportPath" DEBUG_INFORMATION_FORMAT="dwarf-with-dsym" DWARF_DSYM_FOLDER_PATH="${ArchiveLocation}/dSYM" archive -archivePath "$exportPath${appScheme} $Version$AppendedFileName.xcarchive"
/usr/bin/xcrun -sdk iphoneos PackageApplication -v "${exportPath}Applications/${AppName}.app" -o "${exportPath}${appScheme} $Version$AppendedFileName.ipa"
#Now resign and create an internal dev version to test on development ipads
rm -r Payload SwiftSupport
unzip -q "${exportPath}${appScheme} $Version$AppendedFileName.ipa"
BUNDLE_ID=$(/usr/libexec/PlistBuddy -c "Print :CFBundleIdentifier" "Payload/${AppName}.app/Info.plist")
. ./CreateInternalEntitlements.sh "$CompanyAppIdentifierPrefix.$BUNDLE_ID"
. ./resign.sh "${exportPath}${appScheme} $Version$AppendedFileName.ipa" "${exportPath}${appScheme} ${Version}${AppendedFileName} Internal" internalDev.mobileprovision "(My developer account tied to the internal provisioning profile" Entitlements.plist
rm -r Payload SwiftSupport
#Now delete the intermediate files from the installs directory
rm -rf "$exportPath${appScheme} $Version$AppendedFileName.xcarchive"
rm -rf "${exportPath}Applications"
I have left out a few things in there (resign.sh, createentitlements.sh), but those processes I found on stackoverflow, so it shouldn't be too hard for you to find as well.
I've never done it, but I'm relatively certain that you can change the command line arguments for the xcodebuild to build either release or debug like you are trying to do. You could also run the xcodebuild twice, once for debug and once for release, and save the builds to different locations.
I hope this helps you at least a little with your goals. This took my a week or two to put all together and get working for my needs. Good luck.
I am facing the following issue when trying to compile a c program using openssl for android x-86. I set up the environment variables as follows using the following script:
setenv-android.sh
After running the script I have the following environment.
./setenv-android_x86.sh
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
Error: FIPS_SIG does not specify incore module. Please edit this script.
ANDROID_NDK_ROOT: /opt/android-ndk-r9d-x86
ANDROID_ARCH: arch-x86
ANDROID_EABI: x86-4.6
ANDROID_API: android-18
ANDROID_SYSROOT: /opt/android-ndk-r9d-x86/platforms/android-18/arch-x86
ANDROID_TOOLCHAIN: /opt/android-ndk-r9d-x86/toolchains/x86-4.6/prebuilt/darwin-x86_64/bin
FIPS_SIG:
CROSS_COMPILE: i686-linux-android-
ANDROID_DEV: /opt/android-ndk-r9d-x86/platforms/android-18/arch-x86/usr
However when trying to compile with the following command I get the following error:
pwd
/opt/android-ndk-r9d-x86/bin
./i686-linux-android-gcc Test.c -o test -lcrypto
fatal error: openssl/conf.h: No such file or directory
When I locate for the openssl/conf.h I see the file is available in many places:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/openssl/conf.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/openssl/conf.h
/data/local/arm/ssl/include/openssl/conf.h
/data/local/ssl/include/openssl/conf.h
/openssl/include/openssl/conf.h
/openssl-1.0.1e/include/openssl/conf.h
/openssl-1.0.1g/include/openssl/conf.h
/opt/local/include/openssl/conf.h
/usr/include/openssl/conf.h
/usr/local/include/openssl/conf.h
/usr/local/ssl/android-14/include/openssl/conf.h
/usr/local/ssl/android-18/include/openssl/conf.h
/usr/local/ssl/android-18-x86/include/openssl/conf.h
/usr/local/ssl/include/openssl/conf.h
I think the proper path is this:
/usr/local/ssl/android-18-x86/include/openssl/conf.h
I tried with the -L option but with no luck.
-L/usr/local/ssl/android-18-x86/include/
Can anybody please explain how to setup the path properly to solve this issue. (finally there is no issue with the C code as it compiles properly with gcc)
You are compiling your code using the Android NDK, which creates a "virtual" compiling environment by placing everything you need to compile for android into the ${ANDROID_SYSROOT} directory.
Those directories you listed are outside that sysroot, and therefore are not available to the build system, hence the errors. You must install OpenSSL for Android and putting the resulting headers and binaries there. You might follow this guide to assist you in that.
For all you C4DROID dev's, I have a question about the Makefile option. I have been toying around with it for some time now, and just can't seem to get it to function properly. I've tried implementing the tutorial from http://mrbook.org/tutorials/make/ as well as http://www.gnu.org/software/make/manual/make.html#Implicit-Rules . These sites have helped me at least get different error messages (which in some ways is helpful) but I'm just not sure what else I need to be doing. I'm not sure how much detail I should include about what I'm doing (first time posting), so here goes.
At the moment, I have the files (for test purposes only) main.cpp, hello.cpp, factorial.cpp, and functions.h . I created a Makefile with:
all:
g++ main.cpp hello.cpp factorial.cpp functions.h -o MyFile
I have also tried variations like:
all: testTwo
testTwo:
g++ main.cpp hello.cpp factorial.cpp functions.h -o MyFile
In the compilation settings, I have selected Makefile, I have not modified the "Commands Before Make" code (I don't even understand it), my Make command is
make -f Makefile
I have selected the Native Activity in Run Mode, and that's all that I can think of explaining. My error message I get on compilation is "Failed to copy file".
Any help with the process would be greatly appreciated and thank you in advance.
In the compile options screen, section 'Result binary filename': enter the name of the executable that your Makefile generates i.e. MyFile for your example.
I also tend to use 'Run mode: Terminal app' for simple console programs. NativeActivity is intended for Android NDK app development and hence is a bit more involved.
I tried building i686-linux-android-gfortran using build-gcc.sh following this
(it's for androdindk-7b) but I get error about link.h. I added link.h from here, but it gives further more errors.
Has anyone tried enabling i686-linux-android-gfortran for x86 Android?
From https://groups.google.com/forum/#!msg/android-ndk/QR1qiN0jIpE/g0MHkhTd4YMJ as selalerer suggested. I didn't try this, so I'm posting as a community wiki for reference purposes.
Fortran for x86 Android
=================
The guide is based on this one, many thanks to Phil:
Compiling Android NDK with Objective-C-enabled gcc errors
1) Download and unpack Android NDK 'android-ndk-r8c', (the older -r8b NDK won't work due to missing link.h!):
wget http://dl.google.com/android/ndk/android-ndk-r8c-linux-x86.tar.bz2
2) Create somewhere a folder called 'toolchain-src' (e.g. inside the folder android-ndk-r8c),
'cd' to this new folder
3) Make sure to have git installed ('yum install git' or whatever..) and download
the toolchain sources:
git clone https://android.googlesource.com/toolchain/build.git
git clone https://android.googlesource.com/toolchain/gmp.git
git clone https://android.googlesource.com/toolchain/gdb.git
git clone https://android.googlesource.com/toolchain/mpc.git
git clone https://android.googlesource.com/toolchain/mpfr.git
git clone https://android.googlesource.com/toolchain/expat.git
4) Create the folder 'binutils', 'cd' to this directory, unpack
binutils-2.23 there:
wget ftp.gnu.org/gnu/binutils/binutils-2.23.tar.gz
tar -xvzf binutils-2.23.tar.gz
You should now have a folder toolchain-src/binutils/binutils-2.23
5) Change to folder toolchain-src/build, edit the Makefile.in, changing the line:
--with-gnu-as --with-gnu-ld --enable-languages=c,c++
to
--with-gnu-as --with-gnu-ld --enable-languages=c,c++,fortran
6) In the file android-ndk-r8c/build/tools/build-mingw64-toolchain.sh change the line:
var_append GCC_CONFIGURE_OPTIONS "--enable-languages=c,c++"
to
var_append GCC_CONFIGURE_OPTIONS "--enable-languages=c,c++,fortran"
7) In the file android-ndk-r8c/build/tools/build-gcc.sh, change the line:
EXTRA_CONFIG_FLAGS=$EXTRA_CONFIG_FLAGS" --disable-libquadmath --disable-plugin"
to
EXTRA_CONFIG_FLAGS=$EXTRA_CONFIG_FLAGS" --disable-libquadmath --disable-libquadmath-support --disable-plugin"
8) In the file android-ndk-r8c/build/tools/build-host-gcc.sh, change the line:
ARGS=$ARGS" --enable-languages=c,c++"
to
ARGS=$ARGS" --enable-languages=c,c++,fortran"
And change the line
ARGS=$ARGS" --disable-libquadmath --disable-plugin --disable-libitm --disable-bootstrap"
to
ARGS=$ARGS" --disable-libquadmath --disable-libquadmath-support --disable-plugin --disable-libitm --disable-bootstrap"
9) Build your new toolchain:
/your/path/to/android-ndk-r8c/build/tools/build-gcc.sh -j1 --gmp-version=5.0.5 --mpfr-version=2.4.2 --mpc-version=0.8.1 --binutils-version=2.23 --gdb-version=7.3.x /your/path/to/toolchain-src /your/path/to/android-ndk-r8c x86-4.7
(don't worry about messages like 'expr: warning: unportable BRE:')
10) And go down to your knees in front of the screen, praying to the Lord that somehow these
countless configure scripts doing checks that nobody needs, using an ugly shell language
that cooks your brain with indentation going from right to left, will somehow manage to
compile a zillion of far too small files (so that 10% of the time is spent on compilation
and 90% on starting up GCC), and after an hour of watching progress with
tail -F /tmp/ndk-YourUserName/build/toolchain/config.log
your toolchain will be magically ready. You'll find it in the android-ndk-r8c/toolchains folder.
11) Finally, 'cd' to the folder
'/your/path/to/android-ndk-r8c/toolchains/x86-4.7/prebuilt/linux-x86/i686-linux-android'
and run this command:
ln -s ../libexec libexec
Without this command, it may happen that g++ raises the error message
"g++: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found".
Using strace, I found that g++ looks in the wrong folder, but the link
above lets it find the file liblto_plugin.so nevertheless.
And here are a few lessons learned on the way, so that Google finds this page:
*) To speed up the compilation, you can remove the '-j1'. But only after you got
it to work once, since building in parallel on multiple CPU cores was reported to
cause additional troubles.
*) The error message "Link tests are not allowed after GCC_NO_EXECUTABLES" shows up
when linking fails for x86 (works for ARM). The reason is that GCC does not include
the proper ANDROID_STARTFILE_SPEC and ANDROID_ENDFILE_SPEC from
gcc-4.6.1/gcc/config/linux-android.h. GCC 4.6.1 only specifies them for ARM, but not
for i386, GCC 4.8.0 however does. The GCCs downloaded from Google also do,
so best use Google's GCC.
*) The error message "fatal error: link.h: No such file or directory" also happens
with Google's GCC, and apparently (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50877)
only when you enable additional languages like objc or fortran.
The bug thread is here: http://gcc.gnu.org/ml/gcc-bugs/2012-08/msg00494.html
MIPS has link.h in android-ndk-r8b/platforms/android-9/arch-mips/usr/include
In android-ndk-r8c, link.h is now also present in android-9/arch-x86/usr/include/link.h,
so this bug was fixed.
*) The error message "fatal error: quadmath_weak.h: No such file or directory":
It also happens with the latest gcc-4.8, so we can just continue using Googles GCC 4.7.
Google itself uses --disable-libquadmath, but we additionally need --disable-libquadmathsupport
(see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47648). So this needs to be added in
android-ndk-r8c/build/tools/build-gcc.sh
and
android-ndk-r8c/build/tools/build-host-gcc.sh
*) The error message "error: Pthreads are required to build libatomic"
Happens when building the ARM version of gcc-4.8 downloaded from gnu.org,
better stay with Google's GCCs.
*) The GCC that came with android-ndk-r8c didn't work for me (error message about
libstdc++.so.6 being too old), while the one in android-ndk-r8b worked
without problems. Since the android-ndk should support as many environments
as possible, I'm not sure why the Googlers decided to depend on a newer libstdc++,
but the good news are that building your own toolchain solves the issue.
*) If you get an error while compiling generic-morestack.c, then replace
#ifdef linux
// On Linux, the first two real time signals are used by the NPTL
with
#if defined(GLIBC) && defined(linux)
// On Linux, the first two real time signals are used by the NPTL