Develop Reference

Assemble error while using latest gRPC plugin on Android -> Input is shadowed in the --proto_path (Gradle 7.0.1)

I am trying to update an Android project to use the latest gradle plugin (7.0.1), from the current 3.6.4 that it is using. In order to do this, considering the project is using protobuf, I need to update the protobuf and gRPC dependencies, as the current ones are not compatible with the latest plugin.
I have followed https://github.com/grpc/grpc-java in order to use the latest dependency versions. I updated the dependencies to the following versions:
implementation 'io.grpc:grpc-okhttp:1.40.1'
implementation 'io.grpc:grpc-protobuf-lite:1.40.1'
implementation 'io.grpc:grpc-stub:1.40.1'
compileOnly 'org.apache.tomcat:annotations-api:6.0.53'
protobuf "com.google.protobuf:protobuf-java:3.17.3"
I am using the latest protobuf plugin
plugins {
id 'com.google.protobuf' version '0.8.17'
}
And use the following block for code-gen
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.17.3"
}
plugins {
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:1.40.1"
}
}
generateProtoTasks {
all().each { task ->
task.builtins {
java { option 'lite' }
}
task.plugins {
grpc { option 'lite' }
}
}
}
}
The gradle sync succeeds while using those, the problem is when I try to assemble the project, I get the following error:
Execution failed for task ':App:generateDebugProto'.
protoc: stdout: . stderr: C:\Users\phantom\AndroidStudioProjects\Protobuf\App\build\extracted-protos\main\google\protobuf\any.proto: Input is shadowed in the --proto_path by "C:/Users/phantom/AndroidStudioProjects/Protobuf/App/build/extracted-include-protos/debug/google/protobuf/any.proto". Either use the latter file as your input or reorder the --proto_path so that the former file's location comes first.
From what I understand while reading the error, the problem is that the proto files are generated now in both extracted-protos and extracted-include-protos build files, and the latter shadows the first one. I have checked, in the previous version, the files were generated solely in the extracted-protos build files.
Is there a way to skip generating the files in the extracted-include-protos? Or what would be the course of action to be able to assemble the project?

I ran into this same issue yesterday. This is more of a workaround than a full answer. It got me working with Google speech-to-text, but it doesn't work if you add in a non-beta version of text-to-speech, so if anybody has a better answer please post.
protobuf {
protoc {
artifact = 'com.google.protobuf:protoc:3.17.3'
}
plugins {
javalite {
artifact = "com.google.protobuf:protoc-gen-javalite:3.0.0"
}
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:1.40.1"
}
}
generateProtoTasks {
all().each { task ->
task.builtins {
// In most cases you don't need the full Java output
// if you use the lite output.
remove java
}
task.plugins {
javalite {}
grpc {
// Options added to --grpc_out
option 'lite'
}
}
}
}
}
implementation 'io.grpc:grpc-okhttp:1.40.1'
implementation 'io.grpc:grpc-protobuf-lite:1.25.0'
implementation 'io.grpc:grpc-stub:1.40.1'
compileOnly 'org.apache.tomcat:annotations-api:6.0.53'
protobuf "com.google.protobuf:protobuf-java:3.17.3"
implementation("com.google.cloud:google-cloud-speech:1.22.1") {
exclude group: 'com.google.protobuf', module: 'protobuf-java'
exclude group: 'com.google.api.grpc'
}
Note the versions of grpc-protobuf-lite and google-cloud-speech. I had to downgrade both of them from the latest.

Had the same issue,
After trying every possible gradle configuration for 2 days.
I just manually deleted all the proto files under ../build/extracted-include-protos/debug/google/protobuf/
and the error is gone

How do you add gRPC to Android Studio with Kotlin?

Task
I need to connect an Android client with a python server using gRPC. Making the server and generating the protos was easy in Python, but the lack of tutorials and confusing documentation for the Kt client makes it appear overwhelmingly complicated.
Background
Until now I've made some simple Android apps using Kotlin, I got used to adding dependencies to either the module or app level build.gradle.
What have I tried?
My first thought was to go to the official documentation as I did with Python.
I found the guide from there pretty confusing (I felt like there's something missing from that article), so I went to see the full examples from their GitHub. I also cloned the repo and compiled the protos with the gradlew installDist command. Then the things got awfully complicated:
When you create an Android Studio project, you get a bunch of gradle things(module and app level build.gradle's, gradlew and gradlew.bat, settings, etc)
After you clone the repo, you get another bunch of gradle things inside the grpc-kotlin folder.
You also get build.gradle.kts which seem to be the same build logic/package manager helper files, but with other dependencies and with the Kotlin Script syntax.
This is when I went off to YouTube in order to search for a simple implementation and found out that there's only a handful of videos on the gRPC with Kotlin subject, and most of those are presentation videos about the features of gRPC in Kotlin when using Coroutines.
What I have until now
I migrated all my build.gradle's to .kts ones.
This is how my module-level build.gradle.kts looks like:
buildscript {
val kotlin_version = "1.5.10"
repositories {
google()
mavenCentral()
}
dependencies {
classpath("com.android.tools.build:gradle:4.2.1")
classpath("org.jetbrains.kotlin:kotlin-gradle-plugin:${kotlin_version}")
classpath("org.jetbrains.kotlin:kotlin-android-extensions:${kotlin_version}")
classpath("com.google.gms:google-services:4.3.8")
classpath ("com.google.protobuf:protobuf-gradle-plugin:0.8.14")
}
}
allprojects {
repositories {
google()
mavenCentral()
}
}
tasks.register("clean",Delete::class){
delete(rootProject.buildDir)
}
This is how my app level build.gradle.kts looks like:
import com.google.protobuf.gradle.generateProtoTasks
import com.google.protobuf.gradle.id
import com.google.protobuf.gradle.plugins
import com.google.protobuf.gradle.protobuf
import com.google.protobuf.gradle.protoc
plugins {
id("com.android.application")
id("com.google.protobuf")
kotlin("android")
}
android {
compileSdkVersion(30)
buildToolsVersion = "30.0.3"
defaultConfig {
applicationId = "com.example.myapplication"
minSdkVersion(26)
targetSdkVersion(30)
versionCode = 1
versionName = "1.0"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
buildTypes {
getByName("release") {
isMinifyEnabled = false
proguardFiles(getDefaultProguardFile("proguard-android.txt"), "proguard-rules.pro")
}
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
}
kotlinOptions {
jvmTarget = "1.8"
}
buildFeatures {
viewBinding = true
}
}
protobuf {
protoc { artifact = "com.google.protobuf:protoc:3.12.0" }
plugins {
id("grpc") {
artifact = "io.grpc:protoc-gen-grpc-java:1.35.0"
}
}
generateProtoTasks {
all().forEach { task ->
task.plugins.create("java") {
option("lite")
}
task.plugins {
id("grpc") {
this.option("lite")
}
}
}
}
}
dependencies {
val kotlin_version = "1.5.10"
implementation("org.jetbrains.kotlin:kotlin-stdlib:${kotlin_version}")
implementation("androidx.core:core-ktx:1.5.0")
implementation("androidx.appcompat:appcompat:1.3.0")
implementation("com.google.android.material:material:1.3.0")
implementation("androidx.constraintlayout:constraintlayout:2.0.4")
testImplementation("junit:junit:4.13.2")
androidTestImplementation("androidx.test.ext:junit:1.1.2")
androidTestImplementation("androidx.test.espresso:espresso-core:3.3.0")
// GRPC Deps
implementation("io.grpc:grpc-okhttp:1.37.0")
implementation("io.grpc:grpc-protobuf-lite:1.37.0")
implementation("io.grpc:grpc-stub:1.36.0")
implementation("org.apache.tomcat:annotations-api:6.0.53")
}
I could generate the protos but something was off about them.
Problem
When implementing the request functions, respectively a bi-directional stream, I found out that all my rpc functions asked for an extra StreamObserver parameter(which was absent in all of the tutorials I've found on the internet). At a closer look I observed that all the generated files were in java and on the official docs, the generated files are both POJOs and Kotlin.
This is how my generated Stub class looks like:
public static final class ChatServiceStub extends io.grpc.stub.AbstractAsyncStub<ChatServiceStub> {
private ChatServiceStub(
io.grpc.Channel channel, io.grpc.CallOptions callOptions) {
super(channel, callOptions);
}
#java.lang.Override
protected ChatServiceStub build(
io.grpc.Channel channel, io.grpc.CallOptions callOptions) {
return new ChatServiceStub(channel, callOptions);
}
/**
* <pre>
* This bi-directional stream makes it possible to send and receive Notes between 2 persons
* </pre>
*/
public void chatStream(grpc.Chat.Empty request,
io.grpc.stub.StreamObserver<grpc.Chat.Note> responseObserver) {
io.grpc.stub.ClientCalls.asyncServerStreamingCall(
getChannel().newCall(getChatStreamMethod(), getCallOptions()), request, responseObserver);
}
/**
*/
public void sendNote(grpc.Chat.Note request,
io.grpc.stub.StreamObserver<grpc.Chat.Empty> responseObserver) {
io.grpc.stub.ClientCalls.asyncUnaryCall(
getChannel().newCall(getSendNoteMethod(), getCallOptions()), request, responseObserver);
}
}
I do not know how to replicate a gradle script for my project, I found no one on the internet explaining how are all those build.gradle's linked together(I figured out that module level build.gradle's are describing how the module they're in is supposed to build and app level build.gradle's are idem but for the entire app). Most of the articles I found are the same as the official docs.
What I want
I just want a simple-simple project or a step by step tutorial, without "clone this and run a command in the terminal, it just works".
I do not blame the devs or whoever wrote the official docs, I actually bet I'm the stupid one here, but I struggle to understand these concepts and I would be grateful if someone can explain to me what I did wrong or where to learn.
Also, sorry for the long question, I tried to expose my POV the best I could, this is my second question since I started learning programming and I'm sorry if the problem and my goals aren't clear enough, I'll edit anything if it's needed.

I do not have a step-by-step process I can share and do not want to trivialize an excellently asked question. However, I wanted to respond that in researching a similar problem, I found that Square has a library that seems to be more Kotlin friendly:
https://square.github.io/wire/#wire-kotlin

I created the simplest project I could. The minimal configuration of Android project can be found in this commit.
Start a new project from Basic activity template in Android Studio and then:
Modify app/build.gradle
Add Gradle plugin, so it can generate all stubs in the build step:
plugins { // this section should be already at the top of the file
// ...
id 'com.google.protobuf' version "0.9.1" // "0.9.2 causes compilation errors!"
}
WARNING: At the time of writing, the latest version of
protobuf gradle plugin is 0.9.2. Using this version causes
build errors I wasn't able to deal with using information I found
on the Internet.
Add project dependencies. Lack of one can cause usually
non-understandable error messages.
dependencies { // this section should be already in the file
// ...
implementation 'io.grpc:grpc-stub:1.52.1'
implementation 'io.grpc:grpc-protobuf:1.52.1'
implementation 'io.grpc:grpc-okhttp:1.52.1'
implementation 'io.grpc:protoc-gen-grpc-kotlin:1.3.0'
implementation 'io.grpc:grpc-kotlin-stub:1.3.0'
implementation 'com.google.protobuf:protobuf-kotlin:3.21.12'
}
Add protobuf section - the one that is responsible for generating protobuf data classes and gRPC client stubs:
protobuf { // this section needs to be added
protoc {
artifact = "com.google.protobuf:protoc:3.21.12"
}
plugins {
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:1.52.1"
}
grpckt {
// I don't really know what ":jdk8#jar" does...
artifact = "io.grpc:protoc-gen-grpc-kotlin:1.3.0:jdk8#jar"
// ...but it doesn't work without it.
}
}
generateProtoTasks {
all().forEach {
it.plugins {
grpc {}
grpckt {}
}
it.builtins {
kotlin {}
java {}
}
}
}
}
4*. I placed my proto file in app/src/main/proto directory. In case you store your protos in some other directory you can configure it by following instructions from protobuf-gradle-plugin repo readme.
Update AndroidManifest.xml
You also need to add proper permission to app/src/main/AndroidManifest.xml:
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools">
<uses-permission android:name="android.permission.INTERNET" />
<!-- ... -->
</manifest>
Use gRPC client
class FirstFragment : Fragment() {
// ...
override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
super.onViewCreated(view, savedInstanceState)
val port = 50051
val channel = ManagedChannelBuilder.forAddress("192.168.0.13", port).usePlaintext().build()
val stub = PingPongGrpcKt.PingPongCoroutineStub(channel)
binding.buttonFirst.setOnClickListener {
runBlocking {
val request = Pingpong.PingPongMsg.newBuilder().setPayload("World").build()
val response = stub.ping(request)
Log.i("result", response.toString())
}
}
}
// ...
}

Compiler cannot resolve classes in io.ktor.client.features.logging

I'm trying to add logging for Ktor http requests in Android application. According to docs I have to add gradle dependency
implementation "io.ktor:ktor-client-logging:$ktor_version"
and just use this snippet
val client = HttpClient() {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.HEADERS
}
}
Problem is that compiler "ignores" package 'io.ktor.client.features.logging' added as a dependency. What's strange is that JsonFeature (added as similar dependency) works just fine.
install(JsonFeature) { // perfectly works
...
}
install(Logging) { // unresolved reference
...
}
I already checked .jar file that gradle added to the project, it contains all expected classes, I can open them and see the source code, but magically just can't use in my app. After hours of research I guess it may be somehow related to gradle metadata or that logging feature is multiplatform and some additional gradle configuration is required, but unfortunately I'm not a gradle expert.
I tried adding enableFeaturePreview("GRADLE_METADATA") to settings.gradle, but no effect. Even tried to add "-jvm" to dependency.
implementation "io.ktor:ktor-client-logging-jvm:$ktor_version"
With this dependency Android Studio finding package successfully, but fails to compile with following error
More than one file was found with OS independent path 'META-INF/ktor-http.kotlin_module'
Can anyone please clarify how to properly configure dependency for Ktor logger?

For the ktor-client-logging you have to have the dependency set for each platform:
commonMain {
dependencies {
implementation "ch.qos.logback:logback-classic:1.2.3"
implementation "io.ktor:ktor-client-logging:$ktor_version"
}
}
androidMain {
dependencies {
implementation "io.ktor:ktor-client-logging-jvm:$ktor_version"
}
}
iosMain {
dependencies {
implementation "io.ktor:ktor-client-logging-native:$ktor_version"
}
}
as for the meta META-INF/ktor-http.kotlin_module add to the app/build.gradle inside the android {} block:
android {
packagingOptions {
exclude 'META-INF/common.kotlin_module'
exclude 'META-INF/*.kotlin_module'
}
}

Could not determine the dependencies of task ':app:dokka'

I'm trying to use dokka on my android project to generate kdoc.
But I have this error when I'm running the script 'modules:app [dokka]' :
Could not determine the dependencies of task ':app:dokka'.
kotlin.KotlinNullPointerException (no error message)
I added the following lines on my gradle files :
Project build.gradle
buildscript {
ext {
dokka_version = '0.9.18'
}
dependencies {
classpath "org.jetbrains.dokka:dokka-android-gradle-plugin:$dokka_version"
}
}
app build.gradle
plugins {
id 'org.jetbrains.dokka-android'
}
dokka {
outputFormat = 'html'
sourceDirs = files('src/main')
outputDirectory = "$buildDir/javadoc"
}
Could not determine the dependencies of task ':app:dokka'.
kotlin.KotlinNullPointerException (no error message)

The issue is that it's a multiplatform project. In the app level gradle file, I'm also applying the org.jetbrains.kotlin.multiplatform plugin. As described in the dokka github release page:
Experimental Kotlin Multiplatform support is scheduled for 0.9.19
Looks like there's no other solution than wait for the next release of dokka.
Edit: There's a workaround described on the kolinlang forum
dokka {
impliedPlatforms = ["common"] // This will force platform tags for all non-common sources e.g. "JVM"
kotlinTasks {
// dokka fails to retrieve sources from MPP-tasks so they must be set empty to avoid exception
// use sourceRoot instead (see below)
[]
}
sourceRoot {
// assuming there is only a single source dir...
path = kotlin.sourceSets.commonMain.kotlin.srcDirs[0]
platforms = ["common"]
}
}

Gradle - add dependency to tests of another module

I have a multi-module gradle project that looks like this:
Parent
|--server
|--application (android module)
+--common
The server tests have a dependency on the common module tests. For this, I added
testCompile files(project(':common').sourceSets.test.output.classesDi
compileTestJava.dependsOn tasks.getByPath(':common:testClasses')
and it worked great. Unfortunately, when I tried to do the same thing for the application module that also has a dependency on the common module tests, it wouldn't work. It fails with:
Build file 'application\build.gradle' line: 103
A problem occurred evaluating project ':application'.
Could not find property 'sourceSets' on project ':common'
After googling a bit I also tried
project.evaluationDependsOn(':common')
testCompile files(project(':common').sourceSets.test.output.classesDir)
But fails with another exception:
Project application: Only Jar-type local dependencies are supported. Cannot handle: common\build\classes\test
Any ideas on how to fix this?

There's a couple of approaches solving the problem of importing test classes in this article. https://softnoise.wordpress.com/2014/09/07/gradle-sub-project-test-dependencies-in-multi-project-builds/ The one I used is:
code in shared module:
task jarTest (type: Jar) {
from sourceSets.test.output
classifier = 'test'
}
configurations {
testOutput
}
artifacts {
testOutput jarTest
}
code in module depending on the shared module:
dependencies{
testCompile project(path: ':common', configuration: 'testOutput')
}
And there seems to be a plugin for it as well! https://plugins.gradle.org/plugin/com.github.hauner.jarTest/1.0

Following the approach from sakis, this should be the configuration you need to get the tests available from another project in the Android platform (done for debug variant).
Shared module:
task jarTests(type: Jar, dependsOn: "assembleDebugUnitTest") {
classifier = 'tests'
from "$buildDir/intermediates/classes/test/debug"
}
configurations {
unitTestArtifact
}
artifacts {
unitTestArtifact jarTests
}
Your module:
dependencies {
testCompile project(path: ":libName", configuration: "unitTestArtifact")
}

The solution mentioned by droidpl for Android + Kotlin looks like this:
task jarTests(type: Jar, dependsOn: "assembleDebugUnitTest") {
getArchiveClassifier().set('tests')
from "$buildDir/tmp/kotlin-classes/debugUnitTest"
}
configurations {
unitTestArtifact
}
artifacts {
unitTestArtifact jarTests
}
Gradle for project that is going to use dependencies:
testImplementation project(path: ':shared', configuration: 'unitTestArtifact')

I know it's kinda an old question but the solution mentioned in the following blog solves the problem very nicely and is not a sort of hack or a temporary workaround:
Shared test sources in Gradle multi-module project
It works something like this:
// in your module's build.gradle file that needs tests from another module
dependencies {
testCompile project(path: ':path.to.project', configuration: 'test')
}
Also you should note that in the very last paragraph he mentioned that you need to enable Create separate module per source set in IntelliJ settings. But it works fine without using that option too. Probably due to changes in the recent IntelliJ versions.
EDIT: IntelliJ recognizes this fine as of 2020.x versions.

I think you could use gradles java test fixtures. This will automatically create a testFixtures source set, in which you can write your test that you want to reuse.
Test fixtures are configured so that:
they can see the main source set classes
test sources can see the test fixtures classes
For example, if you have some class in common module:
public class CommonDto {
private final Long id;
private final String name;
// getters/setters and other methods ...
}
Then in the common module, you could write into src/testFixtures/java following utils:
public class Utils {
private static final CommonDto A = new CommonDto(1, "A");
private static final CommonDto B = new CommonDto(2, "B");
public static CommonDto a() { return A; }
public static CommonDto b() { return B; }
}
Then in you other modules you could add this to reuse Utils class
dependencies {
// other dependencies ...
testImplementation(testFixtures(project(":common")))
}
All of this is better explained in the documentation that I provided initially. There are some nuances that you need to take into account until you create this not to leak test classes into production.