Develop Reference

curl_multi_wakeup doesn't seem to wakeup the associated curl_multi_poll - Android (but may not be limited to)

Curl version: 7.71.0 with c-ares
Background
We are building a library that's being integrated into mobile apps. We are targeting both iOS and Android.
Curl initialisation happens in a static block inside the library.
The iOS version of the library is bundled into a framework, which is loaded at app startup, if i'm not mistaken.
The Android version of the library is bundled in a module, which is lazily loaded. (I know this is an issue, especially since we link against OpenSSL, but it's probably important for context).
We built a small HTTP client with curl, that allows use to download some data blob from trusted servers.
Quick architecture review
The HTTP client is running on its own thread. It holds a curl_multi_handle, and any transfer started append a curl_easy_handle to it, and return a handle to a Response that contains a buffer to read the received bytes from, and is used to control the transfer if needed.
Since cURL handles are not thread safe, any action (referred to as Tasks from now on) to the handle is dispatched to the HTTP client's thread, and a boost::shared_future is returned (we might want to block or not depending on the use case).
Here is a rough idea of how the main loop is structured:
while (!done) {
deal_with_transfer();
check_transfer_status();
cleanup_any_orphan_transfer();
execute_all_queue_tasks();
curl_multi_poll(multi, nullptr, 0, very_large_number, nullptr);
}
Appending to the task queue also performs a curl_multi_wakeup(multi) to make sure that task is executed (e.g. adding a new download is also a dispatched task).
The issue
We've only thus far tested on Android, and we've seen in some cases, HTTP client tasks that are blocking are sometimes never returning.
Logs and stacktraces show that we wait on a task being executed on by the HTTP client, but the client is still polling. Everything seems to indicate that it was't woken up when appending a task.
I can't seem to replicate the issue locally, on a device, but it happens often enough to be a blocker issue.
I'm a bit at a loss here, and I don't really know where to start looking to find a way to reproduce the issue, let alone fixing it.
I hope I gave enough context to start making educated guess, or even find a the source of error!
Thanks for reading!

Limitations on network activities for background processes
Mobile operating systems such as Android and iOS have a different scheduling strategies for background processes compared to traditional *nix operating systems. The mobile OS tends to starve the background processes in order to preserve the battery time. This is know as background process optimization and will be applied to the processes/threads of the application the moment application enters in background.
As of Android 7, the background processes are no longer informed about the network events with the CONNECTIVITY_ACTION broadcasts unless they register in the manifest that they want to receive the events.
Although the lobcurl is used in android native code, the threads created from the native library will be subject of the entitlements that the application declared in the manifest (which need to be granted).
Workaround to try
I know how frustrating a blocking issue can be so I can offer you a quick workaround to try until the problem is resolved.
curl_multi_poll() can receive a timeout that in your code is set to a very_large_number. In addition, the last parameter of the function call is a pointer to an integer numfds that will be populated with the number of file descriptors on which an event occurred while the curl_multi_pool() was pooling.
You can use this in your favor to construct a workaround in the following way:
Make the very_large_number a reasonably_small_number
replace the nullptr with &numfds
Surround the curl_multi_poll with a do ... while loop
So you will have something like this:
int numfds;
while (!done) {
deal_with_transfer();
check_transfer_status();
cleanup_any_orphan_transfer();
execute_all_queue_tasks();
numfds = 0;
do {
curl_multi_poll(multi, nullptr, 0, reasonably_small_number, &numfds);
numfds += check_for_other_conditions();
} while ( numfds == 0 );
}
Select the timeout to be a reasonable number (ex. 1s, 10s, 60s ...) that will allow you to break the pooling forcefully, at the same time, not to drain the battery.
I am adding the check_for_other_conditions() so you can use is to check additional conditions. By example the size of the task queue, assuming that there are situations in which the curl_multi_poll() can miss an event, although the event occurred, this extra check can help to break the loop and start executing the tasks.

Why in Google App Engine Always 11 secs of Latency coming only for the first requests?

I am deploying my Nodejs sample app to Google App Engine Flexible env and when I am using google app engine URL which is in the form appspot.com to hit my API, it is taking around 11 secs to send response from my mobile data, but other APIs are sending response in milisecs.
Also, the time delay is only happening when I am opening my android app and sending request to the server after that all requests are taking normal time, and again delay is coming when I again open the app and send request to the server.
Edit - I found that
This can be a caused when your application is still booting up or warming up instances to serve the request and can be called as loading latency. To avoid such scenarios you can implement health check handler like readiness check so that your application will only receive traffic when its ready
That's why I checked in my Logs that readiness check is performed sometimes around 1 sec
and sometimes around 200 ms
Can anyone please tell me is there anything wrong in warming up my instances because I don't think cold boot time is causing this problem.
Edit 2
I have also tried to set min_num_instances: 2 so that once loaded atleast my 2 instances will again not get boot up, but the thing is delay is again same.
Edit 3
runtime: nodejs
#vm: true
env: flex
automatic_scaling:
min_num_instances: 2
max_num_instances: 3
Edit 4
I am noticing a strange behaviour that when I am using this app Packet Capture to capture traffic, then all https requests (if I am not enabling SSL Proxying) and all Http requests are executing in milisecs whereas without using this app all Http/Https requests are taking 11-16 secs of delay.
I don't know how but is there any certificate kind of issue here?
Edit 5
Below I have attached Network Profiler where delay is coming 15 secs
Please Help

Depends on which App Engine you are using and how you setup the scaling, there's always a loading time if you don't have a ready instance to serve a request. But if you have readiness check to ensure your instance is ready (and not cold started for the request), then there shouldn't be a problem.
Can you find a loading request or any corresponding slow request in your logs? If not, then it's likely an issue with the app. If possible, instead of calling this API on your app, do it from two apps (one is already open, one is not). So you make calls from both apps and if you notice that the one that's already open is getting a response faster than the other one, that means that's a problem with the app itself. App Engine can't determine whether or not your app is pre-opened so any difference would be client side.
=== Additional information ===
In the your logs, there's no delay at all. The request enter Google and was processed within a few milliseconds. I am sure there's something application-side. Maybe your app is constructing the request URL (first request) from some other source that results in the delay? App Engine has no knowledge of whether or not your app is opened or not or whether it's sending a first request after being opened, it cannot act differently based on it. As long as your App Engine instance is ready and available, it will treat your request the same way regardless of whether or not it's your first request after the app is opened.

The issue is resolved now, it was happening because of network service provider which is Bharti Airtel, their DNS lookup was taking the time to resolve the hostname. After explicitly using alternative DNS like Google 8.8.8.8 the issue got completely resolved. Maybe it's a compatibility issue of Airtel with Google Cloud.

Last time I checked I remember having to put a warmup request handler so that Google would know that the instance is up and running and can be used to answer calls. Keep in mind that code has to be EXACTLY under the endpoint you specify in the handler under the yaml file. (Wouldn't be the first time someone forgets that)
Here are the docs https://cloud.google.com/appengine/docs/standard/python/configuring-warmup-requests this is python specific, but you can also check other languages like Go, Java, and such in the docs.
If the problem is client dependant (each time a new clients spawns and makes a call it gets the latency) then it is most likely, either a problem with the client app itself or with initialization, registration or DNS resolution.
You could also try to reproduce the requests with CURL or similar, and see if also with those you see the mentioned delay.

Unexpected end of stream ksoap2-android with gsoap

We have an android app that uses soap to communicate with an API implemented in C++ with gsoap. The android application uses the ksoap2-android library for soap communications.
When the android app runs on an actual device and communicates over wifi everything seems to work as it should. However when run in the emulater (virtualized x86 image) requests with longer replies often fail with a ProtocolException unexpected end of stream.
This error and because it only tends to happen with long responses let me to the believe that on the gsoap side the socket gets closed to soon. I have tried somethings.
default accept_flags on the gsoap side didn't work
explicitly remove SO_LINGER flags from accept_flags in gsoap, didn't solve the issue (I was worried it might set linger with a timeout of zero which would result in exactly my problem)
explicitly specify the SO_LINGER flag in the accept flags and a timeout of 10, didn't solve the issue
Then i tried adding a sleep for 1 second right before shutdown in gsoaps tcp_disconnect function. That solved the problem but sleeping is no good ofcourse.
As sleeping is not a good solution I ended up using select which seems to work ok but I'm not 100% confident in this solution.
Here is the actual code change. These are the last lines of the tcp_disconnect function in stdsoap2.cpp. The tcp_select call is what I have added.
if (soap_valid_socket(soap->socket) && !(soap->omode & SOAP_IO_UDP))
{
tcp_select(soap, soap->socket, SOAP_TCP_SELECT_RCV | SOAP_TCP_SELECT_ERR, 5);
soap->fshutdownsocket(soap, soap->socket, SOAP_SHUT_RDWR);
soap->fclosesocket(soap, soap->socket);
soap->socket = SOAP_INVALID_SOCKET;
}
return SOAP_OK;
}
Is this the right solution?

c++11 multithreading issues with Android where some threads are not scheduled properly

I am developing a VoIP based application which is multithreaded. For every socket there is a c++11 std::thread (including SSL read & write). The core module for data communication is in C++ which is called through JNI interface.
My observation is that, once initializing the application after few seconds, some threads which were running earlier normally are not getting running time. If a certain thread is running then it keeps running for a while ranging from 3-4 seconds to 30-40 seconds.
After referring change native thread priority on Android, I also tried changing "nice" value to -10 for all the threads, but no luck.
Important to note that exactly same C++ code is working perfectly fine for iOS.
Is there an issue with Android Native thread scheduling, or am I missing something?

while (...) {
int selectResult = select( fd, ...);
if ( selectResult > 0 ) DoSomeWork( fd );
else nanosleep(...); /* this is the new line which solved my stalling threads */
}
I had a similar problem, and found that for my case the solution was to verify that those threads that have an eternal loop (constantly performing select until there is something on a socket to spawn a handler thread), always call nanosleep at least once in their loop.
Like you, I experienced this issue on Android, and no issue on iOS.
I can only hypothesise that the JVM on Android sometimes would give full priority to one thread which is only polling a socket without pausing, such that other threads which had actual work to be done would stall. But I wouldn't know how to verify that hypothesis.
It's also weird that my solution works, given that select already uses a user-chosen timeout, such that I would think it internally sleeps anyway. Apparently not on Android.

getaddrinfo() blocks the application in android if no Internet

I am using getaddrinfo() linux API in my android app (in C++ NDK). Everything works fine except for one case.
If there is no internet connection in the android system, this API blocks the application for more than 50 seconds.
Is there any resolution of this problem?
A lot of posts suggest to use this API in a thread. Is that the only solution? I guess there might be an easy or alternative solution.

Try using connection timeout as the value lesser than 50 seconds.
Also, you can check network before making any API. If there is no network do not proceed. If network goes away while executing API, connection timeout will come into picture.

getaddrinfo() is a blocking API, so this is expected. You'll probably want to do task in a worker thread and/or look into using getaddrinfo_a() which can perform asynchronous queries.