Develop Reference

Bluetooth Low Energy (between ESP32 and Android smartphone): Data transmission quite slow

This is my first question ever asked on this board
The project explained short:
5 sensors, connected with an esp32 board are transmitting 1000 samples/second, each sample has 16 bit. Those values should be transmitted via BLE (With the BLE Arduino library and an ESP32). The connected device (Smartphone) should read those values and do something with them (Also via BLE, with the following library: https://github.com/RobotPajamas/Blueteeth). The ESP32 is the Server! Java is used in Android Studio!
The problem:
While testing the BLE connection a simple "hello world" was transmitted as the value for a characteristic. Every time i received the "hello world" on the android-device-side, a variable was incremented: The problem is, the variable only got incremented 4 times in one second. This means (assuming 1 char in a string equals 1 byte) 11byte*4(1/s)=44byte/s are being transmitted. -> This clearly is not enough (should not BLE transmit ~2MBit/s (minus the protocol-data))
Code Fragments
ESP32: BLE-Server that transmits value
#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>
#define SERVICE_UUID "4fafc201-1fb5-459e-8fcc-c5c9c331914b"
#define CHARACTERISTIC_UUID "beb5483e-36e1-4688-b7f5-ea07361b26a8"
class MyCallbacks: public BLECharacteristicCallbacks {
void onWrite(BLECharacteristic *pCharacteristic) {
std::string value = pCharacteristic->getValue();
if (value.length() > 0) {
Serial.println("*********");
Serial.print("New value: ");
for (int i = 0; i < value.length(); i++)
Serial.print(value[i]);
Serial.println();
Serial.println("*********");
}
}
};
void setup() {
Serial.begin(115200);
BLEDevice::init("MyESP32");
BLEServer *pServer = BLEDevice::createServer();
BLEService *pService = pServer->createService(SERVICE_UUID);
BLECharacteristic *pCharacteristic = pService->createCharacteristic(
CHARACTERISTIC_UUID,
BLECharacteristic::PROPERTY_READ |
BLECharacteristic::PROPERTY_WRITE
);
pCharacteristic->setCallbacks(new MyCallbacks());
pCharacteristic->setValue("Hello World");
pService->start();
BLEAdvertising *pAdvertising = pServer->getAdvertising();
pAdvertising->start();
}
void loop() {
// put your main code here, to run repeatedly:
delay(2000);
}
Android Studio Code (Snippet of the receiving source):
try
{
while(sampleBluetoothData)
{
this.selectedDevice.readCharacteristic(MainActivity.characteristicUUID, MainActivity.serviceUUID, (response, data) ->
{
if (response != BlueteethResponse.NO_ERROR) {
return;
}
Log.d("AUSGANG", new String(data) + "times: "+ i);
i++;
});
}
}
catch (Exception e)
{
e.printStackTrace();
}
The write on the ESP32 side is a blank example code of the Arduino IDE, the read on the Android-side is made by the BLE-Library publisher. Yes the Log.d effects the performance, but it does not drop it that much.
The variable "data" of the Android code is the received char-array. The bluetooth-reading runs on a background thread.
Question I ask myself now:
Is the Android-Studio library the problem or the Arduino library
Is this a normal behaviour, that if a value of a characteristic does not change, it is being transmitted quite slowly.
How fast can you update a value of a characteristic
Thank you in advance!

BLE can definitely transfer much more than 4 portions of 11 bytes per second.
Approach of reading:
Generally, continuos reading all the time is NOT the expected BLE way - it's better to subscribe to data changes, so ESP32 will notify only when needed (e.g. do selectedDevice.subscribeToCharacteristic once, instead of reading in a loop, but then ESP32 code should be changed accordingly)
I guess selectedDevice.readCharacteristic requests asynchronous BLE read, and when you call it in while(sampleBluetoothData), your Bluetooth library is adding more and more read requests. Maybe it would be wise to request new read only after the previous read is done - in read callback add if(sampleBluetoothData) { this.readAgain(); }
Consider making a testing prototype from this kickstart example: BLEProof on github - Android & ESP32, read, write, notify (but it uses just system API without Bluetooth library, you approach is better, it's easier and safer to use the library).
What else to check:
Android side: are you sure that your code doesn't go inside of if (response != BlueteethResponse.NO_ERROR) ?
Android side: to ensure Bluetooth library is not overloaded with read requests, try adding a delay 50 milliseconds in the reading loop (just to check, it's not a solution)
Android side: are you sure that you don't have other BLE read/writes while you read those data?
ESP32 side: use shorter BLE connection interval (BLE throughput article) - add pAdvertising->setMinPreferred(0x06); and pAdvertising->setMaxPreferred(0x20); before pAdvertising->start(); (but that sets only "preferred" interval, Android may ignore that)

Using read requests, you are mainly limited by the connection interval for transfer speeds - that is 2 intervals for request + response.
If for example your client has a connection interval of 50ms, you should expect to read a characteristic of up to 20 bytes 10 times per second.
If another client has a connection interval of 30ms, this rate improves to 16.6 reads per second.
The fastest negotiatiable connection interval is 7.5ms for a maximum of 66.6 reads per second (10.7kbps with 20 byte reads).

I was reading BLE wikipedia page and the minimum data rate is 125Kbit/s , so I think that in your case is viable, because you only will transmit 16Kbit/s. Take a look in BLE wikipedia.

Underrun in Oboe/AAudio playback stream

I'm working on an Android app dealing with a device which is basically a USB microphone. I need to read the input data and process it. Sometimes, I need to send data the device (4 shorts * the number of channels which is usually 2) and this data does not depend on the input.
I'm using Oboe, and all the phones I use for testing use AAudio underneath.
The reading part works, but when I try to write data to the output stream, I get the following warning in logcat and nothing is written to the output:
W/AudioTrack: releaseBuffer() track 0x78e80a0400 disabled due to previous underrun, restarting
Here's my callback:
oboe::DataCallbackResult
OboeEngine::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames) {
// check if there's data to write, agcData is a buffer previously allocated
// and h2iaudio::getAgc() returns true if data's available
if (h2iaudio::getAgc(this->agcData)) {
// padding the buffer
short* padPos = this->agcData+ 4 * playStream->getChannelCount();
memset(padPos, 0,
static_cast<size_t>((numFrames - 4) * playStream->getBytesPerFrame()));
// write the data
oboe::ResultWithValue<int32_t> result =
this->playStream->write(this->agcData, numFrames, 1);
if (result != oboe::Result::OK){
LOGE("Failed to create stream. Error: %s",
oboe::convertToText(result.error()));
return oboe::DataCallbackResult::Stop;
}
}else{
// if there's nothing to write, write silence
memset(this->agcData, 0,
static_cast<size_t>(numFrames * playStream->getBytesPerFrame()));
}
// data processing here
h2iaudio::processData(static_cast<short*>(audioData),
static_cast<size_t>(numFrames * oboeStream->getChannelCount()),
oboeStream->getSampleRate());
return oboe::DataCallbackResult::Continue;
}
//...
oboe::AudioStreamBuilder *OboeEngine::setupRecordingStreamParameters(
oboe::AudioStreamBuilder *builder) {
builder->setCallback(this)
->setDeviceId(this->recordingDeviceId)
->setDirection(oboe::Direction::Input)
->setSampleRate(this->sampleRate)
->setChannelCount(this->inputChannelCount)
->setFramesPerCallback(1024);
return setupCommonStreamParameters(builder);
}
As seen in setupRecordingStreamParameters, I'm registering the callback to the input stream. In all the Oboe examples, the callback is registered on the output stream, and the reading is blocking. Does this have an importance? If not, how many frames do I need to write to the stream to avoid underruns?
EDIT
In the meantime, I found the source of the underruns. The output stream was not reading the same amount of frames as the input stream (which in hindsight seems logical), so writing the amount of frames given by playStream->getFramesPerBurst() fix my issue. Here's my new callback:
oboe::DataCallbackResult
OboeEngine::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames) {
int framesToWrite = playStream->getFramesPerBurst();
memset(agcData, 0, static_cast<size_t>(framesToWrite *
this->playStream->getChannelCount()));
h2iaudio::getAgc(agcData);
oboe::ResultWithValue<int32_t> result =
this->playStream->write(agcData, framesToWrite, 0);
if (result != oboe::Result::OK) {
LOGE("Failed to write AGC data. Error: %s",
oboe::convertToText(result.error()));
}
// data processing here
h2iaudio::processData(static_cast<short*>(audioData),
static_cast<size_t>(numFrames * oboeStream->getChannelCount()),
oboeStream->getSampleRate());
return oboe::DataCallbackResult::Continue;
}
It works this way, I'll change which stream has the callback attached if I notice any performance issue, for now I'll keep it this way.

Sometimes, I need to send data the device
You always need to write data to the output. Generally you need to write at least numFrames, maybe more. If you don't have any valid data to send then write zeros.
Warning: in your else block you are calling memset() but not writing to the stream.
->setFramesPerCallback(1024);
Do you need 1024 specifically? Is that for an FFT? If not then AAudio can optimize the callbacks better if the FramesPerCallback is not specified.
In all the Oboe examples, the callback is registered on the output stream,
and the reading is blocking. Does this have an importance?
Actually the read is NON-blocking. Whatever stream does not have the callback should be non-blocking. Use a timeoutNanos=0.
It is important to use the output stream for the callback if you want low latency. That is because the output stream can only provide low latency mode with callbacks and not with direct write()s. But an input stream can provide low latency with both callback and with read()s.
Once the streams are stabilized then you can read or write the same number of frames in each callback. But before it is stable, you may need to to read or write extra frames.
With an output callback you should drain the input for a while so that it is running close to empty.
With an input callback you should fill the output for a while so that it is running close to full.
write(this->agcData, numFrames, 1);
Your 1 nanosecond timeout is very small. But Oboe will still block. You should use a timeoutNanos of 0 for non-blocking mode.

According to Oboe documentation, during the onAudioReady callback, you have to write exactly numFrames frames directly into the buffer pointed to by *audioData. And you do not have to call Oboe "write" function but, instead, fill the buffer by yourself.
Not sure how your getAgc() function works but maybe you can give that function the pointer audioData as an argument to avoid having to copy data again from one buffer to another one.
If you really need the onAudioReady callback to request the same amount of frames, then you have to set that number while building the AudioStream using:
oboe::AudioStreamBuilder::setFramesPerCallback(int framesPerCallback)
Look here at the things that you should not do during an onAudioReady callback and you will find that oboe write function is forbidden:
https://google.github.io/oboe/reference/classoboe_1_1_audio_stream_callback.html

Android via NFC IsoDep(ISO14443-4) communication with chip's FIFO cache area no way to more than 12 bytes

I'm pretty troubled with android NFC transacting with chip's FIFO cache area. Another side(B) and android app(A): B was electrify to chip,and B write data to chip's FIFO cache area by C language. The chip's FIFO cache area can not save data after outage, and when chip's FIFO cache area send all data, the chip's FIFO cache area will clear.
The situation is, A close to chip, send a APDU command, now chip receives the command and produces a signal. Then, the B detects the chip's signal, and grabs the command, then, B gets the command's first byte(fb), more then, B write [fb+data+9000] to chip's FIFO cache area. Finally, the transaction of send data back to A manage with chip self and we don't know the chip how to manage the send back.
The problem is, when B's write [fb+data+9000] less than 15 bytes(means data only 12 bytes), A can receive the [fb+data+9000] from chip. But the [fb+data+9000] more than 15 bytes, A throws TagLostException.
The chip use ISO14443-4 protocol.
The command:
The transact code:
`
try {
isoDep.close();
isoDep.connect();
}catch (IOException e){
errorfound = true;
onMessageReceived.onError(e);
}
if (!errorfound) {
if (isoDep.isConnected()) {
try {
isoDep.setTimeout(1200);
response = isoDep.transceive(newtest1_apdu);
int status = ((0xff & response[response.length - 2]) << 8) | (0xff & response[response.length - 1]);
if (status != 0x9000) {
log.error("retrieve data ,read failure");
}else {
log.info("retrieve data, result=" + numeralParse.toReversedHex(response));
}
onMessageReceived.onMessage(response);
}
catch (TagLostException e) {
log.info("catch tag lost exception, tag isConnected=" + isoDep.isConnected());
onMessageReceived.onError(e);
}
catch (IOException e) {
log.info("catch IOException, isoDep isConnected=" + isoDep.isConnected());
onMessageReceived.onError(e);
}
}else {
log.error("isoDep not connect");
}
}
`
Android app(A) try to a variety of commands, contains this format: .
And another side(B) only gets first byte in command and write [fb+data+9000] to chip's FIFO cache area. this isn't timeout reason, except setTimeOut(1200), also try setTimeOut(5000) or not setTimeOut. Other, A and B were not appoint the APDU command specific meaning. Other, by different APDU command, A work well with read Public transportation card(may be this read to block area, and now work with cache area, both work way not same). Other, the configuration of chip is basic default. Other, with other card reader test, chip's send data out success.
I go to Google,Bing,Baidu,Android office issues,stackoverflow and so on to search answer, but cannot find. This problem very bothered us. Apologetic with my poor English. Please help, extremely thank you.
(the chip is FM11NC08.)
New progress, We found, giving up using APDU command, if A send 1 byte, A can receive maximum 16 bytes. And if A send 2 bytes, A can receive maximum 15 bytes. And if A send 15 bytes, A can receive maximum 2 bytes. The chip's FIFO cache area has 32 bytes space. After B receive A's data, B will clear FIFO cache area, then write data to FIFO cache area.
Thanks in advance.

Today, B changes the chip's communication rate(from 1M to 2M) and a part of codes. Then A work well with chip! So, we found the communication rate has an impact on NFC communication. If you have the same trouble with NFC communication, might to try our way!
Thanks for people who consider this problem in not-solve days.

Cannot send large APDU commands with libnfc using nfc_initiator_transceive_bytes()

I'm trying to build a C++ wrapper around libnfc to make a communication between my Android and the PN532 RFID module.
This helped me a lot: http://nfc-tools.org/index.php/Libnfc:APDU_example
This code is meant to send an APDU command where the body is contained in message (I'm not sending any header bytes etc.) and read the response into response.
Problem: If message exceeds 262 characters then I get a buffer overflow detected error. Otherwise it works perfectly well. I don't even think the error is thrown by the NFC library.
bool send(const std::string &message, std::string &response){
std::vector<uint8_t> apduCmd(message.begin(), message.end());
uint8_t *capdu = &apduCmd[0];
size_t capdulen = apduCmd.size();
uint8_t rapdu[10];
size_t rapdulen = 10;
// BUFFER OVERFLOW HERE
int res = nfc_initiator_transceive_bytes(m_nfcDevice, capdu, capdulen, rapdu, rapdulen, 500);
if (res<0) {
return false;
}
if(res<2 || rapdu[res-2] != 0x90 || rapdu[res-1] != 0x00){
return false;
}
// byteArrayToString omitting the last two bytes
response = byteArrayToString(rapdu, 0, res-2);
return true;
}

The limit of 262 bytes is a hard limit imposed the PN532 NFC chip. This is the maximum size of the raw data that can be sent (and received) in one InDataExchange command. libnfc explicitly enforces this limit for the method nfc_initiator_transceive_bytes() (see the definition of abtCmd in pn53x_initiator_transceive_bytes() and the definition of PN53x_EXTENDED_FRAME__DATA_MAX_LEN).
What you could do to overcome this limit, is to compose your own ISO/IEC 14443-4 blocks (using InCommunicateThru, i.e. nfc_initiator_transceive_bytes() with m_nfcDevice->bEasyFraming turned off. While each frame would still be limited to 263 bytes (the PN532 actually allows 264 bytes for InCommunicateThru but libnfc seems to limit this to 263 bytes), you can then pack your extended length APDUs into multiple ISO/IEC 14443-4 I-blocks. However, you would need to handle the whole ISO/IEC 14443-4 framing on your own (which means that you also have to take care of receive acknowledgements, etc.)
Finally, since the other communication endpoint is an Android device: Many Android devices do not support extended length APDUs. Consequently, even if you send longer APDUs, you might not be able to receive and process them on the Android side. Also, be aware that you should send proper APDUs conforming to the structures defined in ISO/IEC 7816-4 (i.e. APDUs with valid header and lengths fields), otherwise you might run into issues when talking to some devices.

android TrafficStats getUidRxBytes inaccurate

I write a little android app, sends Http request, receives response from server, and count how many bytes transmitted and received.
and the code is simply as follow
long receivedBytes = TrafficStats.getUidRxBytes(uid)-lastNumer
i find that the receivedBytes is always larger the size http Header+http Body, for example
the actual http frame's size i caught( use wireshark) in server is 1645 bytes(header+body), but the android API returns receivedBytes is 1912, so as the transmission.
the TrafficStats getUidRxBytes itself is inaccurate(may be this problem is specific to my platform samsung i9300 with cynogenmod 10.3)
finally, i find the correct way to count the data usage i find other way to count the data usage which seems more accurate than TrafficStats API.(many thanks to here)
private long[] getStat() {
String line, line2;
long[] stats = new long[2];
try {
File fileSnd = new File("/proc/uid_stat/"+uid+"/tcp_snd");
File fileRcv = new File ("/proc/uid_stat/"+uid+"/tcp_rcv");
BufferedReader br1 = new BufferedReader(new FileReader(fileSnd));
BufferedReader br2 = new BufferedReader(new FileReader(fileRcv));
while ((line = br1.readLine()) != null&& (line2 = br2.readLine()) != null) {
stats[0] = Long.parseLong(line);
stats[1] = Long.parseLong(line2);
}
br1.close();
br2.close();
} catch (Exception e) {
e.printStackTrace();
}
return stats;
}

I see that you've already found a solution, but I'll add my thoughts on your question as it might be useful to other people (ended up here myself after googling how to use the TrafficStats API).
The API documentation states:
Statistics are measured at the network layer, so they include both TCP and UDP usage.
The documentation could indeed be more thorough, but I'm inclined to say that one can assume that the returned byte count also includes the bytes making up the transport layer header and the network layer header.
HTTP is an application layer protocol. When you're calculating your expected bytes to be the HTTP header bytes plus the HTTP body bytes, you're only dealing with application layer bytes, hence not accounting for transport and network layer header bytes. I assume TCP is used for the download. This adds a header ranging from 20 to 60 bytes. Moreover, let's assume you're using IPv4 for the download. This also adds a header ranging from 20 to 60 bytes.
Obviously this won't account for the entire 1912 - 1645 = 267 bytes, but it might give you/other people some leads.
A bit off-topic, but still related. It's not quite clear if the TrafficStats API actually count header bytes or not. According to this answer, the API does not count header bytes. However, given the API documentation listed above, the linked answer may be stipulating something that is not true (at least not for API level 21). Moreover, this question also hints at TrafficStats actually counting network and transport layer header bytes (check comments).
TrafficStats actually counts network and transport layer header bytes. See kernel source and TrafficStatsTest.

From my understanding, you should combine getUidRxBytes with getUidRxPackets.
You should have something like : getUidRxBytes = getUidRxPackets * (tcp/ip header size)