Develop Reference

TCP client for Android: text is not received in full

I am converting a Java desktop project to Android. Part of it includes a TCP connection to a server and parsing a long text from the server to the client (the Android application). This is the code that I have for the desktop project that I also try to use in Android:
// Method is called when a button is tapped
public void tapButton() {
// Create a message to the server that requests for the Departure navdata
String messageToServer = someMethodToMakeHandshakeMessage();
// Connect to the server
if (!messageToServer.equals("")) {
String finalMessageToServer = messageToServer;
new Thread(() -> {
String navdata = connectClient(finalMessageToServer);
getActivity().runOnUiThread(() -> messageReceived(navdata));
// I am also using messageReceived(navdata) without runOnUiThread with the same result
}).start();
}
}
public String connectClient(String messageOut) {
Socket socket = null;
DataInputStream input = null;
DataOutputStream output = null;
BufferedReader br = null;
// Final message from the server
String data = "";
// Message from the server that should terminate TCP connection
String terminator = "END_DATA";
try {
// Create socket and streams
socket = new Socket(someIPAddress, somePort);
input = new DataInputStream(socket.getInputStream());
output = new DataOutputStream(socket.getOutputStream());
//Send message to the server
output.writeBytes(messageOut);
//Read Response
br = new BufferedReader(new InputStreamReader(socket.getInputStream()));
StringBuilder sb = new StringBuilder();
String s = "";
int value = 0;
// Process the message from the server and add to the StringBuilder
while((value = br.read()) != -1) {
// converts int to character
char c = (char)value;
sb.append(c);
if(sb.toString().contains(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
catch (UnknownHostException e) {
// Dealing with exception
}
catch (EOFException e) {
// Dealing with exception
}
catch (IOException e) {
// Dealing with exception
}
finally {
try {
if(socket!=null) { socket.close();}
if(input != null) { input.close();}
if(output != null) { output.close();}
if(br != null) { br.close();}
}
catch (IOException ex) {
// Dealing with exception
}
socket = null;
input = null;
output = null;
br = null;
}
return data;
}
public void messageReceived(String message) {
// Method to deal with received data
}
Whereas the code works fine in the desktop Java application, I have problems with Android (using an emulator). The text is not sent in full length and is cut somewhere in the middle (only 20-50% received by the client; the number of parsed characters differs all the time). Besides, I have noticed that it is taking too long to connect to the server, but, I guess, this is due to working with an emulator.
Should a TCP client receiving long texts from the server be implemented in Android somewhat differently?
EDIT: Implemented the following code using a suggestion by #blackapps:
String line = br.readLine();
while (line != null) {
sb.append(line);
line = br.readLine();
if (line.trim().isEmpty()) {
Log.i("EMPTY LINE>>>>>>>>>>>>>>>>>",line);
}
if(line.equals(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
Two issues. I would like to keep the empty lines in the received text. The terminator is not detected. I think, it is separated from the main text with two empty lines. However, after the first empty line, it goes to indefinite loop and connection never terminated.
EDIT #2.
After having spent several hours trying to figure out what is going on, making changes to the server, and comparing the number of bytes sent and received, I have noticed that this is not the problem with the code. It appears that the client receives the full text. The problem is with how the text is written in the console using the Log.i(String, String) method. I have added the good old System.out.println() in the code, and the whole text was shown in the console. However, the text from Log.i() was cut off in the middle. As this is my first experience with Android Studio, what the heck is going on?
Thanks a lot!

Let talk about TCP socket first.
When talking about TCP socket, it's a stream of data.
TCP views data as an unstructured, but ordered, stream of bytes. It's different from the kinds of socket.io.
From time to time, TCP will grab chunks of data from the send buffer and pass the data to the network layer. The maximum amount of data that can be grabbed and placed in a segment is limited by the maximum segment size (MSS). The MSS is typically set by first determining the length of the largest link-layer frame.
So it depends on the device.
For example, you have two messages, each of them has 1000 bytes data, and you call:
-------------- client side ----------------
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
socket.onReceived(data => {
// process(data)
})
With above pseudocode you should note that:
The data which received and called on onReceived block couldn't be 1000 bytes of theFirstMessage.
It could be first 400 bytes, then on other event you receive 400 bytes, then more 400 bytes (200 of the first one and 200 of the second one).
It could be 1200 bytes (1000 of the first one and 200 of the second one).
TCP views data as an unstructured, but ordered, stream of bytes. Socket.io is a wrapper, when it uses TCP socket, it collect and combine/split the data for you, so that you received the events with exactly the data was sent from other side.
When you work with TCP, you have to do it your self, you have to define the application protocol to do it.
There're two common ways to send/receive TCP requests:
Splitter, you choose a splitter. For example, we choose 32 bits AABBCCDD as the splitter (same as you choose END_DATA string), but keep in mind it's binary data. Then you have to ensure that the data in request doesn't contains the splitter. To do that, you have to encode the request. For example we can encode request as base64, then use the character which isn't included in base64 table as the splitter.
Prefix length, the above method has its overhead as we have to encode request data. The prefix length method is a better choice.
We can prefix the length of request before.
The pseudocode:
// use Int32, 4 bytes to indicate the length of message after it
-------------- client side ----------------
client.send(theFirstMessage.length) // Int32
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage.length)
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
var buffer = Buffer()
socket.onReceived(data => {
buffer.append(data)
let length = Int32(buffer[0...3])
if (buffer.length >= length + 4) {
let theRequest = buffer[4 ... 4 + length - 1]
process(theRequest)
buffer = buffer.dropFirst(4 + length)
}
})
One more thing, when working with TCP socket, it's just stream of bytes, so the endianness is important https://en.wikipedia.org/wiki/Endianness
For example, an android device is little endian and server side (or other android device) is big endian. Then 4 bytes of Int32 from the android device, when received on server side, it will be decoded wrongly if you don't care about it.
So, the prefix length should be encoded by specific endianness.

Arduino/Android Bluetooth delay

We are developping an app that uses Bluetooth library to communicate with an Arduino in bluetooth via an HC-05 module. We made a dummy configuration to test the delay without any computation from eather the Arduino or the app and we have a huge delay of about 1 second between a request and an answer...
Protocol looks easy : Android send byte -2 and if byte received is -2, Arduino send -6, -9 and Android answer again and again.
Android Code :
h = new Handler() {
public void handleMessage(android.os.Message msg) {
switch (msg.what) {
case RECIEVE_MESSAGE: // if receive massage
byte[] readBuf = (byte[]) msg.obj;
for(int i=0;i < readBuf.length;i++)
{
if((int) readBuf[i] != 0) {
txtArduino.append(String.valueOf((int) readBuf[i]) + ", ");
}
}
byte[] msg = {-2};
mConnectedThread.writeByte(msg);
break;
}
};
};
Arduino Code :
const int receveidBuffLen = 8*4;
void setup() {
Serial.begin(115200);
}
void loop() {
if (Serial.available() > 0)
{
byte buff[receveidBuffLen];
Serial.readBytes(buff, receveidBuffLen);
for(int i=0; i < receveidBuffLen;i++)
{
if(buff[i] == (byte) -2) // 254
{
byte message[2] = {(byte) -6, (byte) -9};
Serial.write(message, 2);
Serial.flush();
}
}
}
delay(3);
}
Does anyone know where the delay comes from?
We changed the HC05 baudrate (from 9600 to 115 200) : nothing happened. We changed HC05 with another : nothing happened. We used the Blue2Serial library (Bluetooth as SPP) before and delay was the same... We used another controler (ESP8266) and delay still was 1 second...

Looks like this string is an issue:
Serial.readBytes(buff, receveidBuffLen);
Where receveidBuffLen is 32.
Although you get single byte at a time, you're trying to read 32 of them. Of course, if there are no more bytes, the code will be stuck until timeout.
Furthermore, after bytes is read, you never check how many bytes were actually read, but do scan whole the array from bottom to top:
for(int i=0; i < receveidBuffLen;i++)
instead, you have to do something like this:
int bytesAvailable = Serial.available();
if (bytesAvailable > 0)
{
byte buff[receveidBuffLen];
int bytesToRead = (bytesAvailable < receveidBuffLen) ? bytesAvailable : receveidBuffLen;
// Read no more than the buffer size, but not more than available
int bytesActuallyRead = Serial.readBytes(buff, bytesToRead);
for(int i=0; i < bytesActuallyRead;i++)
...

There are a couple problems with the code that might cause delays:
delay function at end of loop - This will slow down the processing that the Ardunio can keep up with
Calling Serial.flush() - This will block the processing loop() until the internal TX serial buffer is empty. That means the Arduino is blocked and new RX data can pile up, slowing the response time.
Calling Serial.readBytes() - You should focus on the smallest unit of data and process that each loop() iteration. If you are trying to deal with multiple message per loop, that will slow now the loop time causing a delay.
You can try to implement a SerialEvent pattern on the Arduino. We will only read one byte at a time from the serial buffer, keeping the processing that the loop() function has todo to a bare minimum. If we receive the -2 byte we will mark a flag. If the flag is marked the loop() function will call the Serial.write() function but will not block for the data to transmit. Here is a quick example.
bool sendMessage = false;
byte message[2] = {(byte) -6, (byte) -9};
void loop()
{
if (sendMessage == true)
{
Serial.write(message, 2);
sendMessage = false;
}
}
/*
SerialEvent occurs whenever a new data comes in the hardware serial RX. This
routine is run between each time loop() runs, so using delay inside loop can
delay response. Multiple bytes of data may be available.
*/
void serialEvent()
{
while (Serial.available())
{
// get the new byte:
byte inChar = ((byte) Serial.read());
if (inChar == ((byte) -2))
{
sendMessage = true;
}
}
}

We just find some solutions by ourselves and want to share them :
Initial situation : 1050 ms for an answer. Alls solutions are independent and done with the initial situation.
Remove Serial.flush() : 1022 ms.
Add a simple Serial.setTimeout(100) in Arduino Code : 135 ms. (Oh man!)
Add a simple timeout to inputStream of 100ms in Android : 95 ms.
Which solution is the best, we can't say but it works now...

Android + Arduino bluetooth communication

I have a question about android bluetooth working principle. I have an app that connects to arduino with HC-06 shield. And on receive i get new line after first char. Lets say i send from arduino
12345
and in android i see in ListView
1
2345
I believe this is about android not arduino, because i connected arduino to PC and i received correct data.
case MESSAGE_READ:
byte[] readBuf = (byte[]) msg.obj;
// construct a string from the valid bytes in the buffer
String readMessage = new String(readBuf, 0, msg.arg1);
mConversationArrayAdapter.add(readMessage);
break;
Any ideas how to debug is it really android or maybe its arduino failing?
If you are interested i post arduino code:
void loop() // run over and over
{
if (mySerial.available())
Serial.write(mySerial.read());
if (Serial.available())
mySerial.write(Serial.read());
}
EDIT
The problem about "\n" comes out only with arduino+HC-06 bluetooth shield. While connected to pc the android tablet shows correct data.

The answer was to check string ending.
private void onBluetoothRead(byte[] buffer, int len) {
String output = new String(buffer, 0, len); // Add read buffer to new string
Log.i(LOGGER_TAG, String.format("Received: "+ output + " , " + "%d bytes", len));
outputTemp += output;
if (outputTemp.endsWith("\n")){
m_deviceOutput.append(outputTemp); // Add (not replace) string to TextView
StringTokenizer splitStr = new StringTokenizer(outputTemp, ","); // split string by comma
String numberOne = splitStr.nextToken().replaceAll("\\D+",""); // First split string
String numberTwo = splitStr.nextToken().replaceAll("\\D+",""); // Second split string
m_deviceOutputPrs.setText(numberOne);
m_deviceOutputSpeed.setText(numberTwo);
outputTemp = "";
}
}

Seems like you need more time to receive the whole message. You may either append data to the same buffer, not the new one; or you may add a small delay between the time you detected new input data and the time you read the whole input data.

Read advertisement packet in Android

I'm working on a BLE sensor that is advertising manufacturer specific data. Is there any sample code that demonstrates how to receive an advertisement packet in Android and parse its payload?

This is what I was looking for:
The BLE scan API BluetoothAdapter.startLeScan(ScanCallback) requires a call back function for the scan results. the method needs to look like the following:
private BluetoothAdapter.LeScanCallback ScanCallback =
new BluetoothAdapter.LeScanCallback()onLeScan(final BluetoothDevice device,
int rssi,
final byte[] scanRecord)
{...}
And the scanRecord variable is a byte array which contains the Advertisement packet payload.
Per the BLE specification the structure of the payload is very simple as follows:
The packets can be up to 47 bytes in length and consist of:
1 byte preamble
4 byte access address
2-39 bytes advertising channelPDU
3 bytes CRC
For advertisement communication channels, the access address is always 0x8E89BED6.
The PDU in turn has its own header (2 bytes: size of the payload and its type – whether the device supports connections, etc.) and the actual payload (up to 37 bytes).
Finally, the first 6 bytes of the payload are the MAC address of the device, and the actual information can have up to 31 bytes.
the format of the actual information is as follows:
first byte is length of the data and second byte is type followed by the data.
This is a clever way to allow any application to skip entire data records if they don't care about the contents.
Here is the sample code to determine the contents of the Advertisement packet:
parseAdvertisementPacket(final byte[] scanRecord) {
byte[] advertisedData = Arrays.copyOf(scanRecord, scanRecord.length);
int offset = 0;
while (offset < (advertisedData.length - 2)) {
int len = advertisedData[offset++];
if (len == 0)
break;
int type = advertisedData[offset++];
switch (type) {
case 0x02: // Partial list of 16-bit UUIDs
case 0x03: // Complete list of 16-bit UUIDs
while (len > 1) {
int uuid16 = advertisedData[offset++] & 0xFF;
uuid16 |= (advertisedData[offset++] << 8);
len -= 2;
uuids.add(UUID.fromString(String.format(
"%08x-0000-1000-8000-00805f9b34fb", uuid16)));
}
break;
case 0x06:// Partial list of 128-bit UUIDs
case 0x07:// Complete list of 128-bit UUIDs
// Loop through the advertised 128-bit UUID's.
while (len >= 16) {
try {
// Wrap the advertised bits and order them.
ByteBuffer buffer = ByteBuffer.wrap(advertisedData,
offset++, 16).order(ByteOrder.LITTLE_ENDIAN);
long mostSignificantBit = buffer.getLong();
long leastSignificantBit = buffer.getLong();
uuids.add(new UUID(leastSignificantBit,
mostSignificantBit));
} catch (IndexOutOfBoundsException e) {
// Defensive programming.
Log.e("BlueToothDeviceFilter.parseUUID", e.toString());
continue;
} finally {
// Move the offset to read the next uuid.
offset += 15;
len -= 16;
}
}
break;
case 0xFF: // Manufacturer Specific Data
Log.d(TAG, "Manufacturer Specific Data size:" + len +" bytes" );
while (len > 1) {
if(i < 32) {
MfgData[i++] = advertisedData[offset++];
}
len -= 1;
}
Log.d(TAG, "Manufacturer Specific Data saved." + MfgData.toString());
break;
default:
offset += (len - 1);
break;
}
}
thanks to
how-ibeacons-work
bluetooth org specs
mass for putting me on the right direction!

ADPayloadParser in nv-bluetooth parses the payload of an advertising packet and returns a list of AD structures. The AD structure format is described in "11 ADVERTISING AND SCAN RESPONSE DATA FORMAT" of "Bluetooth Core Specification 4.2".
The following code snippet is an implementation example of onLeScan method.
public void onLeScan(
BluetoothDevice device, int rssi, byte[] scanRecord)
{
// Parse the payload of the advertising packet.
List<ADStructure> structures =
ADPayloadParser.getInstance().parse(scanRecord);
// For each AD structure contained in the advertising packet.
for (ADStructure structure : structures)
{
if (structure instanceof IBeacon)
{
// iBeacon packet was found.
handleIBeacon((IBeacon)structure);
}
......
}
}
You can register a parser of your own for your manufacturer-specific format into ADPayloadParser. Refer to the following links for more information.
Blog: http://darutk-oboegaki.blogspot.jp/2015/03/ibeacon-as-kind-of-ad-structures.html
GitHub: https://github.com/TakahikoKawasaki/nv-bluetooth
JavaDoc: http://takahikokawasaki.github.io/nv-bluetooth/
Maven: http://search.maven.org/#search|ga|1|a%3A%22nv-bluetooth%22

edit 21.02.2016
The library i linked below seems to have been moved;
see https://github.com/AltBeacon/android-beacon-library
You can use Android iBeacon Library for a start.
There is a reference application which you can use for the basics and with simulated data.
~https://github.com/RadiusNetworks/ibeacon-reference-android~
Once you get it up and running you may wish to import the library and use it with your real device, there is also some example code on the site:
http://developer.radiusnetworks.com/ibeacon/android/samples.html

Serial BT data from Arduino is chopped up in Android, how to solve this?

The exact problem I am having is the same as in this thread:
Why does the serial BT data I received get chopped out?
So I know I need to make delimiters and parse, which I what I need, but sadly the answer to that thread wasn't specific enough.
I need to send analog data (from 0-1023) in the Arduino over to the Android device, so I added a "n" as a delimiter to the end of each string before sending over as such:
#include <SoftwareSerial.h>
int bluetoothTx = 2;
int bluetoothRx = 3;
boolean toggle = true;
SoftwareSerial bluetooth(bluetoothTx, bluetoothRx);
void setup()
{
//Setup usb serial connection to computer
Serial.begin(9600);
//Setup Bluetooth serial connection to android
bluetooth.begin(115200);
bluetooth.print("$$$");
delay(100);
bluetooth.println("U,57600,N");
bluetooth.begin(57600);
}
void loop()
{
//Read from serial to bluetooth
while(1) //to reduce jitters
{
String sensorString = String(analogRead(A0), DEC);
sensorString = sensorString + "n";
bluetooth.println(sensorString);
delay(100);
}
}
This is for the Arduino side.
For the Android side, I used the BluetoothChat example, so in the mHandler and in the switch-case of MESSAGE_READ, the codes are as such:
case MESSAGE_READ:
byte[] readBuf = (byte[]) msg.obj;
// construct a string from the valid bytes in the buffer
String readMessage = new String(readBuf, 0, msg.arg1);
mConversationArrayAdapter.add(readMessage);
Where mConversationArrayAdapter is a String ArrayAdapter. May i know how I can modify the code within the MESSAGE_READ case so as to solve this problem?

Send a string of ascii characters formed by each nibble of 0-1023 code (2 bytes), and append it by newline character (i.e. /n). As an example if hex code to send is 0x03FF (i.e. 1023 decimal), then string to send will be 0x30,0x33,0x46,0x46,0x0A.