Develop Reference

TCP client for Android: text is not received in full

I am converting a Java desktop project to Android. Part of it includes a TCP connection to a server and parsing a long text from the server to the client (the Android application). This is the code that I have for the desktop project that I also try to use in Android:
// Method is called when a button is tapped
public void tapButton() {
// Create a message to the server that requests for the Departure navdata
String messageToServer = someMethodToMakeHandshakeMessage();
// Connect to the server
if (!messageToServer.equals("")) {
String finalMessageToServer = messageToServer;
new Thread(() -> {
String navdata = connectClient(finalMessageToServer);
getActivity().runOnUiThread(() -> messageReceived(navdata));
// I am also using messageReceived(navdata) without runOnUiThread with the same result
}).start();
}
}
public String connectClient(String messageOut) {
Socket socket = null;
DataInputStream input = null;
DataOutputStream output = null;
BufferedReader br = null;
// Final message from the server
String data = "";
// Message from the server that should terminate TCP connection
String terminator = "END_DATA";
try {
// Create socket and streams
socket = new Socket(someIPAddress, somePort);
input = new DataInputStream(socket.getInputStream());
output = new DataOutputStream(socket.getOutputStream());
//Send message to the server
output.writeBytes(messageOut);
//Read Response
br = new BufferedReader(new InputStreamReader(socket.getInputStream()));
StringBuilder sb = new StringBuilder();
String s = "";
int value = 0;
// Process the message from the server and add to the StringBuilder
while((value = br.read()) != -1) {
// converts int to character
char c = (char)value;
sb.append(c);
if(sb.toString().contains(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
catch (UnknownHostException e) {
// Dealing with exception
}
catch (EOFException e) {
// Dealing with exception
}
catch (IOException e) {
// Dealing with exception
}
finally {
try {
if(socket!=null) { socket.close();}
if(input != null) { input.close();}
if(output != null) { output.close();}
if(br != null) { br.close();}
}
catch (IOException ex) {
// Dealing with exception
}
socket = null;
input = null;
output = null;
br = null;
}
return data;
}
public void messageReceived(String message) {
// Method to deal with received data
}
Whereas the code works fine in the desktop Java application, I have problems with Android (using an emulator). The text is not sent in full length and is cut somewhere in the middle (only 20-50% received by the client; the number of parsed characters differs all the time). Besides, I have noticed that it is taking too long to connect to the server, but, I guess, this is due to working with an emulator.
Should a TCP client receiving long texts from the server be implemented in Android somewhat differently?
EDIT: Implemented the following code using a suggestion by #blackapps:
String line = br.readLine();
while (line != null) {
sb.append(line);
line = br.readLine();
if (line.trim().isEmpty()) {
Log.i("EMPTY LINE>>>>>>>>>>>>>>>>>",line);
}
if(line.equals(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
Two issues. I would like to keep the empty lines in the received text. The terminator is not detected. I think, it is separated from the main text with two empty lines. However, after the first empty line, it goes to indefinite loop and connection never terminated.
EDIT #2.
After having spent several hours trying to figure out what is going on, making changes to the server, and comparing the number of bytes sent and received, I have noticed that this is not the problem with the code. It appears that the client receives the full text. The problem is with how the text is written in the console using the Log.i(String, String) method. I have added the good old System.out.println() in the code, and the whole text was shown in the console. However, the text from Log.i() was cut off in the middle. As this is my first experience with Android Studio, what the heck is going on?
Thanks a lot!

Let talk about TCP socket first.
When talking about TCP socket, it's a stream of data.
TCP views data as an unstructured, but ordered, stream of bytes. It's different from the kinds of socket.io.
From time to time, TCP will grab chunks of data from the send buffer and pass the data to the network layer. The maximum amount of data that can be grabbed and placed in a segment is limited by the maximum segment size (MSS). The MSS is typically set by first determining the length of the largest link-layer frame.
So it depends on the device.
For example, you have two messages, each of them has 1000 bytes data, and you call:
-------------- client side ----------------
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
socket.onReceived(data => {
// process(data)
})
With above pseudocode you should note that:
The data which received and called on onReceived block couldn't be 1000 bytes of theFirstMessage.
It could be first 400 bytes, then on other event you receive 400 bytes, then more 400 bytes (200 of the first one and 200 of the second one).
It could be 1200 bytes (1000 of the first one and 200 of the second one).
TCP views data as an unstructured, but ordered, stream of bytes. Socket.io is a wrapper, when it uses TCP socket, it collect and combine/split the data for you, so that you received the events with exactly the data was sent from other side.
When you work with TCP, you have to do it your self, you have to define the application protocol to do it.
There're two common ways to send/receive TCP requests:
Splitter, you choose a splitter. For example, we choose 32 bits AABBCCDD as the splitter (same as you choose END_DATA string), but keep in mind it's binary data. Then you have to ensure that the data in request doesn't contains the splitter. To do that, you have to encode the request. For example we can encode request as base64, then use the character which isn't included in base64 table as the splitter.
Prefix length, the above method has its overhead as we have to encode request data. The prefix length method is a better choice.
We can prefix the length of request before.
The pseudocode:
// use Int32, 4 bytes to indicate the length of message after it
-------------- client side ----------------
client.send(theFirstMessage.length) // Int32
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage.length)
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
var buffer = Buffer()
socket.onReceived(data => {
buffer.append(data)
let length = Int32(buffer[0...3])
if (buffer.length >= length + 4) {
let theRequest = buffer[4 ... 4 + length - 1]
process(theRequest)
buffer = buffer.dropFirst(4 + length)
}
})
One more thing, when working with TCP socket, it's just stream of bytes, so the endianness is important https://en.wikipedia.org/wiki/Endianness
For example, an android device is little endian and server side (or other android device) is big endian. Then 4 bytes of Int32 from the android device, when received on server side, it will be decoded wrongly if you don't care about it.
So, the prefix length should be encoded by specific endianness.

Stream video frame Android-to-android

I currently work on an app where I use the phone camera and open CV to process the frames. Now I thought it would be cool to be able to send the frames to another Android client. I thought frame by frame with steamer could work, but don't know how to setup the host and if it's not efficient. Any suggestions?

If you just want to send each frame as a raw set of data you can use sockets.
This code below is old now but it worked fine when last tested - it sends an entire video but you can use the same to send whatever file you want:
//Send the video file to helper over a Socket connection so he helper can compress the video file
Socket helperSocket = null;
try {
Log.d("VideoChunkDistributeTask doInBackground","connecting to: " + helperIPAddress + ":" + helperPort);
helperSocket = new Socket(helperIPAddress, helperPort);
BufferedOutputStream helperSocketBOS = new BufferedOutputStream(helperSocket.getOutputStream());
byte[] buffer = new byte[4096];
//Write the video chunk to the output stream
//Open the file
File videoChunkFile = new File(videoChunkFileName);
BufferedInputStream chunkFileIS = new BufferedInputStream(new FileInputStream(videoChunkFile));
//First send a long with the file length - wrap the BufferedOutputStream in a DataOuputStream to
//allow us send a long directly
DataOutputStream helperSocketDOS = new DataOutputStream(
new BufferedOutputStream(helperSocket.getOutputStream()));
long chunkLength = videoChunkFile.length();
helperSocketDOS.writeLong(chunkLength);
Log.d("VideoChunkDistributeTask doInBackground","chunkLength: " + chunkLength);
//Now loop through the video chunk file sending it to the helper via the socket - note this will simply
//do nothing if the file is empty
int readCount = 0;
int totalReadCount = 0;
while(totalReadCount < chunkLength) {
//write the buffer to the output stream of the socket
readCount = chunkFileIS.read(buffer);
helperSocketDOS.write(buffer, 0, readCount);
totalReadCount += readCount;
}
Log.d("VideoChunkDistributeTask doInBackground","file sent");
chunkFileIS.close();
helperSocketDOS.flush();
} catch (UnknownHostException e) {
Log.d("VideoChunkDistributeTask doInBackground","unknown host");
e.printStackTrace();
return null;
} catch (IOException e) {
Log.d("VideoChunkDistributeTask doInBackground","IO exceptiont");
e.printStackTrace();
return null;
}
The full source code is at: https://github.com/mickod/ColabAndroid/tree/master/src/com/amodtech/colabandroid
You may also find there are more up to date socket libraries available which might be better for you to use, but the general principles should be similar.
If you want to stream your video so that the other app can play it like a regular video it streams from the web, then you would want to set up a web server on the 'sending' device. At this point it might be easier to send it to a server and stream from there instead.

What is the use to send spitting/chunk files to server

I have to upload big video files to a server, but it's taking too long to upload, so I decided to split/chunk the files and then send them to the server
After splitting my files, I get a response like the following:
[ /storage/emulated/0/1493357699.mp4.001, /storage/emulated/0/1493357699.mp4.002, /storage/emulated/0/1493357699.mp4.003, /storage/emulated/0/1493357699.mp4.004, /storage/emulated/0/1493357699.mp4.005, /storage/emulated/0/1493357699.mp4.006, /storage/emulated/0/1493357699.mp4.007, /storage/emulated/0/1493357699.mp4.008 ]
My thought is what is the use to upload spitting/chunk file to server?
My code for splitting files:
public static List<File> splitFile(File f) {
try {
int partCounter = 1;
List<File> result = new ArrayList<>();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles];
// create a buffer of bytes sized as the one chunk size
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
String name = f.getName();
int tmp = 0;
while ((tmp = bis.read(buffer)) > 0) {
File newFile = new File(f.getParent(), name + "." + String.format("%03d", partCounter++));
// naming files as <inputFileName>.001, <inputFileName>.002, ...
FileOutputStream out = new FileOutputStream(newFile);
out.write(buffer, 0, tmp);//tmp is chunk size. Need it for the last chunk,
// which could be less then 1 mb.
result.add(newFile);
}
return result;
} catch (Throwable throwable) {
throwable.printStackTrace();
}
return null;
}

I have implemented in one of my projects. I see two primary reasons:
To achieve multi-threaded / multiple connection for uploading chunks. You can upload multiple chunks at the same time.
Stop/Resume uploading of rest of the chunks if either of the chunk fails to upload (depending on server response)

Retrofit2 Streaming GET method OutOfMemoryError

My interface:
public interface Downloader {
#Streaming #GET Call<ResponseBody> get(#Url final String url);
}
My download method:
private void download(final String url, final File zip) throws IOException {
AsyncHelper.assertNotMainThread();
final Call<ResponseBody> call = downloaderInstance.get(url);
final Response<ResponseBody> response = call.execute();
if (!response.isSuccessful()) {
downloadFailed(response.code());
return;
}
final ResponseBody body = response.body();
downloadedBytes = 0;
totalBytes = body.contentLength();
final InputStream in;
final OutputStream fout;
in = body().byteStream();
fout = new FileOutputStream(zip);
int read;
while ((read = in.read(BUFFER)) != -1) {
fout.write(BUFFER, 0, read);
downloadedBytes += read;
LOG.d("wrote %d bytes (total: %d)", downloadedBytes, totalBytes);
}
body.close();
fout.close();
}
This whole thing runs on a background thread and I don't get any log lines with "wrote x bytes (total: y)" in the log, which tells me that it's not streaming.
In a previous implementation I was running this stream directly into a ZipInputStream to unzip on the fly, I split the process into download and decompress steps thinking that the ZipInputStream might be the problem. The process fails during download, which means once I fix this I can revert back to unzip on the fly.
What am I doing wrong?
I've also tried final InputStream in = response.body().source().inputStream(); with the same results. I've also tried enqueue instead of execute with the same results.

So, this is embarrassing, but I'll leave it here in case others come across a similar scenario.
The answer to my question is: "Nothing". I'm doing everything right. At least according to the above code.
What I were doing wrong though was that for debugging purposes I had logging level set to BODY, which means it has to put the whole thing in a buffer. That's also why no incremental progress was being reported.

URLConnection.getContentLength() returns -1

I have a URL which, when I enter in browser, opens the image perfectly. But when I try the following code, I get getContentLength() as -1:
URL url = new URL(imageUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// determine the image size and allocate a buffer
int fileSize = connection.getContentLength();
Please guide me what can be the reason behind this?

If the server is sending down the response using Chunked Transfer Encoding, you will not be able to pre-calculate the size. The response is streamed, and you'll just have to allocate a buffer to store the image until the stream is complete. Note that you should only do this if you can guarantee that the image is small enough to fit into memory. Streaming the response to flash storage is a pretty reasonable option if the image may be large.
In-memory solution:
private static final int READ_SIZE = 16384;
byte[] imageBuf;
if (-1 == contentLength) {
byte[] buf = new byte[READ_SIZE];
int bufferLeft = buf.length;
int offset = 0;
int result = 0;
outer: do {
while (bufferLeft > 0) {
result = is.read(buf, offset, bufferLeft);
if (result < 0) {
// we're done
break outer;
}
offset += result;
bufferLeft -= result;
}
// resize
bufferLeft = READ_SIZE;
int newSize = buf.length + READ_SIZE;
byte[] newBuf = new byte[newSize];
System.arraycopy(buf, 0, newBuf, 0, buf.length);
buf = newBuf;
} while (true);
imageBuf = new byte[offset];
System.arraycopy(buf, 0, imageBuf, 0, offset);
} else { // download using the simple method
In theory, if the Http client presents itself as HTTP 1.0, most servers will switch back to non-streaming mode, but I don't believe this is a possibility for URLConnection.

I am late here but this might help someone. I was facing same issue i was always getting -1 value, when ever i was trying get the content length.
previously i was using below method to get content length.
long totalByte=connection.getContentLength();
Below fixed my problem:-
long totalByte=connection.getHeaderFieldLong("Content-Length",-1);