I am converting a Java desktop project to Android. Part of it includes a TCP connection to a server and parsing a long text from the server to the client (the Android application). This is the code that I have for the desktop project that I also try to use in Android:
// Method is called when a button is tapped
public void tapButton() {
// Create a message to the server that requests for the Departure navdata
String messageToServer = someMethodToMakeHandshakeMessage();
// Connect to the server
if (!messageToServer.equals("")) {
String finalMessageToServer = messageToServer;
new Thread(() -> {
String navdata = connectClient(finalMessageToServer);
getActivity().runOnUiThread(() -> messageReceived(navdata));
// I am also using messageReceived(navdata) without runOnUiThread with the same result
}).start();
}
}
public String connectClient(String messageOut) {
Socket socket = null;
DataInputStream input = null;
DataOutputStream output = null;
BufferedReader br = null;
// Final message from the server
String data = "";
// Message from the server that should terminate TCP connection
String terminator = "END_DATA";
try {
// Create socket and streams
socket = new Socket(someIPAddress, somePort);
input = new DataInputStream(socket.getInputStream());
output = new DataOutputStream(socket.getOutputStream());
//Send message to the server
output.writeBytes(messageOut);
//Read Response
br = new BufferedReader(new InputStreamReader(socket.getInputStream()));
StringBuilder sb = new StringBuilder();
String s = "";
int value = 0;
// Process the message from the server and add to the StringBuilder
while((value = br.read()) != -1) {
// converts int to character
char c = (char)value;
sb.append(c);
if(sb.toString().contains(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
catch (UnknownHostException e) {
// Dealing with exception
}
catch (EOFException e) {
// Dealing with exception
}
catch (IOException e) {
// Dealing with exception
}
finally {
try {
if(socket!=null) { socket.close();}
if(input != null) { input.close();}
if(output != null) { output.close();}
if(br != null) { br.close();}
}
catch (IOException ex) {
// Dealing with exception
}
socket = null;
input = null;
output = null;
br = null;
}
return data;
}
public void messageReceived(String message) {
// Method to deal with received data
}
Whereas the code works fine in the desktop Java application, I have problems with Android (using an emulator). The text is not sent in full length and is cut somewhere in the middle (only 20-50% received by the client; the number of parsed characters differs all the time). Besides, I have noticed that it is taking too long to connect to the server, but, I guess, this is due to working with an emulator.
Should a TCP client receiving long texts from the server be implemented in Android somewhat differently?
EDIT: Implemented the following code using a suggestion by #blackapps:
String line = br.readLine();
while (line != null) {
sb.append(line);
line = br.readLine();
if (line.trim().isEmpty()) {
Log.i("EMPTY LINE>>>>>>>>>>>>>>>>>",line);
}
if(line.equals(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
Two issues. I would like to keep the empty lines in the received text. The terminator is not detected. I think, it is separated from the main text with two empty lines. However, after the first empty line, it goes to indefinite loop and connection never terminated.
EDIT #2.
After having spent several hours trying to figure out what is going on, making changes to the server, and comparing the number of bytes sent and received, I have noticed that this is not the problem with the code. It appears that the client receives the full text. The problem is with how the text is written in the console using the Log.i(String, String) method. I have added the good old System.out.println() in the code, and the whole text was shown in the console. However, the text from Log.i() was cut off in the middle. As this is my first experience with Android Studio, what the heck is going on?
Thanks a lot!
Let talk about TCP socket first.
When talking about TCP socket, it's a stream of data.
TCP views data as an unstructured, but ordered, stream of bytes. It's different from the kinds of socket.io.
From time to time, TCP will grab chunks of data from the send buffer and pass the data to the network layer. The maximum amount of data that can be grabbed and placed in a segment is limited by the maximum segment size (MSS). The MSS is typically set by first determining the length of the largest link-layer frame.
So it depends on the device.
For example, you have two messages, each of them has 1000 bytes data, and you call:
-------------- client side ----------------
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
socket.onReceived(data => {
// process(data)
})
With above pseudocode you should note that:
The data which received and called on onReceived block couldn't be 1000 bytes of theFirstMessage.
It could be first 400 bytes, then on other event you receive 400 bytes, then more 400 bytes (200 of the first one and 200 of the second one).
It could be 1200 bytes (1000 of the first one and 200 of the second one).
TCP views data as an unstructured, but ordered, stream of bytes. Socket.io is a wrapper, when it uses TCP socket, it collect and combine/split the data for you, so that you received the events with exactly the data was sent from other side.
When you work with TCP, you have to do it your self, you have to define the application protocol to do it.
There're two common ways to send/receive TCP requests:
Splitter, you choose a splitter. For example, we choose 32 bits AABBCCDD as the splitter (same as you choose END_DATA string), but keep in mind it's binary data. Then you have to ensure that the data in request doesn't contains the splitter. To do that, you have to encode the request. For example we can encode request as base64, then use the character which isn't included in base64 table as the splitter.
Prefix length, the above method has its overhead as we have to encode request data. The prefix length method is a better choice.
We can prefix the length of request before.
The pseudocode:
// use Int32, 4 bytes to indicate the length of message after it
-------------- client side ----------------
client.send(theFirstMessage.length) // Int32
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage.length)
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
var buffer = Buffer()
socket.onReceived(data => {
buffer.append(data)
let length = Int32(buffer[0...3])
if (buffer.length >= length + 4) {
let theRequest = buffer[4 ... 4 + length - 1]
process(theRequest)
buffer = buffer.dropFirst(4 + length)
}
})
One more thing, when working with TCP socket, it's just stream of bytes, so the endianness is important https://en.wikipedia.org/wiki/Endianness
For example, an android device is little endian and server side (or other android device) is big endian. Then 4 bytes of Int32 from the android device, when received on server side, it will be decoded wrongly if you don't care about it.
So, the prefix length should be encoded by specific endianness.
Related
I am trying to create a chat application between Android and a Windows 10 device.
I have successfully sent text from Android using DataOutputStream and read it in Windows 10 using a data reader class.
My problem is Android is not able to recognize the text from Windows. It displays the result of the datainputstream.available() function but the application hangs in case I use the readString() or the readbyte() function.
Code in Android for receiving:
DataInputStream dIn = new DataInputStream(clientSocket.getInputStream());
if(dIn.available()>0)
{
int length = dIn.readInt(); // app hangs in here
byte[] byteReceived = new byte[length];
dIn.readFully(byteReceived, 0 , length); // sometimes app hangs here
String textReceived = new String(byteReceived);
text.setText(Client Says: "+ textReceived + "\n");//
}
Data sent from Windows through datawriter:
DataWriter writer = new DataWriter(socket.OutputStream))
{
writer.UnicodeEncoding=windows.Storage.Streams.UnicodeEncoding.Utf8;
writer.ByteOrder = windows.Storage.Streams.ByteOrder.LittleEndian;
uint size =writer.MeasureString(message);
writer.WriteUint32(size);
writer.WriteString(message);
try
{
await writer.StoreAsync();
}
catch (Exception exception)
{
switch (SocketError.GetStatus(exception.HResult))
{
case SocketErrorStatus.HostNotFound:
// Handle HostNotFound Error
throw;
default:
throw;
}
}
await writer.FlushAsync();
writer.DetachStream();
}
What is the issue here?
Your dIn.readFully expects bytes and not String. Moreover, it expects the exact number of bytes, as length variable. You need to create bytes from String on the windows size and send the length of byte array as Int in first transaction. Then you need to transfer this byte array unchanged in second transaction. Try it.
I am trying to create a client-server android app in which I want to transfer a file using a UDP protocol. Till now I am able to transfer the file and receive the acknowledgements for the packets.
Now I want to add the sequence numbers to the with the data in the packet. I have tried to do the following:
Create a ByteArrayOutputStream.
Wrap it in an ObjectOutputStream
Write data to the object using writeObject()
Serialized class includes:
public class Message implements Serializable {
private int seqNo;
private byte[] data;
private boolean ack;
public Message(int seqNo, byte[] data, boolean ack) {
this.seqNo = seqNo;
this.data = data;
this.ack = ack;
}
Client Side
byte[] fileBytes = new byte[500];
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(outStream);
while((numBytesRead = inputBuf.read(fileBytes)) != -1) {
//DatagramPacket packet = new DatagramPacket(fileBytes, fileBytes.length);
if (os == null) {
os = new ObjectOutputStream(outStream);
}
Message msg = new Message(++seqNo, fileBytes, false);
os.writeObject(msg);
os.flush();
os.reset();
byte[] data = outStream.toByteArray();
atagramPacket packet = new DatagramPacket(data, data.length);
clientSocket.send(packet);
}
Server Side
byte[] incomingData = new byte[1024];
while (true) {
try{
DatagramPacket incomingPacket = new DatagramPacket(incomingData, incomingData.length);
serverSocket.receive(incomingPacket);
byte[] data = incomingPacket.getData();
ByteArrayInputStream in = new ByteArrayInputStream(data);
ObjectInputStream is = new ObjectInputStream(in);
if (is == null) {
is = new ObjectInputStream(in);
}
Message msg = (Message) is.readObject();
System.out.println(msg.getSeqNo());
out.write(msg.getData(),0,msg.getData().length);
}
The problem that I am facing is
I am receiving the same sequence number for each packet (i.e. 1)
I am not sure about the buffer size for the incoming packet, as I am using 500 bytes at Client side and 1024 at
the Sever. And if I take 500 bytes at both the sides I get an EOFexception.
I would really appreciate if you could suggest better ways to implement the same thing! Thanks :)
Message msg = new Message(++seqNo, fileBytes, false);
Here you are assuming that the prior read() filled the buffer. On the last read() before end of file it almost certainly won't, and it isn't guaranteed to fill it any time, only to transfer at least one byte.
You should be passing the read count 'numBytes' to this constructor, and it should create a byte array of that size, and copy only that many bytes into it.
Other issues:
It is impossible for 'os' to be null at the point you're testing it.
Ditto 'is'.
You should be creating a new ObjectOutputStream and ByteArrayOutputStream per datagram.
Java Datagrams keep shrinking to the size of the shortest datagram payload received so far. You must either create a new one per receive, or at least reset its length before each receive.
you need a larger buffer at the receiver because of ObjectOutputStream overheads.
I don't believe this code presently works at all, let alone that you keep getting the same sequence number. More likely you keep getting the same message, because you're ignoring an exception somewhere.
So I am facing a problem from a while now . Any suggestion would be good.
First I used my code to receive data from arduino , then I used the bluetoothChat and changed the uuid , I can pair , everything is good , but if I send an entire string from arduino to android I get only parts of that string.
If I use bluetooth terminal from google play everything is ok, and on the description it says it is made from the bluetooth Chat sample .
Code Arduino
#include <SoftwareSerial.h>
SoftwareSerial mySerial(10, 9); //RX,TX
long int i = 0;
void setup(){
mySerial.begin(9600);
}
void loop(){
mySerial.print("This is a message n. ");
mySerial.println(i);
i++;
delay(100);
}
Android code : Bluetooth Chat Sample
Exemple of message received on Android:
Message to be sent!
So first messages I think are waiting while the module is paired .
because every time I get .
is is a message n. 466
This is a message n.467
.
. ( here I get correct messages )
.
This is a message n.470
message n. 495
.
.
and after the first messages I get messages like
ssage n.534
t
essage n.m
essage n.
535
( I neved again get an entire message )
Handler :
h = new Handler() {
public void handleMessage(Message msg) {
switch (msg.what) {
case RECIEVE_MESSAGE: // if receive massage
byte[] readBuf = (byte[]) msg.obj;
String strIncom = new String(readBuf, 0, msg.arg1); // create string from bytes array
sb.append(strIncom); // append string
int endOfLineIndex = sb.indexOf("\r\n"); // determine the end-of-line
if (endOfLineIndex > 0) { // if end-of-line,
String sbprint = sb.substring(0, endOfLineIndex); // extract string
sb.delete(0, sb.length()); // and clear
Log.d("Arduino", "Mesaj:"+ sbprint.toString());
}
Log.d("Arduino", "...Mesaj:"+ sb.toString() + " Byte:" + msg.arg1 + "...");
break;
}
};
};
Listener to InputStream
public void run() {
byte[] buffer = new byte[256]; // buffer store for the stream
int bytes; // bytes returned from read()
// Keep listening to the InputStream until an exception occurs
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer); // Get number of bytes and message in "buffer"
h.obtainMessage(RECIEVE_MESSAGE, bytes, -1, buffer).sendToTarget(); // Send to message queue Handler
} catch (IOException e) {
break;
}
}
}
note you are using a software emulation of serial port, hence timing is not as good as it would be with a hardware UART.
It is likely one or both of the following two possible issues:
1) the start and stop bit are not properly timed, causing back to back bytes. Which occur when a string is set, rather then pecking in keys one at a time.
The solution would be to space out each key.
2) baud rates do not match with in tolerance. Either SLOWING DOWN or SPEEDING UP the baud rate on both the HC05 and Arduino will better match the timing.
I would also recommend ensuring your library is SoftwareSerial, states that it is NewSoftSerial. It has many issues fixed. It was implemented in to the Arduino IDE 1.0.+ core libraries, so if you have recent IDE you should have it.
I am trying to post a large video (nearly 1 GB).
I am using FTP to send video to a server, but the upload stops after a while. On the server the video crashes, but I am able to upload a smaller sized video.
I've also used HTTP to send video to the server, sent as a Base64 enoded string, but there is an out of memory exception while encoding.
I've tried to upload the video as a file, but without success. What is the best way to upload a large video to a server?
Use HTTP POST, and post content as Form-based File Upload (mime type: multipart/form-data). This system is standard on web for sending forms and/or uploading files.
Use HTTP chunked post mode, so that size doesn't need to be known beforehand, and you can stream any file in small parts. You still have to make some code on server (e.g. in PHP) to accept the file and do what is needed.
Use HttpURLConnection to initiate connection. Then use my attached class to send the data. It will create proper headers, etc, and you'll use it as OutputStream to write your raw data to it, then call close, and you're done. You can overrite its onHandleResult to handle resulting error code.
public class FormDataWriter extends FilterOutputStream{
private final HttpURLConnection con;
/**
* #param formName name of form in which data are sent
* #param fileName
* #param fileSize size of file, or -1 to use chunked encoding
*/
FormDataWriter(HttpURLConnection con, String formName, String fileName, long fileSize) throws IOException{
super(null);
this.con = con;
con.setDoOutput(true);
String boundary = generateBoundary();
con.setRequestProperty(HTTP.CONTENT_TYPE, "multipart/form-data; charset=UTF-8; boundary="+boundary);
{
StringBuilder sb = new StringBuilder();
writePartHeader(boundary, formName, fileName==null ? null : "filename=\""+fileName+"\"",
"application/octet-stream", sb);
headerBytes = sb.toString().getBytes("UTF-8");
sb = new StringBuilder();
sb.append("\r\n");
sb.append("--"+boundary+"--\r\n");
footerBytes = sb.toString().getBytes();
}
if(fileSize!=-1) {
fileSize += headerBytes.length + footerBytes.length;
con.setFixedLengthStreamingMode((int)fileSize);
}else
con.setChunkedStreamingMode(0x4000);
out = con.getOutputStream();
}
private byte[] headerBytes, footerBytes;
private String generateBoundary() {
StringBuilder sb = new StringBuilder();
Random rand = new Random();
int count = rand.nextInt(11) + 30;
int N = 10+26+26;
for(int i=0; i<count; i++) {
int r = rand.nextInt(N);
sb.append((char)(r<10 ? '0'+r : r<36 ? 'a'+r-10 : 'A'+r-36));
}
return sb.toString();
}
private void writePartHeader(String boundary, String name, String extraContentDispositions, String contentType, StringBuilder sb) {
sb.append("--"+boundary+"\r\n");
sb.append("Content-Disposition: form-data; charset=UTF-8; name=\""+name+"\"");
if(extraContentDispositions!=null)
sb.append("; ").append(extraContentDispositions);
sb.append("\r\n");
if(contentType!=null)
sb.append("Content-Type: "+contentType+"\r\n");
sb.append("\r\n");
}
#Override
public void write(byte[] buffer, int offset, int length) throws IOException{
if(headerBytes!=null) {
out.write(headerBytes);
headerBytes = null;
}
out.write(buffer, offset, length);
}
#Override
public void close() throws IOException{
flush();
if(footerBytes!=null) {
out.write(footerBytes);
footerBytes = null;
}
super.close();
int code = con.getResponseCode();
onHandleResult(code);
}
protected void onHandleResult(int code) throws IOException{
if(code!=200 && code!=201)
throw new IOException("Upload error code: "+code);
}
}
I guess it failed because of a timeout by the big size.
Since
Small size video uploaded successfully
, My suggestion is
Split one big file to several small files.
Upload one by one or several together based on the condition of network.
Join all of parts (after all of those uploaded successfully) at server.
Because of small size, Re-upload failed part will be easy.
Just a theroy.
This site may help .
Added 08.01.2013
It has been a while, don't know if you still need this. Anyway, I wrote some simple codes implement the theory above, because of interest mainly.
Split one big file to several small files. Read the big file into several small parts.
ByteBuffer bb = ByteBuffer.allocate(partSize);
int bytesRead = fc.read(bb);
if (bytesRead == -1) {
break;
}
byte[] bytes = bb.array();
parts.put(new Part(createFileName(fileName, i), bytes));
Upload one by one or several together based on the condition of network.
Part part = parts.take();
if (part == Part.NULL) {
parts.add(Part.NULL);// notify others to stop.
break;
} else {
uploader.upload(part);
}
Join all of parts (after all of those uploaded successfully) at server.
Because it is via HTTP, so it can be in any language, such as Java, PHP, Python, etc. Here is a java example.
...
try (FileOutputStream dest = new FileOutputStream(destFile, true)) {
FileChannel dc = dest.getChannel();// the final big file.
for (long i = 0; i < count; i++) {
File partFile = new File(destFileName + "." + i);// every small parts.
if (!partFile.exists()) {
break;
}
try (FileInputStream part = new FileInputStream(partFile)) {
FileChannel pc = part.getChannel();
pc.transferTo(0, pc.size(), dc);// combine.
}
partFile.delete();
}
statusCode = OK;// set ok at last.
} catch (Exception e) {
log.error("combine failed.", e);
}
I put all codes on GitHub. And made a Android example too.
Please have a look if you still need.
private HttpsURLConnection conn = null;
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setChunkedStreamingMode(1024);
I'm trying to transfer about a megabyte of arbitrary data at a time from one android phone to another. Currently, I write the size, a command code and the data to a DataOutputStream around a BufferedOutputStream, around the OutputStream returned from bluetoothSocketInstance.getOutputStream().
The receiving phone reads the size and command code and then reads from the input stream until it has gotten all the data it is expecting. This works for short strings, but for larger files not all the data is transferred. Running the app in the debugger shows that the write returns without any exceptions and the read reads a fraction of the bytes expected and then blocks indefinitely. It also does not throw any exceptions.
Is there a buffer somewhere that is filling up? Is there something else I need to do to ensure that all the data gets transferred?
My code for the sender and receiver are below:
Sender:
try {
DataOutputStream d = new DataOutputStream(new BufferedOutputStream(mmOutStream,buffer.length+8));
//int b= buffer.length;
d.writeInt(buffer.length);
d.writeInt(command);
d.write(buffer);
d.flush();
} catch (IOException e) {
Log.e(TAG, "Exception during write", e);
}
}
Receiver:
try {
// Read from the InputStream
int messageSize= inStream.readInt();
int messageCode = inStream.readInt();
bytes=0;
buffer =new byte[messageSize];
while(bytes < messageSize)
{
bytes += inStream.read(buffer,bytes,messageSize - bytes);
}
message = bytes;
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
After some more testing on my end, I changed my sending code to look like this:
for(int i=0; i<buffer.length;i+=BIG_NUM)
{
int b = ((i+BIG_NUM) < buffer.length) ? BIG_NUM: buffer.length - i;
d.write(buffer,i,b);
d.flush();
}
The files now get sent. Does anyone have an idea why? Does the call to flush() block until the data has actually been transferred? Is there any documentation about the size of the send and receive buffers that would help me to decide how large I can safely make BIG_NUM?
I have similar problem, when sending file there are some parts missing. I try BufferedOutputStream but problem still exist.
Finally i find simple solution:
You don't need to send buffer length, just split sending buffer to byte array (for example [8192]) and on receive side make sure that this buffer is much bigger about 4 or 8 times than sending buffer. This worked for me and file is sent completed.