Develop Reference

TCP client for Android: text is not received in full

I am converting a Java desktop project to Android. Part of it includes a TCP connection to a server and parsing a long text from the server to the client (the Android application). This is the code that I have for the desktop project that I also try to use in Android:
// Method is called when a button is tapped
public void tapButton() {
// Create a message to the server that requests for the Departure navdata
String messageToServer = someMethodToMakeHandshakeMessage();
// Connect to the server
if (!messageToServer.equals("")) {
String finalMessageToServer = messageToServer;
new Thread(() -> {
String navdata = connectClient(finalMessageToServer);
getActivity().runOnUiThread(() -> messageReceived(navdata));
// I am also using messageReceived(navdata) without runOnUiThread with the same result
}).start();
}
}
public String connectClient(String messageOut) {
Socket socket = null;
DataInputStream input = null;
DataOutputStream output = null;
BufferedReader br = null;
// Final message from the server
String data = "";
// Message from the server that should terminate TCP connection
String terminator = "END_DATA";
try {
// Create socket and streams
socket = new Socket(someIPAddress, somePort);
input = new DataInputStream(socket.getInputStream());
output = new DataOutputStream(socket.getOutputStream());
//Send message to the server
output.writeBytes(messageOut);
//Read Response
br = new BufferedReader(new InputStreamReader(socket.getInputStream()));
StringBuilder sb = new StringBuilder();
String s = "";
int value = 0;
// Process the message from the server and add to the StringBuilder
while((value = br.read()) != -1) {
// converts int to character
char c = (char)value;
sb.append(c);
if(sb.toString().contains(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
catch (UnknownHostException e) {
// Dealing with exception
}
catch (EOFException e) {
// Dealing with exception
}
catch (IOException e) {
// Dealing with exception
}
finally {
try {
if(socket!=null) { socket.close();}
if(input != null) { input.close();}
if(output != null) { output.close();}
if(br != null) { br.close();}
}
catch (IOException ex) {
// Dealing with exception
}
socket = null;
input = null;
output = null;
br = null;
}
return data;
}
public void messageReceived(String message) {
// Method to deal with received data
}
Whereas the code works fine in the desktop Java application, I have problems with Android (using an emulator). The text is not sent in full length and is cut somewhere in the middle (only 20-50% received by the client; the number of parsed characters differs all the time). Besides, I have noticed that it is taking too long to connect to the server, but, I guess, this is due to working with an emulator.
Should a TCP client receiving long texts from the server be implemented in Android somewhat differently?
EDIT: Implemented the following code using a suggestion by #blackapps:
String line = br.readLine();
while (line != null) {
sb.append(line);
line = br.readLine();
if (line.trim().isEmpty()) {
Log.i("EMPTY LINE>>>>>>>>>>>>>>>>>",line);
}
if(line.equals(terminator)) {
break;
}
}
// Create the final string
data = sb.toString();
}
Two issues. I would like to keep the empty lines in the received text. The terminator is not detected. I think, it is separated from the main text with two empty lines. However, after the first empty line, it goes to indefinite loop and connection never terminated.
EDIT #2.
After having spent several hours trying to figure out what is going on, making changes to the server, and comparing the number of bytes sent and received, I have noticed that this is not the problem with the code. It appears that the client receives the full text. The problem is with how the text is written in the console using the Log.i(String, String) method. I have added the good old System.out.println() in the code, and the whole text was shown in the console. However, the text from Log.i() was cut off in the middle. As this is my first experience with Android Studio, what the heck is going on?
Thanks a lot!

Let talk about TCP socket first.
When talking about TCP socket, it's a stream of data.
TCP views data as an unstructured, but ordered, stream of bytes. It's different from the kinds of socket.io.
From time to time, TCP will grab chunks of data from the send buffer and pass the data to the network layer. The maximum amount of data that can be grabbed and placed in a segment is limited by the maximum segment size (MSS). The MSS is typically set by first determining the length of the largest link-layer frame.
So it depends on the device.
For example, you have two messages, each of them has 1000 bytes data, and you call:
-------------- client side ----------------
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
socket.onReceived(data => {
// process(data)
})
With above pseudocode you should note that:
The data which received and called on onReceived block couldn't be 1000 bytes of theFirstMessage.
It could be first 400 bytes, then on other event you receive 400 bytes, then more 400 bytes (200 of the first one and 200 of the second one).
It could be 1200 bytes (1000 of the first one and 200 of the second one).
TCP views data as an unstructured, but ordered, stream of bytes. Socket.io is a wrapper, when it uses TCP socket, it collect and combine/split the data for you, so that you received the events with exactly the data was sent from other side.
When you work with TCP, you have to do it your self, you have to define the application protocol to do it.
There're two common ways to send/receive TCP requests:
Splitter, you choose a splitter. For example, we choose 32 bits AABBCCDD as the splitter (same as you choose END_DATA string), but keep in mind it's binary data. Then you have to ensure that the data in request doesn't contains the splitter. To do that, you have to encode the request. For example we can encode request as base64, then use the character which isn't included in base64 table as the splitter.
Prefix length, the above method has its overhead as we have to encode request data. The prefix length method is a better choice.
We can prefix the length of request before.
The pseudocode:
// use Int32, 4 bytes to indicate the length of message after it
-------------- client side ----------------
client.send(theFirstMessage.length) // Int32
client.send(theFirstMessage) // 1000 bytes
client.send(theSecondMessage.length)
client.send(theSecondMessage) // 1000 bytes
-------------- server side -----------------
var buffer = Buffer()
socket.onReceived(data => {
buffer.append(data)
let length = Int32(buffer[0...3])
if (buffer.length >= length + 4) {
let theRequest = buffer[4 ... 4 + length - 1]
process(theRequest)
buffer = buffer.dropFirst(4 + length)
}
})
One more thing, when working with TCP socket, it's just stream of bytes, so the endianness is important https://en.wikipedia.org/wiki/Endianness
For example, an android device is little endian and server side (or other android device) is big endian. Then 4 bytes of Int32 from the android device, when received on server side, it will be decoded wrongly if you don't care about it.
So, the prefix length should be encoded by specific endianness.

Sending multiple files - end of each file in the byte[] buffer

I am sending files between 2 devices, so I established a socket communication. Right now, I am just trying to send one file, but in the future I want to send multiple files (selected by the user from a gridview).
The problem is that when I send one file, on the server side (that receives the file) the socket.getInputStream().read(buffer) does not detect the end of the file. It just waits for "more" data to be sent.
After searching a bit on this issue, I reached some topics that kind of gave me some options, but I am still not satisfied with it because I dont know if those options would be efficient to send multiple files. This is an example : How to identify end of InputStream in java
I could close the socket or the stream objects after sending a file, but if I want to send a lot of files, it wouldn't be efficient to be always closing and opening the sockets.
Code on the receiver :
File apkReceived = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS) + "/testeReceiveServerComm.apk");
byte[] buffer = new byte [8192];
FileOutputStream fos=new FileOutputStream(apkReceived);
int count=0;
int total=0;//so apra ir vendo quanto recebi.
while((count = in.read(buffer)) != -1){
fos.write(buffer,0,count);
total+=count;
System.out.println("Server Comm receive thread - already received this ammount : "+total);
}
Code on the client (sender) :
File apkToSend=new File(filePath);
byte[] buffer = new byte [8192];
BufferedInputStream bis=new BufferedInputStream(new FileInputStream(apkToSend));
int count;
int total=0;
while((count=bis.read(buffer))!=-1){
out.write(buffer,0,count);
total+=count;
out.reset();
System.out.println("send thread - already sent this ammount : "+total);
}
out.flush();
bis.close();

DataInputStream hangs on reading

I am trying to create a chat application between Android and a Windows 10 device.
I have successfully sent text from Android using DataOutputStream and read it in Windows 10 using a data reader class.
My problem is Android is not able to recognize the text from Windows. It displays the result of the datainputstream.available() function but the application hangs in case I use the readString() or the readbyte() function.
Code in Android for receiving:
DataInputStream dIn = new DataInputStream(clientSocket.getInputStream());
if(dIn.available()>0)
{
int length = dIn.readInt(); // app hangs in here
byte[] byteReceived = new byte[length];
dIn.readFully(byteReceived, 0 , length); // sometimes app hangs here
String textReceived = new String(byteReceived);
text.setText(Client Says: "+ textReceived + "\n");//
}
Data sent from Windows through datawriter:
DataWriter writer = new DataWriter(socket.OutputStream))
{
writer.UnicodeEncoding=windows.Storage.Streams.UnicodeEncoding.Utf8;
writer.ByteOrder = windows.Storage.Streams.ByteOrder.LittleEndian;
uint size =writer.MeasureString(message);
writer.WriteUint32(size);
writer.WriteString(message);
try
{
await writer.StoreAsync();
}
catch (Exception exception)
{
switch (SocketError.GetStatus(exception.HResult))
{
case SocketErrorStatus.HostNotFound:
// Handle HostNotFound Error
throw;
default:
throw;
}
}
await writer.FlushAsync();
writer.DetachStream();
}
What is the issue here?

Your dIn.readFully expects bytes and not String. Moreover, it expects the exact number of bytes, as length variable. You need to create bytes from String on the windows size and send the length of byte array as Int in first transaction. Then you need to transfer this byte array unchanged in second transaction. Try it.

CryptographicException: Bad PKCS7 padding

I am seeing a small percentage of production users randomly report this exception related to encrypting/decrypting strings with Xamarin.Android but unfortunately I cannot reproduce it.
What could cause this and/or how could I reproduce the exception so that I can figure out a fix/workaround?
[CryptographicException: Bad PKCS7 padding. Invalid length 147.]
Mono.Security.Cryptography.SymmetricTransform.ThrowBadPaddingException(PaddingMode padding, Int32 length, Int32 position):0
Mono.Security.Cryptography.SymmetricTransform.FinalDecrypt(System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount):0
Mono.Security.Cryptography.SymmetricTransform.TransformFinalBlock(System.Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount):0
System.Security.Cryptography.CryptoStream.FlushFinalBlock():0
com.abc.mobile.shared.Security+PasswordEncoder.DecryptWithByteArray(System.String strText, System.String strEncrypt):0
EDIT: Here's the code I am using to encrypt/decrypt
private string EncryptWithByteArray(string inPassword, string inByteArray)
{
byte[] tmpKey = new byte[20];
tmpKey = System.Text.Encoding.UTF8.GetBytes(inByteArray.Substring(0, 8));
DESCryptoServiceProvider des = new DESCryptoServiceProvider();
byte[] inputArray = System.Text.Encoding.UTF8.GetBytes(inPassword);
MemoryStream ms = new MemoryStream();
CryptoStream cs = new CryptoStream(ms, des.CreateEncryptor(tmpKey, mInitializationVector), CryptoStreamMode.Write);
cs.Write(inputArray, 0, inputArray.Length);
cs.FlushFinalBlock();
return Convert.ToBase64String(ms.ToArray());
}
private string DecryptWithByteArray (string strText, string strEncrypt)
{
try
{
byte[] tmpKey = new byte[20];
tmpKey = System.Text.Encoding.UTF8.GetBytes (strEncrypt.Substring (0, 8));
DESCryptoServiceProvider des = new DESCryptoServiceProvider ();
Byte[] inputByteArray = Convert.FromBase64String (strText);
MemoryStream ms = new MemoryStream ();
CryptoStream cs = new CryptoStream (ms, des.CreateDecryptor (tmpKey, mInitializationVector), CryptoStreamMode.Write);
cs.Write (inputByteArray, 0, inputByteArray.Length);
try {
cs.FlushFinalBlock();
} catch (Exception ex) {
throw(ex);
}
System.Text.Encoding encoding = System.Text.Encoding.UTF8;
return encoding.GetString(ms.ToArray());
}
catch (Exception ex)
{
throw ex;
}
}
EDIT 2:
The encryption key is always the local Device ID. Here's how I am getting this:
TelephonyManager telephonyMgr = Application.Context.GetSystemService(Context.TelephonyService) as TelephonyManager;
string deviceId = telephonyMgr.DeviceId == null ? "UNAVAILABLE" : telephonyMgr.DeviceId;
Here's an example of how it's called:
string mByteArray = GetDeviceId();
string mEncryptedString = EncryptWithByteArray(stringToEncrypt, mByteArray);
string mDecryptedString = DecryptWithByteArray(mEncryptedString, mByteArray);

You have not provided much details about your use case but I would say this is happening because you are not using the same cipher settings during the encryption and decryption operations. Symmetric ciphers require you to use exactly the same settings/parameters during the data encryption and also decryption. For example for AES CBC you would need to use exactly the same key, IV, cipher mode and padding on both devices. It is best to set these setting explicitly in the code:
System.Security.Cryptography.RijndaelManaged aes = new System.Security.Cryptography.RijndaelManaged();
aes.Key = new byte[] { ... };
aes.IV = new byte[] { ... };
aes.Mode = CipherMode.CBC;
aes.Padding = PaddingMode.PKCS7;
If you are sure you are using the same settings then you should also consider scenario that some data get corrupted or altered during the network transfer.
Edit after some code fragments have been provided:
Decryption method you have provided does not work for me at all so I have put together all your samples and turned them into the code which does the same thing as yours but uses IMO a slightly cleaner approach. For example this code uses more robust "key derivation" (please forgive me cryptoguys) and it has also passed basic code analysis.
You should be able to easily use public methods to do what you need:
string plainData = "This information should be encrypted";
string encryptedData = EncryptStringified(plainData);
string decryptedData = DecryptStringified(encryptedData);
if (plainData != decryptedData)
throw new Exception("Decryption failed");
Implementation and private methods follows:
/// <summary>
/// Encrypts string with the key derived from device ID
/// </summary>
/// <returns>Base64 encoded encrypted data</returns>
/// <param name="stringToEncrypt">String to encrypt</param>
public string EncryptStringified(string stringToEncrypt)
{
if (stringToEncrypt == null)
throw new ArgumentNullException("stringToEncrypt");
byte[] key = DeviceIdToDesKey();
byte[] plainData = Encoding.UTF8.GetBytes(stringToEncrypt);
byte[] encryptedData = Encrypt(key, plainData);
return Convert.ToBase64String(encryptedData);
}
/// <summary>
/// Decrypts Base64 encoded data with the key derived from device ID
/// </summary>
/// <returns>Decrypted string</returns>
/// <param name="b64DataToDecrypt">Base64 encoded data to decrypt</param>
public string DecryptStringified(string b64DataToDecrypt)
{
if (b64DataToDecrypt == null)
throw new ArgumentNullException("b64DataToDecrypt");
byte[] key = DeviceIdToDesKey();
byte[] encryptedData = Convert.FromBase64String(b64DataToDecrypt);
byte[] decryptedData = Decrypt(key, encryptedData);
return Encoding.UTF8.GetString(decryptedData);
}
private byte[] DeviceIdToDesKey()
{
TelephonyManager telephonyMgr = Application.Context.GetSystemService(Context.TelephonyService) as TelephonyManager;
string deviceId = telephonyMgr.DeviceId ?? "UNAVAILABLE";
// Compute hash of device ID so we are sure enough bytes have been gathered for the key
byte[] bytes = null;
using (SHA1 sha1 = SHA1.Create())
bytes = sha1.ComputeHash(Encoding.UTF8.GetBytes(deviceId));
// Get last 8 bytes from device ID hash as a key
byte[] desKey = new byte[8];
Array.Copy(bytes, bytes.Length - desKey.Length, desKey, 0, desKey.Length);
return desKey;
}
private byte[] Encrypt(byte[] key, byte[] plainData)
{
if (key == null)
throw new ArgumentNullException("key");
if (plainData == null)
throw new ArgumentNullException("plainData");
using (DESCryptoServiceProvider desProvider = new DESCryptoServiceProvider())
{
if (!desProvider.ValidKeySize(key.Length * 8))
throw new CryptographicException("Key with invalid size has been specified");
desProvider.Key = key;
// desProvider.IV should be automatically filled with random bytes when DESCryptoServiceProvider instance is created
desProvider.Mode = CipherMode.CBC;
desProvider.Padding = PaddingMode.PKCS7;
using (MemoryStream encryptedStream = new MemoryStream())
{
// Write IV at the beginning of memory stream
encryptedStream.Write(desProvider.IV, 0, desProvider.IV.Length);
// Perform encryption and append encrypted data to the memory stream
using (ICryptoTransform encryptor = desProvider.CreateEncryptor())
{
byte[] encryptedData = encryptor.TransformFinalBlock(plainData, 0, plainData.Length);
encryptedStream.Write(encryptedData, 0, encryptedData.Length);
}
return encryptedStream.ToArray();
}
}
}
private byte[] Decrypt(byte[] key, byte[] encryptedData)
{
if (key == null)
throw new ArgumentNullException("key");
if (encryptedData == null)
throw new ArgumentNullException("encryptedData");
using (DESCryptoServiceProvider desProvider = new DESCryptoServiceProvider())
{
if (!desProvider.ValidKeySize(key.Length * 8))
throw new CryptographicException("Key with invalid size has been specified");
desProvider.Key = key;
if (encryptedData.Length <= desProvider.IV.Length)
throw new CryptographicException("Too short encrypted data has been specified");
// Read IV from the beginning of encrypted data
// Note: New byte array needs to be created because data written to desprovider.IV are ignored
byte[] iv = new byte[desProvider.IV.Length];
Array.Copy(encryptedData, 0, iv, 0, iv.Length);
desProvider.IV = iv;
desProvider.Mode = CipherMode.CBC;
desProvider.Padding = PaddingMode.PKCS7;
// Remove IV from the beginning of encrypted data and perform decryption
using (ICryptoTransform decryptor = desProvider.CreateDecryptor())
return decryptor.TransformFinalBlock(encryptedData, desProvider.IV.Length, encryptedData.Length - desProvider.IV.Length);
}
}
It is really hard to tell what exactly was problem with your code because your decryption method did not work for me at all - most likely because it is using CryptoStream in write mode for decryption which seems a little odd to me.
So much for the code. Now let's get to encryption which is really really weak. It is more just an obfuscation that should protect the data from being accidentally displayed in plain text form (some people use BASE64 encoding for the same thing). The main cause of this is relatively old encryption algorithm and easily predictable encryption key. AFAIK every application running on the same device can read device ID without any privileges. That means any application can decrypt your data. Of course your SQLite database is probably accessible only to your application but that can no longer be true if you remove the SD card or root your phone. To make this a little better you could for example ask user to provide a password and then use it to derive unique encryption key but that is completely different problem. Anyway I am not really sure what you are trying to achieve with this encryption - it may be fully sufficient for your needs even if it can be considered to be weak.
Hope this helps.

Converting int to byte[] in android

I am using the write() method in order to write in a file of the external storage. This method only accepts byte[] as an input. I have tried passing a String and I get an error message ("The method write(int) in the type FileOutputStream is not applicable for the arguments String"). If I pass an int, I don't get error but in the file nothing is written. The value I get from calling getNumSentPackets() is an int and I need to convert it to byte[]. I have been looking at other questions already answered here and I have tried the ByteBuffer option but the result I get in the file is not what I want, this means, I don't get the number of sent packets. Can anybody help me, please?
This is my code:
public void createFile(String name) {
try {
String filename = name;
File myFile = new File(Environment.getExternalStorageDirectory(), filename);
if (!myFile.exists())
myFile.createNewFile();
String title = "FLOODING RESULTS FILE\n\n";
String sent = "Number of sent packets\n";
FileOutputStream fos;
byte[] data = title.getBytes();
byte[] intSent = sent.getBytes();
int numSent = mSender.getNumSentPackets();
byte[] numSentBytes = ByteBuffer.allocate(10).putInt(numSent).array();
try{
fos = new FileOutputStream(myFile);
fos.write(data);
fos.write(intSent);
fos.write(numSentBytes);
fos.flush();
fos.close();
} catch (Exception e) {
e.printStackTrace();
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static int getNumSentPackets() {
return nSentPackets;
}
The expected output file would be as follows:
FLOODING RESULTS FILE
Number of sent packets 200
200 is only an example, meaning with this that I would like to see there a number which would correspond to the total number of sent packets.
Thank you in advance.

As I am a lazy developer, I like to use the existing facilities in my languages of choice, for example, for java, a PrintWriter.
public void createFile(String filename) {
try {
File myFile = new File(Environment.getExternalStorageDirectory(), filename);
PrintWriter out = new PrintWriter(myFile); // this will create the file if necessary
out.println("FLOODING RESULTS FILE");
out.println();
out.print("Number of sent packets ");
out.println(mSender.getNumSentPackets());
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
This is much easier to read and maintain than your current approach, and looks more idiomatic.

ByteBuffer.allocate(capacity).putInt(yourInt).array();

The text representation of "200" requires you to write 3 characters. All files are just a bunch of bytes in the end so there needs to be a mapping from character to some byte value. Assuming ASCII(*) the data to write into the file would be
// '2','0','0'
byte[] textVersion = { 50, 48, 48 }
int on the other hand is a 32bit numeric value, i.e. has 4 bytes and 200 is equivalent to
byte[] intVersion = { 0, 0, 0, 200 }
When using a ByteBuffer, you'll get this. If you write that into a file and a text viewer tries to display that it would display something like ◻◻◻Č if you're lucky. A 0 is actually a non printable control character and therefore often either skipped when printing or replaced with strange looking character like boxes. The 200 would be equivalent to Č in Windows-CP1250. It has no meaning on it's own when interpreted as UTF8 - it's the start of a 2 byte sequence and so the next 2 byte are required to determine which character to display.
You could have used
String.valueOf(200).getBytes( /* you should specify which encoding to use here */ );
which will create the "200" string first, then return you the required bytes for those 3 characters.
You should however use Java's character based IO facility: The numerous (and confusing) Reader & Writer implementations. They all(*^2) wrap an InputStream or OutputStream in the end and do the text to byte conversion for you.
PrintWriter is probably the most convenient to use but not without flaw: https://stackoverflow.com/a/15803472/995891
FileWriter should be avoided because you can't specify the encoding
The longer alternative route would be
BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(file), encoding));
writer.write("Hello ");
writer.write(String.valueOf(200));
writer.newLine();
(*) most encodings are ASCII compatible for the first 127 characters which basically covers normal english text.
(*^2) nothing forces a Writer to output the characters into a stream, e.g. StringWriter. But they are used mostly that way.