I'm using sockets for data transfer from one android phone and PC, I am using DataInputStream.
But it takes a long time in data transfer, about 10 minutes to transfer a 4 MB file.
Could anyone suggest any better way to do that?
I did some changes in my code and now it is taking 15 seconds to read about 1 Mb of data. I want to improve its performance.
My Code is:
InputStream is= socket.getInputStream();
DataInputStream inChannel= new DataInputStream(new BufferedInputStream(in));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int oneByte;
while ((oneByte = inChannel.read()) != -1) {
if (oneByte == 0) {
break;
}
baos.write(oneByte);
byteCount++;
}
byte[] inData = baos.toByteArray();
baos.close();
Are you sure you are not sending empty packets in between there or packets doing something else? If you are using TCP if a packet doesn't reach its destination (a buffer is overfilled with other packets somewhere in between your connection on the router or on one of the devices) the packet will have to be resent. This might be because of your setup. Given the little information you have given us, I can give the the following advice:
Look into more advanced typologies, I am going to assume you are not using any of these:
A send a receive thread; the send thread pushes out packets which you put into a queue:
|P| - packet held inside a Que part of your program
send out thread <-| P | P | P | <-App/program
reveive -> | P | P | P| P -> App/Program deques and analyses data
because your send and receive queuing system works along side each other they are constantly buffering up the received and sent packets. What seems to be happening with your setup (as far as I assume) is that you have a loop which grabs the latest packets and analyses them. This means that the hardware (part of your network hardware) buffer is overfilling with packets which your program is doing other things and by the time your program comes around to collect them, some of them are missing. This causes one side to go "I didn't get those packets, send them again", in other words, your resending the packets you should of sucked up into a Que which are waiting to be dequeued and analysed. The queue can grow regardless of how quickly the program is back to grab that data and do stuff with it (of course you are confined to the RAM). That way you ensure that you have possession of packets that you are supposed to receive rather than rely on your network card/receiver to hold on to them for you, possibly overflowing its buffer.
Another approach is to do a handshake system were one of the programs waits until the last packet is trasmitted, recieved and the other side goes "Cool, send me the next one". This slows down your download/upload speed but is a little bit quicker than packets falling off the end of the end of one of the buffers (each node, such as your router buffers your packets in case more come in than it can handle at a cycle) in the network.
You should utilize a state machine if you can on one or both ends. When your app is downloading the file, lock it into a receive state so its not trying to send/receive other stuff at the same time. Once the download is complete, switch to any other state (say, open file state). If you don't know much about state machines, I recommend you look at the wikipedia article:
http://en.wikipedia.org/wiki/Finite-state_machine
Related
I am working on an android app that has continuous communication between the phone and a local server on the computer through cable and Android Open Accessory connection(AOA), the server(the computer) keeps sending packets of bytes to the android, then android parse them, all kinds of data gets received successfully.
In the implementation I am using BufferedInputStream like this:
val bufferedInputStream = BufferedInputStream(inputStream, 8192) // 8192 is the def buffer size
As I said everything works fine UNTIL android received a packet that is larger than 8192 which is the buffer size, in this case I need to loop until I read the whole packet is read(I know the size of the packets because I send that in the beginning of the packet)
When this kind of packet gets received, the call
bufferedInputStream.read(anyByteArrayOfTheWriteSize)
fails with message "Invalid argument", the annoying thing is that I can read the beginning of the packet which has the size and parse it successfully, so for example it tells android that the size of this packet is for ex 10192 bytes, this means that it needs to read 8192 then will do another iteration and read the remaining 200 bytes.
but it does not, it just fails once it reaches the call of the method read.
I was trying to use inputStream instead of bufferedInputStream but it did not work at all, bufferedInputStream kinda worked like magic, not sure why?! but this is not the topic of the question, just adding this info if incase it is needed.
For receiving UDP broadcast packets from the server to an android device, i used a service class and listen for packets in a thread. It receives the packet successfully. The problem is that if multiple packets are being sent from the server in the same time then packet loss will be the result.
I even tried with a queue and processing the received packets in separate thread then also i am not getting the packet. I am completely new to network programming any help would be widely appreciated
void startListenForUdpBroadcast() {
UDPBroadcastThread = new Thread(new Runnable() {
public void run() {
try {
InetAddress broadcastIP = InetAddress.getByName(UdpConstants.IP_ADDRESS);
Integer port = UdpConstants.RECEIVER_PORT;
while (shouldRestartSocketListen) {
listenAndWaitAndThrowIntent(broadcastIP, port);
}
} catch (Exception e) {
Log.i("UDP", "no longer listening for UDP broadcasts cause of error " + e.getMessage());
}
}
});
UDPBroadcastThread.setPriority(Thread.MAX_PRIORITY); //Setting The Listener thread to MAX PRIORITY to minimize packet loss.
UDPBroadcastThread.start();
}
This code listens for new packets and pushes to queue
private void listenAndWaitAndThrowIntent(InetAddress broadcastIP, Integer port) throws Exception {
byte[] recvBuf = new byte[64000];
if (socket == null || socket.isClosed()) {
socket = new DatagramSocket(port, broadcastIP);
socket.setBroadcast(true);
}
//socket.setSoTimeout(1000);
DatagramPacket packet = new DatagramPacket(recvBuf, recvBuf.length);
socket.receive(packet);
messQueue.add(packet);
}
This checks the queue for new messages and process it
/**
* #purpose Checking queue and If anything is added to the queue then broadcast it to UI
*/
private void checkQueue() {
queueThread = new Thread(new Runnable() {
public void run() {
try {
while (shouldRestartSocketListen) {
if (!messQueue.isEmpty()) {
broadcastIntent(messQueue.poll());
}
}
} catch (Exception e) {
}
}
});
queueThread.start();
}
The problem with UDP is that your sender (your server) does not know you (your android device) missed some. It's not lost because you can't read it fast enough, sometimes just over the air interference/congestion or a busy socket.
The receiver would only know if:
you get an error while processing data since you're missing data
OR your UDP packets are numbered sequentially in its header and you detect a missing number (eg. 1,2,4 - missing 3)
Once the packet is lost, it's lost. You got two options:
implement a resend request: upon detection of a missing packet, the receiver would need to notify the sender to resend that missing packet until it does get it, and your packet processing might be halted until it does
OR be able to ignore it, "hey, we can do it without him", and fill-in with blank data (eg. a bitmap would have some blank pixels, like a broken image)
throttle your sending speed down so the packets wont jam up and get lost
the smaller your packets, the more likely they'll live
(option 1: all this resend requesting is just pseudo-TCP, so you might just consider abandoning UDP and go TCP)
I think your problem is mainly that you use Udp Broadcast over wifi.
Their are two very well documented answers why this is a very slow way to operate and why there are much more packet losts:
answer number one.
answer number two.
The thing I did to solve the extremely slow bandwidth was some kind of multi-unicast protocol:
Manage the list of clients you have connected.
Send each packet you have in your server to all of your clients separately with send call.
This is the code in java:
DatagramPacket packet = new DatagramPacket(buffer, size);
packet.setPort(PORT);
for (byte[] clientAddress : ClientsAddressList) {
packet.setAddress(InetAddress.getByAddress(clientAddress));
transceiverSocket.send(packet);
}
If you receive multiple datagrams in a short burst, your receiver loop may have trouble keeping up, and the OS-level RCVBUF in the socket may overflow (causing the OS to drop a packet it indeed did receive).
You might get better handling of short bursts if you increase the RCVBUF size. Prior to doing this, get an idea of how big it is already via socket.getReceiveBufferSize. Also bear in mind that the number of bytes in the receive buffer must accommodate not just the payload but also the headers and the sk_buff structure that stores packets in the kernel (see, e.g. lxr.free-electrons.com/source/include/linux/…).
You can adjust the recieve buffer size via socket.setReceiveBufferSize - but bear in mind that this message is just a hint, and may be overridden by settings in the kernel (if you request a size bigger than the max size allowable by the current kernel network settings, then you'll only get the max size).
After requesting a bigger receive buffer, you should double check what the kernel has allowed by calling socket.getReceiveBufferSize.
If you have the right permissions, you should be able to tweak the max buffer size the kernel will allow - see e.g. https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html
[ Note that, in general, this will accommodate for short bursts where datagrams arrive faster than your client loop can read them - but if the client loop is chronically slower than datagram delivery, you'll still get occasional drops due to overflow. In this case, you need to speed up your client loop, or slow down the sender.
Also, as otherwise noted in other answers, packets may actually be dropped by your network - and mobile networks especially may be prone to this - so if you absolutely need guaranteed delivery you should use TCP. However, if this were your primary problem, you would expect to see dropped packets even when your server sends them slowly, rather than in a burst.]
I suppose that you are capturing only a single packet by saying
socket.receive(packet);
This is a Blocking I/O call which will wait infinitely until it receives a packet so once first packet arrives it is done waiting and next command executes i.e
messQueue.add(packet);
However when multiple packets are been received you need to continue receiving packets. in your case you just stopped receiving packages after arrival of first package
Note: UDP being a un-reliable protocol doesn't guarantee packet delivery so there might be a case a packet is lost , However this can't be a problem on every run of your program , However a nice way to check whether the packet is hitting your machine and problem is within your application (application is not able to handle the packets recieved) use tcpdump (it's a command-line utility for linux-based OS or MAC) use the following command
sudo tcpdump -i <interface name(one that handles traffic) eth0, eth1 .. en0, en1 (for MAC)> host <ipaddress or hostname of UDP Broadcast server>
Example:
sudo tcpdump -i en1 host 187.156.13.5
(if tcpdump command not found then go forward and install it)
By using this command you will see packets pumping in from destination ump server on terminal if you see more then one packets arriving then you would be sure that packets are arriving at machine , However application falls short to address the packet
it might help
With reference to above explanation by me following changes you can make to make code behave according to requirement
I suppose you can make following changes to make your problem code work instead of creating socket into listenAndWaitAndThrowIntent(InetAddress broadcastIP, Integer port ) method create it in startListenForUdpBroadcast() as follows
socket = new DatagramSocket(port, broadcastIP);
socket.setBroadcast(true);
while (shouldRestartSocketListen) {
listenAndWaitAndThrowIntent(broadcastIP, port, socket);
}
Now you also need to change implementation of listenAndWaitAndThrowIntent method as follows
private void listenAndWaitAndThrowIntent(InetAddress broadcastIP,
Integer port, DatagramSocket socket) throws Exception {
byte[] recvBuf = new byte[64000];
//socket.setSoTimeout(1000);
//change value of condition in for loop to desired number of packets you would like to receive in below example it is 2 so it captures two packets
for (int i=0 ; i <= 2 ; i++ ){
DatagramPacket packet = new DatagramPacket(recvBuf, recvBuf.length);
socket.receive(packet);
messQueue.add(packet);
}
}
Try this it should work !!
it might help
I just try to use the recieve() method of the socket.
When I send the data of a short String,as "send data",for 100
times,the recieve() function performs well.
While I send the data of a long String,as "send data to the
client,send data to the client,send data to the client,send data to
the client,send data to the client,send data to the client,send data
to the client,send data to the client",for 100 times,the recieve()
function cannot performs well.
In android project:
It just recieved about 30 packets, that is, other 70 packets are lost when I use another computer to send pakcets. But I checked the recieve buffer size, it is adequate to contain 100 packets above.
It recieves all the 100 packets when I use the localhost address to test. And I used the capturing tool named wireshark to capture the packets and the capturing tool can capture all the 100 packets everytime.
So I can eliminate the possibility of that another computer did not send all the packets.So I included that the problem exists in the emulator.
The above two cases both happened in android project,so the recieve buffer size is the same,
why in the former it will lose packets??
In java project:It recieves all the 100 packets too.
If you need all the packets then you should use TCP/IP protocol. In UDP protocol you can lose data because it is doesnot support reliable connection like Tcp/ip. UDP is designed to be faster at transferring data at the cost of unreliable connection. Based on network routing the packets you received could also arrive in different order.
Emulator TCP/IP stack's udp buffer size is smaller than the computer. So if the UDP buffer fills up, the emulator TCP/IP stack starts dropping packet. The PC TCP/IP stack pushes data at faster rate which the emulator can't handle. You can try reducing the rate at sender end to avoid filling the emulator buffer or increase the receive socket buffer size of android app to higher value by using the SO_RCVBUF socket option.
I'm in the process of developing an application which reads data from a DAQ that streams its data over Bluetooth. The packet sizes can change, as can the sampling rate (1Hz - 512Hz), and I'm able to loop through and read the data off the device using a buffer.
My question is, how do I correctly process the data when there is such a variable of packet size and sampling rate? How do I determine the buffer size?
Currently I'm simply opening a socket, opening an input stream, and then using a while loop (while the socket is open == true) to read from the stream, and process the data (simple decoding, not an extraneous task).
As an example, there are 23 bytes in a packet, and I have the sampling rate very low at the moment at 1Hz. I have a buffer of 256 bytes, which means that it wont accommodate a full packet at the end of the buffer, and I've written code for it to run over onto the next buffer. Once the data is read grab one packet from the buffer, decode and store it, do the next one, etc.
How should I be streaming, and manipulating the data correctly? Eventually I'll be grabbing something in the region of 44 packets at 512Hz, right at Bluetooths transfer limit, and I want to be able to process it as effective as possible, and display errors when a packet is dropped in the process, etc.
TL;DR: how do I correctly stream data using buffers and/or interrupts.
I need to programmatically write data of say 1 to 100 MB in chunks of 1024 bytes to the remote Bluetooth device. Both are android devices. Here is a sample code snippet in my client program to transfer data –
bTSocket.connect(); //connect to remote BT device
DataOutputStream outStream = new DataOutputStream(bTSocket.getOutputStream());
byte[] buffer = new byte[1024];
int bytesToTransfer = 1000000;
while (bytesToTransfer > 0) {
outStream.write(buffer);
outStream.flush();
bytesToTransfer -= 1024;
}
outStream.close();
While running this piece of code on Android 2.2(Froyo), it works fine. However in case of Android 2.3.4 and 4.0.4, outStream.write(buffer) blocks infinitely after transfer of some data (say of 100 KB). Worth mentioning, the remote device is not configured for listening data. Is there any limitation on the amount of data that can be written?
The Bluetooth socket operates in blocking mode for both reads and writes.
If you fill up the send buffer, then the only thing that .write() can do to stop you trying to send any more data is to block. The alternative to it blocking would be to return an "operation would block!" error code, just like TCP sockets can do when placed in non-blocking mode. But the Bluetooth socket doesn't provide any such non-blocking mode.
You state that the remote Bluetooth device is not reading from its socket. With this being the case, the local sending buffer and remote receive buffer, with each only being of a certain finite size, will eventually fill up. At this point, your .write() operation is going to block until the remote end reads something from its socket. You can't just keep pumping in megabytes of data and expect it to just buffer it all somewhere.
The differences you experience between different Android platforms are probably down to the different amounts of buffer space available in the related Bluetooth stacks.