can anybody confirm what are the currently allowed methods for peer-to-peer communications within the Android framework? I need to transfer json strings and I'm currently using SMS which works ok but the problem is that the data also ends up as lots of text messages. I've read Reto Meier's first edition of Professional Android Application Development where he says that the data transfer options were not implemented due to security concerns.
Has this changed at all and how would you do peer-to-peer transfer of data?
Have you looked at Qualcomm's AllJoyn library? It is designed to work over Bluetooth or wifi, so may fit, though if you're connecting over 3G or wider range networks it won't work.
Given the variation and reliability of networks between two remote devices not on the same network, I'd question whether peer-to-peer is the best solution, and would venture to suggest considering using an application server in between, so you could then use Cloud to Device Messaging [deprecated] (perhaps in tandem with Google App Engine). i.e. to send a message the sender passes it to the server, and the server then passes it on to the recipient.
In theory all devices on the net have a unique IP address and can talk to each other, but it's rarely that simple as routers/firewalls are configured differently so you'd need to pay great attention to the ports you use, especially considering many inbound ports are blocked by default for security reasons.
You can simply use UDP/TCP sockets. In a separate thread you set up the server-side listener socket and that's it. Of course your application has to be started first (or should run in the background all the time). Here's an example:
http://thinkandroid.wordpress.com/2010/03/27/incorporating-socket-programming-into-your-applications/
If you also need peer discovery that will make the thing more difficult.
You should also take a look at peerdroid, an open source project available here. I've been looking in to peer communication options from the point of view of having a collection of federated devices (pre-paired, if you like, similar to Bluetooth pairing); this library looks like it may give you the foundation for what you are trying to do.
If you are on your own network (for example a home or office WiFi) then you should be able to query for other connected devices. If the network you are on is not under your control (the mobile network or a public wifi) then the network will have been configured to isolate each device from everything else. In this case you will have no choice but to put a server up to act as the man in the middle. That brings its own architectural compromises - every device has to regularly poll the server or keep a connection open - unless you use Google App Engine which supports push notifications over Google's own infrastructure.
Thanks for your answer ldx but I would need peer discovery as you indicated. Some further research appears to indicate XMPP as a suitable technology and there are now some services on offer, although these appear to be aimed at 'server' to client notifications. There is a good discussion here on XMPP and some more here although it would appear that there are still some issues to deal with such as polling v push, long-running open http connections and battery life. Xtify looks promising, especially their web service. I hope this provides suitable information to others looking at the topic of peer-to-peer data communication.
Related
I've always used GCM for push notifications in android apps, but for security constraint i'm obliged to avoid GCM and use local push server, i've found a lot of answers for this question but none of them is well structured, i hope to get a clear answer for step by step process to use some XMPP server and client libraries for this topic
You may not fully understand what is GCM and what GCM was designed for.
Traditionally, each desktop applications establish its own TCP connection to its own server, just like you can connect to your Openfire instance and send/receive messages when you want without any third-party services.
This approach have many issues in mobile environment:
mobile devices are resource limited: when you have many TCP connections open it will drain your battery fast
mobile devices are not have persistent network connection: you should care about cellular and/or wifi network changes (and it also have battery impact) and you should not lose any messages in these unstable environment.
To solve these problems, Apple, Google (and also their minor competitors) provide "notification services". They are consists of:
System-wide background application ("service") which keeps connection to Apple/Google "notifications" infrastructure servers.
API for developers, to "register" their applications both on server and client side: your server-side application send all events to Apple/Google, and you client app receive "pushes" from that system service, you no need to care about network outages and battery impacts - all issues are handled by OS manufacturer.
But when you are trying to "avoid GCM" you getting all things to be cared by you, plus:
Your "handmade" persistent network service will interfere with Google's one, and will at least double network and battery usage (or even more, just because your implementation will have errors/other corner cases which are already where solved by Apple/Google big engineering teams)
Your application will dramatically more complex, and you will have much more security issues (as security issues are now handled by you, too) and poor user experience (applications with "own" networking are often have bad UX design, prompting and boring end user with network error messages, for example).
So, to sum up:
Yes, technically, you can take smack library and connect to Openfire server. Basically it the same as GCM works.
No, in 99.9999% of cases you should not go this way. Unless you are a big company which develops competing mobile platform.
Is it possible to set up the Android Bluetooth Chat sample app to connect more than one person at a time, and have a mini chat room? What would that entail?
tl;dr version: Bluetooth sucks for this, don't use it, use wifi instead, probably backed by a web backend.
I have investigated this issue thoroughly throughout the years in the interests of a social wireless network research project. My general advice is: it doesn't work with more than two / three people. Bluetooth just isn't designed with wireless peer to peer networks in mind.
In general, it seems that the cheap Bluetooth controllers included on Android devices (especially HTC's devices, iirc) don't really handle any more than two or three connections at a time. I'm unsure if this is a hardware or firmware problem, but I can recount some basic anecdotes. I was working to implement this idea at the SDK level (i.e., without firmware modifications) around the beginning of 2011, and was able to get a peer to get two additional connections (i.e., three devices, each connecting to the other two) to work for a period of a few minutes to an hour before the connections would suddenly die and the socket would become unusable, requiring reconnection. Unfortunately, 20 minutes was an upper bound, and generally it was impossible to get connections to more than one other device at all reliably.
The goal of the project was to support multiple people interacting with each other silently in the background, but this never materialized, instead we ditched Bluetooth and went with wifi instead, which worked much much better. In the abstract, I think people view Bluetooth as a possible medium for reliable peer to peer communication, but it wasn't really designed that way: it's more of a medium used for short range communication between small devices (think headsets).
Be aware that if you want to do this, the maximum number of devices to which you can connect is fixed, because as per the Bluetooth spec, a piconet supports a maximum of seven devices. (See the wikipedia article.)
The required change is simple: you use a different UUID for each device. This can be implemented a number of ways, using an out of band exchange mechanism, or simple scheme where you assign UUIDs in an increasing fashion and when connecting to the network, try each in succession.
Here are some relevant Google groups threads:
Bluetooth peer to peer networks
Multiple connections on Android Bluetooth
I remember posting a more elaborate one detailing how to do this (with code) that I might dig up as well.., if I can find it. It should be from late 2010 or early 2011.
So the answer is, in the abstract, yes, you can try to do this, by using multiple UUIDs (after you use one, that's it, and you have to try another using some assignment protocol). However, in practice, after a lot of trial and error, this doesn't really work for what you probably want to use it for, and it's a lot better to go with an internet backend instead. By the way, this is also good for another reason, most users don't really like to turn on their Bluetooth for fear of their battery being drained..
Leaving this here, in case it helps someone else.
I was able to make my custom chat room following official bluetooth tutorial and modifying it a little.
Unfortunately, I cannot provide most of my code, but main idea is:
Every device is acts both as server and as a client. When Chat is started, device starts its server thread. Server thread is the same as official but doesn't ends when accept connection. It just keep listening.
Client thread is identical as in tutorial.
Both server and client thread manages connection same. I created separated threads for accepting messages following this tutorial and one for sending them.
private void manageConnectedSocket(BluetoothSocket socket) {
//create thread responsible for sending messages.
SendingThread w = new SendingThread(socket);
MainActivity.addSendingThread(w);
//Creates listener for messages to accept.
MainActivity.addListener(socket);
}
Now in main activity always when user click send button, for each worker (sending thread) send message to remote device. Listening is running asynchronously.
IMPORTANT:
You need to handle exceptions when message send fails and remove sending and recieving thread for device when you detect it is disconected. In my case I used well known UUID "00001101-0000-1000-8000-00805f9b34fb". For every device.
You need to wait 3 second between atempts to connect as client because some devices has weak bluetooth hardware and it is refusing connect as client.
Bt connection is supporting up to 7 -10 connections. So you will be limited in that range. I think it is designed for extensions of main device and not for random comunication
Source: search "bluetooth programming" on google
I'm building a multi-OS mirroring system which I would like to implement using a hybrid client-server and p2p communication method (at least that's the best way I have of describing it).
My issue is that at some point I have a central server (appengine, so there are limitations to what I could do because of time and networking capability constraints) that would need to get a message to a host of different devices which are not necessarily running the same OS (Windows, Android, iOS, Linux, etc...).
Android and iOS (or any other mobile platform) are the main problems it looks like I will be having on 2 levels.
1 - They are both limited by battery power (more so than a laptop and desktops shouldn't have that issue at all), so whichever method I use needs to take that into account.
2 - NAT (harder because the user has relatively less control over their firewall than on a network that they are running). My central server will maintain a table of which device has what IP address, but from what I understand if there is NAT or a firewall it won't be able to get to it if the port was not forwarded.
Since I will be writing a specific client for each OS I prefer a solution that is more universal. I have been leaning towards writing an extremely simple HTTP server that sits on each client and takes requests (which appengine is able to send) and treats them as messages that alert the client to perform an action (either with the server or another client). However, I run into the issue of NAT/firewall. For instance if appengine needs to send a message to AndroidDevice1 it would grab its IP address from a table and make a request to it. However this doesn't work if the ports aren't forwarded correctly, and if the user is on 3g/4g the firewall is controlled by the data provider.
Because of this, I started thinking about using Android C2DM but I want a solution I could implement across platforms.
The only other method I could think of is to just have the client poll the server for messages. This has the battery and network consumption issue though.
Would there be any other way to implement this, and if not, which one of the above methods are best in terms of balancing usability, power and data consumption and user input (the less the user has to do to get the client set up (ie port forwarding, etc...) the better)? Please note that I do not intend for this to become a discussion/flame war but a logical presentation of facts.
Thanks in advance!
You can create a persistent TCP connection from the device to the server and then communicate over this open connection. This would be a very simple connection with keepalive packets for the most part.
In theory this would consume some battery through the radio, but in practice I have experienced that the battery is not affected much at all. Key is to keep the communication over this line to a minimum.
If AppEngine does not allow this approach, you can run your own socket server and then communicate between this server and the appengine server using REST. A socket server I have used is Apache MINA and had no issues with scalability.
Another problem you will have with this approach or any other approach is that on iOS (afaik) you cannot keep a tcp socket open when the App goes into background. The only thing to communicate with the iOS device is Apple Push Notification Service
I would prefer rather than having HTTP Connection you should create TCP/IP tunnel and make communication fast and reliable. I have one Chat application which runs perfact for me using TCP/IP. If you use this you will have same logic for multiple platforms. Only thing you need to write is different code for iOS and android.
Hi all I was wondering what options do we have to exchange data between two different android devices?
For example, User-A and User-B both installs my app. I would like User-A to send data (possibly just a simple message or user-A's location info) to User-B.
The functionality I would need is similar to the functionality that WhatsApp has. However unlike WhatsApp, I do not have a server and I was wondering if we could do data exchange between two different android devices without a server?
I was thinking we build it atop SMS or something.
Options for exchanging information between devices are the following:
Bluetooth - this would be between two devices in the near vicinity
TCP/UDP IP connection - this would be using TCP to open a socket directly to another server socket. That could be hosted on the phone or a shared server. There are pros and cons to both.
The pros of bluetooth would be no need for a central server. The big downside is this means you can only exchange data between two people standing within 20 meter range. The other downside is you have to pair the devices which not everyone finds easiest.
You can use TCP/IP connections to exchange data just like any client-server program you write on a traditional computer. This could be used no matter if your phone is using 3G/4G/WIFI/EDGE or future radio protocols. The problem is the IP address of the phone might not be globally reachable. The IP address of the phone might be a non-routable like a private IP. They might be behind a firewall or NAT address.
This is where a central server is probably needed to either exchange IP addresses for users, or serve as a common location for clients behind infrastructure that could block. This is where protocols like SWIFT come in handy for jumping firewalls. Even with things like P2P you still run into these types of issues with non-accessible devices, and tricks like this have to be used to crawl around them. Unfortunately, that means you probably need a central server even with the P2P model.
Without an external server to keep a list of all connected clients, you would need to implement communication in a P2P fashion. Depending on the needs of your app, you could have the user type in the IP address/email/phone number of the other user they want to exchange data with.
If you wish to use a server approach, you can sign up for Google's App Engine which has good Eclipse integration as well as a plugin to easily interface with an Android app. This would give you an infrastructure option without initially (or maybe never depending on how high you scale) having to put down any money.
Google gave a good IO talk showing an example of a web app that can easily communicate with an Android app. You could extend this to do what you are looking to do.
I'm building native mobile applications in both iOS and Android. These apps require "realtime" updates from and to the server, same as any other network-based application does (Facebook, Twitter, social games like Words with Friends, etc)
I think using HTTP long polling for this is over kill in the sense that long polling can be detrimental to battery life, especially with a lot of TCP setup/teardown. It might make sense to have the mobile applications use persistent TCP sockets to establish a connection to the server, and send RPC style commands to the server for all web service communication. This ofcourse, would require a server to handle the long-lived TCP connection and be able to speak to a web service once it makes sense of the data passed down the TCP pipe. I'm thinking of passing data in plain text using JSON or XML.
Perhaps an Erlang based RPC server would do well for a network based application like this. It would allow for the mobile apps to send and receive data from the server all over one connection without multiple setup/teardown that individual HTTP requests would do using something like NSURLConnection on iOS. Since no web browser isn't involved, we don't need to deal with the nuances of HTTP at the mobile client level. A lot of these "COMET" and long-polling/streaming servers are built with HTTP in mind. I'm thinking just using a plain-text protocol over TCP is good enough, will make the client more responsive, allow for receiving of updates from the server, and preserve battery life over the traditional long polling and streaming models.
Does anyone currently do this with their native iOS or Android app? Did you write your own server or is there something open sourced out there that I can begin working with today instead of reinventing the wheel? Is there any reason why using just a TCP based RPC service is a worse decision than using HTTP?
I also looked into HTTP pipelining, but it doesn't look to be worth the trouble when it comes to implementing it on the clients. Also, I'm not sure if it would allow for bi-directional communication in the client<->server communication channel.
Any insight would be greatly appreciated.
Using TCP sockets with your own protocol rolled down is quite better than HTTP especially with the nature of resources on the mobile devices. Erlang will do quite well, however lets start from your protocol. Erlang excels well at this especially with the Bit Syntax expressions. However still, you could use plain text as you wish. JSON (would need a parser: Mochijson2.erl found in Mochiweb library) and XML (will need a parser: Erlsom).
I have personally worked on a project in which we were using raw TCP Sockets with our Erlang Servers and Mobile Devices. However, depending on the Port numbers you choose, Routers along the way would block/Drop packets depending on the security policies of service providers. However, i still think that HTTP can work. People chat on Facebook Mobile, send Twits e.t.c from their devices and am sure these social engines use some kind of Long Polling or Server Push or whatever but using HTTP. The mobile devices have advanced in capability of late.
Rolling your own TCP Based protocol comes with a number of challenges: Port selection, Parsing of data both at the client and server, Security issues e.t.c. Using HTTP will let you think of the actual problem than spending time correcting protocol issues at client or server. The Devices you've mentioned above like Android and IOS (Ipad, Iphone e.t.c) are very capable of handling HTTP COMET (Long polling). Am sure when you follow the standards for Web Applications on Mobile devices as well as these W3C Mobile Web Best Practices, your app will function well using HTTP.
Using HTTP methods will quicken the work and there are a lot of libraries on the SDKs of these Devices which would assist you prototype the solution you want as compared to the situation of rolling your own TCP-based plain text protocol. To back up this reasoning, look through these W3C findings.
Let me finally talk of the HTTP benefits on these Devices. If you are to use Web technologies for Mobile devices, such as Opera Widgets, Phone Gap, Sencha Touch, and JQuery Mobile, their SDKs and Libraries have Optimizations already done for you or have well documented ways in which your app can be made efficient. Further still, these technologies have the APIs to access the native Devices' resources like Battery check, SMS, MMS, GSM broadcast channels, Contacts, Lighting, GPS , and Memory; all as APIs in the JavaScript classes. It would become hard (inflexible) if you use native programming languages like J2ME, Mobile Python or Symbian C++ / Qt as compared to using Web technologies like CSS3, HTML5 and JavaScript tools mentioned above. Using the Web tools mentioned above will make your app easily distributable by say Ovi Store or Apple Store, from experience.
Take note that if you use HTTP, testing will be easy. All you need is a public Domain so the Widgets on the mobile device locates your servers over the Internet. If you role your own TCP/IP protocol, the Network Routers may be disruptive against the Port number you use unless you plan on using port 80 or another well known port, but then still your Server IP would have to be made Public. There is a short cut to this: if you put your TCP Server behind the same ISP as your testing Mobile's Internet connection, the ISP routers will see both source and destination as behind its Network. But all in all, there are challenges with rolling your own protocol.
Edit: Using HTTP, you will benefit from REST. Web Servers implemented in Erlang (especially Yaws and Mochiweb) excel at REST services. Look at this article: RESTFUL services with Yaws. For mochiweb, there is an interesting article about: A million User comet application using Mochiweb which is broken into 3 parts. Further still, you could look at the solution given to this question.
There are ZeroMQ builds for android and iOS. Java and ObjC bindings exist as well.
HTTP was created for infrequent requests with large responses. It is highly inefficient for transferring very big amounts of small data chunks. In typical situation, http headers can be twice in size of actual payload. The only strong side of HTTP is its habitualness, its 'One size fits all' karma.
If you want lightweight and fast solution, I guess ZeroMQ can be a perfect solution.
One reason to go with HTTP instead of a custom service is that it's widely supported on a transport level.
With mobile devices, a user might be on Wi-Fi at a hotel, airport, coffee shop, or corporate LAN. In some cases this means having to connect via proxy. Your application's users will be happiest if the application is able to use the device's proxy settings to connect. This provides the least surprise -- if web browsing works, then the application should work also.
HTTP is simple enough that it isn't difficult to write a server that will accept HTTP requests from a custom client. If you decide to go this route, the best solution is the one that you don't have to support. If you can write something in Erlang that is supportive of application changes, then it sounds like a reasonable solution. If you're not comfortable doing so then PHP or J2EE gets bonus points for the availability of cheap labor.
While HTTP does benefit from being widely supported, some successful projects are based on other protocols. The Sipdroid developers found that persistent TCP connections do greatly improve battery life. Their article on the topic doesn't address the server side but it does give a high-level description of their approach on the client.
Erlang is very well suited for your use case. I'd prefer using TCP over HTTP for the sake of saving battery life on the phone as you noted already.
Generally getting the communication between device and server up and running will be very easy. The protocol which you are using between the two is what will require most work. However writing protocols in Erlang is strikingly straight forward when using gen_fsm
You should checkout metajack's talk at the Erlang Factory which highlights his solution to a very similar use case for his iPhone game Snack Words.
I work on a application that connects to a Microsoft http server with long lived http/https connections to mobile devices to allow for push type data to be sent to the mobile. It works but there are lots of little gotcha's on the mobile side.
For the client to get 'packets' of data, we put the http connection into Chucked Encoding mode so that each packet is in one chucked packet.
Not all native http API services on each mobile will support calling you back when a 'chuck' of data has arrived, on the ones that don't normally wait until all the data from the server has arrived before calling the application back with the data. Platforms that support callbacks with partial data are (that I have found):
Symbian
Windows Mobile
Platforms that don't support partial data callbacks:
IOS
Blackberry
For the platforms that don't support partial callbacks, we have written our own http connection code with chucked encoding support using the native sock support. It's actually not very hard.
Don't rely on the fact that one chuck is one of your packets, http proxies or the native http api implementations may break that assumption.
On IOS with this background multitasking rules, means you can't keep this connection going while your application is in the background. You really need to use Apples Push Notification service and live by it's limitations.
Never trust mobile cellular networks, I have seen the weirdest stuff going on like the server side seeing the http connection drop and then reconnect (and replay of the original http request) while on the mobile end you don't see any drop in the connection. Basically treat the connection as unreliable where data can go missing. We ended up implementing a 'tcp' like sequence number scheme to ensure we didn't lose data.
Using http/https makes it easier to get past firewall rules on customer sites.
I'm not sure using http/https long-lived connections was the wisest decision we ever made, but it was made long before I turned up so I have to live with the fall-out of it.
As a alterative, we are looking at web sockets as well, but with the web-socket spec in the state of flux atm and generally being not to good to follow, I don't know if it will work out or not.
So that is my experience with using http/https as a long-lived realtime connection.
Your milage may vary.
It all depends on what data you are sending - the size of it, the criticality of timeliness, frequency of update etc.
If you are looking for a reasonably lazy update and verbose data (JSON say) then go with a HTTP comet pattern, as you will find it much easier to navigate standard network gear as other answers have highlighted. If you are behind a corporate firewall/proxy for example, http will be a much safer bet.
However, if you are doing fast things with small data sizes then go with something homegrown and leverage a TCP connection. It's much more to the point and you'll find the performance in real terms much better. Simple data structures and use fast operators to slice you data up as you need it.
Again as other posters have noted, battery usage is a big concern. You will eat a battery by literally burning a hole in your pocket if you are not careful. It is very easy to turn a battery that lasts 2 days into one that will last 6hours.
Lastly, don't trust the network if you are time sensitive. If you are not then a long poll over HTTP will be just fine for you. But if you are looking for high performance messaging, then be acutely aware that a mobile network is not an end-to-end TCP connection. Your requests will varying in trip time and latency.
So back to what you want to do with the app. As you are building for iOS (native obviously dictated) and Andriod, I would leverage Apple Push Services and their notification framework. Build you back end services to talk to that and also provide interfaces for non-apple devices (i.e. http or tcp level listeners). That way one platform and multiple 'gateways' for your apps. You can then do RIM via their push service too if you wanted to.