Develop Reference

Best practices for securing MQTT access from mobile applications?

I would like to create an app (which will be public) that would send and receive some data in real-time to/from a server. Because for this particular use case the PUB/SUB pattern fits naturally, I was thinking about using the MQTT protocol with TLS. What are some best practices for securing access in such cases?
Would it be safe to only have one user with password for the mobile apps registered in the broker (like Mosquitto or RabbitMQ) and use ACLs for limiting client by ID for topics and store these secrets in the mobile apps? If not, what would be a suitable solution? (If using individual user in broker for each app account wouldn't it be a performance issue, given the potential big number of users?)
Are there any other schemes for allowing secure access from the mobile app to the broker?

No, a single username/password is a REALLY bad idea. (All it takes is somebody to disassemble the app and the username/password are compromised and anybody can impersonate anybody else). Also linking clientIds to users probably means users can only have a single device.
Having a user sign in with a username/password should have minimal impact on performance as it will only be checked at connection time.
It also means you can revoke the username/password of a single user and not have to ship a new version of the app.
An alternative to username/password is to issue a certificate to user for them to authenticate with the broker. The CN from the certificate can be used to identify the user.

If you use EMQ X MQTT Broker, the application can send messages to the application server through an HTTP request, the application server will authenticate, record, check, and then invoke REST API of EMQ X Broker to publish the message for the application agent.
The application can be sent only after it has passed the verification, and the sending operation history can be recorded

Maintain persistent login without using SSL encryption for every web service call

First, my sincere apologies for asking a newbie security question. This is my first real foray in the world of online security and I'm kind of lost.
I am in the process of developing an Android app and an accompanying website which both require the users to login to be able to use the services my web servers provide. The services are all written in a session-less fashion, meaning authenticated requests (request requiring user credentials) all need to provide their accompanying security tokens to function and that every authenticated request first has to validate user's credentials.
How I've developed such an architecture is to have the users login using email and password. This information is sent to an authentication server via SSL and an authentication token (an independent hash to the password hash) is provided. This token is then stored on the client (cookies for website and private shared preferences in android). For all future calls, unless the user logs out, this token is valid and can be used to authenticate the user. Each device (different Android devices or web clients) also get their own independent token so that the authentication token is a pair of hashed token + device id.
In addition, I would like to avoid using SSL for every authenticated call. Ideally I would like only the initial authentication (with the email/password) to be encrypted and the rest of the calls to go via HTTP using the authentication token that was obtained when the user signed in. My reason for this avoidance is the triple handshake cost and that maintaining persistent or long lived connections are not preferred.
Not using SSL however leaves me open to a man-in-the-middle attack (MIM). If anyone intercepts the calls and gets a hold of the [device id + authentication token], for all intents and purposes, they will be able to impersonate the user and have access to everything the user can access until the user logs out, at which point the token will be invalidated.
I know my implementation doesn't handle MIM attacks so I was wondering if you could suggest another way to implement this that doesn't include SSL for each and every call and yet avoids MIM attacks.
In short, my requirements are:
Do not maintain sessions on the server
Use SSL only for initial login (email/password pair)
Do not use SSL for subsequent calls that provide authentication token and device id
Somehow avoid MIM attacks if possible (this is the real requirement)
Is it at all possible to have all 4 of these requirements together? Can I avoid using SSL connections and still maintain secure, session-less servers? Where am I going wrong with my implementation and how can I avoid issues with MIM attacks?
Many thanks in advance and apologies if this is a duplicate. I couldn't find the answer anywhere. Perhaps I was searching the wrong thing. If so, please let me know and I'll close/remove the question.

SSL keeps a state at OSI session layer, and it encrypts the whole message using shared key. If you want a cheap solution to prevent MIM and with no server state, the only (low-security) solution I can think of is to use a server-wide global secret and send it to the client during the initial SSL process. The following HTTP roundtrips will have requests/responses encrypted with this shared secret.
But it's simply a badly implemented SSL, and the application level encryption/decryption processes will be probably more costly then the built-in SSL. And you can't encrypt the headers!

Secured RESTful API that can be used by Web App (angular), iOS and Android

I have to lay out a plan to develop a RESTful API (Python/Flask) that could be used by our future web app (Angularjs) and mobile apps (iOS/Android).
I have been researching for three days and have come across several scenarios:
Using HTTPS is one way on top of the methods below to keep it safer. But https is slower, which could mean we need faster and more expensive servers.
Using Basic-Http-Auth and sending username/password in plain (yet https) over the wire for every request to the API.
Using Digest-Auth, which is a hash of the password and the tracking would be automatic This would work for the web app, however I wasn't able to confirm if iPhones and Android would support this natively. If they do, that could be an easy solution!
Using a custom http header, where I would send a custom Auth string in http header upon a successful authentication. But then I have to make sure I am sending this auth code for every request that the user makes. This makes it exactly like 1) with the difference that plain passwords aren't used and the auth code can expire without any risk. Also problematic is the tracking of the auth code, which is no longer automated as in 2)
Using OAuth is an option. But its quite difficult to set up. If there is no better way, maybe thats the only way?
Securing the API like Amazon S3 as described in this great article. In short, he says that both server and client would know of a private key, which they would use to hash the communication. It will be like gangster handshake, that you only would trust the delivery boy, if he knows the gangsta handshake. Further down the comments someone asks:
How to keep the private key “secure” in a pure HTML5 app ?
You are exactly right; in a pure HTML5 (JS/CSS/HTML) app,
there is no protecting the key. You would do all communication over
HTTPS in which case you wouldn’t need a key since you could safely
identify a client using a standard API_KEY or some other friendly
identifier without the need or complexity of an HMAC.
So in other words there is even no point of using the method for an web app in first place. And honestly I don't understand how this should work on the mobile device either. A user downloads our app and how do I send the private key from the iphone to the server? The moment I transferred it, it will be compromised.
The more I am researching the more indecisive I am getting.
I was hoping to ask some pros who have done this previously and could share their experience. Many Thanks

You seem to be confusing/merging two different concepts together. We start of talking about encrypting traffic (HTTPS) and then we start talking about different ways to manage authenticated sessions. In a secure application these are not mutually exclusive tasks. There also seem to potentially be a misunderstanding how session management can impact authentication. Based on that I will provide a primer on web application/web api session management, authentication, and encryption.
Introduction
Session Management
HTTP transactions are stateless by default. HTTP does not specify any method to let your application know that a HTTP request has been sent from a specific user (authenticated or not).
For robust web applications, this is not acceptable. We need a way to associate requests and data made across multiple requests. To do this, on initial request to the server a user needs to be assigned a "session". Generally sessions have some kind of unique id that is sent to the client. The client sends that session id with every request and the server uses the session id sent in every request to properly prepare a response for the user.
It is important to remember that a 'session id' can be called many other things. Some examples of those are: session token, token, etc. For consistency I will use 'session id' for the rest of this response.
Each HTTP request from the client needs to include the session id; this can be done in many ways. Popular examples are:
It can be stored in a cookie - cookies for the current domain are automatically sent on every request.
It can be sent on the URL - each request could send the session id on the URL, not suggested since session ids will stay in the clients history
It can be sent via as a HTTP header - each request would need to specify the header
Most web application frameworks use cookies. However application that rely on JavaScript and single page designs may opt to use a HTTP header/store it in some other location that is observable by the server.
It is very important to remember that the HTTP response that notifies the client of their session id and the client's requests that contain the session id are completely plain text and 100% unsafe. To battle that, all HTTP traffic needs to be encrypted; that is where HTTPS comes in.
It is also important to point out we have not talked about linking a session to a specific user in our system. Session management is just associating data to a specific client accessing our system. The client can be in both authenticated and unauthenticated states, but in both states they generally have a session.
Authentication
Authentication is where we link a session to a specific user in our system. This is generally handled by a login process where a user supplies credentials, those credentials are verified, and then we link a session to a specific user record in our system.
The user is in turn associated with privileges for fine grained access control via access control lists and access control entries (ACL and ACE). This is generally referred to as "Authorization". Most system always have both Authentication and Authorization. In some simple systems all authenticated users are equals in which case you won't have authorization past simple authentication. Further information on this is out of scope for this question, but consider reading about ACE/ACL.
A specific session can be flagged as representing an authenticated user in different ways.
Their session data stored server side could store their user id / some other flag that denotes that the use is authenticated as a specific user
Another user token could be send to the client just like a session id (which over unencrypted HTTP is just as unsafe as sending a session id unencrypted)
Either option is fine. It generally comes down to the technology you are working in and what they offer by default.
A client generally initiates the authentication process. This can be done by sending credentials to a specific url (e.g. yoursite.com/api/login). However if we want to be 'RESTful' we generally would referencing a resource by some noun and doing the action of 'create'. This could be done by requiring a POST of the credentials to yoursite.com/api/authenticatedSession/. Where the idea would be to create an authenticated session. Most sites just POST the credentials to /api/login or the like. This is a departure from "true" or "pure" RESTful ideals, but most people find this a simpler concept rather than thinking of it as "creating an authenticated session".
Encryption
HTTPS is used to encrypt HTTP traffic between a client and server. On a system that relies on authenticated and unauthenticated users, all traffic that relies on a user being authenticated needs to be encrypted via HTTPS; there is no way around this.
The reason for this is that if you authenticate a user, share a secret with them (their session id, etc) and then begin to parade that secret in plain HTTP their session can be hijacked by man-in-the-middle attacks. A hacker will wait for for the traffic to go through an observed network and steal the secret (since its plain text over HTTP) and then initiate a connection to your server pretending to be the original client.
One way people combat this is by associating the requests remote IP address to an authenticated session. This is ineffective alone as any hacker will be able to spoof their requests remote IP address in their fake requests and then observe the responses your sever is sending back. Most would argue that this is not even worth implementing unless you are tracking historical data and using it to identify a specific user's login patterns (like Google does).
If you need to split up your site between HTTP and HTTPS sections, it is imperative that the HTTP traffic does not send or receive the session id or any token used to manage the authentication status of a user. It is also important that you do not send sensitive application data within non-HTTPs requests/responses.
The only way to secure data within web applications/APIs is to encrypt your traffic.
Your Topics One By One
Basic-Http-Auth
Authentication: YES
Session Management: NO
Encryption: NO
This is a method for authenticating by web resource only. Basic authentication authenticates uses by resource identified by URL. This was most popularly implemented by Apache HTTP Web Server with the use of .htaccess based directory/location authentication. Credentials have to be sent with each request; clients generally handled this transparently for users.
Basic authentication can be used by other systems as a mode of authentication. However, the systems that utilize Basic-Http-Auth are providing authentication and session management, not the Basic-Http-Auth itself.
This is not session management.
This is not encryption; content and credentials are nearly 100% plain text
This does not secure the contents of the application's HTTP request/responses.
Digest-Auth
Authentication: YES
Session Management: NO
Encryption: NO
This is exactly the same as Basic-Http-Auth with the addition of some simple MD5 digesting. This digesting should not be relied upon instead of using encryption.
This is not session management.
This is not encryption; the digest is easily broken
This does not secure the contents of the application's HTTP request/responses.
OAuth
Authentication: YES
Session Management: NO
Encryption: NO
OAuth just lets you have an external service validate credentials. After that it is up to you to manage/work with the result of authentication request to your OAuth provider.
This is not session management.
This is not encryption; your sites traffic is still plain text. The authentication process will be secure due to HTTPS restrictions, but your application is still vulnerable.
This does not secure the contents of the application's HTTP request/responses.
Gangster Handshake / Custom HTTP header
Authentication: YES, potentially
Session Management: YES, potentially
Encryption: NO
"Custom HTTP header" is a type of "Gangster Handshakes"; as such I will use the same section to discuss them. The only difference is that a "Custom HTTP header" is specifying where the hanshake (session id, token, user authentication toke, etc) will be stored (i.e. in a HTTP header).
It is important to note that these do not specify how authentication will be handled, nor do they specify how session management will be handled. They essentially describe how and where session ids/authentication tokens will be stored.
Authentication would need to be handled by your application or via a third party (e.g. OAuth). Session management will still need to be implemented as well. The interesting thing is you can choose the merge the two if you wish.
This is not encryption; your sites traffic is still plain text. The authentication process will be secure due to HTTPS restrictions if you use OAuth, but your application is still vulnerable.
This does not secure the contents of the application's HTTP request/responses.
What You Need To Do
...I highly suggest you make sure that you understand that a robust web application that is secure needs the following:
Encryption (HTTPS is pretty much your only choice)
Session Management
Authentication / Authorization
Authorization relies upon Authentication. Authentication relies upon Session Management and Encryption makes sure the session isn't hijacked and that the credentials are not intercepted.
Flask-Login
I think you should look into flask-login as a way to avoid re-implementing the wheel. I have personally never used it (I use pyramid for web applications in python). However, I have seen it mentioned before in web application/python boards. It handles both authentication and session management. Throw your web api/application through HTTPS and you have all three (Encryption, Session Management, and User Authentication).
If you do not / can not use flask-login, be prepared to write your own, but do research first on how to create secure authentication mechanisms.
If at all possible, if you do not understand how to write an authentication procedure please do not attempt it without first learning how hackers use pattern based attacks, timing attacks, etc.
Please Encrypt Your Traffic
...move past the idea that you can avoid using HTTPS with some "clever" token use. Move past the idea that you should avoid using HTTPS/encryption because "its slow", process intensive, etc. It is process intensive because it is an encryption algorithm. The need to ensure the safety of your user's data and your applications data should always be your highest priority. You do not want to go through the horror of notifying your users that their data was compromised.

The https it is slower, but not a not.
Only the handshaking is slower. For us the biggest problem it is to upkeep the key pair on server-mobiles side and the rights.
We have implemented a message digest too. The problem it is: is hard to set up the php-android-ios version properly. After this is done ( a parameter need to changes what is suggesting Google at first results only at android side) the problem will be with low-end devices: to much CPU usage, slow on decrypt-encrypt process, a lot slower than https, especially when you need to transform 10kb String(can take several minutes).
If I don't transfer Nasa data to Hamas, than I would go with a very simple encryption over simple HTTP: like invert the bits or so...

Go with HTTPS. It's (marginally) slower, but the security you get from it for the relatively short investment time (purchasing the SSL cert and just changing your URLs from http to https) is worth it. Without HTTPS, you run the risk of your users' sessions getting hijacked on unsecured public networks, which is extremely easy for someone to do.

How to validate the origin of a web service invokation

Suppose you have a mobile application (Windows Phone or Android) that connects yo your back-end using SOAP.
For making it easy, let's say that we have a Web Service implemented in C#. The server exposes the following method:
[WebMethod]
public string SayHallo() { return "Hallo Client"; }
From the server perspective, you can't tell if the caller is your mobile application or a developer trying to debug your web service or a hacker trying to reverse engineer/exploit your back-end.
How can one identify that the origin of the web service call is THE application? as anyone with the WSDL can invoke the WS.
I know I can implement some standard security measures to the web service like:
Implement HTTPS on the server so messages travel encrypted and the danger of eavesdropping is reduced.
Sign the requests on the client-side using a digest/hashing algorithm, validate the signature in the server and reject the messages that have not been signed correctly.
Write custom headers in the HTTP request. Anyways headers can be simulated.
However, any well experienced hacker or a developer who knows the signing algorithm, could still generate a well signed, well, formatted message. Or a really good hacker could disassemble the application and get access to the hidden know-how of my "top secret" communications protocol.
Any ideas how to make the SayHallo() method to answer ONLY to request made from my mobile application?
We are running in the context of a mobile application, with hardware access, there could be something that can be done exploiting the hardware capabilities.
If someone wonders, I'm thinking on how to make a mobile app secure enough for sensitive applications like banking, for example.
Thanks for your ideas.

What you are describing is bi-directional authentication. It can only be done by storing a signed public key (certificate) on boths sides of the communication. Meaning that each app would need to authenticate your server with your servers public key and the server would need to authenticate each instance of your app. The app's public key would need to be produced and stored on the server at the deployment time with each instance of your app. Think of this as 2 way HTTPS, in general the only authentication that needs to be done is one direction, with the browser authenticating the server with a trusted signing key. In your case this would need to be done on both sides. Normally you would have a service like VeriSign sign each instance of a public key, this can get quite spendy with multiple deployments of an app. In your case you could just create an in house signing application using something like OPENSSL to sign your app every time it is distributed. This does not mean that someone still could not hack your code and extract the signing key on the app side. In general any code can be hacked, it's just a question of how hard can you make it before they give up? If you were to go the custom hardware route, there are things such as crypto chips and TMP's that can serve as a key sotre and make it harder for people to gain access to the private keys on the device.
A quick google search turned up the following:
http://www.codeproject.com/Articles/326574/An-Introduction-to-Mutual-SSL-Authentication
If you are thinking about using rewards points, and are really worried about someone gaming the system from the outside a better solution is to have each person make an account that is stored securely on the server and their points are saved and tallied there. This centralizes all the data and allows you complete control over it with out worrying about a malicious app reporting non-existent points. (this is how banks work)

If you want to verify that a user is both mobile and who they say they are then the best way is to leverage the network. Send a push notification with the hashed key that you want the user to use via:
APN for iOS
something like urban airship for windows phone
GCM for Android.

In general, the model looks like:
Server authenticates itself to the many clients with a certified public key (this is the whole Public Key Infrastructure, Certificate Authorities, etc)
Each client identifies itself to the server via some other authentication system (in 99.9% of cases, this is a password)
So if you're wondering how this sort of thing works in the case of banking apps, etc that's basically how it breaks down: (1) Client and server establish a secure channel such as a shared secret key, using the server's public key, (2) Client authenticates via this secure channel using some other mechanism.
Your question specifically, however, seems more aimed at the app authenticating itself (i.e., any request from your app is authentic) with the thought that if only your app can be authenticated, and your app is well-behaved, then everything should be safe. This has a few implications:
This implies that every user of your app is trusted. In security terms, they are part of your "trusted computing base".
Attempting to achieve this sort of goal WITHOUT considering the user/user's computing platform as trusted is essentially the goal of DRM; and while it's good enough to save money for music publishers, it's nowhere close to good enough for something really sensitive.
In general:
The problem that you're specifically looking at is VERY hard to solve, if you're looking for very strong security properties.
You likely don't need to solve that problem.
If you give us some more context, we might be able to give you more specific advice.

In addition to the answers already given, how about using a login type scheme with the unique id of the phone and a password? Get the user to register with your "back-end" and every time a transaction has to be made with the back-end, require the password or have an option to automatically log in.

You can use the following way to secure and to track your requests to server.
You can force the mobile or web clients to send you a Custom Headers of the device type when accessing your webservice through REST methods
Use basic http authentication by forcing each client to have their own username and passwords provided by you as a authorized webservice provider..
If you need more advanced protection you can use OAuth 2.0 to secure your webservice.

Since your originating app is going to be Android or Windows Phone apps, either one of them will be relatively easy for the wanna be hacker to debug. in any case you're going to be running the code on a machine that you have no control over so no ssl tricks or checking signing will solve your fundamental problem.
the only way you can combat threat from it is to NOT TRUST THE CLIENT. verify that the input coming from the clients is valid before acting on it if you're making a game - that it's accompanied by a valid security token etc.
in essence build your service so that it doesn't matter if the user is using an unofficial client.

How should I properly impliment HTTP(S) auth (REMOTE_AUTH) in django?

I am in the planning phase a new project. I want to be able to control multiple relays from my android powered phone over the internet. I need to use an HTTP based server as a middleman between the phone and the relays. Django is my preferred platform because Python is my strongest skill set. This would not be a "web app" (with the exception of the admin interface for managing the user and their access to the relays). Rather, the server would simply provide an API in the form of HTTPS requests and JSON encoding. Though, I should note that I have never done any web development in my life, so I don't know best practices (yet). The authentication method should meet the following criteria:
Works over HTTPS (self-signed SSL)
Provides multi-factor authentication (in the form of something you have and something you know)
Be reasonably secure (Would be very difficult to fool, guess at. or otherwise bypass)
Is simple in implementation for the server operator and end user on the mobile client
Is lightweight in in terms of both CPU cycles and bandwidth
I plan to use the following scheme to solve this:
An administrator logs into the web interface, creates a user, and sets up his/her permissions (including a username and a password chosen by the user).
The user starts the client, selects add server, and enters the server URL and his/her credentials.
The client attempts to authenticate the the user via HTTP auth
(over SSL). If the authentication was successful, the server will generate an API key in the form of a UUID and sends it to the client. The client will save this key and use it in all API calls over HTTPS. HTTP auth is only used for the initial authentication process prior to reviving a key, as a session scheme would not be nessessary for this application. Right? The client will only work if the phone is configured to automatically lock with a PIN or pattern after a short timeout. The server will only allow one key to be generated per user, unless an administrator resets the key. Hence, simple, mobile, multifactor authentication.
Is this sound from a security standpoint? Also, can anyone point me to an example of how to use the HTTP auth that is built into Django? From a Google search, I can find a lot of snipits witch hack the feature together. But, none of them implement HTTP auth in the wayit was added to Django in 1.1. The official documentation for REMOTE_AUTH can be found here, but I am having difficulty understanding the documentation as I am very new to Django.

I'm not entirely sure of how basic auth would work on Django, but I can take a shot.
The basic auth article on wikipedia covers a pretty standard usecase for logging in. For Android I've personally skipped the first part (401) and just pass my credentials in right away.
With your auth request you will have to just grab the user credentials from the request headers (WWW-Authenticate) and then do all the necessary work for that. With the credentials you can then just use the authentication framework provided in Django to verify that the user then generate their UUID (I guess).
As for basic auth on Android it's a little bit tricky at first and may leave you pulling your hair. I've found this article on Basic HTTP auth for android which helps explain how to do it.
As for the security part of it, I'm not too sure. It's pretty simple, which I'd say is a good thing :)