Develop Reference

accessing aws private bucket in android without cognito

I am new to using aws and I am having some troubles. I have a set of videos that are present in multiple folders in an aws s3 bucket . I am creating an android app which would be reading the videos of bucket ,display them as list and allow the users to stream those videos.
The issue is that my app is using a non aws seperate server for authentication and thus i don't want to use aws-cognito authentication. When i tried searching for a native sdk for s3, i was pointed to aws-amplify framework and when i tried to follow the docs here , I got stuck at this step which enforces to set up the cognito authentication.
? You need to add auth (Amazon Cognito) to your project in order to add storage for user files. Do you want to add auth now?
`Yes` // <------------------------ this is for either now or later, can't say no to adding auth at all
? Do you want to use the default authentication and security configuration?
`Default configuration` //<-------------------------------------- can't say no here
? How do you want users to be able to sign in?
`Username`
? Do you want to configure advanced settings?
`No, I am done.`
? Please provide a friendly name for your resource that will be used to label this category in the project:
`S3friendlyName`
? Please provide bucket name:
`storagebucketname`
? Who should have access:
`Auth and guest users`
? What kind of access do you want for Authenticated users?
`create/update, read, delete`
? What kind of access do you want for Guest users?
`create/update, read, delete`
? Do you want to add a Lambda Trigger for your S3 Bucket?
`No`
I am not sure about this, but I believe there is a way to access some private bucket data using just an iam user access keys. So which sdk or custom code would help me access the whole private bucket for all users without needing to authenticate?

Currently, Amplify's AWSS3StoragePlugin is hardcoded to require use of Amazon Cognito for authentication/authorization.
If you would like to use IAM only, please create a feature request on the Amplify Android GitHub repository.
If you would like to auth with your own credential provider, you could federated an OpenID Connect provider with Cognito.
The nuclear option is to use the low-level AmazonS3Client, in the AWS SDK for Android. This will allow you to supply your own AWSCredentialsProvider. There are a variety of valid ways to provide credentials. Note: the AWSMobileClient is a utility for doing Cognito auth, and it is what is used in the AWSS3StoragePlugin.

If a user is supposed to be able to access the bucket without the need for authentication, you could make the objects in that bucket publicly accessible. That way you skip the extra work involved in accessing a private bucket for objects you intend to be easily accessible (no authentication).
If the objects in the bucket and the bucket itself must remain private, then you could follow this guide: "Restricting Access to Amazon S3 Content by Using an Origin Access Identity for controlling access to S3 via CloudFront. The theory is that users won't be able to access your S3 content unless is came from your OAI configured CloudFront distribution. Afterwards you can figure out a way to access your CloudFront distribution and fetch content from there using your application only.

How to grant access to android app users for AWS S3 Objects?

I have a bucket in Amazon S3 and I have Lambda functions that generates JSON files for this bucket. I am using the S3 files in my mobile app. Until recently, I gave public access to these S3 files for simplicity. But now I want these S3 objects to be accessible with a simple authentication. I’ve examined all AWS tutorials but couldn’t find an easy way to implement this. I don’t want to use Cognito service since my app doesn’t need authentication and since my S3 files are not user-related, they are used for app. I want these S3 objects to be accessible by http request to a url which includes simple key like this:
https://s3.eu-central-1.amazonaws.com/<bucket name>/<object name>?<key>
where key can be a combination of region, aws access key, secret access key or other values of the user that i define (I am using Retrofit to fetch json data from S3 bucket) I’ve looked at the presigned url option but an example for android-sdk doesn’t exists there, and most of the methods that can be used for this purpose are deprecated. Isn’t there an easy way for this? Or should I host my json files in other service/place?

If you say no to Cognito then you are just complicating things for yourself. This is exactly the case where you want to use Cognito. The fact that you don't require your users to authenticate and that the S3 content is not user specific doesn't mean that Cognito is not suitable in this scenario.
All you have to do is to create Cognito identity pool and choose to support unauthenticated entities. Create an IAM policy that allows reading from that specific S3 bucket and let unauthenticated users to assume that IAM role by attaching it to those unauthenticated entities.
Authentication then happens automatically during the initialization of SDK in your application. That is all that you need to do to allow access to that S3 bucket only from your application.
And you get access metrics even for unauthenticated users as a bonus. And if you later decide that you want your application to support authentication as well, then you don't need to change almost anything in your setup.

AWS S3 on multiple devices

Ok, so I am new to AWS. I want to make an app that will store a small amount of user data. S3 seems to be the way to store data. Is there a way to make multiple storage spaces automatically with S3? Lets say I make an android app and people install it on their phone. Will they each automatically get an S3 storage space? how do I do that? thanks

You can create a S3 Bucket to your project with folders (depending on the architecture, can be one for customer). On this way, you will have an instance of the s3 service with all your user data.

Amazon S3 is simply an object-storage system. How you use it is totally up to you.
If you wish to store information on a per user basis, then you need to consider security in addition to how the data is stored.
If the intention is that a user can access some information that is private to them (as opposed to being publicly visible to anyone), then you first need to control access to data.
For public information, no authentication is necessary
For private information, something needs to determine what they are allowed to access, and then grant access
You should not give permanent AWS Credentials (Access Key, Secret Key) to every user. These credentials are only for your IT operations staff (you!) and for your applications.
This leaves two options:
Your central server could generate temporary access credentials using the AWS Secure Token Service, while specifying what access rights they have (eg access to a particular S3 bucket and path, or to other AWS services such as DynamoDB). OR
Generate pre-signed URLs for specific objects stored in Amazon S3.
Based upon your use-case, it seems a better fit to use pre-signed URLs. Basically, the flow is:
Your app would send a request to your central server, requesting access to an object.
The server (or rather, the app you have written that is running on a central server) verifies their identity and confirms that they should be allowed access to the object stored in Amazon S3.
The central server then generates a pre-signed URL that grants time-limited access to an object in Amazon S3 and sends the URL back to the client app
The client app then uses the URL to retrieve the data from Amazon S3
Only the app running on your central server requires AWS credentials. It then uses those credentials to generate pre-signed URLs that can be used by the client apps.
By the way, the app on the central server doesn't actually need to be running on a server. You could use AWS API Gateway to send requests to AWS Lambda functions, which can perform the logic and send back the response. This would be a serverless solution, but still with centralized logic.

What is the preferred way of using AWS (specifically S3) from mobile apps?

Adding the AWS access key and secret key directly in app code is definitely not a good approach, primarily because the app resides on the users device (unlike server side code), and can be reverse engineered to get the credentials, which can then be misused.
Though I find this information everywhere, but am unable to find a definitive solution to this problem. What are my options? I read about the token vending machine architecture for temporary credentials, but I am not convinced that it is any better. If I can reverse engineer the secret key, then I can reverse engineer the code which requests for temporary credentials. And once I have a set of temporary credentials to access S3, I am as good as if I had the key. I can request the temporary credentials again and again, even if they expire pretty quickly. To summarize, if an app can do something, I can do the same as a malicious user. If anything, the TVM can be a bit better at management (rotating credentials, and changing key in case of breach, etc.). Please note we can put the same access restrictions on the secret key, as we plan to do in case of TVM temporary credentials.
Additionally, if Amazon doesn't want people to use the secret key directly in the App, why don't they block it in their SDK, and enforce TVM or the correct solution. If you will leave a path, people are going to use it. I read several articles like these, and wonder why?: http://blog.rajbala.com/post/81038397871/amazon-is-downloading-apps-from-google-play-and
I am primarily from web background, so my understanding of this may be a bit flawed. Please help me understand if this is better, and whether there is a perfect (or may be good) solution available to this problem.
PS: Is there a rails implementation of TVM?

Embedding S3 keys in App code is very risky. Anyone can easily get that key from your app code (no reverse engineering or high skill set required), even if that is stored encrypted it is still compromised just that someone need to try harder (depending on how do you encrypt).
I hope that you understand the advantages of using temporary credentials to access Amazon (S3 etc) resources (mainly security + some others like no app update etc). I think you are more confused about the process to get the temporary credentials from TVM and how that is safer than embedding keys in code.
Every client using TVM first need to register with the TVM server implementation hosted by you. The communication between App (using TVM client) and TVM server is over SSL.
First the app register with TVM by providing UUID and a secret key. Please note that the secret key is not embedded in App code (which I think is main reason for your confusion) but generated randomly (using SecRandomCopyBytes which generates an array of cryptographically secure random bytes) at the time of registration (and hex encoded).
Once the device is registered successfully with TVM, the client TVM store the generated UDID and secret key in a storage called Keychain in iOS and Shared Preferences in Android. The keychain in iOS is the shared storage provided by iOS to securely (encrypted) store information (mainly keys, password etc).
After registration and UDID/Secret Key storage, App can get the token from TVM by sending the UDID, cryptographic signature, and a timestamp. The cryptographic signature is an HMAC hash generated from the timestamp using the secret key. The TVM can use the UDID to lookup the secret key and uses it to verify the signature. The TVM then responds by sending back temporary credentials, which are encrypted using the secret key (uses AES). The application decrypts the temporary credentials using the key and can then use them to access any AWS services for which the temporary credentials are authorized. Eventually, the expiration time of these temporary credentials will be reached, at which point the application can get the fresh temporary credentials, if required.

I am not sure how signed URLs relate to TVM, because I don't understand the concepts 100% but signed URLs really solved the problem for me. I needed a mechanism that would feed web app and mobile app data without allowing for misuse of the credentials. Putting the key in the code is indeed a very bad idea as it may generate a huge bill for the company.
After 3 days of extensive research, I found a simple and, what seems to be, a reliable and relatively safe solution: signed URLs. The idea is, that a very light-weight back-end can generate a temporary URL that will grant the user access to the specific resource for a limited time. So the idea is simple:
the user asks our back-end with a Rest call that he wants a specific resource
the back-end is already authorized with AWS S3
the back-end generates a temporary URL for the user and sends it in the Rest response
the user uses the URL to fetch the data directly from the AWS
A plug-and-play Python implementation can be found here and with a slight modification that I had to use: here.
Of course one more thing to figure out would be how do we authorize the user before we know that we can grant it the URL but that's another pair of shoes.

You should ideally use Cognito Identity for achieving this along with appropriate policies. It should be used with S3TransferUtility and S3TransferManager in iOS and Android SDKs. That would allow for background uploads and downloads as well. Cognito vends temporary credentials for access to AWS resources and is free. Also, you could federate it using UserPools or providers like Google, Facebook if you want secure access.
Thanks,
Rohan

Storing API key on mobile device

I've read many, if not all, answers to previously asked questions about the same topic, but questions themselves are not exactly about my case.
I have an OAuth 2.0 server running. It has an endpoint that provides access tokens to users. Programs and websites requesting the access token may or may not be owned by me, in other words, I may add a tool for website users and locate it on the same or neighboring website and my users may create an app and with their API key request access to user's data. Now I am developing the app that will operate user's data.
I realize that storing the API secret on the device is not a good solution. I have read about creating an end-point to which I make request directly from the app and then the endpoint makes API requests, but the endpoint will have to exist on the same host and probably server as OAuth server.
Is there a way to authorize mobile application to access user data when I am in controll of the application and the OAuth server? Should I create a separate end-point? Should I pass it device ID and/or any other information?
P.S. I know that plain old authorization would work here, but then what if some user wants to create his own mobile extension (not allowed currently for security reasons)? Also, current system has a button that is recognized by many people and they know what exactly will happen after clicking it. With the app, it may be an issue when the user sees a login dialog instead of the "Login with *" button. I really hope there is a clever solution to this.

Your concern is spot on. Your API Secret should not be stored on the device.
However, I am not sure why you would be concerned to build a separate endpoint, since OAuth 2 has a authorization flow for these use cases.
https://www.rfc-editor.org/rfc/rfc6749#section-9
In your use case, I'd suggest using the implicit grant flow to fetch the access token and store that on the local device. There would be no refresh tokens and the access_token can have an expiration date. Of course, the token on the device can be compromised, but the damage will be limited to a particular user and not the entire application.
If this level of security is not acceptable, then you can look at splitting up your API Secret in different parts of your app and then assemble it at run time in your app.