Develop Reference

Android FileBody equivalent encoding in iOS

We have some code posting an image and some other data as part of a Multipart in a POST request. In Android is working just fine, but I can't seem to make it work on iOS, where I keep getting a 500 Internal Server Error. The working Android code looks like
String uploadURL = "http://someServer.com/upload";
String imageToUploadPath = "imgFilePath";// path de la imagen a subir
String userId = "123";
String token = "abcd";
HttpClient httpClient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(uploadURL);
File f = new File(imageToUploadPath);
FileBody fileBody = new FileBody(f);
MultipartEntity reqEntity = new MultipartEntity();
Charset chars = Charset.forName("ISO-8859-1");
reqEntity.addPart("id", new StringBody(userId, chars));
reqEntity.addPart("token", new StringBody(token, chars));
reqEntity.addPart("image", fileBody);
httppost.setEntity(reqEntity);
HttpResponse response = httpClient.execute(httppost);
My iOS attempt using AFNetworking is as follows
uploadURLStr = #"http://someServer.com/upload";
NSString *token = #"abcd";
NSString *userID = #"123";
AFHTTPSessionManager *manager = [AFHTTPSessionManager manager];
manager.responseSerializer = [AFHTTPResponseSerializer serializer];
NSMutableSet *supportedContentTypes = [manager.responseSerializer.acceptableContentTypes mutableCopy];
supportedContentTypes addObject:#"text/html"];
manager.responseSerializer.acceptableContentTypes = supportedContentTypes;
NSURLSessionTask *task = [manager POST:uploadURLStr
parameters:nil
constructingBodyWithBlock:^(id<AFMultipartFormData> formData) {
NSDictionary *tokenHeaders = #{#"Content-Disposition": #"form-data",
#"name": #"token",
#"Content-Type": #"text/plain",
#"charset": #"ISO-8859-1",
#"Content-Transfer-Encoding": #"8bit"
};
[formData appendPartWithHeaders:tokenHeaders body:[token dataUsingEncoding:NSISOLatin1StringEncoding]];
NSMutableDictionary *userIDHeaders = [tokenHeaders mutableCopy];
[userIDHeaders setObject:#"id" forKey:#"name"];
[formData appendPartWithHeaders:[userIDHeaders copy] body:[userID dataUsingEncoding:NSISOLatin1StringEncoding]];
NSDictionary *imgHeaders = #{#"Content-Disposition": #"form-data",
#"name": #"image",
#"filename": fileName,
#"Content-Type": #"application/octet-stream",
#"Content-Transfer-Encoding": #"binary"
};
[formData appendPartWithHeaders:imgHeaders
body:[imgData base64EncodedDataWithOptions:0]];
}
progress:^(NSProgress *uploadProgress) {
NSLog(#"progress: %.2f", uploadProgress.fractionCompleted);
}
success:^(NSURLSessionTask *task, id responseObject) {
NSLog(#"responseObject = %#", responseObject);
} failure:^(NSURLSessionTask *task, NSError *error) {
[self showUploadError];
NSLog(#"error = %#", error);
}];
Headers for each part we managed to log it from the Android app so we replicated it -for instance, I was sending the image with a Content-Type of image/jpeg and Android sends it as application/octet-stream- I assume that the image data is not being encoded as in Android. I have tried with base64 encoding as it's in the code now -tried different options other than 0-, just leaving the NSData returned by UIImageJPEG representation but I can't hit the nail. Any help is appreciated.

How to upload media (.mp4) file using Azure SAS url from Android device

I am building rich video library on android and trying to upload video, as i do not want client to access my media key and other details i have making call to my server and return SAS url as a response which client will use to upload media
Just wondering what is best way to achieve uploading here, i tried with okHttp library and scuceeded most of the time ()few times i got scoket timeout exception for large file) but not sure how performant okHttp with azure,
intheory it is always good to chunk media file and upload, so could anyone one tell me what is the best way to upload file in azure blob from android using SAS url

Instead of uploading large files, you could try to cut them into blocks and then upload separate pieces of no larger than 4Mb. And once all the pieces (Blocks) are uploaded, you need to commit them all and give the order in which they should appear.
For more info, you can refer to this similar topic from SO.
Edit:
Here is an example Java code that works with OkHttp 3.6.0, and SAS URL.
public class UploadBlobUsingSASUrl {
public static void main(String[] args) throws IOException {
UploadBlobUsingSASUrl uploadHandler = new UploadBlobUsingSASUrl();
String sasSignature = "?sv=2016-05-31&ss=bfqt&srt=sco&sp=rwdlacup&se=2017-03-30T23:39:00Z&st=2017-03-20T15:39:00Z&spr=https&sig=A2m1O5qZf79L7925DHMJtfkBRI6GVFhYcrGLStBo%2BL0%3D";
String blobStorageEndpoint = "https://<your-account>.blob.core.windows.net/<your-container>/<your-blob>" + sasSignature;
InputStream inStream = new FileInputStream("<file-path>");
BufferedInputStream bis = new BufferedInputStream(inStream);
List<String> blockIds = new ArrayList<String>();
int counter = 1;
while (bis.available() > 0) {
int blockSize = 4 * 1024 * 1024; // 4 MB;
int bufferLength = bis.available() > blockSize ? blockSize : bis.available();
byte[] buffer = new byte[bufferLength];
bis.read(buffer, 0, buffer.length);
String blockId = Base64.getEncoder().encodeToString(("Block-" + counter++).getBytes("UTF-8"));
uploadHandler.UploadBlock(blobStorageEndpoint, buffer, blockId);
blockIds.add(blockId);
}
uploadHandler.CommitBlockList(blobStorageEndpoint, blockIds);
bis.close();
inStream.close();
}
public void UploadBlock(String baseUri, byte[] blockContents, String blockId) throws IOException {
OkHttpClient client = new OkHttpClient();
MediaType mime = MediaType.parse("");
RequestBody body = RequestBody.create(mime, blockContents);
String uploadBlockUri = baseUri + "&comp=block&blockId=" + blockId;
Request request = new Request.Builder()
.url(uploadBlockUri)
.put(body)
.addHeader("x-ms-version", "2015-12-11")
.addHeader("x-ms-blob-type", "BlockBlob")
.build();
client.newCall(request).execute();
}
public void CommitBlockList(String baseUri, List<String> blockIds) throws IOException {
OkHttpClient client = new OkHttpClient();
StringBuilder blockIdsPayload = new StringBuilder();
blockIdsPayload.append("<?xml version='1.0' ?><BlockList>");
for (String blockId : blockIds) {
blockIdsPayload.append("<Latest>").append(blockId).append("</Latest>");
}
blockIdsPayload.append("</BlockList>");
String putBlockListUrl = baseUri + "&comp=blocklist";
MediaType contentType = MediaType.parse("");
RequestBody body = RequestBody.create(contentType, blockIdsPayload.toString());
Request request = new Request.Builder()
.url(putBlockListUrl)
.put(body)
.addHeader("x-ms-version", "2015-12-11")
.build();
client.newCall(request).execute();
}

using blobstore with google cloud endpoint and android

I am developing an app-engine connected android project using the eclipse plugin. One aspect of the app is to allow user Alpha to send pictures to user Bravo. To do that I have the following setup:
User Alpha posting:
send image to my app engine server through endpoints
server stores image in blob store
server stores blobkey in datastore
User Bravo getting:
server gets blobkey from datastore
server gets image using blob key
server sends image to android app using endpoints
This setup takes upward of two (2) minutes from when my android app sends an image to when I can see it in the blob sore. Needless to say this is completely unacceptable.
My server is processing the image programmatically, thru the following code:
public static BlobKey toBlobstore(Blob imageData) throws FileNotFoundException, FinalizationException, LockException, IOException {
if (null == imageData)
return null;
// Get a file service
FileService fileService = FileServiceFactory.getFileService();
// Create a new Blob file with mime-type "image/png"
AppEngineFile file = fileService.createNewBlobFile("image/jpeg");// png
// Open a channel to write to it
boolean lock = true;
FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
// This time we write to the channel directly
writeChannel.write(ByteBuffer.wrap
(imageData.getBytes()));
// Now finalize
writeChannel.closeFinally();
return fileService.getBlobKey(file);
}
Does anyone know how I can either adapt the official example to use endpoints (in the case where I must use my app-engine instances) or use getServingUrl (bypassing my instances) to store and serve my blobs? Please, instead of words, include the code. Thanks.

I'll share how I'm doing this. I'm not using the google-cloud-endpoints, but just my own rest based api, but it should be the same idea either way.
I'll lay it out step by step with code, hopefully it will be clear.
You'd simply adapt the way you send your requests to use endpoints instead of doing it more generic like in this example. I'm including some boilerplate, but excluding try/catch,error checking etc for brevity.
Step 1 (client)
First client requests an upload url from server:
HttpClient httpclient = new DefaultHttpClient();
HttpConnectionParams.setConnectionTimeout(httpclient.getParams(), 10000); //Timeout Limit
HttpGet httpGet = new HttpGet("http://example.com/blob/getuploadurl");
response = httpclient.execute(httpGet);
Step 2 (server)
On the server side the upload request servlet would look something like this:
String blobUploadUrl = blobstoreService.createUploadUrl("/blob/upload");
res.setStatus(HttpServletResponse.SC_OK);
res.setContentType("text/plain");
PrintWriter out = res.getWriter();
out.print(blobUploadUrl);
out.flush();
out.close();
note the argument to createUploadUrl. This is where the client will be
redirected once the actual upload has been completed. That's where
you'll handle storing the blobkey and/or serving url and returning it to the client. You'll have to map a servlet to that url, which will handle step 4
Step 3 (client)
Back to the client again to send the actual file to the upload url using the url returned from step 2.
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(uploadUrlReturnedFromStep2);
FileBody fileBody = new FileBody(thumbnailFile);
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("file", fileBody);
httppost.setEntity(reqEntity);
HttpResponse response = httpclient.execute(httppost)
Once this request is sent to the servlet in step 2, it will be redirected to the servlet you specified in the createUploadUrl() earlier
Step 4 (server)
Back to the server side:
This is the servlet handling the url mapped to blob/upload. We will here return the blobkey and serving url to the client in a json object:
List<BlobKey> blobs = blobstoreService.getUploads(req).get("file");
BlobKey blobKey = blobs.get(0);
ImagesService imagesService = ImagesServiceFactory.getImagesService();
ServingUrlOptions servingOptions = ServingUrlOptions.Builder.withBlobKey(blobKey);
String servingUrl = imagesService.getServingUrl(servingOptions);
res.setStatus(HttpServletResponse.SC_OK);
res.setContentType("application/json");
JSONObject json = new JSONObject();
json.put("servingUrl", servingUrl);
json.put("blobKey", blobKey.getKeyString());
PrintWriter out = res.getWriter();
out.print(json.toString());
out.flush();
out.close();
Step 5 (client)
We'll get the blobkey and serving url from the json and then send it along with user id etc to store in the datastore entity.
JSONObject resultJson = new JSONObject(resultJsonString);
String blobKey = resultJson.getString("blobKey");
String servingUrl = resultJson.getString("servingUrl");
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("userId", userId));
nameValuePairs.add(new BasicNameValuePair("blobKey",blobKey));
nameValuePairs.add(new BasicNameValuePair("servingUrl",servingUrl));
HttpClient httpclient = new DefaultHttpClient();
HttpConnectionParams.setConnectionTimeout(httpclient.getParams(), 10000);
HttpPost httppost = new HttpPost(url);
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = httpclient.execute(httppost);
// Continue to store the (immediately available) serving url in local storage f.ex
Step 6 (server)
Actually storing everything in the datastore (using objectify in this example)
final String userId = req.getParameter("userId");
final String blobKey = req.getParameter("blobKey");
final String servingUrl = req.getParameter("servingUrl");
ExampleEntity entity = new ExampleEntity();
entity.setUserId(userId);
entity.setBlobKey(blobKey);
entity.setServingUrl(servingUrl);
ofy().save().entity(entity);
I hope this makes things more clear. If someone wants to edit the answer to use cloud endpoints instead of this more generic example, feel free :)
About the serving url
The serving url is a great way to serve images to your clients, because of the way it can dynamically scale images on the fly. For example you can send smaller images to your LDPI users by simply appending =sXXX at the end of the serving url. Where XXX is the pixel size of the largest dimension of your image. You completely avoid your instances and only pay for bandwidth, and the user only downloads what she needs.
PS!
It should be possible to stop at step 4 and just store it directly there, by passing along userId f.ex in step 3. Any parameters are supposed to be sent along to Step 4, but I did not get that to work, so this is how I do it at the moment, so I'm sharing it this way since i know it works.

I used the answer of this question to build my own system that uses AppEngine Endpoints. Unlike the posts above, I want to have a clean API that directly transmits the image (as byte array) to Google Endpoint and the upload to BlobstorageService is done on the backend side. The benefit of that is that i have an atomic API. The drawback obviously the load on the server as well as the heavy marshalling operations on the client.
Android - load, scale and serialize image and upload to endpoints
void uploadImageBackground(Bitmap bitmap) throws IOException {
// Important! you wanna rescale your bitmap (e.g. with Bitmap.createScaledBitmap)
// as with full-size pictures the base64 representation would not fit in memory
// encode bitmap into byte array (very resource-wasteful!)
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] byteArray = stream.toByteArray();
bitmap.recycle();
bitmap = null;
stream = null;
// Note: We encode ourselves, instead of using image.encodeImageData, as this would throw
// an 'Illegal character '_' in base64 content' exception
// See: http://stackoverflow.com/questions/22029170/upload-photos-from-android-app-to-google-cloud-storage-app-engine-illegal-char
String base64 = Base64.encodeToString(byteArray, Base64.DEFAULT);
byteArray = null;
// Upload via AppEngine Endpoint (ImageUploadRequest is a generated model)
ImageUploadRequest image = new ImageUploadRequest();
image.setImageData(base64);
image.setFileName("picture.png");
image.setMimeType("image/png");
App.getMyApi().setImage(image).execute();
}
Backend API Endpoint - Upload image to BlobstorageService
#ApiMethod(
name = "setImage",
path = "setImage",
httpMethod = ApiMethod.HttpMethod.POST
)
public void saveFoodImageForUser(ImageUploadRequest imageRequest) throws IOException {
assertNotEmpty(userId, "userId");
assertNotNull(imageRequest, "imageRequest");
// create blob url
BlobstorageService blobService = BlobstoreServiceFactory.getBlobstoreService();
String uploadUrl = blobService.createUploadUrl("/blob/upload");
// create multipart body containing file
HttpEntity requestEntity = MultipartEntityBuilder.create()
.addBinaryBody("file", imageRequest.getImageData(),
ContentType.create(imageRequest.getMimeType()), imageRequest.getFileName())
.build();
// Post request to BlobstorageService
// Note: We cannot use Apache HttpClient, since AppEngine only supports Url-Fetch
// See: https://cloud.google.com/appengine/docs/java/sockets/
URL url = new URL(uploadUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.addRequestProperty("Content-length", requestEntity.getContentLength() + "");
connection.addRequestProperty(requestEntity.getContentType().getName(), requestEntity.getContentType().getValue());
requestEntity.writeTo(connection.getOutputStream());
// BlobstorageService will forward to /blob/upload, which returns our json
String responseBody = IOUtils.toString(connection.getInputStream());
if(connection.getResponseCode() < 200 || connection.getResponseCode() >= 400) {
throw new IOException("HTTP Status " + connection.getResponseCode() + ": " + connection.getHeaderFields() + "\n" + responseBody);
}
// parse BlopUploadServlet's Json response
ImageUploadResponse response = new Gson().fromJson(responseBody, ImageUploadResponse.class);
// save blobkey and serving url ...
}
Servlet that handles callback from BlobstorageService
public class BlobUploadServlet extends HttpServlet {
#Override
public void doPost(HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException {
BlobstorageService blobService = BlobstoreServiceFactory.getBlobstoreService();
List<BlobKey> blobs = blobService.getUploads(req).get("file");
if(blobs == null || blobs.isEmpty()) throw new IllegalArgumentException("No blobs given");
BlobKey blobKey = blobs.get(0);
ImagesService imagesService = ImagesServiceFactory.getImagesService();
ServingUrlOptions servingOptions = ServingUrlOptions.Builder.withBlobKey(blobKey);
String servingUrl = imagesService.getServingUrl(servingOptions);
res.setStatus(HttpServletResponse.SC_OK);
res.setContentType("application/json");
// send simple json response (ImageUploadResponse is a POJO)
ImageUploadResponse result = new ImageUploadResponse();
result.setBlobKey(blobKey.getKeyString());
result.setServingUrl(servingUrl);
PrintWriter out = res.getWriter();
out.print(new Gson().toJson(result));
out.flush();
out.close();
}
}
The only thing left to do is to bind /blob/upload to UploadBlobServlet.
Note: This doesn't seem to work when AppEngine is running locally (if executed locally, then the POST to BlobstorageService would always return a 404 NOT FOUND)

Since I tried with many way to do the callback service in the api of endpoint, I abort that aproach. However, I could solve that problem making a parallel servlet to the api endpoint, it only needs define the class server and add it web.xml configuration. Here my solution:
1 Enpoint Service for get the URL for upload:
Then the service coudl be protected with clientId
#ApiMethod(name = "getUploadURL", httpMethod = HttpMethod.GET)
public Debug getUploadURL() {
String blobUploadUrl = blobstoreService.createUploadUrl("/update");
Debug debug = new Debug();
debug.setData(blobUploadUrl);
return debug;
}
2. Now the Client can call to endpoint for get the upload URL:
Maybe some like this (for android use you client library enpoint too):
gapi.client.debugendpoint.getUploadURL().execute();
3. The next step is todo a post to url catched in last step:
You can do that with a httpClient of android, again, in my case I need upload from a web then I use a form, and onChangeFile() event callback for get the uploadurl (using step 3) then when it response to change the form parameters "action" and "codeId" before that someone decide do click on submit button:
<form id="submitForm" action="put_here_uploadUrl" method="post" enctype="multipart/form-data">
<input type="file" name="image" onchange="onChangeFile()">
<input type="text" name="codeId" value='put_here_some_dataId'>
<input type="submit" value="Submit"></form>
4 Finally the paralele servlet class:
#SuppressWarnings("serial")
public class Update extends HttpServlet{
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
String userId = req.getParameter("codeId");
List<BlobKey> blobs = BSF.getService().getUploads(req).get("image");
BlobKey blobKey = blobs.get(0);
ImagesService imagesService = ImagesServiceFactory.getImagesService();
ServingUrlOptions servingOptions = ServingUrlOptions.Builder.withBlobKey(blobKey);
String servingUrl = imagesService.getServingUrl(servingOptions);
resp.setStatus(HttpServletResponse.SC_OK);
resp.setContentType("application/json");
JSONObject json = new JSONObject();
try {
json.put("imageUrl", servingUrl);
json.put("codeId", "picture_of_"+userId);
json.put("blobKey", blobKey.getKeyString());
} catch (JSONException e){
e.printStackTrace();
}
PrintWriter out = resp.getWriter();
out.print(json.toString());
out.flush();
out.close();
}
}
and add to web.xml, where com.apppack is the package of Update Class
<servlet>
<servlet-name>update</servlet-name>
<servlet-class>com.apppack.Update</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>update</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>

How to upload videos using multi part post entity?

I use this code that i found in some page but I only can upload images from my android application to the server and is working, but when i upload a video(.mp4) its saved as "file" like unknown.
public void upload() throws Exception {
//Url of the server
String url = "";
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url);
MultipartEntity mpEntity = new MultipartEntity();
//Path of the file to be uploaded
String filepath = "";
File file = new File(filepath);
ContentBody cbFile = new FileBody(file);
//Add the data to the multipart entity
mpEntity.addPart("image", cbFile);
mpEntity.addPart("name", new StringBody("Test", Charset.forName("UTF-8")));
mpEntity.addPart("data", new StringBody("This is test report", Charset.forName("UTF-8")));
post.setEntity(mpEntity);
//Execute the post request
HttpResponse response1 = client.execute(post);
//Get the response from the server
HttpEntity resEntity = response1.getEntity();
String Response=EntityUtils.toString(resEntity);
Log.d("Response:", Response);
//Generate the array from the response
JSONArray jsonarray = new JSONArray("["+Response+"]");
JSONObject jsonobject = jsonarray.getJSONObject(0);
//Get the result variables from response
String result = (jsonobject.getString("result"));
String msg = (jsonobject.getString("msg"));
//Close the connection
client.getConnectionManager().shutdown();
}
There is any way to make this work to upload videos too?

There is not issue with this code works fine for upload videos and images, the problem was i´m missing the file extension in the name, so when the video was uploaded it should be NAME.EXTENSION not only NAME.
NOTE:
For all the people who is trying to upload a large(2MB-10GB) image or video to the server the only solution i found was encode the file in chunks and upload each chunk to the server, from there you only have to encode the chunks again. NO SIZE LIMITATION :)!!!!

How to upload an image to BlobStore using MediaHttpUploader?

I use the following function in android but I get error "500 Java Heap Space" when calling uploader.upload().
If I called uploader.setDirectUploadEnabled(true) before uploader.upload() I get error "411 Length required" on upload() instead of error 500.
public static String uploadImage(Context context, String file, MediaHttpUploaderProgressListener progressListener)
throws Exception {
// The file is something like : /mnt/sdcard/wallpapers/image.jpg
File mediaFile = new File(file);
HttpTransport transport = new NetHttpTransport();
HttpRequestFactory factory = transport.createRequestFactory(null);
//obtain the upload url
GenericUrl getUploadUrl = new GenericUrl(Util.getBaseUrl(context)
+ "/myapp/blobservice?get-key");
HttpRequest request = factory.buildGetRequest(getUploadUrl);
HttpResponse response1 = request.execute();
String postUrl = convertStreamToString(response1.getContent()).trim();
// the returned post url when using the development server is something like : //http://192.168.1.4:8888/_ah/upload/agd5YWRnZXQychwLEhVfX0Jsb2JVcGxvYWRTZXNzaW9uX18Y2wUM
InputStreamContent mediaContent = new InputStreamContent("image/jpeg",
new BufferedInputStream(new FileInputStream(mediaFile)));
mediaContent.setLength(mediaFile.length());
MediaHttpUploader uploader = new MediaHttpUploader(mediaContent,transport, null);
uploader.setProgressListener(progressListener);
//the following line produces exception "500 Java heap space" on the local server
//on the remote server I get "500: Internal server error"
HttpResponse response = uploader.upload(new GenericUrl(postUrl));
if (!response.isSuccessStatusCode()) {
throw new Exception("Uploading image failed.");
} else {
String blobKey = convertStreamToString(response.getContent()).trim();
return blobKey;
}
}

It seems that MediaHttpUpload uses resumable protocol, which is not compatible with multipart/form-data file upload of GAE blobstore.
You should create a custom data upload servlet, that knows how to handle the resumable protocol.