I want to write a android application which needs data from the web. This information is stored in a json-file. The data from the json-file is saved on the device. To keep it up to date, I need to check for changes in the file every hour.
As the remote File can get quite large I want to download it only if it is different from the version which was previously downloaded. I thought about using the Last-Modified-Header of HTTP for this.
I came up with the following workflow (pseudo-code):
data = null; data_timestamp = null;
Every hour repeat:
Issue a HTTP Head-Request to the URL and option new_timestamp from Last-Modified Header.
If either data==null or new_timestamp > data_timestamp then
Issue a normal HTTP-Request to the URL
Save to data and set data_timestamp = new_timestamp
Do you think this is a reasonable approach? I could use the if-modified-since HTTP Header to get the data only if it has changed since the last download. This would save me one request. If it has changed, a body containing the new data is provided, if it hasn't, the body is empty.
I also thought about using ETags, as I typically want to download if the file has new content (and not if the modified-date has changed), but my webserver (nginx) doesn't support the creation of etags and I don't want to involve another layer on the server-side for performance-reasons.
You should look into using ETags instead of relying on HTTP HEAD. They are supported in javax.ws.rs.core with the EntityTag class.
You can see a Java-based example using Spring to help explain some of the concepts as well.
I solved the problem as described by me above: Download the file by using the if-modified-since HTTP Header. The ngnix webserver can be configured to return the right information regarding this header.
Related
One of the requirements for our native mobile apps is to retrieve a config JSON that can easily be changed during production. For example, in our apps, we will hit up a URL that returns a JSON response containing features that the business wants enable, disabled or list of base URLs:
GET https://ourwebsite.com/mobile_config/
RESPONSE
{
"enableFeatureA": true,
"enableFeatureB": false,
"baseUrls": [
"foo",
"bar"
]
}
To meet the requirement of it being configurable during runtime, we're planning to use a content management system, which has a publishing feature. We've recently been looking into AEM as our new publishing tool. My idea is to save a JSON file into the AEM content manager, and when a GET request is made to the URL, the backend will read the JSON file and return it as an 'application/json' content type. However, I can't find any obvious way to achieve the result of returning the contents of the JSON file when making a request using AEM.
Being fairly new to AEM, my guess is just create a jsp file to read the contents. But how do I return it? How do I manage the 'routes'? Do I need to set up a template? Is this the best approach?
Any guide is appreciated.
AEM reserves the use of .json as an extension; it uses it for RESTful views of content in a JSON format. You might have luck using a different file extension, such as filename.foo - but it'll probably give you the wrong mimetype unless you configure Dispatcher to fix this. This probably isn't the most considerate approach.
In short, if you upload a .json file into the DAM, you won't get what you expect. You'll end up with a path that looks something like /content/xyz/abc.json - but it'll render with the AEM JSON code, and give you the properties from the datastore instead of your file.
It is possible to turn off AEM JSON output, but it tends to break things as noted in the Security Checklist. Besides it tends to be blocked in Dispatcher configuration anyway most of the time (depending on local policy).
One approach, assuming that Dispatcher does allow JSON requests (or that you are able to permit this safely) is to create a JSON.jsp a page or component, and have the JSON data as a property. This also can have side effects in the authoring interface, as the JSON can interfere with the edit dialogues.
I think the best option might be to use static replication to publish the file onto a static webserver. This is done as a separate replication queue. There's a related question here that may help.
I have recently done something pretty much identical to what you are asking. I created a featureToggling AEM component to allow runtime authoring of specific features of standalone angular applications. The angular application looks to the JSON content stored for my featureToggle component as the app initialises and enables/disabled features based on the values stored.
The component itself is a multifield that allows me to add multiple feature types with a toggle switch to turn them on or off. The JSON for ALL AEM pages is actually exposed by appending .infinity.json to the end of the url. My angular app simply reads from and interprets the JSON on initialise then shows/hides features of the app.
EG: http://[mysite]/[featureTogglePagename].infinity.json
In Formhub, in the form management page, there was a button on the right to connect to bamboo.io . Doing that gave me a handy URL having all the collected data in a JSON, through which I could dynamically load latest collected survey data in my application. See screenshot.
As an example, all entries submitted on this form (and its ODK Collect counterpart), were collecting at this URL in JSON format which I could then import into my javascript and do mapping, data visualization, etc.
I'm using formhub's successor, ona.io now because formhub.org isn't loading half the time, and I read in some article that they don't intend to keep it up and one should go to ona.io now to do the same thing. It's the same thing (at much better speed, I can see that formhub.org was having server problems), But it's missing the easy bamboo.io integration. There is a "REST Services" tab mentioned below in the form's dashboard..
..that I can't make sense out of. What am I supposed to enter as "service URL" ?
I tried looking up ona.io's documentation but it's all for API and involved CURL commands etc. I'm checking out bamboo.io and even there I'm not getting any help. Everywhere they're going into advanced usage of custom servers, API's CURL GET etc. But I just want the simple live JSON data through a simple link that I was getting in formhub.
I'm open to any other methods if available.. the end goal is to have a URL that carries all the accumulated data submitted by the form in JSON or CSV format. I want to import JSON (or CSV) from this set URL in my javascript on a web page hosted on my own server.
Note that it's not possible for me to self-host an ODK Aggregate instance. Ona.io is doing that for me.
In the traditional way of exporting data at ona.io (and formhub before it), the system creates a new file each time you export, with a timestamped file name. It doesn't auto-generate any single URL having latest data.. one has to go there and generate manually. That won't do... need a fixed URL which will have all latest data JSON or CSV.
Found a way. ona.io provides the same thing natively; I don't need bamboo.io (besides, even that is being retired same as formhub!)
All you need to do to get started, is to go here:
https://ona.io/api/v1/data?owner=[your usename]
From there you'll find URL's to the data collected by your forms, and you'll get the direct URLs for getting the data in JSON / CSV / many other formats.
I am in the process of creating an android app (my first) that consumes a REST API.
I use a background job to fetch content and I plan to use a GET request with a from_id parameter in order to get more content. Of course anything fetched from the API gets stored in the SQLite db (I am using greendao) and the app only uses data that is already present there, in order to be snappy.
So, the question is: What happens if a given record is updated on the server? If records once read are cached, how come the app will notice that there are changes to sync? Which strategies are feasible solutions?
Thanks.
EDIT:
As Satish P points out in his answer, the client-server communication is handled with ETag (and I must add the possibility of using If-Modified-Since).
But my main concern, is how to mix this with the app UI. Given this example:
A list of elements, which have been retrieved from the REST service but client-side are read from the local database to make the app more responsive.
User clicks in one of those elements and a detailed view is show. Again, the data is loaded from the local database. I guess that at this point a GET request for the specific record is requested, either with ETag or If-Modified-Since headers.
It happens that the server returns a modified record, thus the local data is modified, so now it's time to update whatever the user is seeing.
Problem: If the detailed view is already populated because the local database read was already done when the remote request returns, how can I update the view? I don't think that just replacing current data with the fresher one is acceptable, the user would see a change out of the blue.
Satish's answer is absolutely right in terms of what you need your server to do. The gist is that it needs to support ETags and 304 response codes in case the content hasn't changed since the last time you got it from the server. On the client side now, there are essentially three strategies you can follow (each with it's own pros and cons):
Only use the cache if the content hasn't changed. That means you will always do a request and will display a progress bar to the user. If the server returns 304, then your content hasn't changed, and the request will be pretty fast (the moment you see that, you display the cached content). If the server actually returns new content, you continue showing the progress bar, and when the content is loaded you display the new content. The good thing about this is that the user will only ever see valid content, therefore avoiding a lot of headaches on your part. The bad thing is that the app does not appear that fast (especially if the content has changed and you are in a very slow connection).
Use only the cache for a predefined period and then fallback to first case. There are a couple of cache header to define that period ('max-age' and 'Expires'). Before that period you always use the cache (without doing a request), and after that you do a request and see if the content has changed. The good thing about this method is that for during the period mentioned above, the app is really fast. The bad thing is that there is a possibility that the user is looking at incorrect content.
Use both the cache and the network for a predefined period, and then fallback to the first case. You can use the cache headers mentioned earlier in a different way. Instead of only showing the cached content, you can actually display the cached content AND do a request in the background. If that request comes back with a 304, fine, else you will have to update you UI with the new data (expect two responses, one with the cached data and one with the newly retrieved data). The positive with this is that you get both a fast experience and valid data (most of the time). The negative is that you add a lot of complexity to your app (what happens if the user interacts with the stale data, and then a second response comes in etc).
All in all, every strategy is valid depending on the use case. For example, if the user can't interact with the screen that displays the data (like a tv program), the third option is pretty good. If it is crucial that the user sees correct data (a financial app let's say), then the first option is best. If speed is more important than having the latest data (a game or something) then the second option is your best choice.
How efficient the client can do caching is solely dependent on how much support you get from the REST API your client is accessing.
Using ETag is the industry standard to make caching on the client side more efficient and also server to serve the request faster. In short ETag is LIKE an MD5 hash of the content returned. More about ETag here: http://en.wikipedia.org/wiki/HTTP_ETag
If it is a popular API like Google, Facebook etc they inherently support ETags.
Please look at links below:
ETag usage best explained here: https://developers.facebook.com/docs/reference/ads-api/etags-reference
When the client does a GET on the a particular resource, the server when responding back with the content should include an ETag.
The client should store the ETag for that resource against the data cached.
Whenever the client is using the cache information, it should verify the cache using the ETag. Can work in multiple ways depending on the service implementation again
Make a usual GET on the resource and include the ETag as part of the request. If the content did not change the service will ideally no return any data but will give a specific code like (304 - Not Modified). Client knows that the cache is still valid and continues to use it
Make a HEAD call on the resource and the ETag is returned. It is part of the standard HTTP Specification. http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (Refer 9.4). In this case Client will verify the ETag and decide whether or not to make the GET call.
Sample of a resource in the above explaination is like below
GET http://serverapi.com/employees/2312312312
The screen update could be handled fairly gracefully by Javascript, rewriting specific elements in the DOM -- optionally applying CSS formatting to call attention to the change in the UI -- if each datum in the UI has a unique ID/container ID or can otherwise be targeted by JS.
A simple/istic way to defeat some caching is to append a query string to the resource you want. For example, a file named testfile.csv can be accessed as easily at testfile.csv?12345 -- and the next time you want to bypass the cache, just update the query string, e.g., testfile.csv?23456. If updating a query string manually is arduous in your context, get a bit more clever at the cost of a modest hit in performance via PHP: call on the resource as testfile.csv to cause the query string to auto-update on every query after the resource is modified; the updated version gets served instead of the cached one.
Lately what I've been doing is using GraphQL with Apollo. It handles all this stuff automatically which is plain awesome.
I have some questions about developing a Android application which shall be able to communicate with a NodeJS server.
The Android application gathers some data and saves everything in a .csv file.
This file now needs to be uploaded to a NodeJS server. The NodeJS server should save the file as well as storing the content in a MongoDB.
My question now is how I should implement the communication between the Android device and the server.
I know how to upload a single file to a NodeJS server using a HttpURLConnection with a DataOutputStream.
But I need more than just uploading the file because I need a unique identification of each Android device.
I thought about using the (encrypted) Google account E-Mail address of the user to distinguish the devices. I am not interested in knowing who uploads which data but I need to store the data for each device separately.
The problem is that I don't know how to communicate between the device and the server.
If I upload a file via HttpURLConnection and DataOutptStream it seems that I can only upload the file without any additional information like the unique key for the device.
I also thought about uploading the file via sockets. But I am not sure how to handle huge file sizes (5 MB or more).
I am not looking for code fragments. I rather need some hints to the right direction. Hopefully my problem was stated clearly and someone can help me with this.
Using a HttpUrlConnection on the Android side, and a RESTful server on the Node side would be a straightforward option.
You can embed information into the URL in a RESTful way:
pathParam: www.address.com/api/save/{clientId}/data
queryParam: www.address.com/api/save/data?c={clientID}
each uniquely identifying the client. This can be whatever scheme you choose. You will have to build the HttpUrlConnection each time as the URI is unique, and important!
The server side can then route the URL however you see fit. Node has a number of packages to aid in that (Express, Restify, etc.). Basically you'll grab the body of the request to store into your DB, but the other parameters are available too so it's all a unique and separated transaction.
Edit: The package you use for RESTful handling can stream large files for you as well. Processing of the request can really begin once the data is fully uploaded to the server.
Using a socket would be nearly just as easy. The most difficult part will be 'making your own protocol' which in reality could be very simple.
Upload 1 file at at time by sending data to the socket like this:
54::{filename:'myfilename.txt',length:13023,hash:'ss23vd'}xxxxxxxxxxx...
54= length of the JSON txt
:: = the delimiter between the length and the JSON
{JSON} = additional data you need
xxx... = 13023 bytes of data
Then once all the data is sent you can disconnect... OR if you need to send another file, you know where the next set of data should be.
And since node.js is javascript you already have wonderful JSON support to parse the JSON for you.
Would I suggest using a socket? Probably not. Because if you ever have to upload additional files at the same time, HTTP and node.js HTTP modules might do a better job. But if you can guarantee nothing will ever change, then sure, why not... But that's a bad attitude to have towards development.
Good day, SO.
Using Trigger.IO's javascript SDK, is it possible to create a Forge File object from Base64 data suitable for passing to Forge's forge.request.ajax method?
I am attempting to upload an image to a server that requires submission in the form of an oldschool multipart/form-data request (can't be changed, unfortunately) from my TriggerIO mobile app. The image comes from canvas.toDataURL() in Base64 format.
Android 2.2 lacks both FormData, BlobBuilder and typed arrays -- making this exercise difficult via XMLHttpRequest (though I have it working well on all other modern platforms). I'm wondering if I might be able to get some joy using Trigger IO's Forge's forge.request.ajax?
Per #James Brady's comment, this cannot be done as of the current version (v1.4).
" The File object has to come from the native side originally "