I'm using OkHttp on my project for downloading a binary file on separate chunks using the Range HTTP header.
Say I want to download a file with content length of 9968738 bytes. Splitting it on equally size chunks with the exception of the last chunk request, where I force it to get all that's left:
Range: bytes=9138008-9968737 <-- request for last chunk
Despite I make a request for this range though, the response I'm getting is like:
Content-Range: bytes 9138008-9968735/9968738
I tried the same request with cURL pointing to the samer URL and I get the expected response.
Is this an issue from OkHttp or my misunderstanding of the HTTP Range header?
Related
We are struggling to get our server set up to stream videos with react-native-video on Android. All the files to be streamed are in .MP4 format and vary in size from 50-100 MB. I am not able to provide the URL to reproduce or share any of the MP4 files. I have tried Android player and Exoplayer and I am not seeking the video as we speak.
For small videos there doesn't seem to be problem, as the server receives HTTP request with the following headers from the player:
{
"name":"test.mp4",
"User-Agent":"stagefright/1.2 (Linux;Android 10)",
"Connection":"close",
"X-Forwarded-Proto":"https",
"X-Forwarded-For":**********,
"Host":**********,
"Accept-Encoding":"gzip",
"ssl_session_id":**********,
}
And we serve the test.mp4 file. In this case everything works on the client side (video plays fine), even though we are getting the following error on server side as react-native-video is sending the same identical HTTP request again. Why is it sending multiple requests?
Failed to handle request.: java.io.IOException: Broken pipe
....
Failed to send error response: java.lang.IllegalStateException: UT010019: Response already commited
For both identical requests we return a response with the file content as body, 200 STATUS OK and the following headers:
{
"Server": "nginx",
"Date": <date>,
"Content-Length": <length in bytes>,
"Connection": "keep-alive",
"Expires": 0,
"Cache-Control": "no-cache, no-store, max-age=0, must-revalidate",
"Pragma": "no-cache",
"Content-Disposition": "attachment; filename="test.mp4",
}
What type of headers is react-native-video expecting?
The situation changes when files get larger (~100 MB). At some point react-native-video starts to send range requests. When? However the first request is without Range header parameter and identical to the one before. As usual we send the same response and produce same errors. However from there on the player starts to send repetitive range requests similar to the one described above, but with the Range header parameter like this:
{
...,
"Range":"bytes=1752325-"
}
All the following requests follow the same pattern: bytes=<start-bytes>-, which means requesting bytes from some value to end. The value <start-bytes> increases with every new request (sometimes there are requests with the same value multiple times). Why is it requesting the same bytes over and over again and not requesting for specific ranges?
We have range requests supported on our server and thus responding with valid file data,
206 STATUS PARTIAL CONTENT and the same headers with the following addition (values are for example):
{
....,
"Content-Range": "bytes 0-11546/115461173"
}
What type of response is react-native-video expecting?
With those range requests the server is also producing the errors described above and the video is getting freezed very often on the client side for minutes. After for some time the video starts to play fine and the errors are not being produced on the server side.
I have several questions marked with bold in the text that I am confused of. Also I could not find any documentation describing how the communication with the player works and what kind of requests is it sending and responses is it expecting.
Is there any documentation describing the protocol or could you hint anything I might be doing wrong here?
"mp4" does not tell the player very much about the file. The player needs information about the codecs, codec features, file layout, and a dozen or so other things. With mp4 this metadata may be at the beginning or end of the file. Its also possible the file you point to is not actually an mp4 so the player needs to check for that as well.
So how does the player get this information? Its starts to download the file. Once is has a few bytes the player may need to cancel the request because the data it needs was not at that location in the file. But there is no cancel mechanism in HTTP, so it just terminates the TCP session; hence java.io.IOException: Broken pipe. It may appear the player is downloading the same data multiple times, But its not because the request was canceled before the request was finished. As the player is scanning the file it can leap-frog over large blocks using the data it has learned so far seeking for the metadata needed.
To give the player the best chance of finding the metadata quickly, make sure it is art the start of the file. This is called "fast start". Searching google for "fast start mp4" will tell you how to do that.
I have an api that returns an object as the response, and an etag and max age as headers. The response looks like this:
HEADERS:
'x-frame-options': 'SAMEORIGIN',
'x-xss-protection': '1; mode=block',
'x-content-type-options': 'nosniff',
'x-download-options': 'noopen',
'strict-transport-security': 'max-age=15778476000; includeSubDomains',
'access-control-allow-origin': '*',
etag: 'c69148a0489a95058e729bde7fd4bf32bf2077b1cba8d4fcf0c2da6e696fa33e',
'cache-control': 'private,max-age=43200'
BODY:
{
id: 1985,
url: "https://example.com",
...
}
The desired scenario is that an android application makes the request to ask for this data. The Api returns the data, along with an max age of 43200 secs.
If a request is made before 43200 secs pass, the application has the data from the last response cached. The application makes the request nevertheless, the back-end service compiles the response data, uses the request's etag to decide whether the response data has changed. If the data has changed, it returns a 200 http status and the data. Otherwise it returns a 403 status and no data.
The application receives the response. It uses fast networking to handle caching (says my android teammate). If a 200 status code was returned, the data are updated. Otherwise the application keeps the old data.
If a request is made after 43200 secs have passed, the application no longer has the cached response or it's etag. The request is made, the data are considered as 'new' even if nothing has changed in the data, the status code 200 is returned along with max-age header as above.
What actually happens:
For some reason, after the first request is processed and the application receives the data, no request is made until 43200 secs have passed. The android developer says they see that the request is made and 0 bytes are returned, but when I monitor the requests in the server I don't see any made towards this API.
This doesn't make sense, since max-age does not imply that no requests are made. It simply instructs the application to keep the data in the cache for the duration.
Am I missing the idea of how cache, etags and max-age work?
Back-end is built in node js, and uses express for routing.
You've only set the max-age and private directives in the Cache-Control header. The actual behaviour you have described is the correct behaviour since max-age directive has no bearing on forcing the cache to validate responses each time a request is made. For that, you have to add the no-cache directive as well to the Cache-Control header.
The no-cache directives tells the cache to always validate the stored response with the origin server before serving it (i.e., the desired behaviour you have described). Upon revalidation, the stored response will be valid for another 43200 secs (max-age). Without the no-cache directive, the HTTP client is free to make use of cached responses. Which I guess is why your friend said the request was made, but 0 bytes were returned (browsers also show 0 bytes for responses served from the cache). And which is also why you didn't observe any incoming requests to the server.
Have a look at this article from Google for a good overview on HTTP caching:
https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching
If you need in-depth detail on how responses are constructed from caches, have a look at the RFC7234 spec: https://www.rfc-editor.org/rfc/rfc7234#section-4
I'm using Volley to make a server call, which can in some cases return a large response - a few megabytes JSON.
When the server response is more "normal" size, everything works, but with a large payload Volley throws a NetworkError without any details attached and logs a weird message - BasicNetwork.performRequest: Unexpected response code 200.
I'm pretty sure the issue is with response size, so my question is whether there a known limit for HTTP request/response size in Android/Volley, or a setting to change it. I know some HTTP libraries have it.
I know the server is fine as there are other clients using the same server endpoint in the same way.
I started debugging inside the library code and found that the response code is indeed 200 and the correct data is being received. The request content is being copied from response stream in chunks, but at some point something goes wrong in the copying.
This is the closest I could get to the original exception. Apparently, a java.net.ProtocolException is getting thrown sometime during the while loop at first breakpoint shown on the picture. You can also see the actual JSON content is there.
Then this exception gets handled and re-thrown a couple more times:
I facing problem with downloading zip file from server means by using Url connection i am able to download a file but my requirement is whenever i downloading that time may internet connection may lost or app may close for some other reason . that time again i need to download that file but not from starting, means if started downloading and 40% download and then connection, then again i need to download that from 40%.
how to do that? i browsed but i did not find my requirement? please any on e can help?
if(new File(filePath).exists()){
//ucon.disconnect();
// ucon.setAllowUserInteraction(true);
ucon.setRequestMethod("GET");
ucon.setRequestProperty("Range", "bytes=" + new File(filePath).length() + "-");
inStream.skip(new File(filePath).length());
ucon.connect();
}
You should use Range request of HTTP protocol to request part of the file.
First you have to send HEAD to see if the server you are talking to supports range requests by sending HEAD with Accept-Ranges: bytes header.
If server supports it, it will return 200 OK response and then you can safely send GET request with byte offsets in Range: 10-100 header.
The above GET request will query the server for data starting from 10th byte to 100th byte and if request succeeds, it will return 206 Partial Document response
I am study about doPost and get method to send data from android app to tomcat server.
I am trying to check the size of http request and response because I am sure that request and response was sending data and catching data from tomcat server.
I used this example ,
http://www.avajava.com/tutorials/lessons/how-do-i-determine-the-content-length-of-a-request.html
to check the contents length.
After I get contents length, i get two questions
First, contents-length represents whole size of http request ?
if it is not, then how can i check the size of http request (by bytes)?
second , I am still having a trouble to getting size of response ?
I tried
How to Get the HTTP Response Size in Java (in Bytes)
Determine size of HTTP Response?
but i was getting error to get size of http response
is there any way to get size of http response ?
thanks
No, it is not the size of the whole request. It is the length of the message body, excluding status line and headers.
Not all servers provide the content length. If you use chunked transfer encoding, the content length cannot be determined without downloading the whole response. If the server sends the "Content-Length" header, you can get the size of the message body. You need to hook the socket stream to get the actual (full) response size.