I am facing a strange issue, and I am not able to debug it out. I have implemented a logic for uploading stream of data and am using Volley for the same, I have customized a logic little bit in HurlStack, addBodyIfExists api,so that body of type "application/octet-stream" can be handled.
My logic is to post progress to user, so that UI can be updated indicating user progress in upload, below my logic for same.
int toRead = length; // File length
byte[] data = new byte[4096];
connection.setDoOutput(true);
if(length != -1) {
connection.setFixedLengthStreamingMode(length);
} else {
connection.setChunkedStreamingMode(4096);
}
OutputStream os;
int i;
int count;
os = connection.getOutputStream();
int progress= 0;
try {
for(i = 0; (count= is.read(data)) > 0; ++i) { // is, is not null and contains a valid input stream
os.write(data, 0, count); // at this line am getting unexpected end of stream
progress+= count;
if(i % 20 == 0) {
rs.deliverProgress(progress, 0L);
progress= 0;
}
}
os.flush();
} finally {
if(is != null) {
is.close();
}
if(os != null) {
os.close();
}
}
on executing above code am getting this, although I have verified, output stream is not null, neither do input stream, it fails in first iteration of read loop itself, am seeing it has read 4096 bytes and then trying to write the same.
java.net.ProtocolException: unexpected end of stream
at com.android.okhttp.internal.http.HttpConnection$FixedLengthSink.close(HttpConnection.java:326)
at com.android.okio.RealBufferedSink.close(RealBufferedSink.java:174)
at com.android.okio.RealBufferedSink$1.close(RealBufferedSink.java:142)
any help in debugging above will he highly appreciated.
This may help you :
That exception is thrown by FixedLengthInputStream when the expected number of bytes (usually set in the content-length header of the response) is larger than the actual data in the response.
Check that the content-length header is correct. (If you're supplying your own value for the content length, make sure it is correct.)
It would help to see your code that sets up the input stream.
Already Fixed it, please add "Accept-Encoding", "identity" in header, then the server-side will get command that it will not modify the response, then send back to Clients.
If you have checked everywhere in your code and tried every solution in stackoverflow and github but the issue still occurs, and you have only tested your code on emulator, then, you should try to run your code on your real device instead. Maybe it will work, or maybe it won't, but if you feel desperate, just have a try, seriously. I was astonished when I happened to find that my code ran with bugs on emulator everytime but successfully on my mobile phone. Besides, the code also ran sucessfully on others' android emulators. So I guess there is something wrong in my android studio configuration that I can't find out. I have no idea why this happen, just like we don't know why "Clean Project/Invalidate caches" sometimes works better than any solution.
It is a little strange that your data length might be unknown.
Is it a media file? Or a live stream?
Anyway, I tried to upload my live stream data. And it happened in the same error.
I added this setting to the Connection and solved my problem.
Transfer-Encoding : chunked
("setChunkedStreamingMode" didn't work. I still don't know why.)
This happens for me on android emulator and doesn't happen on my physical android device.
I was doing GET request to flask server running on 0.0.0.0 on my laptop from the android app.
To fix it on the emulator, add the servers ip address in the emulators proxy.
see How to set up Android emulator proxy settings
The exact problem i had was unexpected end of stream retrofit
Related
Essentially the question was in title. In detail: I am using Realm for storing
users in device storage. I am doing one request to server which returns list with 110k objects, gson parses this, all actions from out of memory were done. On emulator this operation continues for about 3 minutes and all saved to db successfully. All json data is valid, if it wasn't, on emulator it would crashed too. But, when I checked on real android device I got this
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected ':' at line 2 column 3 path $[70816].info2000
when gson was trying to parse 70816 object it couldn't, because it's not valid. I re-checked this user in database. All was good. I ran again and got this exception on another object
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected name at line 2 column 1091 path $[56000].accountCrm
So it's randomly.
I got confused and began researching, and I couldn't find anything. These objects come in response body(okhttp), I parse it like this:
Gson gson = GsonBase.getGlobalGson();
JsonReader reader = new JsonReader(responseBody.charStream()); //these are my 110k objects
reader.setLenient(true);
reader.beginArray();
List<ContactCRMRealm> contactCRMRBuffer = new ArrayList<>();
int counter = 0;
while (reader.hasNext()) {
if (counter < BUFFER_SIZE) {
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
counter++;
} else {
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
counter = 0;
contactCRMRBuffer.clear();
}
}
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
contactCRMRBuffer.clear();
reader.endArray();
Exceptions are thrown in this line:
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
I can't figure out but It seems like response body is being broken unexpectedly. Why It works on emulator and doesn't on android device? Few things that I can say: when I comment this line in else block:
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
all parsing side was performed without exceptions. I think it's because of realm. But what exactly? I re-wrote this parsing using createOrUpdateAllFromJson() realm method:
defaultInstance.beginTransaction();
try {
defaultInstance.createOrUpdateAllFromJson(ContactCRMRealm.class, responseBody.byteStream());
defaultInstance.commitTransaction();
} catch (IOException e) {
defaultInstance.close();
} finally {
defaultInstance.close();
}
On emulator was out of memory(wtf), on real device nothing happened for 10 minutes and login exception was thrown(read timeout 10 minutes).Also I was trying to make offset, by 10 thousands users on one request, nothing changed. Please help.
Ok, after a day of investigation, the problem was solved. Realm was 2.3.0 version in the app(yes, really old), so I was thinking maybe need to upgrade to newer verison. I did it, I upgraded realm to 3.7.0(not the last one because rxjava1 is using in the app). That's it. There are a lot of improvements, especially speed improvements, so now all is fine.
My code used to work, it does not work anymore, I tried troubleshooting and can't figure out why.
I have this piece of code in my PHP:
$android_id_01 = $_GET['pmysql_room_id'];
$android_id_02 = "";
$f = fopen("00_android_id_01.txt", "w");
fwrite($f, print_r($android_id_01, true));
fclose($f);
$f = fopen("00_android_id_02.txt", "w");
fwrite($f, print_r($android_id_02, true));
fclose($f);
For troubleshooting I created two android IDs ($android_id_01 and $android_id_02) which are both empty (The first one is From Android and the second one I created directly from PHP).
Now when I launch my Android device, the PHP file is executed from server side and both the text files are created empty and identical. Now my code only works when I use $android_id_02 and not $android_id_01 from the code below:
if ($android_id == '')
{
//my code
}
(Yes when I use either one of the $android_id_01 OR $android_id_02 I rename it to $android_id and comment out the other one)
My question is, although this was working yesterday, why does it work with $android_id_02 = ""; and not $android_id_01 = $_GET['pmysql_room_id']; even though they are both empty????
I don't know what changed from yesterday to today.
Ok after a bit of troubleshooting I found a solution, strange though.
On the server side "display_errors" under PHP settings must be turned off. Somehow having this on interferes with the json_encode sent back to android client. (even though my code is not generating any errors)
I opened a socket between an Android app and a python server. The combination is that the Server listens, and android connects to the Server.
Here is the server code. The problematic part takes place in the definition of handle :
import SocketServer
from time import sleep
import sys
HOST = '192.168.56.1'
PORT = 2000
class SingleTCPHandler(SocketServer.StreamRequestHandler):
def handle(self):
try:
while(1):
sleep(0.03)
data = self.rfile.readline().strip()
print data
except KeyboardInterrupt:
sys.exit(0)
class SimpleServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
allow_reuse_address = True
def __init__(self, server_address, RequestHandlerClass):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)
server = SimpleServer((HOST, PORT), SingleTCPHandler)
try:
server.serve_forever()
except KeyboardInterrupt:
sys.exit(0)
The connection is established normally, and the Android app sends the following data to the socket:
'0:0'
But the data is received on the Server as:
'\x000\x00:\x000\x00'
The variable that receives the data is:
data = self.rfile.readline().strip()
and printing gives the regular format:
In [2]: print data
0:0
I didn't manage to step into the print function with pdb to see what it does.
I'm looking for a way to convert the '\x000\x00:\x000\x00' to '0:0'.
Please advise on a way to convert the variable. You are welcome to comment/criticize the whole implementation. This is my first project in dealing with sockets so i don't know the pitfalls.
Update
This was the original Android code:
String podaci = "0:0";
public void Socketic() throws IOException {
Socket mojSocket = new Socket(urlServer, port);
DataOutputStream izlazdata = new DataOutputStream(
mojSocket.getOutputStream());
while (podaci != "end") {
try {
Thread.sleep(60);
} catch (InterruptedException e) {
e.printStackTrace();
}
izlazdata.writeChars(podaci);
izlazdata.flush();
}
izlazdata.close();
mojSocket.close();
};
And the problem was, as you suspected in:
izlazdata.writeChars(podaci);
writeChars uses the method writeChar. The API documentation for writeChar states:
Writes a char to the underlying output stream as a 2-byte value, high byte first...
The two bytes represent the 16bits which UTF-16 uses for encoding.
When we changed it to everything started working:
izlazdata.writeBytes(podaci);
Update
Based on the answers given, here is how the unwanted string is to be interpreted in terms of characters.
This solves my concrete problem, however, if someone would give a more generic solution to what happend here so that a larger lesson can be learned.
If not, i will accept Esailijas answer in a few days.
You need to show the code happening Android but it strongly seems like it's sending data in UTF-16BE. You should specify the encoding on the Android end. The characters are not hexadecimal literally, but because the NUL character is unprintable, python shows \x00 instead.
Another option is to decode it:
self.rfile.readline().decode("utf_16_be").strip()
note that the result of this is an unicode string.
I've been trying all day to make this thing works but it's still not right yet. I've checked so many posts around here and tested so many different implementations that I'dont know where to look now...
Here is my situation, I have a small php test file (gz.php) on my server wich looks like this :
header("Content-Encoding: gzip");
print("\x1f\x8b\x08\x00\x00\x00\x00\x00");
$contents = gzcompress("Is it working?", 9);
print($contents);
This is the simplest I could do and it works fine with any web browser.
Now I have an Android activity using Jsoup that has this code :
URL url = new URL("http://myServerAdress.com/gz.php");
doc = Jsoup.parse(url, 1000);
Which cause an empty EOFException on the "Jsoup.parse" line.
I've read everywhere that Jsoup is supposed to parse gzipped content without having to do anything special, but obviously, there's something missing.
I've tried many other ways like using Jsoup.connect().get() or InpuStream, GZipInputStream and DataInpuStream. I did try the gzDeflate() and gzencode() methods from PHP as well but no luck either. I even tried not to declare the header-encoding in PHP and try to deflate the content later...but it was as clever as effective...
It has to be something "stupid" I'm missing but I just can't tell what... anybody has an idea?
(ps : I'm using Jsoup 1.7.0, so the latest one as of now)
The asker indicated in a comment that gzcompress was writing a CRC that was both incorrect and incomplete, according to information from here, the operative code being:
// Display the header of the gzip file
// Thanks ck#medienkombinat.de!
// Only display this once
echo "\x1f\x8b\x08\x00\x00\x00\x00\x00";
// Figure out the size and CRC of the original for later
$Size = strlen($contents);
$Crc = crc32($contents);
// Compress the data
$contents = gzcompress($contents, 9);
// We can't just output it here, since the CRC is messed up.
// If I try to "echo $contents" at this point, the compressed
// data is sent, but not completely. There are four bytes at
// the end that are a CRC. Three are sent. The last one is
// left in limbo. Also, if we "echo $contents", then the next
// byte we echo will not be sent to the client. I am not sure
// if this is a bug in 4.0.2 or not, but the best way to avoid
// this is to put the correct CRC at the end of the compressed
// data. (The one generated by gzcompress looks WAY wrong.)
// This will stop Opera from crashing, gunzip will work, and
// other browsers won't keep loading indefinately.
//
// Strip off the old CRC (it's there, but it won't be displayed
// all the way -- very odd)
$contents = substr($contents, 0, strlen($contents) - 4);
// Show only the compressed data
echo $contents;
// Output the CRC, then the size of the original
gzip_PrintFourChars($Crc);
gzip_PrintFourChars($Size);
Jonathan Hedley commented, "jsoup just uses a normal Java GZIPInputStream to parse the gzip, so you'd hit that issue with any Java program." The EOFException is presumably due to the incomplete CRC.
I have a curious problem, respectively a weird effect of using my self-programmed android App.
My app reads out the HTML-source-code of a website and parse it for my desired information. And it work... oh well, not really consistent.
Scenario 1: I use my WLan at home and run my app -> All is working fine. All desired items can be seen in my ListView
Scenario 2: I use my mobile Internet, like Edge or HSDPA -> My ListView is only presenting 1 Item. All of the others are vanished...
I don' t know why. Could there be any time-out, that detain the app to read out the whole HTML-site? But all of the other items would directly follow in the next line of the HTML-source-code...
I have no idea how could I fix it. On google I didn' t find anyone else with the same problem.
Regards, Julian
Here is some code
// With this I get the HTML-source-code
URL url = new URL("http://www.area4.de);
URLConnection conn = url.openConnection();
DataInputStream dataIn = new DataInputStream(conn.getInputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(dataIn, "UTF-8"));
String line;
// Then I parse the code with
while ((line=reader.readLine()) != null)
{
if (line.contains(searchPattern))
al.add(line); //al is an ArrayList
}
That was all I do in my app till now (besides presenting the arrayList in a ListView).
The source code of the site you can see in your browser (Ctrl + u). I search for these lines
THIRTY SECONDS TO MARS //
DROPKICK MURPHYS //
With 3G I only get thirty-seconds-to-mars...
Ah, I solved it. I searched, as it can be seen above, with this code-snippet
while ((line=reader.readLine()) != null)
{
if (line.contains(searchPattern))
al.add(line); //al is an ArrayList
}
With WLan (and my emulator) I really have a new line for each band e. g.:
line1
line2
line3
....
But with Edge or HDSPA all lines I get with Wlan are written in one line.
line1line2line3.... And with my regex i delte all before and after the line when I find a desired target. Hope you understand, it' s difficult to explain it in a foreign language.
A simple
while (line.contains(searchPattern))
fixed it.
You can always try reading whole http response before sending it for parsing. This way you get to see whole document is loaded properly.