I am finding that reading one line at a time from a text file on the SD card is rather slow. I imagine that it might be quicker if the file is in internal memory, so I want to copy files from the SD card to internal storage.
The file copy examples I can find on the web seem to involve copying one byte at a time from an InputStream to an OutputStream or from a FileReader to a FileWriter. Is this really the quickest and most efficient method?
If you are pulling the file in for use in your application what I suggest you do is read in the data then stuff the in memory data you have collected into some kind of reader (BufferedReader perhaps) so that you can then read the lines from there.
Here is an example of what I typically do:
// Assumption: I already have the file object I want to read
// Note: I'm not doing any error handling.
InputStream input = new FileInputStream(file);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int bytesRead = 0;
while( (bytesRead = input.read(buffer)) > 0){
baos.write(buffer, 0, bytesRead);
}
StringReader stringReader = new StringReader( new String(baos.toByteArray()) );
BufferedReader bufferedReader = new BufferedReader( stringReader );
for(String line : bufferedReader.readLine()){
// TODO: Handle each line appropriately or something
Log.d("Reading Data Example", line);
}
One of the truisims of CS that only becomes more true with time as CPUs get faster is: I/O is slow.
If you want speed, generally your best bet is to do as few I/O's as possible. Ideally, find out how big that file is, allocate that much memory, and then read the entire thing in one big I/O. Then you can just access the data from program memory. If you might not have enough RAM for every concievable file size then you might have to do a bit more work, but this is what you should strive for.
Related
I wrote an android application that part of it is to handle upload and download documents. Currently I am using the Microsoft Azure server to save the files on.
The way I am currently doing it is by turning the files to a string and saving it that way on the Azure server:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
FileInputStream fis;
try {
fis = new FileInputStream(new File(Uridata.getPath()));
byte[] buf = new byte[1024];
int n;
while (-1 != (n = fis.read(buf)))
baos.write(buf, 0, n);
} catch (Exception e) {
e.printStackTrace();
}
byte[] bbytes = baos.toByteArray();
item.setStringFile(Base64.encodeToString(bbytes, Base64.URL_SAFE));
item.setName(Uridata.getLastPathSegment());
where item is my class that saves the string representation and the name of the file and is being loaded to the Azure, Uridata is an Uri instance of the file chosen.
I have one main problem with this solution and it is the limit on the file size.
I am searching for a good server to use instead of the Azure (maybe a RESET one) and if there is a better way to save files of all kinds (pdf, word...).
I will also want in the future to use the same data in a web interface
Does anybody have any suggestions on how to do it?
Thanks in advance!
To start, you don't have to transform the file into a string, you can just save it as a file. You have the possibility of losing data by continuing to do that. See: How do I save a stream to a file in C#?
If you're looking for another service to save files, then you should look into Azure Blob Storage. It will allow you to upload as much data as you want to a storage service for arbitrary files. See for example:
https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/
I have an app that needs to implement a 'share' option to transfer data contained in the device's local database to another device, where that device can take this data and insert it into the appropriate tables in the device's database.
I have communication working over bluetooth, I'm just looking for a good way to transfer this data across. Is there an easy way to do an sqlite dump on device A, transfer this across to device B using bluetooth, and have device B reinsert this data?
Implementing aContentProvider is how you leverage the Android framework to do CRUD operations on a database.
A good way could be to transform a row into Json and unpackage that string on the receiving side to insert it. This way is more testable, easier to recover from errors during send, but might be very slow depending on the size of your data. For speeding it up I would send multiple rows in batches and test to see what batch size would be best for speed and reliability.
A fast way might be to dump the data to flat file then use the FileProvider to provide access to that file as a URI and try something like the following from here
public void sendFile(Uri uri, BluetoothSocket bs) throws IOException {
BufferedInputStream bis = new BufferedInputStream(getContentResolver().openInputStream(uri));
OutputStream os = bs.getOutputStream();
try {
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
// we need to know how may bytes were read to write them to the byteBuffer
int len = 0;
while ((len = bis.read(buffer)) != -1) {
os.write(buffer, 0, len);
}
} finally {
bis.close();
os.flush();
os.close();
bs.close();
}
}
My android program crashes on this line when the file size is very large. Is there any way I can prevent the program from crashing ?
byte[] myByteArray = new byte[(int)mFile.length()];
Additional details :-
I am trying to send a file to server.
error log-
E/dalvikvm-heap(29811): Out of memory on a 136309996-byte allocation.
You should use a stream when reading the file. Since you've mentioned sending to a server, you should stream that file to the server.
As others have mentioned, you should consider your data size (1GB seems excessive). I haven't tested this, but the basic approach in code would look something like:
// open a stream to the file
FileInputStream fileInputStream = new FileInputStream(filePath);
// open a stream to the server
HttpURLConnection connection = new URL(url).openConnection();
DataOutputStream outputStream = new DataOutputStream(connection.getOutputStream());
byte[] buffer = new byte[BUFFER_SIZE]; // pick some buffer size
int bytesRead = 0;
// continually read from the file into the buffer and immediately write that to output stream
while ((bytesRead = fileInputStream.read(buffer)) != -1) {
outputStream.write(buffer);
}
Hope that is clear enough for you to fit to your needs.
Yep. Don't try to read the whole file into memory at once...
If you really need the whole file in memory you might have more luck with allocating dynamic memory for each line and storing the lines in a list. (you might be able to get a bunch of smaller chunks of memory but not one big piece)
Without knowing the context we can't tell, but normally you would parse the file into data structs rather than just storing the whole file in memory.
In JDK 7 you can use Files.readAllBytes(Path).
Example:
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
Path path = Paths.get("path/to/file");
byte[] myByteArray = Files.readAllBytes(path);
Don't try reading the complete file into memory. Instead open a stream and process the file line by line (is it's a text file) or in parts. How that has to be done depends on the problem you are trying to solve.
EDIT: You say you want to upload a file, so please check this question. You don't need to have the complete file in memory.
I am reading a file with:
char [] buffer = new char[300];
FileInputStream istream = new FileInputStream(path);
InputStreamReader file = new InputStreamReader(istream);
size = file.read(buffer);
file.close();
After a few tries, it turns out that the file.read(buffer) reads exactly the number of chars allocated for buffer (in this case, 300, even that the file has much more characers in it).
Can I rely on read() always reading as much as it can, without generating any exception?
Or is this an undocumented feature?
The read method description says:
Reads characters from this reader and stores them in the character
array buf starting at offset 0. Returns the number of characters
actually read or -1 if the end of the reader has been reached.
There is no mention of the buffer allocation issue.
This is very important, and a good thing that it works this way, because it allows you to define the size of the buffer as you want/need and there is no need to guess, no need to code for exceptions. Actually, it is read(char[] buffer) but it works as read(char[] buffer, int size).
Yes you can rely on this call, unless an I/O error occurs, which is already mentionned in the api.
If you look at the code of read(char cbuf[]) you'll notice it calls the method public int read (char[] buffer, int offset, int length).
From Android source code:
public int read(char cbuf[]) throws IOException { read(cbuf, 0, cbuf.length);}
In your implementation, you need to continue reading the file with file.read(buffer) to obtain remaining bytes. The content of buffer needs to be appended to another buffer that will grow, depending on the size of the file you're reading.
You could also allocate that buffer with the size of the file with the method getTotalSpace()
I'm writing a simple budget app for myself, and I'm having trouble figuring out how to write to internal storage. I don't seem to be writing to the file properly, and I can't find any more in depth examples than the Data Storage article on developer.android.com
Basically, I'm trying to write a test float to the MyBalance file, then read it into balance. In my actual code I use try/catch statements around the file in/out operations, but I skipped them to make the code more readable.
float test = 55;
float balance;
byte[] buffer = null;
FileOutputStream fos = openFileOutput( "MyBalance", Context.MODE_PRIVATE );
fos.write(Float.floatToRawIntBits(balance));
fis.read(buffer); //null pointer
ByteBuffer b = ByteBuffer.wrap(buffer);
balance=b.getFloat();
That's the gist of it, anyone see what I'm doing wrong?
Edit:
Thanks for the reply, I went ahead and converted to/from String like you suggested, but I still don't think the file is being created. I have an if statement that reads from it if it exists in onResume() and it isn't being run. Lemme post some of my code.
Here's how I'm writing the file, (setbal is an EditText and balanceview is a TextView):
balance = Float.valueOf(setbal.getText().toString());
balanceview.setText(setbal.getText());
balstring = String.valueOf(balance);
for (int i = 0; i < balstring.length(); ++i)
try {
fos.write((byte)balstring.charAt(i));
} catch (IOException e) {
e.printStackTrace();
}
I check if the file exists in onResume() like so:
File file = new File("data/data/com.v1nsai.mibudget/balance.txt");
Is that where an internal file for that context would be stored?
So this isn't exactly what you asked for, but this is how I have it working for Strings, and it may be helpful to you to see. (You could box the primatives and toString them of course if you wanted to use this code.)
Writing
FileOutputStream fos = context.openFileOutput("savedstate.txt", 0);
for (int i = 0; i < out.length(); ++i)
fos.write((byte)out.charAt(i));
Reading
StringBuilder inb = new StringBuilder();
FileInputStream fis = this.mContext.openFileInput("savedstate.txt");
int ch;
while((ch = fis.read()) != -1)
inb.append((char)ch);
Update
One thought that springs to mind is that you may not want to trust using a File object with a hand typed full path to the file. Instead, just use the FileInputStream with the context object and a relative path like in my code, then see if you get a String back of some length or something like that, or an exception that the file doesn't exist.
If you are really curious of where the file is created, or want to see it with your own eyes, I believe you can browse to it on your phone through the file manager in DDMS.
One last thing, I would suggest moving the try/catch block outside of your writing loop. Since it is an identical task being repeated, there is no need for the overhead of that approach, though it is typically good practice to minimize the size of your try/catch blocks.
Ok really one last thing, if you want to use the File object with the full path, you might want to have the path be the following:
File file = new File("/data/data/com.v1nsai.mibudget/balance.txt");
The beginning slash may make all the difference.