I wrote an android application that part of it is to handle upload and download documents. Currently I am using the Microsoft Azure server to save the files on.
The way I am currently doing it is by turning the files to a string and saving it that way on the Azure server:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
FileInputStream fis;
try {
fis = new FileInputStream(new File(Uridata.getPath()));
byte[] buf = new byte[1024];
int n;
while (-1 != (n = fis.read(buf)))
baos.write(buf, 0, n);
} catch (Exception e) {
e.printStackTrace();
}
byte[] bbytes = baos.toByteArray();
item.setStringFile(Base64.encodeToString(bbytes, Base64.URL_SAFE));
item.setName(Uridata.getLastPathSegment());
where item is my class that saves the string representation and the name of the file and is being loaded to the Azure, Uridata is an Uri instance of the file chosen.
I have one main problem with this solution and it is the limit on the file size.
I am searching for a good server to use instead of the Azure (maybe a RESET one) and if there is a better way to save files of all kinds (pdf, word...).
I will also want in the future to use the same data in a web interface
Does anybody have any suggestions on how to do it?
Thanks in advance!
To start, you don't have to transform the file into a string, you can just save it as a file. You have the possibility of losing data by continuing to do that. See: How do I save a stream to a file in C#?
If you're looking for another service to save files, then you should look into Azure Blob Storage. It will allow you to upload as much data as you want to a storage service for arbitrary files. See for example:
https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/
Lots of Intent actions, like ACTION_VIEW, take a Uri pointing to the content the action should be performed upon. If the content is backed by a file -- whether the Uri points directly to the file, or to a ContentProvider serving the file (see FileProvider) -- this generally works.
There are scenarios in which developers do not want to have the content reside in a file for sharing with other apps. One common scenario is for encryption: the decrypted data should reside in RAM, not on disk, to minimize the risk of somebody getting at that decrypted data.
My classic solution to sharing from RAM is to use ParcelFileDescriptor and createPipe(). However, when the activity responding to ACTION_VIEW (or whatever) gets an InputStream on that pipe, the resulting stream is limited compared to the streams you get when the ContentProvider is serving up content from a file. For example, this sample app works fine with Adobe Reader and crashes QuickOffice.
Based on past related questions, my assumption is that createPipe() is truly creating a pipe, and that pipes are non-seekable. Clients that attempt to "rewind" or "fast forward" run into problems as a result.
I am seeking a reliable solution for sharing in-memory content with a third-party app that gets around this limitation. Specifically:
It has to use a Uri syntax that is likely to be honored by client apps (i.e., ACTION_VIEW implementers); solutions that involve something obtuse that client apps are unlikely to recognize (e.g., pass such-and-so via an Intent extra) do not qualify
The data to be shared cannot be written to a file as part of the sharing (of course, the client app could wind up saving the received bytes to disk, but let's ignore that risk for the moment)
Ideally it does not involve the app looking to share the data opening up a ServerSocket or otherwise exacerbating security risks
Possible suggested ideas include:
Some way to reconfigure createPipe() that results in a seekable pipe
Some way to use a socket-based FileDescriptor that results in a seekable pipe
Some kind of RAM disk or something else that feels like a file to the rest of Android but is not persistent
A key critierion, if you will, of a working solution is if I can get a PDF served from RAM that QuickOffice can read.
Any suggestions?
Thanks!
You've posed a really difficult combination of requirements.
Lets look at your ideas for solutions:
Possible suggested ideas include:
Some way to reconfigure createPipe() that results in a seekable pipe
Some way to use a socket-based FileDescriptor that results in a seekable pipe
Some kind of RAM disk or something else that feels like a file to the rest of Android but is not persistent
The first one won't work. This issue is that the pipe primitive implemented by the OS is fundamentally non-seekable. The reason is supporting seek that would require the OS to buffer the entire pipe "contents" ... until the reading end closes. That is unimplementable ... unless you place a limit on the amount of data that can be sent through the pipe.
The second one won't work either, for pretty much the same reason. OS-level sockets are not seekable.
At one level, the final idea (a RAM file system) works, modulo that such a capability is supported by the Android OS. (A Ramfs file is seekable, after all.) However, a file stream is not a pipe. In particular the behaviour with respect to the end-of-file is different for a file stream and a pipe. And getting a file stream to look like a pipe stream from the perspective of the reader would entail some special code on that side. (The problem is similar to the problem of running tail -f on a log file ...)
Unfortunately, I don't think there's any other way to get a file descriptor that behaves like a pipe with respect to end-of-file and is also seekable ... short of radically modifying the operating system.
If you could change the application that is reading from the stream, you could work around this. This is precluded by the fact that the fd needs to be read and seeked by QuickOffice which (I assume) you can't modify. (But if you could change the application, there are ways to make this work ...)
By the way, I think you'd have the some problems with these requirements on Linux or Windows. And they are not Java specific.
UPDATE
There have been various interesting comments on this, and I want to address some here:
The OP has explained the use-case that is motivating his question. Basically, he wants a scheme where the data passing through the "channel" between the applications is not going to be vulnerable in the event that the users device is stolen (or confiscated) while the applications are actually running.
Is that achievable?
In theory, no. If one postulates a high degree of technical sophistication (and techniques that the public may not know about ...) then the "bad guys" could break into the OS and read the data from shared memory while the "channel" remained active.
I doubt that such attacks are (currently) possible in practice.
However, even if we assume that the "channel" writes nothing to "disc" there could still be traces of the channel in memory: e.g.
a still mounted RAMfs or still active shared memory segments, or
remnants of previous RAMfs / shared memory.
In theory, this data could in theory be retrieved, provided that the "bad guy" doesn't turn of or reboot the device.
It has been suggested that ashmem could be used in this context:
The issue of there being no public Java APIs could be addressed (by writing 3rd-party APIs, for example)
The real stumbling block is the need for a stream API. According the "ashmem" docs, they have a file-like API. But I think that just means that they conform to the "file descriptor" model. These FDs can be passed from one application to another (across fork / exec), and you use "ioctl" to operate on them. But there is no indication that they implement "read" and "write" ... let alone "seek".
Now, you could probably implement a read/write/seekable stream on top of ashmem, using native and Java libraries on both ends of the channel. But both applications would need to be "aware" of this process, probably to the level of providing command line options to set up the channel.
These issues also apply to old-style shmem ... except that the channel setup is probably more difficult.
The other potential option is to use a RAM fs.
This is easier to implement. The files in the RAMfs will behave like "normal" files; when opened by an application you get a file descriptor that can be read, written and seeked ... depending on how it was opened. And (I think) you should be able to pass a seekable FD for a RAMfs file across a fork/exec.
The problem is that the RAMfs needs to be "mounted" by the operating system in order to use it. While it is mounted, another (privileged) application can also open and read files. And the OS won't let you unmount the RAMfs while some application has open fds for RAMfs files.
There is a (hypothetical) scheme that partly mitigates the above.
The source application creates and mounts a "private" RAMfs.
The source application creates/opens the file for read/write and then unlinks it.
The source application writes the file using the fd from the open.
The source application forks / execs the sink application, passing the fd.
The sink application reads from the (I think) still seekable fd, seeking as required.
When the source application notices that the (child) sink application process has exited, it unmounts and destroys the RAMfs.
This would not require modifying the reading (sink) application.
However, a third (privileged) application could still potentially get into the RAMfs, locate the unlinked file in memory, and read it.
However, having re-reviewed all of the above, the most practical solution is still to modify the reading (sink) application to read the entire input stream into a byte[], then open a ByteArrayInputStream on the buffered data. The core application can seek and reset it at will.
It's not a general solution to your problem, but opening a PDF in QuickOffice works for me with the following code (based on your sample):
#Override
public AssetFileDescriptor openAssetFile(Uri uri, String mode) throws FileNotFoundException {
try {
byte[] data = getData(uri);
long size = data.length;
ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
new TransferThread(new ByteArrayInputStream(data), new AutoCloseOutputStream(pipe[1])).start();
return new AssetFileDescriptor(pipe[0], 0, size);
} catch (IOException e) {
e.printStackTrace();
}
return null;
};
private byte[] getData(Uri uri) throws IOException {
AssetManager assets = getContext().getResources().getAssets();
InputStream is = assets.open(uri.getLastPathSegment());
ByteArrayOutputStream os = new ByteArrayOutputStream();
copy(is, os);
return os.toByteArray();
}
private void copy(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
in.close();
out.flush();
out.close();
}
#Override
public Cursor query(Uri url, String[] projection, String selection, String[] selectionArgs, String sort) {
if (projection == null) {
projection = new String[] { OpenableColumns.DISPLAY_NAME, OpenableColumns.SIZE };
}
String[] cols = new String[projection.length];
Object[] values = new Object[projection.length];
int i = 0;
for (String col : projection) {
if (OpenableColumns.DISPLAY_NAME.equals(col)) {
cols[i] = OpenableColumns.DISPLAY_NAME;
values[i++] = url.getLastPathSegment();
}
else if (OpenableColumns.SIZE.equals(col)) {
cols[i] = OpenableColumns.SIZE;
values[i++] = AssetFileDescriptor.UNKNOWN_LENGTH;
}
}
cols = copyOf(cols, i);
values = copyOf(values, i);
final MatrixCursor cursor = new MatrixCursor(cols, 1);
cursor.addRow(values);
return cursor;
}
private String[] copyOf(String[] original, int newLength) {
final String[] result = new String[newLength];
System.arraycopy(original, 0, result, 0, newLength);
return result;
}
private Object[] copyOf(Object[] original, int newLength) {
final Object[] result = new Object[newLength];
System.arraycopy(original, 0, result, 0, newLength);
return result;
}
I believe you're looking for StorageManager.openProxyFileDescriptor, function added in API 26. This will give you ParcelFileDescriptor, needed for your ContentProvider.openAssetFile to work. But you can also grab its file descriptor and use it in file I/O: new FileInputStream(fd.getFileDescriptor())
In function description is :
This can be useful when you want to provide quick access to a large file that isn't backed by a real file on disk, such as a file on a
network share, cloud storage service, etc. As an example, you could
respond to a ContentResolver#openFileDescriptor(android.net.Uri,
String) request by returning a ParcelFileDescriptor created with this
method, and then stream the content on-demand as requested. Another
useful example might be where you have an encrypted file that you're
willing to decrypt on-demand, but where you want to avoid persisting
the cleartext version.
It works with ProxyFileDescriptorCallback, which is your function to provide I/O, mainly read pieces of your file from various offsets (or decrypt it, read from network, generate, etc).
As I tested, it's well suited also for video playback over content:// scheme, because seeking is efficient, no seek-by-read as is the option for pipe-based approach, but Android really asks relevant fragments of your file.
Internally Android uses some fuse driver to transfer the data between processes.
I've been experimenting with #josias code. I found some of the query(...) calls came with a projection of _data. Including the data for that column and setting the actual length means more file types can be opened in more apps. Always including _data even when not in the passed in projection allows opening even more file types.
Here is what I ended up with:
private static final String[] PROJECTION = {OpenableColumns.DISPLAY_NAME, OpenableColumns.SIZE, "_data"};
#Override
public Cursor query(Uri url, String[] projection, String selection, String[] selectionArgs, String sort) {
byte[] data = getData(mSourcePath, url);
final MatrixCursor cursor = new MatrixCursor(PROJECTION, 1);
cursor.newRow()
.add(url.getLastPathSegment())
.add(data.length)
.add(data);
return cursor;
}
In my application i want send a file or text over Bluetooth to another Bluetooth device(receiver device may be android,Nokia,LG,etc..). I want to send a file whenever the sever returns the data. For example am checking weather if the climate level falls below any particular given value. It automatically, need to send data over Bluetooth to the receiver device. It wont allow the user to send.
How can I implement it using android Bluetooth API?
And also I need to transfer any file via Bluetooth by converting it to byte array. I have gone through Bluetooth chat example. In that they have given the buffer size of 1024. If the file size more than 1024 byte means how should I transfer. Whether I have to sent each 1024 byte every time and have to merge it at the receiver side or else any other better way is available?
Thanks in advance.
As far as I understand you're asking three questions.
How to send a file whenever a server returns data: You basically open a connection to the server (e.g. http, but might also be any other TCP or UDP-based protocol). Then you listen for incoming data; once you receive data, you trigger whatever action you want. These are some relevant calls for a starting point when your server is not using http (untested, consult the docs for details and alternatives):
Socket s = new Socket('your.server.com', 47000);
s.connect();
SocketChannel c = s.getChannel();
ByteBuffer buffer = new ByteBuffer(1);
c.read(buffer); // blocks until bytes are available
How to initiate a Bluetooth connection automatically: Obtain the target device's BluetoothDevice object, then connect to it - as in the BluetoothChat demo.
How to send a file / more than 1024 bytes via Bluetooth: Yes you have to split your data into blocks on the sending side and reassemble them on the receiving side (mind to send the filesize before the actual data, so the receiver knows when the file is complete). You can also use reasonably larger byte buffers. I'd recommand using a maximum block size of 64 Kb: This allows you to resend blocks without too much (time) cost and doesn't consume too much memory.
As a starter regarding 3., something like this could the core of the sending side (untested and without error handling, just to give the idea):
// Send the file size
OutputStream out = socket.getOutputStream();
ByteBuffer header = ByteBuffer.allocate(8);
header.putLong(file.length();
out.write(header.array());
// Send the file in chunks
byte buffer[1024];
InputStream in = new BufferedInputStream(new FileInputStream(file));
int length = in.read(buffer);
while (length > 0) {
out.write(buffer, 0, length);
if (length < 1024) break;
length = in.read(buffer, 0, sizeof(buffer));
};
... and the receiving side:
// Receive and unmarshal the file size
InputStream in = socket.getInputStream();
ByteBuffer header = ByteBuffer.allocate(8);
byte buffer[1024];
in.read(buffer, 0, 8);
header.put(buffer);
long filesize = header.getLong();
long receivedBytes = 0;
// Receive the file
OutputStream out = new BufferedOutputStream(new FileOutputStream(file));
int length = in.read(buffer);
while ((receivedBytes < filesize) && (length > 0)){
out.write(buffer, 0, length);
receivedBytes += length;
length = in.read(buffer);
}
if (receivedBytes != filesize) ... // Assure the transfer was successful
My app receives data through a serial port... they're typically small chunks of data. For example, sometimes it 40 bytes, sometimes 60 bytes. All the chunks of data are separated by a second, or possibly even a minute.
I read that using BufferedInputStream is good for reading chunks of data so that the app doesn't create a lot of CPU overhead by reading data byte by byte.
So that's what I did - just like this example: http://www.roseindia.net/java/example/java/io/ReadFilterFile.shtml
When it works, it works great!
My app gets a complete chunk of data - I was worried that I would receive incomplete chunks, but no, to my amazement it's complete chunks.
However, sometimes it doesn't work so well
What seems to happen is that a small chunk of data doesn't cause the read() method to complete. When a little bit larger chunk comes along later then finally the read() will return. This is undesirable !
I do not want my app to be denied a chunk of data until another chunk arrives.
Question:
How do I ensure that BufferedInputStream.read() returns shortly after a small chunk of data was received ? Is byte-by-byte read the only way ?
I have it solved - the solution was to use smaller buffers... taking the roseindia.net sample, the following fixes the read() to always return after a chunk of data:
final int BUFFER_SIZE = 128;
byBuffer = new byte[BUFFER_SIZE];
try
{
FileInputStream fin = new FileInputStream("Filterfile.txt");
BufferedInputStream bis = new BufferedInputStream(fin, BUFFER_SIZE*2);
// Now read the buffered stream.
while (bis.available() > 0)
{
int iBytesRead = bis.read(byBuffer, 0, BUFFER_SIZE);
}
}
catch (Exception e)
{
System.err.println("Error reading file: " + e);
}
maybe the 8K default buffer was too large and the small chunks of data didn't reliably pass some threshold ?
I am finding that reading one line at a time from a text file on the SD card is rather slow. I imagine that it might be quicker if the file is in internal memory, so I want to copy files from the SD card to internal storage.
The file copy examples I can find on the web seem to involve copying one byte at a time from an InputStream to an OutputStream or from a FileReader to a FileWriter. Is this really the quickest and most efficient method?
If you are pulling the file in for use in your application what I suggest you do is read in the data then stuff the in memory data you have collected into some kind of reader (BufferedReader perhaps) so that you can then read the lines from there.
Here is an example of what I typically do:
// Assumption: I already have the file object I want to read
// Note: I'm not doing any error handling.
InputStream input = new FileInputStream(file);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int bytesRead = 0;
while( (bytesRead = input.read(buffer)) > 0){
baos.write(buffer, 0, bytesRead);
}
StringReader stringReader = new StringReader( new String(baos.toByteArray()) );
BufferedReader bufferedReader = new BufferedReader( stringReader );
for(String line : bufferedReader.readLine()){
// TODO: Handle each line appropriately or something
Log.d("Reading Data Example", line);
}
One of the truisims of CS that only becomes more true with time as CPUs get faster is: I/O is slow.
If you want speed, generally your best bet is to do as few I/O's as possible. Ideally, find out how big that file is, allocate that much memory, and then read the entire thing in one big I/O. Then you can just access the data from program memory. If you might not have enough RAM for every concievable file size then you might have to do a bit more work, but this is what you should strive for.