I am working with Skobbler SDK 2.3 and I am wondering how to update the map styles in SKMaps.zip.
The problem:
I modified the contained "daystyle", but I noticed that this only takes effect after deleting the app data. This is really not what I want. It looks like SKPrepareMapTextureThread(android.content.Context context, java.lang.String mapTexturesPath,java.lang.String zipName, SKPrepareMapTextureListener listener).start()
only copies the files if SKMaps folder doesn't exists. Does anybody know if there is such a check inside the start() method, or (better) how to deliver modified SKMap styles to the users?
SKPrepareMapTextureThread only copies the files if SKMaps folder doesn't exists and this is how it is intended to be, as the unziping of the map resources takes a rather long time and is intended to be executed only once.
To update a style a workaround will be required:
1) delete the file .installed.txt from map resources path and call SKPrepareMapTextureThread so that the resources will be restored from assets. Although this is the easiest way, it is also the most time consuming:
final File txtPreparedFile = new File(mapResourcesDirPath + "/.installed.txt");
if (!txtPreparedFile.exists()) {
txtPreparedFile.delete();
}
new SKPrepareMapTextureThread(context, mapResourcesDirPath, "SKMaps.zip", skPrepareMapTextureListener).start();
2) a more optimal approach would be to write a routine that replace the old style with the new one
copyFile(new File("path/daystyle.json"), new File("mapResourcesDirPath/SKMaps/daystyle/daystyle.json"));
...
public static void copyFile(File sourceFile, File destFile) throws IOException {
if(!destFile.exists()) {
destFile.createNewFile();
}
FileChannel source = null;
FileChannel destination = null;
try {
source = new FileInputStream(sourceFile).getChannel();
destination = new FileOutputStream(destFile).getChannel();
// previous code: destination.transferFrom(source, 0, source.size());
// to avoid infinite loops, should be:
long count = 0;
long size = source.size();
while((count += destination.transferFrom(source, count, size-count))<size);
}
finally {
if(source != null) {
source.close();
}
if(destination != null) {
destination.close();
}
}
}
Related
I've got an Android app written in Kotlin targeting framework 30+, so I'm working within the new Android 11 file access restrictions. The app needs to be able to open an arbitrary .zip file in the shared storage (chosen interactively by the user) then do stuff with the contents of that .zip file.
I'm getting a URI for the .zip file in what I'm led to understand is the canonical way:
val activity = this
val getContent = registerForActivityResult(ActivityResultContracts.GetContent()) {
CoroutineScope(Dispatchers.Main).launch {
if(it != null) doStuffWithZip(activity, it)
...
}
}
getContent.launch("application/zip")
My problem is that the Java.util.zip.ZipFile class I'm using only knows how to open a .zip file specified by a String or a File, and I don't have any easy way to get to either of those from a Uri. (I'm guessing that the ZipFile object needs the actual file rather than some kind of stream because it needs to be able to seek...)
The workaround I'm using at present is to turn the Uri into an InputStream, copy the contents to a temp file in private storage, and make a ZipFile instance from that:
private suspend fun <T> withZipFromUri(
context: Context,
uri: Uri, block: suspend (ZipFile) -> T
) : T {
val file = File(context.filesDir, "tempzip.zip")
try {
return withContext(Dispatchers.IO) {
kotlin.runCatching {
context.contentResolver.openInputStream(uri).use { input ->
if (input == null) throw FileNotFoundException("openInputStream failed")
file.outputStream().use { input.copyTo(it) }
}
ZipFile(file, ZipFile.OPEN_READ).use { block.invoke(it) }
}.getOrThrow()
}
} finally {
file.delete()
}
}
Then, I can use it like this:
suspend fun doStuffWithZip(context: Context, uri: Uri) {
withZipFromUri(context, uri) { // it: ZipFile
for (entry in it.entries()) {
dbg("entry: ${entry.name}") // or whatever
}
}
}
This works, and (in my particular case, where the .zip file in question is never more than a couple MB) is reasonably performant.
But, I tend to regard programming by temporary file as the last refuge of the terminally incompetent, thus I can't escape the feeling that I'm missing a trick here. (Admittedly, I am terminally incompetent in the context of Android + Kotlin, but I'd like to learn to not be...)
Any better ideas? Is there a cleaner way to implement this that doesn't involve making an extra copy of the file?
Copying from external source (and risking downvoting to oblivion) and this isn't quite an answer, but too long for a comment
public class ZipFileUnZipExample {
public static void main(String[] args) {
Path source = Paths.get("/home/mkyong/zip/test.zip");
Path target = Paths.get("/home/mkyong/zip/");
try {
unzipFolder(source, target);
System.out.println("Done");
} catch (IOException e) {
e.printStackTrace();
}
}
public static void unzipFolder(Path source, Path target) throws IOException {
// Put the InputStream obtained from Uri here instead of the FileInputStream perhaps?
try (ZipInputStream zis = new ZipInputStream(new FileInputStream(source.toFile()))) {
// list files in zip
ZipEntry zipEntry = zis.getNextEntry();
while (zipEntry != null) {
boolean isDirectory = false;
// example 1.1
// some zip stored files and folders separately
// e.g data/
// data/folder/
// data/folder/file.txt
if (zipEntry.getName().endsWith(File.separator)) {
isDirectory = true;
}
Path newPath = zipSlipProtect(zipEntry, target);
if (isDirectory) {
Files.createDirectories(newPath);
} else {
// example 1.2
// some zip stored file path only, need create parent directories
// e.g data/folder/file.txt
if (newPath.getParent() != null) {
if (Files.notExists(newPath.getParent())) {
Files.createDirectories(newPath.getParent());
}
}
// copy files, nio
Files.copy(zis, newPath, StandardCopyOption.REPLACE_EXISTING);
// copy files, classic
/*try (FileOutputStream fos = new FileOutputStream(newPath.toFile())) {
byte[] buffer = new byte[1024];
int len;
while ((len = zis.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
}*/
}
zipEntry = zis.getNextEntry();
}
zis.closeEntry();
}
}
// protect zip slip attack
public static Path zipSlipProtect(ZipEntry zipEntry, Path targetDir)
throws IOException {
// test zip slip vulnerability
// Path targetDirResolved = targetDir.resolve("../../" + zipEntry.getName());
Path targetDirResolved = targetDir.resolve(zipEntry.getName());
// make sure normalized file still has targetDir as its prefix
// else throws exception
Path normalizePath = targetDirResolved.normalize();
if (!normalizePath.startsWith(targetDir)) {
throw new IOException("Bad zip entry: " + zipEntry.getName());
}
return normalizePath;
}
}
This apparently works with pre-existing files; however since you already have an InputStream read from the Uri - you can adapt this and give it a shot.
EDIT:
It seems like it's extracting to Files as well - you could store the individual ByteArrays somewhere then decide what to do with them later on. But I hope you get the general idea - you can do all of this in-memory, without having to use the disk (temp files or files) in between.
Your requirement is a bit vague and unclear however, so I don't know what you're trying to do, merely suggesting a venue/approach to try out
What about a simple ZipInputStream ? –
Shark
Good idea #Shark.
InputSteam is = getContentResolver().openInputStream(uri);
ZipInputStream zis = new ZipInputStream(is);
#Shark has it with ZipInputStream. I'm not sure how I missed that to begin with, but I sure did.
My withZipFromUri() method is much simpler and nicer now:
suspend fun <T> withZipFromUri(
context: Context,
uri: Uri, block: suspend (ZipInputStream) -> T
) : T =
withContext(Dispatchers.IO) {
kotlin.runCatching {
context.contentResolver.openInputStream(uri).use { input ->
if (input == null) throw FileNotFoundException("openInputStream failed")
ZipInputStream(input).use {
block.invoke(it)
}
}
}.getOrThrow()
}
This isn't call-compatible with the old one (since the block function now takes a ZipInputStream as a parameter rather than a ZipFile). In my particular case -- and really, in any case where the consumer doesn't mind dealing with entries in the order they appear -- that's OK.
Okio (3-Alpha) has a ZipFileSystem https://github.com/square/okio/blob/master/okio/src/jvmMain/kotlin/okio/ZipFileSystem.kt
You could probably combine it with a custom FileSystem that reads the content of that file. It will require a fair bit of code but will be efficient.
This is an example of a custom filesystem https://github.com/square/okio/blob/88fa50645946bc42725d2f33e143628e7892be1b/okio/src/jvmMain/kotlin/okio/internal/ResourceFileSystem.kt
But I suspect it's simpler to convert the URI to a file and avoid any copying or additional code.
It's easy to check the .zip and .rar files in the Android-Kotlin FileAdapter(work with file manager), add the bellow function to your code:
private fun isZip(name: String): Boolean {
return name.contains(".zip") || name.contains(".rar")
}
I'm trying to crawl the entire file system of an android device, both directories and files, without the benefit of NIO, to build a tree of it. If I had NIO then I could use WalkTree or similar, but I don't.
The problem I am having (on the Nexus 5 API 23 x86 emulator) is in /sys/bus/pci/devices and possibly other directories (eg /proc/self) - it doesn't complete before the app times out/quits/crashes (unknown which), possibly getting into some kind of loop or something (the path may change in a repetitive fashion but the canonical path varies little or not at all) .
However if I rule out Symbolic links then that problem goes away but I get what is only some of the files on the device rather than all - for example lacking files under /data (or /data/media/0) and those files not showing up elsewhere - not to mention it looks completely different from the file system that most file managers show. The former is strange as I'd understood Symbolic Links pointed to files and folders that were still present in the file system, but just made them look as if they were elsewhere.
What's the solution? Do I have to code exceptions or special handling for /sys/bus/pci/devices, /proc/self and others? I'd prefer to keep Symbolic Links being followed if I can, and I'd prefer to crawl as many files and folders as I can (so starting in a sub-folder is not preferred).
And a few related questions that might affect the approach I eventually take - if I DO keep SymLinks then does that mean that some things will be crawled twice or more? Is there a way to avoid that? Is there a way to detect when something is the TARGET of a SymLink, other than following the SymLink and checking the CanonicalPath?
Here's my code:
I get the root (I understand that in Android, the first and likely only root is the valid one):
File[] roots = File.listRoots();
String rootPath = "";
try {
rootPath = roots[0].getCanonicalPath();
} catch (IOException e) {
// do something
}
Then I start the crawl (note the boolean to choose whether to ignore simlinks or not):
try {
// check if the rootPath is null or empty, and then...
File rootFile = new File(rootPath);
rootNode = new FileFolderNode(rootFile, null, true, false); // last param may be true to ignore sim links
//FileFolderNode(String filePath, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks)
} catch (Exception e) {
// do something
}
That uses the FileFolderNode, which has constructor:
public FileFolderNode(File file, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks) throws IOException {
if (file == null)
throw new IOException("File is null in new FileFolderNode");
if (!file.exists())
throw new IOException("File '" + file.getName() + "' does not exist in new FileFolderNode");
// for now this uses isSymLink() from https://svn.apache.org/repos/asf/commons/_moved_to_git/io/trunk/src/main/java/org/apache/commons/io/FileUtils.java adjusted a bit to remove Java 7 and Windows mentions
if (!ignoreSimLinks)
if (FileUtils.isSymlink(file))
return;
this.name = file.getName();
if (this.name.equals("") && ! file.getCanonicalPath().equals("/"))
throw new IOException("Name is empty in new FileFolderNode");
this.isDirectory = file.isDirectory();
if (this.isDirectory) {
this.children = new ArrayList<FileFolderNode>();
if (addChildren) {
File[] files = file.listFiles();
if (files == null) {
// do something
} else {
// add in children
for (File f : files) {
FileFolderNode child = null;
try {
child = new FileFolderNode(f, this, addChildren, ignoreSimLinks);
} catch (Exception e) {
child = null;
}
if (child != null)
children.add(child);
}
}
}
}
}
Given the lack of answers here, I've broken this question down into areas needing clarification, and am trying to get answers to those - please do see if you can help with those:
Get Android Filing System root
Android SymLinks to hidden or separate locations or partitions
Avoiding Android Symbolic Link loop
I am trying to test a sample project called Android.Routing.Offline from OsmSharp.Samples in Github.
After two taps on the screen (the first one gets just the GeoCoordinate) I get a ProtoBuf.ProtoException in the Router.cs
private static IBasicRouterDataSource<CHEdgeData> _graph;
public static void Initialize()
{
var routingSerializer = new CHEdgeDataDataSourceSerializer();
_graph = routingSerializer.Deserialize(
Assembly.GetExecutingAssembly().GetManifestResourceStream(#"Android.Routing.Offline.kempen-big.contracted.mobile.routing"));
}
public static Route Calculate(GeoCoordinate from, GeoCoordinate to)
{
try
{
lock(_graph)
{
var router = Router.CreateCHFrom(_graph, new CHRouter(), new OsmRoutingInterpreter());
// The exception happens here below
var fromResolved = router.Resolve(Vehicle.Car, from);
var toResolved = router.Resolve(Vehicle.Car, to);
if(fromResolved != null && toResolved !=null)
{
return router.Calculate(Vehicle.Car, fromResolved, toResolved);
}
}
}
catch(Exception ex)
{
OsmSharp.Logging.Log.TraceEvent("Router", OsmSharp.Logging.TraceEventType.Critical, "Unhandled exception occured: {0}", ex.ToString());
}
return null;
}
And the exception:
> {ProtoBuf.ProtoException: Invalid wire-type; this usually means you
> have over-written a file without truncating or setting the length; see
> http://stackoverflow.com/q/2152978/23354 at
> ProtoBuf.ProtoReader.ReadSingle () ...
I didnt overwrite the file (kempen-big.contracted.mobile.routing) just added it as a linked file in the project. Any ideas how I can solve this issue?
Well, the first thing to try is to check that the contents of the Stream you are reading (via GetManifestResourceStream) contains exactly the contents you are expecting, and not some wrapper or otherwise-corrupt mess. If you have some checksum algorithm you can run: great! Checking just the .Length would be a great start. Otherwise, you could cheat (just for the purposes of validating the contents) by getting the hex:
using (var ms = new MemoryStream())
{
stream.CopyTo(ms);
string hex = BitConverter.ToString(
ms.GetBuffer(), 0, (int)ms.Length);
// dump this string, and compare it to the same output run on the
// oringal file; they should be identical
}
Note that this duplicates the contents in-memory, purely so we can get a byte[] (oversized) to get the hex from - it isn't intended for "real" code, but until you are sure that the contents are correct, all other bets are off. I strongly suspect that you'll find that the contents are not identical to the contents in the original file. Note that I'm also implicitly assuming that the original file works fine in terms of deserialization. If the original file doesn't work: again, all bets are off.
This problem is directed to the Google Drive Android API support team. It may be considered a bug or just heads-up note. When testing GDAA I've run into this problem:
I created a file asynchronyously
Before the file was ready, I performed a search and found the file by name
Managed to retrieve it's metadata with no sign of problems
Attempt to use the metadata blew up (obviously)
The point is: Shouldn't the file search or metadata retrieval indicate that file is not ready / does not exist yet? Or is there a method to check that?
Here are code snippets to demonstrate the problem (simplified - not a production level code)
primitive # 1 create a file asynch
void createFileAsync(final DriveFolder fldr, final String name,
final String mime, final byte[] buff) {
final DriveId dId = fldr.getDriveId();
Drive.DriveApi.newContents(_gac).setResultCallback(new ResultCallback<ContentsResult>() {
#Override public void onResult(ContentsResult rslt) {
if (!rslt.getStatus().isSuccess()) return;
DriveFolder folder = Drive.DriveApi.getFolder(_gac, dId);
MetadataChangeSet meta = new MetadataChangeSet.Builder()
.setTitle(name).setMimeType(mime).build();
folder.createFile(_gac, meta, rslt.getContents()).setResultCallback(
new ResultCallback<DriveFileResult>() {
#Override public void onResult(DriveFileResult rslt) {
_dFile = rslt.getStatus().isSuccess() ? rslt.getDriveFile() : null;
}
});
}
});
}
primitive # 2 find the file by name - with wait
public DriveFile findFirst(String name) {
Filter filtr = Filters.and(
Filters.eq(SearchableField.TRASHED, false),
Filters.eq(SearchableField.TITLE, name)
);
Query qry = new Query.Builder().addFilter(filtr).build();
MetadataBufferResult rslt = (Drive.DriveApi.query(_gac, qry).await()
if (rslt.getStatus().isSuccess()) {
MetadataBuffer mdb = null;
try {
mdb = rslt.getMetadataBuffer();
return Drive.DriveApi.getFile(_gac, mdb.get(0).getDriveId());
} finally { if (mdb != null) mdb.close(); }
}
return null;
}
test / problem scenario :
GoogleApiClient _gac;
DriveFolder fldr = Drive.DriveApi.getRootFolder(_gac);
byte[] buffer = ("FooBar ").getBytes();
// create a file async
DriveFile _dFile = null;
createFileAsync(fldr, "foo", "text/plain", buffer);
// file is not ready yet, but FOUND and it's metadata VALID (non-null)
Metadata md = findFirst("foo").getMetadata(_gac).await().getMetadata();
// any attempt to use Metadata methods
// md.isTrashed(), md.getTitle(), ...
// blows up until the createFileAsync() is finished
I am happy to report that the problem disappeared (by intervention ?). The query by name on a file that is being created (asynchronously) does not return valid metadata until the creation is complete. Simplified code chunk from primitive #2 above:
for (Metadata md : rslt.getMetadataBuffer())
; // NO md AVAILABLE until the file creation is completed!!!
works correctly now, not enumerating any metadata until 'the file is ready'.
I'm writing a word game in Android. It's my first app so my knowledge is almost non-existent.
What I would like to do is use JWI to access the WordNet dictionary. This requires specifying the WordNet dictionary's file path.
From what I've read, Android "assets" are not available via a simple file path, but what JWI requires to initialize the WordNet dictionary API is a URL to the disk location of the dictionary files.
So, what is the best course of action? Should I copy the assets at startup-time into a known folder on the android device? I can't think of a better way but that seems entirely stupid to me.
Any help gratefully received.
I have the same problem (for a jetty webapp however and not android) and tried those two approaches, however unsuccessfully:
JWNL.initialize(this.getClass().getClassLoader().getResourceAsStream("wordnet_properties.xml");
dict = Dictionary.getInstance();
Here it successfully loads wordnet_properties.xml but it cannot access the dictionary which is pointed to by the properties file.
Using the dictionary folder directly:
String dictPath = "models/en/wordnet/dict/";
URL url = this.getClass().getClassLoader().getResource(dictPath);
System.out.println("loading wordnet from "+url);
dict = new RAMDictionary(url, ILoadPolicy.NO_LOAD);
Here I get the dictionary URL to be jar:file:/home/myusername/.m2/repository/package/1.0-SNAPSHOT/commons-1.0-SNAPSHOT.jar!/models/en/wordnet/dict/. WordNet however doesn't accept the jar protocol and gives me the error:
java.lang.IllegalArgumentException: URL source must use 'file' protocol
at edu.mit.jwi.data.FileProvider.toFile(FileProvider.java:693)
at edu.mit.jwi.data.FileProvider.open(FileProvider.java:304)
at edu.mit.jwi.DataSourceDictionary.open(DataSourceDictionary.java:92)
at edu.mit.jwi.RAMDictionary.open(RAMDictionary.java:216)
My next investigation will be to create a subclass to RAMDictionary or something similar, please tell me if you have found a solution in the meantime.
P.S.: I just wrote the developer a mail asking for help after I tried to rewrite the FileProvider to use resources instead but after one or two hours I gave up because the code calls so much other code that also only works with files. I will keep you up to date!
P.P.S.: I received an answer from the developer saying that it is principially not possible with streams because they don't offer random access which is necessary. However, he offered to implement a solution to load it all in RAM, if really necessary, but that would use up about 500 MB and I guess that is too much for android apps so I guess it is still best to unpack it somewhere.
P.S.: Here is my unpacking solution (you can replace the System.out.println statements with logger statements if you use logging or remove them if you don't like them):
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URISyntaxException;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
/** Allows WordNet to be run from within a jar file by unpacking it to a temporary directory.**/
public class WordNetUnpacker
{
static final String ID = "178558556719"; // minimize the chance of interfering with an existing directory
static final String jarDir = "models/en/wordnet/dict";
/**If running from within a jar, unpack wordnet from the jar to a temp directory (if not already done) and return that.
* If not running from a jar, just return the existing wordnet directory.
* #see getUnpackedWordNetDir(Class)*/
static File getUnpackedWordNetDir() throws IOException
{return getUnpackedWordNetDir(WordNetUnpacker.class);}
/**If running from within a jar, unpack wordnet from the jar to a temp directory (if not already done) and return that.
* If not running from a jar, just return the existing wordnet directory.
* #param clazz the class in whose classloader the wordnet resources are found.
* #see getUnpackedWordNetDir()**/
static File getUnpackedWordNetDir(Class clazz) throws IOException
{
String codeSource = clazz.getProtectionDomain().getCodeSource().getLocation().getPath();
System.out.println("getUnpackedWordNetDir: using code source "+codeSource);
if(!codeSource.endsWith(".jar"))
{
System.out.println("not running from jar, no unpacking necessary");
try{return new File(WordNetUnpacker.class.getClassLoader().getResource(jarDir).toURI());}
catch (URISyntaxException e) {throw new IOException(e);}
}
try(JarFile jarFile = new JarFile(codeSource))
{
String tempDirString = System.getProperty("java.io.tmpdir");
if(tempDirString==null) {throw new IOException("java.io.tmpdir not set");}
File tempDir = new File(tempDirString);
if(!tempDir.exists()) {throw new IOException("temporary directory does not exist");}
if(!tempDir.isDirectory()) {throw new IOException("temporary directory is a file, not a directory ");}
File wordNetDir = new File(tempDirString+'/'+"wordnet"+ID);
wordNetDir.mkdir();
System.out.println("unpacking jarfile "+jarFile.getName());
copyResourcesToDirectory(jarFile, jarDir, wordNetDir.getAbsolutePath());
return wordNetDir;
}
}
/** Copies a directory from a jar file to an external directory. Copied from Stack Overflow. */
public static void copyResourcesToDirectory(JarFile fromJar, String jarDir, String destDir) throws IOException
{
int copyCount = 0;
for (Enumeration<JarEntry> entries = fromJar.entries(); entries.hasMoreElements();)
{
JarEntry entry = entries.nextElement();
if(!entry.getName().contains("models")) continue;
if (entry.getName().startsWith(jarDir) && !entry.isDirectory()) {
copyCount++;
File dest = new File(destDir + "/" + entry.getName().substring(jarDir.length() + 1));
File parent = dest.getParentFile();
if (parent != null) {
parent.mkdirs();
}
FileOutputStream out = new FileOutputStream(dest);
InputStream in = fromJar.getInputStream(entry);
try {
byte[] buffer = new byte[8 * 1024];
int s = 0;
while ((s = in.read(buffer)) > 0) {
out.write(buffer, 0, s);
}
} catch (IOException e) {
throw new IOException("Could not copy asset from jar file", e);
} finally {
try {
in.close();
} catch (IOException ignored) {}
try {
out.close();
} catch (IOException ignored) {}
}
}
}
if(copyCount==0) System.out.println("Warning: No files copied!");
}
}
You can just copy all dict files from "assets" to the internal directory of your app. Just do it once, on the first app launch.
Since then you can use JWI in a causual way like this:
String path = getFilesDir() + "/dict";
URL url = new URL("file", null, path);
IDictionary dict = new Dictionary(url);