Develop Reference

Audio not working in Android WebView using shouldInterceptRequest()

My Android app is showing an html5 e-book in a WebView.
I have a zipped file containing an e-book with all its resources: text, images and audio (mp3 files).
In order to unzip the book I use shouldInterceptRequest(), which intercepts the file:///... requests, and returns the data via a WebResourceResponse object. The code works fine for text and images.
When I get to audio resources, I get runtime errors, and the audio file is not played.
Note: I do see the unzipped file is returned with the correct size (about 10MB).
Error messages I get:
cr_MediaResourceGetter File does not exist
cr_MediaResourceGetter Unable to configure metadata extractor
My HTML code for the audio :
<div xmlns="http://www.w3.org/1999/xhtml">
<p style="text-align:center;margin:0px;">
<audio controls="controls" src="../Audio/01-AudioTrack-01.mp3">Your browser does not support the audio tag.</audio>
<br />
</p>
</div>
My Android Code:
setWebViewClient(new WebViewClient()
{
#Override
public WebResourceResponse shouldInterceptRequest(WebView view, final String url)
{
String urlWithoutAnchor = URLUtil.stripAnchor(url);
String fileName = urlWithoutAnchor;
try {
byte [] resource = tbxPool.getResource(fileName);
/* SIMPLE VERSION without calling setResponseHeaders():
return new WebResourceResponse(mimeType, "UTF-8", new ByteArrayInputStream(resource));
*/
WebResourceResponse returnedMediaResource = new WebResourceResponse(mimeType, "UTF-8", new ByteArrayInputStream(resource));
if (mimeType.toLowerCase().startsWith("audio")) {
Map<String, String> responseHeaders = new HashMap<String, String>();
responseHeaders.put("Content-Type", mimeType);
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {//2CLEAN
returnedMediaResource.setResponseHeaders(responseHeaders);
Logger.v(TAG, "Response Headers added to audio resource");
}
else {
//TODO: Handle else for API<21. Toast?
}
}
return returnedMediaResource;
} catch (IOException e) {
Logger.e(TAG, "failed to load resource "+fileName,e);
return null;
}
}
}
Environment
Android 6.0.1 (Nexus 5)
Android System WebView version 47
Requirement Clarification
The audio is to play in browser like an html5 document should, without laucnhing external player.
Question:
What am I doing wrong?! Many Thanks in advance!

The workaround I found to this problem is not elegant, but it's the only one that worked for me: Write the audio file to sd card :(.
Stage 1): When shouldInterceptRequest() is called with a chapter url.
The chapter is intercepted first (before the other chapter resources (images, audio, fonts, ..) are intercepted.
When the chapter is intercepted we search the html for the <audio> tag. If found, we replace the relative path (e.g. SRC="../Audio/abc.mp3")
with an absolute path (e.g. SRC="/storage/tmp/abc.mp3")
Stage 2): When shouldInterceptRequest() is called with an audio url.
Your attention. Like all workarounds this is a bit tricky (but works!):
After Stage 1) the audio url will be an absolute url (the absolute url is what is now written in the modified html).
We now have to do 2 things:
a) read the audio file from the zipped epub.
To do this we need to "fool" the code, and read the audio file from its original zipped relative url, e.g. "../Audio/abc.mp3" in our example
(although shouldInterceptRequest has been called with "/storage/tmp/abc.mp3").
b) After reading the zipped audio file, write it to the storage (sdcard)
Stage 3) When shouldInterceptRequest() is called with a chapter url,
We delete the temp audio files
Note: If you follow the code, you will see this is Step 0) in shouldInterceptRequest(), executed before stage 1), but I found it clearer explained as above.
if (isChapterFile(mimeType)) {
deleteTempFiles(); // this line is stage 3)
changeAudioPathsInHtml(tzis); // this line is stage 1)
This is the code:
setWebViewClient(new WebViewClient()
{
private String tmpPath = TbxApplication.getAppPath(null) + "/tmp/";
#Override
public WebResourceResponse shouldInterceptRequest(WebView view, final String url)
{
Logger.d(TAG, "in shouldInterceptRequest for " + url);
String urlWithoutAnchor = URLUtil.stripAnchor(url);
String mimeType = StringUtils.getFileMimeType(urlWithoutAnchor);
String urlWithoutBase; //the url stripped from leading 'epubBaseUrl' (base url for example:"file:///storage/.../123456.tbx")
if (isAudioFile(mimeType)) { //write AUDIO file to phone storage. See AUDIO WORKAROUND DOCUMENTATION
String storagePath = StringUtils.truncateFileScheme(url); //WebView calls shoudlInterceptRequest() with "file://..."
try {
String oEBPSAudioPath = storagePathToOEBPSAudioPath(storagePath); //e.g. change"/storage/tmp" to "OEBPS/Audio/abc.mp3"
byte[] audioBytes = tbxPool.getMedia(oEBPSAudioPath);
FileUtils.writeByteArrayToFile(audioBytes, storagePath); //TODO: To be strict, write in separate thread
Logger.d(TAG, String.format("%s written to %s", oEBPSAudioPath, storagePath));
return null;//webView will read resource from file
//Note: return new WebResourceResponse("audio/mpeg", "UTF-8", new ByteArrayInputStream(audioBytes));
//did NOT work,so we had to change html for audio to point to local storage & write to disk
//see AUDIO WORKAROUND DOCUMENTATION in this file
} catch (Exception e) {
Logger.e(TAG,e.getMessage());
return null;
}
}
.....
else {
if (isChapterFile(mimeType)) { //This is a CHAPTER
deleteTempFiles(); //Loading a new chapter. Delete previous chapter audio files. See AUDIO WORKAROUND DOCUMENTATION in this file
InputStream htmlWithChangedAudioPaths = changeAudioPathsInHtml(tzis); //see AUDIO WORKAROUND DOCUMENTATION in this file
WebResourceResponse webResourceResponse = new WebResourceResponse(mimeType, "UTF-8", htmlWithChangedAudioPaths);
return webResourceResponse;
}
//Changes relative paths of audio files, to absolute paths on storage
//see AUDIO WORKAROUND DOCUMENTATION in this file
private InputStream changeAudioPathsInHtml(InputStream inputStream) {
String inputString = StringUtils.inputStreamToString(inputStream);
String outputString = inputString.replaceAll("\"../Audio/", "\"" + tmpPath);// e.g. SRC="../Audio/abc.mp3" ==>SRC="/sdcard/tmp/abc.mp3" //where '*' stands for multiple whitespaces would be more elegant
return StringUtils.stringToInputStream(outputString);
}
/** Example:
* storagePath="/storage/tmp/abc.mp3
* Returns: "OEBPS/Audio/abc.mp3"*/
private String storagePathToOEBPSAudioPath(String storagePath){
String fileName = StringUtils.getFileName(storagePath);
String tbxOEBPSAudioPath = "OEBPS/Audio/" + fileName;
return tbxOEBPSAudioPath;
}
public static void writeByteArrayToFile(byte[] byteArray, String outPath) {
try {
File file = new File(outPath);
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArray);
fos.close();
} catch (IOException e) {
Logger.e(TAG, String.format("Could not write %s", outPath));
}
}

Application Error on Android Cordova/Phonegap Application

I have a cordova (2.7.0) android app that is crashing with an Application Error when it tries to load an iframe where the source has a protocol relative (network-path reference) src.
For instance, if the iframe is:
<iframe src="//instagram.com/p/beGdCuhQYl/embed/?wmode=opaque&wmode=opaque" width="800" height="928" style="border:0;" frameborder="0"></iframe>
Then the app tries to load the source from
file://instagram.com/p/beGdCuhQYl/embed/?wmode=opaque&wmode=opaque
Since the html page that loads this iframe is loaded from the file system, it makes sense that it is doing this. However, is there a way to stop the app from crashing? The same cordova app on iOS just doesn't load anything, and has a blank iframe. I would be nice if the android app behaved the same way.
It would be even nicer if there was a way to tell the cordova app to load these types of urls from http:// and not file:// but I think that is asking too much.

Ok, so I ended up doing this in two parts. First part, try to fix as many protocol relative urls as possible in javascript, and the second part was to provide some java code to ignore any that I missed.
First part (uses jQuery)
/**
* Takes text, looks for elements with src attributes that are
* protocol relative (//) and converts them to http (http://)
* #param {String} text the text that you want to fix urls in
* #returns {String} the updated text with corrected urls
*/
fixProtocolRelativeUrlsInText: function(text) {
var $html, $elements;
try {
$html = $('<div>' + text + '</div>');
$elements = $html.find('[src^="//"]');
if ($elements.length) {
$elements.each(function() {
var $this = $(this);
$this.attr('src', 'http:' + $this.attr('src'));
});
return $html.html();
} else {
return text;
}
} catch(ex) {
return text;
}
},
Second part:
/**
* Override the default makeWebViewClient and provide a custom handler for protocol
* relative urls.
*/
#Override
public CordovaWebViewClient makeWebViewClient(CordovaWebView webView) {
//
// We already try to fix protocol relative urls in the javascript. But this is a safety net in case anything
// gets through. So, in order to not crash the app, lets handle these types ourself and just swallow them up
// for now. The url won't load but at least it won't crash the app either. By the time the protocol relative
// url gets in here, it has the file: appended to it already. If it was a true file:// path to something on the
// device, then it will have file:///some/path, and if it was a protocol relative url that was converted to a
// file:// then it will have file://some.domain, so we look for urls that don't have the three /'s
//
final Pattern pattern = Pattern.compile("^file://[^/].*$");
CordovaWebViewClient webViewClient;
if(android.os.Build.VERSION.SDK_INT < android.os.Build.VERSION_CODES.HONEYCOMB) {
webViewClient = new CordovaWebViewClient(this, webView) {
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
Matcher matcher = pattern.matcher(url);
if (matcher.matches()) {
Log.i(LOG_TAG, "swallowing url '" + url + "'");
return true;
} else {
return super.shouldOverrideUrlLoading(view, url);
}
}
};
} else {
webViewClient = new IceCreamCordovaWebViewClient(this, webView) {
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
Matcher matcher = pattern.matcher(url);
if (matcher.matches()) {
Log.i(LOG_TAG, "swallowing url '" + url + "'");
return true;
} else {
return super.shouldOverrideUrlLoading(view, url);
}
}
};
}
return webViewClient;
}

Cordova doesn't support protocol relative src, it expects you to specify either file, or http.

WebView and Assets problems

My current monodroid project is having two problems - one in loading from Assets and the other is a generated webpage being passed to be displayed and their inter-related which makes things more annoying!
The assets problem is simple enough, the file can't be found. My current code looks like this
Android.App.Application.SynchronizationContext.Post(delegate
{
string uri = "file:///android_asset/StyleSheet.css";
string settings = string.Empty;
using (var input = c.Assets.Open(uri))
using (StreamReader sr = new System.IO.StreamReader(input))
{
settings = sr.ReadToEnd();
}
string CSS = "<html><head><style>" + settings + "</style></head><body style='height:600px;background: url(Back-AQHA.jpg)' >";
retStr = CSS + tableData.Content + "</body></html>";
callback(retStr);
}, null);
This dies at the sr.ReadToEnd() line as the file can't be found. All the permissions are correctly set for me to be able to access the assets directory and read in.
c is just a Context passed to the method
Now, assuming that the file had be read in and passed back to the caller, it now fails to be displayed (tested this by generating a page within the app and passing it back)... The webviewer code is bog standard
public class webservice_webview : Activity
{
WebView web_view;
protected override void OnCreate(Bundle bundle)
{
base.OnCreate(bundle);
SetContentView(Resource.Layout.webview);
string res = base.Intent.GetStringExtra("content");
if (res.Length > 0)
{
web_view = FindViewById<WebView>(Resource.Id.webviewer);
web_view.Settings.JavaScriptEnabled = true;
web_view.LoadData(res, "text/html", null);
web_view.SetWebViewClient(new websiteviewClient());
}
else
return;
}
private class websiteviewClient : WebViewClient
{
public override bool ShouldOverrideUrlLoading(WebView view, string url)
{
view.LoadData(url, "text/html", null);
return true;
}
}
public override bool OnKeyDown(Android.Views.Keycode keyCode, Android.Views.KeyEvent e)
{
if (keyCode == Keycode.Back && web_view.CanGoBack())
{
web_view.GoBack();
return true;
}
return base.OnKeyDown(keyCode, e);
}
}
}
When the content is passed over and rendered, all that results is a dead webpage saying that the url data;text/html;null,[generated html] may have moved or has gone. How do I get the generated page to display?
Thanks
Paul

It might be worth check your assets are in the assets folder and are marked in the properties pane as being "AndroidAsset" - also please make sure capitalisation is correct.
Assuming they are then you can just pass the assets direct to the webview using code like:
var web = FindViewById<WebView>(Resource.Id.AboutWebView);
web.LoadUrl("file:///android_asset/About/index.html");
from https://github.com/slodge/mobile-samples/blob/master/MWC/MWC.Droid/Views/About/AboutXamarinView.cs
If you do want to read in and manipulate the html text first, then try code like: Assets.Open("About/index.html") - i.e. without the file: scheme attached.
In your second sample, the code for web_view.LoadData(res, "text/html", null); looks correct - but then I'm not sure what your logic inside ShouldOverrideUrlLoading does - that looks like it might just prevent things from loading?

android webview encrypted content

I have an application that uses a webview in order to display content and the Javascript calls are the controller of my application.
In order to provide a level of security I obfuscated the code. This is not enough as I would like to encrypt the html and js files and then decrypt them at runtime. I packed the apk file with these resources encrypted with RC4 algorithm. When loading the files, I am decrypting the javascript files, load them and then decrypt the html file and load it. However this doesn't work as the webcontent displays a message in the form of: the web page at data:text/html might be temporarily down or it may have moved permanently, etc, etc.
I overloaded onLoadResource in order to see what content is loaded and I can see it loads the Javascript content, but the content loaded is html escaped also.
My questions are:
1. How to secure the html and javascript files (located in assets folder) in order to not be accessible?
2. In case my approach is correct, has anyone any idea on what I am doing wrong?
Thanks!
Below is the code that decrypts and loads the resources:
protected void loadWebContent() {
checkEncryptionEnabled();
loadJSFiles();
logger.info("Loaded js ... going for html");
loadAssetFile("www/index.html", "text/html");
}
private void loadJSFiles() {
String[] jsFilesArray = { "app.js", "iscroll.js", "iui.js", "json.js" };
for (String js : jsFilesArray) {
loadAssetFile("www/js/" + js, "application/javascript");
}
}
private void loadAssetFile(String filePath, String mimeType) {
AssetManager assetMgr = getAssets();
InputStream is = null;
try {
is = assetMgr.open(filePath);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] temp = new byte[512];
int bytesRead = -1;
while ((bytesRead = is.read(temp)) > 0) {
baos.write(temp, 0, bytesRead);
}
byte[] encrypted = baos.toByteArray();
String content = null;
/**
* true
* */
if (Config.ENCRYPTION_ENABLED) {
byte[] decrypted = new RC4Encrypter("rc4_key").rc4(encrypted);
content = new String(decrypted, "utf-8");
} else {
content = new String(encrypted, "utf-8");
}
/**
* The webview to use
* */
if("application/javascript".equals(mimeType)) {
webContent.loadUrl("javascript:" + content);
} else {
webContent.loadData(content, mimeType, "utf-8");
}
} catch (IOException ex) {
logger.error(null, ex);
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
}
}
}
}

found the answer for the second question question instead of: webContent.loadData(content, mimeType, "utf-8"); I used: webContent.loadDataWithBaseURL("file:///android_asset/www/", content, mimeType, "utf-8", null); Content is shown with no problems ... However, the first question kind of stands and not; but considering there was no answer for more than a year, I'll consider encrypting data is OK.

Data encryption is OK as long as you can also keep the decryption key confidential, which is not the case in the above code. The hardcoded decryption key can be easily spotted after decompiling the DEX files embedded inside the APK.
If you want to hide the application logic inside the HTML and JavaScript files and if that application logic doesn't require offline capabilities then you could outsource the code of that application logic on a server.
From here you have two choices:
Load the application code dynamically from the server whenever
you need it (and run the application code on the client).
Implement the application logic on the server side, e.g., as a
web service (and run the application code on the server, the client
knows only how to call the web service)
The short answer to your first question is that there is no methodology or technology to perfectly protect your application. I recommend to you to take a look at How to avoid reverse engineering of an APK file? for an overview of possible protection methods.

html5 apps for tablets: possible to load images from device's filesystem?

I created a simple html file that loads some images from my local hard-drive (ubuntu).
It is enough to put
<img src=/home/user/directory/image.jpg></img>
Now I need to know if it is the same when Html5 is displayed on a tablet like Android or iOS, or Html5 is used in offline app.
I mean, if html5 can load an image from the device's filesystem just like on my computer, without localStorage or sessionStorage.

If you deploy the application as native application it is possible (wrap it with Phonegap).
For saved HTML files it is not possible.

On Android, it can be done even though it looks a bit tricky at first. Say you have defined a WebView in your layout.xml, which you want to fill with an html file shipped with your application, which in turn should import a png also shipped with your application.
The trick is to put the html file into res/raw and the png into assets.
Example.
Say you have hello.html which should include buongiorno.png.
Within your project, say MyProject, place buongiorno.png into MyProject/assets.
The hello.html goes into MyProject/res/raw (because we want to avoid having it 'optimised' by the android resource compiler), and could look like this:
<html>
<head></head>
<body>
<img src="file:///android_asset/buongiorno.png"/>
<p>Hello world.</p>
</body>
</html>
In your java code, you would put this code:
WebView w = (WebView) findViewById(R.id.myWebview);
String html = getResourceAsString(context, R.raw.hello);
if (html != null) {
w.loadDataWithBaseURL(null, html, "text/html", "UTF-8", null);
}
where getResourceAsString() is defined as follows:
public static String getResourceAsString(Context context, int resid) throws NotFoundException {
Resources resources = context.getResources();
InputStream is = resources.openRawResource(resid);
try {
if (is != null && is.available() > 0) {
final byte[] data = new byte[is.available()];
is.read(data);
return new String(data);
}
} catch (IOException ioe) {
throw new RuntimeException(ioe);
} finally {
try {
is.close();
} catch (IOException ioe) {
// ignore
}
}
return null;
}