Develop Reference

Uri and WebView classes parsing URLs containing backslashes in authority (host or user information) differently

When using the URIs
String myUri = "https://evil.example.com\\.good.example.org/";
// or
String myUri = "https://evil.example.com\\#good.example.org/";
in Java on Android, the backslash in the host or user information of the authority part of the URI causes a mismatch between how Android’s android.net.Uri and android.webkit.WebView parse the URI with regard to its host.
The Uri class (and cURL) treat evil.example.com\.good.example.org (first example) or even good.example.org (second example) as the URI’s host.
The WebView class (and Firefox and Chrome) treat evil.example.com (both examples) as the URI’s host.
Is this known, expected or correct behavior? Do the two classes simply follow different standards?
Looking at the specification, it seems neither RFC 2396 nor RFC 3986 allows for a backslash in the user information or authority.
Is there any workaround to ensure a consistent behavior here, especially for validation purposes? Does the following patch look reasonable (to be used with WebView and for general correctness)?
Uri myParsedUri = Uri.parse(myUri);
if ((myParsedUri.getHost() == null || !myParsedUri.getHost().contains("\\")) && (myParsedUri.getUserInfo() == null || !myParsedUri.getUserInfo().contains("\\"))) {
// valid URI
}
else {
// invalid URI
}
One possible flaw is that this workaround may not catch all the cases that cause inconsistent hosts to be parsed. Do you know of anything else (apart from a backslash) that causes a mismatch between the two classes?

It's known that Android WebView 4.4 converts some URLs, in the linked issue are some steps described how to prevent that. From your question is not completely clear if your need is based in that issue or something else.
You can mask the backslashes and other signs with there according number in the character-table. In URLs the the number is written in hexademcimal.
Hexadecimal: 5C
Dezimal: 92
Sign: \
The code is the prepended with a % for each sign in the URL, your code looks like this after replacement:
String myUri = "https://evil.example.com%5C%5C.good.example.org/";
// or
String myUri = "https://evil.example.com%5C%5C#good.example.org/";
it might be required still to add a slash to separate domain and path:
String myUri = "https://evil.example.com/%5C%5C.good.example.org/";
// or
String myUri = "https://evil.example.com/%5C%5C#good.example.org/";
Is it possible that the backslashes never shall be used for network-communication at all but serve as escaping for some procedures like regular expressions or for output in JavaScript (Json) or some other steps?
Bonus ;-)
Below is a php-script that prints a table for most UTF-8-signs with the corresponding Numbers in hex and dec. (it still should be wrapped in an html-template including css perhaps):
<?php
$chs = array('0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F');
$chs2 = $chs;
$chs3 = $chs;
$chs4 = $chs;
foreach ($chs as $ch){
foreach ($chs2 as $ch2){
foreach ($chs3 as $ch3){
foreach ($chs4 as $ch4){
echo '<tr>';
echo '<td>';
echo $ch.$ch2.$ch3.$ch4;
echo '</td>';
echo '<td>';
echo hexdec($ch.$ch2.$ch3.$ch4);
echo '</td>';
echo '<td>';
echo '&#x'.$ch.$ch2.$ch3.$ch4.';';
echo '</td>';
echo '</tr>';
}
}
}
}
?>

Is this known, expected or correct behavior?
IMO, it is not. For both URI and WebView. Because RFC won't allow a backslash, they could have warn it. However it is less important because it does not affect the working at all if the input is as expected.
Do the two classes simply follow different standards?
The URI class and WebView strictly follows the same standards. But due to the fact that they are different implementations, they may behave differently to an unexpected input.
For example, "^(([^:/?#]+):)?((//([^/?#]*))?([^?#]*)(\\?([^#]*))?)?(#(.*))?" this is the regular expression in URI which is used to parse URIs. The URI parsing of WebView is done by native CPP methods. Even though they follow same standards, chances are there for them to give different outcome (At least for unexpected inputs).
Does the following patch look reasonable?
Not really (See the answer of next question).
Do you know of anything else (apart from a backslash) that causes a
mismatch between the two classes?
Because you are so concerned about the consistent behavior, I won't suggest a manual validation. Even the programmers who wrote these classes can't list all of such scenarios.
The solution
If I understand correctly, you need to load URLs which is supplied by untrustable external sources (which attackers can exploit if there is a loop hole), but you need to identify it's host correctly.
In that case, you can parse it using URI class itself and use URI#getHost() to identify the host. But for WebView, instead of passing the original URL string, pass URI#toString().

What is the purpose of using double underscore ( __ ) before the start and after the end of an file name in c?

I'm studying the android kernel as a beginner. I can read the messages thrown from the macro ERROR() inside the function main() at system/core/init/init.c using dmesg command through adb. I observed that after calling the function open_devnull_stdio() inside main(), dmesg no longer displays the messages thrown by ERROR().
To find the reason, I started digging into the declaration of open_devnull_stdio() inside system/core/init/util.c and I found this line I can't understand
static const char *name = "/dev/__null__";
Actually there was no file named __null__ inside /dev/ in the device, but there was a file named null and I was able to grab it using adb pull and it was a 0 byte (empty) file.
So why is a file name wrapped with double underscore (__) ?
Here is the link for the util.c

There is no special purpose of using double underscore before the start, after the end or both in C. From the point of view of C the file name is just a string, the operating system is free to interpret in whatever way it chooses. From the point of view of Linux, the same applies. Underscores in file names are just characters. They are not treated differently from the letters b and t.
If I guessed right and I'm reading the same file as you (it might be a good idea to link to the source code you're reading) then it should be pretty obvious what the code does on the lines after the one you mentioned. The next lines are:
if (mknod(name, S_IFCHR | 0600, (1 << 8) | 3) == 0) {
fd = open(name, O_RDWR);
unlink(name);
Which creates the null device which is then opened and immediately deleted again.
I suspect this is done so that programs can run without access to the root filesystem and still be able to open the equivalent of /dev/null.

I don't know the answer but I have an idea:
The following page shows an "strace" output where /dev/__null__ is used:
https://gist.github.com/tetsu-koba/1522515
Under Linux device files have a 33-bit (?) number which identifies the device. (At least under old Linux versions) you could delete some file in /dev and you could restore it or even create it in another directory (!) when you know the 33-bit number! (So you can delete the device /dev/sda2 and create the device (not file!) /home/myuser/sda2 instead.)
The trace in the link above shows the following three lines:
mknod("/dev/__null__", S_IFCHR|0600, makedev(1, 3)) = 0
open("/dev/__null__", O_RDWR|O_LARGEFILE) = 3
unlink("/dev/__null__") = 0
These lines will create the device file /dev/__null__ (with the 33-bit number identifying /dev/null). Then it opens that file and then it removes the file again.
Maybe this is done because the tool shall be able to run both on Linux installations where the device file "/dev/null" is present (in this case the file should not be overwritten) and on installations where that file is missing (in this case a replacement file must be created using the known 33-bit number).

As other people have pointed out this just tells it's the "null device", not a regular file called "null". null is supposed to act like an information sink, not like a normal file where you dump your data to. Hope this helps.

error with pymtp to work on python 3

I want to access a android device from python to download some photos.
libmtp works from the CLI.
Than pymtp. It's been around for a while but it's designed for python 2 and i'm using python 3. Meanwhile fixed several minor issues but i'm stuck at an error from function get_filelisting
specially this section:
ret = []
next = files
while next:
ret.append(next.contents)
if (next(next.contents) is None):
break
next = next(next.contents)
The error is related to the "next".
That section looks strange to me, i've been coding in python for a while but i'm new to ctypes. Tried a lot of variants, they all failed. The "next" could be confusing with python buildin function so i renamed it to nextpointer and came to this code:
ret = []
nextpointer = files
while nextpointer:
ret.append(nextpointer.contents)
nextpointer = nextpointer.contents.next
It seems to work but did it work by accident ? does it have any design flaws ? Could anyone with experience on python ctypes confirm this a solution ? Any suggestion welcome.

From python2.7 documentation
next(iterator[, default])
Retrieve the next item from the iterator by calling its next() method. If default is given, it is returned if the iterator is
exhausted, otherwise StopIteration is raised.
from python3 documentation
next(iterator[, default])
Retrieve the next item from the iterator by calling its __next__() method. If default is given, it is returned if the iterator is
exhausted, otherwise StopIteration is raised.
Notice that next() method was removed from python3 but the function still exists.
This is all I can say about the next function and .next()/__next__() methods.
I downloaded the pymtp module and get_filelisting() is slightly different from what you posted in your ported code, here it is:
ret = []
next = files
while next:
ret.append(next.contents)
if (next.contents.next == None):
break
next = next.contents.next
If none of this helped you (which probably didn't :D), the version of pymtp library that I am using is 0.0.6 download using pip.

why just after initializing the zram read is issued before write?

I am newbie to Linux kernel and just started to know how zram works. Initial testing, I am seeing that READ is issued before WRITE just after the zram is being initialized. But I am just eager to know, why this is so ?
As an activity I took the dump_stack() and followed the path form where to how this zram read is being performed.
zram get to know this info whether it has to do READ or WRITE operation on issued bio->bi_rw. Code flow is like that zram_make_request API is being called from create_device in zram driver. And zram_make_request internally called __zram_make_request which called the zram_bvec_rw API.
In zram_bvec_rw API check the available info of bio->bi_rw and correspondingly issued the READ and WRITE call.
Now, in this case what is happening: READ is being encapsulated inside bio struct itself. As triage I found that submit_bh fills all the entry of bio and issued the submit_bio.
I was wondering who is actually sets the bio->bi_rw as READ. By enabling the few prints I found that ll_rw_block API is being called by __block_write_begin with READ, later ll_rw_block calls the submit_bh API where rest of bio struct entries are filled.
But I am still not getting the answer why READ is issued for ll_rw_block from __block_write_begin ?
zram driver:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/block/zram/zram_drv.c?id=refs/tags/v3.18.14
in file: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/buffer.c?id=refs/tags/v3.18.14
if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
!buffer_unwritten(bh) &&
(block_start < from || block_end > to)) {
ll_rw_block(READ, 1, &bh);
*wait_bh++=bh;
}
buffer_uptodate(bh), /* Contains valid data */
buffer_delay(bh), /* Buffer is not yet allocated on disk */
buffer_unwritten(bh), /* Buffer is allocated on disk but not written */
Please can someone give an explaination/answer to my question ?
How I am concluding that read is perfomed before write ??
I just check the num_reads and num_writes count. And num_reads count is set to 1 while num_writes is found 0 when we do mkswap /dev/block/zram0 and after calling the swapon /dev/block/zram0 the final counts are num_reads = 2 and num_writes=1.
NOTE: This is the case when we don't performing any additional zram activity. We got this behavior in case as explained above.

Because of this as far as i can see: block_start < from || block_end > to (respecting other conditions of course, buffer_uptodate() etc.) .
i.e. bio will write a whole block so if a region to be updated smaller than submited block you obviously need a fresh copy.

Storing android adb shell screencap as variable

Making an app at the moment for my personal use (rooted) and it requires getting certain pixels colors from the screen. I was trying to accomplish this through the Runtime.
Process p = Runtime.getRuntime().exec("screencap");
p.waitFor();
InputStream is = p.getInputStream()
BitmapFactory.decodeStream(is);
and I get factory returned null.
but if I dump the process to my sd card through adb -d shell screencap /sdcard/ss.dump and access it from my app
BitmapFactory.decodeFile("/sdcard/ss.dump");
all goes well.
So it there anyway to dump the stream straight into BitmapFactory within my app?
Thanks SO and please excuse the generally laziness/shortcuts of the example code.

This might help if not too far off your intended path. (I think you are using node / javascript). I spawned the ADB.EXE command producing a stream (and being 'jailed' on Windows the program must transform the stream to account for linefeed ending differences. So with that, I have working the following:
exports.capture = function(filename) {
// you'll need to map your requirement (decodeStream) instead
// of streaming to a file
var strm = fs.createWriteStream(path);
var cv = new Convert();
cv.pipe(strm);
var capture = spawn(cmd, args);
capture.stdout.on('data', function(data) {
cv.write(data);
});
capture.stdout.on('exit', function(data) {
cv.end();
});
}
To explain the process, spawn is running the ADB command, on windows, CR-LF are inserted (being a PNG stream), and stream is chunked / piped through a fs-transformation. Others on the web have described the process as adb.exe shell screencap -p | sed 's/\r$//' > output.file. And it does work. To be clear the conversion is CR-CR-LF => LF for us window jailed birds. And if you don't want to implement a SED and not use javascript regular expressions converting binary->strings->binary, you may follow my path. There is probably a simpler way, I just don't know what it is...
So, Convert() is an object with methods that converts the byte stream on the fly.
See the codewind blog link is: http://codewinds.com/blog/2013-08-20-nodejs-transform-streams.html to build your convert.

When using screencap from inside an app you must use su, i.e. root. When you do this via adb it runs under a different user id, which has more permissions than a normal Android app.
There are several examples how to use screencap, e.g. here.