Develop Reference

Can not execute hexagon_graph_execution on hexagon-sim

I have followed the build_and_run_inception_hexagon.sh and generated the hexagon_graph_execution executable. Now instead of using a real device I would like to test inception model with hexagon-sim available at SDK 3.0. So there is no need to use adb push commands as the SDK can simulate HVX device with hexagon-sim.
I have put the run-time libraries and the inception model plus the image at the same folder. After execution It gives me this error:
~/Qualcomm/HEXAGON_Tools/7.2.12/Tools/bin/hexagon-sim ./hexagon_graph_execution "/home/aashouri/Qualcomm/Hexagon_SDK/3.0/test/common/inception"
Error: Unsupported machine type 0x0 in ELF image "./hexagon_graph_execution" - exiting.
Can anyone comment on this?
Hexagon-readelf:
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: ARM
Version: 0x1
Entry point address: 0xc8c9c
Start of program headers: 52 (bytes into file)
Start of section headers: 39944896 (bytes into file)
Flags: 0x5000000, Version5 EABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 8
Size of section headers: 40 (bytes)
Number of section headers: 36
Section header string table index: 35
XXD info:
0000000: 7f45 4c46 0101 0100 0000 0000 0000 0000 .ELF............
0000010: 0300 2800 0100 0000 9c8c 0c00 3400 0000 ..(.........4...
0000020: c082 6102 0000 0005 3400 2000 0800 2800 ..a.....4. ...(.
0000030: 2400 2300 0600 0000 3400 0000 3400 0000 $.#.....4...4...
0000040: 3400 0000 0001 0000 0001 0000 0400 0000 4...............
0000050: 0400 0000 0300 0000 3401 0000 3401 0000 ........4...4...
0000060: 3401 0000 1300 0000 1300 0000 0400 0000 4...............
0000070: 0100 0000 0100 0000 0000 0000 0000 0000 ................
0000080: 0000 0000 ece6 2a01 ece6 2a01 0500 0000 ......*...*.....
0000090: 0010 0000 0100 0000 20eb 2a01 20fb 2a01 ........ .*. .*.

ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: ARM
Version: 0x1
This binary was not built by hexagon-clang. You must have accidentally built it with the ARM toolchain -- the "Machine" field would say "Qualcomm Hexagon" if it had been built for the hexagon DSP.

FFmpeg camera encode h264 mp4 muxing atom avcC is too small

I tried to encode Android camera preview frame to h264, and mux to mp4 container.
I can created mp4 file successfully. But the mp4 format seems corrupted.
Use the ffprobe, I got the following error.
$ ffprobe o.mp4
[h264 # 0x209fe50] non-existing PPS 0 referenced
[h264 # 0x209fe50] decode_slice_header error
[h264 # 0x209fe50] no frame!
[mov,mp4,m4a,3gp,3g2,mj2 # 0x209ea60] decoding for stream 0 failed
[mov,mp4,m4a,3gp,3g2,mj2 # 0x209ea60] Could not find codec parameters for stream 0 (Video: h264 (avc1 / 0x31637661), none, 568x320, 505 kb/s): unspecified pixel format
Then I use mp4info tool to see if the information is correct. I found this.
$ mp4info o.mp4
mp4info version 2.0.0
o.mp4:
ReadProperties: atom 'avcC' is too small; overrun at property: configurationVersion (src/mp4atom.cpp,386)
mp4info: can't open o.mp4
By hexdump the content of file, I got this
$ xxd o.mp4 |grep -A 5 -B 5 avcC
0039170: 6331 0000 0000 0000 0001 0000 0000 0000 c1..............
0039180: 0000 0000 0000 0000 0000 0238 0140 0048 ...........8.#.H
0039190: 0000 0048 0000 0000 0000 0001 0000 0000 ...H............
00391a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00391b0: 0000 0000 0000 0000 0000 0000 0018 ffff ................
00391c0: 0000 0008 6176 6343 0000 0020 7374 7473 ....avcC... stts
00391d0: 0000 0000 0000 0002 0000 004f 0000 0ea6 ...........O....
00391e0: 0000 0001 0000 0000 0000 0058 7374 7373 ...........Xstss
00391f0: 0000 0000 0000 0012 0000 0001 0000 0005 ................
0039200: 0000 000a 0000 000f 0000 0014 0000 0019 ................
0039210: 0000 001e 0000 0023 0000 0028 0000 0029 .......#...(...)
If I didn't add global header to AVCodecContext,
if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER) {
// oc_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
}
ffprobe can detect the format without error, also ffplay can play it. But the avcC atom still not correct. Other player cannot play it.
Why the muxer didn't write the correct avcC atom?
How can I solve it?

You must set the extradata field in the codec context. The format is referenced here: Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream

Solved. This problem only happened in the new version ffmpeg.
When call ffmpeg api avformat_write_header/av_write_tailer, it uses the data in AVStream->codecpar since AVStream->codec was marked deprecated.
And the extra data is filled after avcodec_open2, so I need to copy extra data back to AVStream->codecpar.
stream->codecpar->extradata = av_malloc(oc_ctx->extradata_size + FF_INPUT_BUFFER_PADDING_SIZE);
stream->codecpar->extradata_size = oc_ctx->extradata_size;
memcpy(stream->codecpar->extradata, oc_ctx->extradata, oc_ctx->extradata_size);
Similar code snippet also can be found in ffmpeg project ffmpeg.c
if (!ost->st->codecpar->extradata && avctx->extradata) {
ost->st->codecpar->extradata = av_malloc(avctx->extradata_size + FF_INPUT_BUFFER_PADDING_SIZE);
if (!ost->st->codecpar->extradata) {
av_log(NULL, AV_LOG_ERROR, "Could not allocate extradata buffer to copy parser data.\n");
exit_program(1);
}
ost->st->codecpar->extradata_size = avctx->extradata_size;
memcpy(ost->st->codecpar->extradata, avctx->extradata, avctx->extradata_size);
}

Android editing ubi/ubfs system image

I have a ubifs system image (https://www.dropbox.com/s/txgye8mu5r3og5y/system.img?dl=0) for a mediatek tablet device and am trying to add and remove some files.
I'm stuck trying to mount/extract files from the image.
Here are the steps I have tried so far on Debian Jessie with kernel 4.1.0-0.bpo.2-amd64:
I tried:
https://github.com/jrspruitt/ubi_reader
$ ubireader_display_info ./system.img
UBI File
---------------------
Min I/O: 16384
LEB Size: 4161536
PEB Size: 4194304
Total Block Count: 122
Data Block Count: 120
Layout Block Count: 2
Internal Volume Block Count: 0
Unknown Block Count: 0
First UBI PEB Number: 0
Image: 1101756791
---------------------
Image Sequence Num: 1101756791
Volume Name:system
PEB Range: 0 - 121
Volume: system
---------------------
Vol ID: 0
Name: system
Block Count: 120
Volume Record
---------------------
alignment: 1
crc: 3336263623
data_pad: 0
errors:
flags: autoresize
name: system
name_len: 6
padding:
rec_index: 0
reserved_pebs: 248
upd_marker: 0
vol_type: dynamic
But when I try and extract files using ubireader_extract_files I get the correct number of files but the resulting files are garbage.
Next I dismantled the tablet to work out what nand flash it was using to try and use nandsim following this post:
https://web.archive.org/web/20150109021228/http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_extract
to emulate the nand and found out it was using SanDisk SDTNRGAMA 64G 3.3V 8-bit which has id bytes of 0x45,0xde,0x94,0x93,0x76,0x50 - from the following post:
http://lists.infradead.org/pipermail/linux-mtd/2014-January/051330.html
Running the following causes a segfault - on earlier kernels the id_bytes option is not recognized:
`modprobe nandsim id_bytes=0x45,0xde,0x94,0x93,0x76,0x50 cache_file=./test.img`
which gives the following segfault:
[ 142.734637] [nandsim] warning: read_byte: unexpected data output cycle, state is STATE_READY return 0x0
[ 142.734637] [nandsim] warning: read_byte: unexpected data output cycle, state is STATE_READY return 0x0
[ 142.734640] nand: device found, Manufacturer ID: 0x45, Chip ID: 0xde
[ 142.734641] nand: SanDisk SDTNRGAMA 64G 3.3V 8-bit
[ 142.734644] nand: 8192 MiB, MLC, erase size: 4096 KiB, page size: 16384, OOB size: 1280
[ 142.734650] nand: No oob scheme defined for oobsize 1280
[ 142.734672] ------------[ cut here ]------------
[ 142.734674] kernel BUG at /build/linux-PoJsUp/linux-4.1.6/drivers/mtd/nand/nand_base.c:3952!
[ 142.734677] invalid opcode: 0000 [#1] SMP
[ 142.734680] Modules linked in: nandsim(+) nand nand_ecc nand_bch bch nand_ids mtd cfg80211 rfkill joydev nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc iosf_mbi coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel hid_generic aes_x86_64 lrw irda gf128mul glue_helper psmouse vmw_balloon crc_ccitt ablk_helper serio_raw vmw_vmci cryptd battery pcspkr 8250_fintek acpi_cpufreq processor thermal_sys ac shpchp evdev i2c_piix4 fuse parport_pc ppdev lp parport autofs4 usbhid hid ext4 crc16 mbcache jbd2 sr_mod cdrom ata_generic sg sd_mod crc32c_intel ata_piix uhci_hcd ehci_pci ehci_hcd usbcore e1000 usb_common button libata vmwgfx ttm mptspi scsi_transport_spi mptscsih drm_kms_helper mptbase scsi_mod drm
[ 142.734731] CPU: 0 PID: 1235 Comm: modprobe Not tainted 4.1.0-0.bpo.2-amd64 #1 Debian 4.1.6-1~bpo8+1
[ 142.734733] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/20/2012
[ 142.734735] task: ffff88007aaf54f0 ti: ffff880079134000 task.ti: ffff880079134000
[ 142.734737] RIP: 0010:[<ffffffffa05d5ff0>] [<ffffffffa05d5ff0>] nand_scan_tail+0xa40/0xac0 [nand]
[ 142.734743] RSP: 0018:ffff880079137c58 EFLAGS: 00010296
[ 142.734745] RAX: 000000000000002c RBX: ffff880077093450 RCX: 0000000000000006
[ 142.734746] RDX: 000000000000002c RSI: 0000000000000246 RDI: ffff88007f60ea10
[ 142.734748] RBP: ffff880077093000 R08: 00000000000094d8 R09: 00000000000044aa
[ 142.734750] R10: 0000000000000086 R11: 20726f662064656e R12: ffff880077093860
[ 142.734751] R13: 0000000000000000 R14: ffffffffa05ec200 R15: ffff88007b67ad40
[ 142.734754] FS: 00007fe945772700(0000) GS:ffff88007f600000(0000) knlGS:0000000000000000
[ 142.734756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 142.734757] CR2: 00007f57a6920040 CR3: 00000000790fa000 CR4: 00000000000406f0
[ 142.734870] Stack:
[ 142.734873] 0000000000000000 0000000000000000 ffff880077093000 ffffffffa05ef54a
[ 142.734877] 0000000000000000 0000000000000018 ffff880079137cd8 ffff880079137c98
[ 142.734879] 0000000000000000 ffffffff81814080 ffff880077211760 ffffffffa05ef000
[ 142.734882] Call Trace:
[ 142.734889] [<ffffffffa05ef54a>] ? ns_init_module+0x54a/0x1000 [nandsim]
[ 142.734896] [<ffffffffa05ef000>] ? 0xffffffffa05ef000
[ 142.734902] [<ffffffff81002148>] ? do_one_initcall+0xd8/0x210
[ 142.734907] [<ffffffff815723c1>] ? do_init_module+0x5a/0x1c2
[ 142.734912] [<ffffffff810f2316>] ? load_module+0x2026/0x24e0
[ 142.734915] [<ffffffff810ede60>] ? store_uevent+0x40/0x40
[ 142.734919] [<ffffffff810ee9d5>] ? copy_module_from_fd.isra.45+0xb5/0x140
[ 142.734923] [<ffffffff810f299d>] ? SyS_finit_module+0x7d/0xa0
[ 142.734928] [<ffffffff815792b2>] ? system_call_fast_compare_end+0xc/0x6b
[ 142.734930] Code: 00 00 30 10 5d a0 e9 f8 f6 ff ff 48 c7 83 88 03 00 00 30 19 5d a0 e9 3c f7 ff ff 89 c6 48 c7 c7 b8 9c 5d a0 31 c0 e8 33 c2 f9 e0 <0f> 0b 48 c7 83 40 03 00 00 40 bb 5d a0 e9 14 f6 ff ff 48 c7 83
[ 142.734959] RIP [<ffffffffa05d5ff0>] nand_scan_tail+0xa40/0xac0 [nand]
[ 142.734964] RSP <ffff880079137c58>
[ 142.734975] ---[ end trace 0270ba33a10a2b05 ]---
So, in short - I need help. I'm not massively familiar with ubi/ubifs method and cannot find any sane well written guides which show you have to mount/extract files from an existing image.
Update: su is installed on the tablet, and I set selinux to permissive mode:
adb shell su -c setenforce 0
from: https://source.android.com/devices/tech/security/selinux/validate.html
Update 03Oct15:
Ran the mdtinfo -a on the tablet and got the following result:
mtd16
Name: system
Type: nand
Eraseblock size: 4194304 bytes, 4.0 MiB
Amount of eraseblocks: 256 (1073741824 bytes, 1024.0 MiB)
Minimum input/output unit size: 16384 bytes
Sub-page size: 16384 bytes
OOB size: 1280 bytes
Character device major/minor: 90:32
Bad blocks are allowed: true
Device is writable: true
Using the information above I tried to create a blank ubifs image on my pc, I get the error that the LEB is too large! Looks like I have a limit of 2MiB on the LEB size!
$ mkfs.ubifs -m 16384 -e 4MiB -c 256 -o ./image.img
Error: too large LEB size 4194304

It looks like that ubi image is using a different compression type for the data. If you run ubireader_extract_files -v system.img the -v is for very verbose, the UBIFS data nodes have a compression type of 3 (compr_type: 3) as far as I know 1 and 2 are the only valid options, LZO and ZLIB respectively. Perhaps they used a custom compression, or somehow it got the wrong number associated with it. But it explains why the files and directories extract okay, but the data is scrambled.

Just in case someone else finds this post. I managed to find a workaround which involved editing the boot.img using mtk-tools to mount the root partition in rw mode rather than ro. (Look in init.rc in the root of boot.img and change any mount options for /system to rw).
I them managed to edit the root image, power down the tablet and then used MTK Droid Tools to image the partition.

ShellScript hex to decimal

I have a file how capture the users events in a txt file :
/dev/input/event2: 0003 0039 0000001d
/dev/input/event2: 0003 0035 00000169
Now I would create a shell script how converts hex data to decimal for example I would like :
/dev/input/event2: 3 57 29
/dev/input/event2: 3 53 361
How can I process ? I have tried to read the file line by line and split the string but It doesn't work..

If you are using gawk, you could do as following. input.txt is the file you capture the events.
$ awk '{printf "%s ", $1;for(i=2;i<=NF;i++){v="0x"$i;printf "%d ",strtonum(v)};printf "\n";}' input.txt
/dev/input/event2: 3 57 29
/dev/input/event2: 3 53 361

What does /proc/pid/maps mean?

When I analyse a app memory problem, I found its maps (cat /proc/pid/maps) like this:
903 5fec1000-5fec2000 r--s 00001000 b3:10 98 /system/app/PacProcessor.apk
904 5fec2000-5fec3000 r--s 00000000 b3:10 98 /system/app/PacProcessor.apk
905 5fec3000-5fed7000 r--s 00560000 b3:10 125 /system/app/iReader.apk
906 5fed7000-5ff09000 r--s 0019a000 b3:10 125 /system/app/iReader.apk
907 5ff09000-5ff0b000 r--s 00043000 b3:10 81 /system/app/Galaxy4.apk
908 5ff0b000-5ff0c000 r--s 00042000 b3:10 81 /system/app/Galaxy4.apk
It seems this app loads other app's code into its stack.
How can I avoid this kind memory allocation ?

From proc manual on Linux:
/proc/[pid]/maps
A file containing the currently mapped memory regions and their access permissions. See
mmap(2) for some further information about memory mappings.