I am trying to read in a large file using asset manager in Android NDK.
The problem is that the code that I have written is not reading the entire content. Rather only a portion of it. When I try to achieve the same functionality using Java, it is giving me correct results.
This is the code that I have written:
std::string file = hats::files::SOURCE_DATASET_FILENAME;
AAssetManager *mgr = AAssetManager_fromJava(env, assetManager);
AAsset *asset = AAssetManager_open(mgr, file.c_str(), AASSET_MODE_BUFFER);
size_t assetLength = AAsset_getLength(asset);
char *buffer = (char *) malloc(assetLength + 1);
int nbytes{0};
while( (nbytes = AAsset_read(asset, buffer, assetLength)) > 0) {
LOGD("%s", buffer);
AAsset_seek(asset, nbytes, SEEK_CUR);
}
AAsset_close(asset);
return env->NewStringUTF(file.c_str());
I am not able to understand why only partial file is being read. I am not able to find any proper tutorial for Android NDK also.
Please help me out.
Related
I am building a forensic tool, that shall read and write from file slacks. At the moment I use JNI to call the C functions read() and write() to write a buffer to the FAT SD card (also EXT4 internal memory would be nice). I use those because you can pass the length to read/write thus ignoring EOF.
I already tried writing by using a standard write and then truncate it in order to write in the file slack, which works on ubuntu 14.04 but not on Android (API 21).
Reading more from a file than the actuall file size does not reflect the flie slack on the SD-card.
There are tools e.g. "bmap" or "slacker.exe" (for NTFS) that manage to access file slacks. I am in need of a way to ignore EOF or handle it myself. I would prefere not to change existing file system drivers.
I appreciate any suggestions.
here is some sample code (that does not work yet):
jstring
Java_com_example_hellojni_HelloJni_write(JNIEnv *env, jobject thiz, jstring path)
{
char *cpath = (*env)->GetStringUTFChars(env, path, NULL);
int raw_file_descriptor = open(cpath,O_WRONLY,0);
lseek(raw_file_descriptor,0,SEEK_END);
char block_buffer [4096] = "hidden";
int buffer_length = 6;
int wret = write(raw_file_descriptor,block_buffer,buffer_length); // increases file size and moves EOF - I don't want that
LOGD(" char written: %d", wret);
free(cpath);
return (*env)->NewStringUTF(env, "write ok ");
}
jbyteArray
Java_com_example_hellojni_HelloJni_read2(JNIEnv *env, jobject thiz, jstring path) {
char *cpath = (*env)->GetStringUTFChars(env, path, NULL);
LOGD("open %s with ", cpath);
int raw_file_descriptor = open(cpath,O_RDONLY,0);
char buffer [4096];
int readretval = read(raw_file_descriptor, buffer, 50); // stops at EOF - I want it to continue reading all 50 bytes from the SD-card - This gives me the file content until EOF and then I get some random characters from the uninitialized buffer.
LOGD("read (%d) buffer with length %d: %s", readretval, sizeof(buffer), buffer);
int i; // Debug code
for (i=0; i < 49; ++i) {
LOGD("%c",buffer[i]);
}
close(raw_file_descriptor);
free(cpath);
return NULL;
}
Apparently this way is impossible. But with root it is possible to write file slack via mounting a loop back device. Still, ignoring EOF is not possible without changing the driver or implementing your own kernel module.
I'm working on a Cocos2dx C++ project and I am trying to load a XMLfile and parse it with rapidxml.
The Win32 build works great with this code.
rapidxml::file<> xmlFile("myxml.xml");
xml_document<> doc; // character type defaults to char
doc.parse<0>(xmlFile.data()); // 0 means default parse flags
xml_node<> *main = doc.first_node(); //Get the main node that contains everything
This does not work for the Android build. It's most likely because "myxml.xml" can't be found.
So my solution was to load the file in a different way.
So I tried to replace the xmlFile.data() with this. The mobile could now locate and read the data but I was unable to parse it trough rapidxml.
const unsigned char* xmldata = CCFileUtils::sharedFileUtils()->getFileData("myxml.xml", "r", &tmpSize);
But then I have no idea how to parse it.
This is another way of getting the data but still no idea how to parse with rapidxml.
std::string xmldata2 = CCFileUtils::sharedFileUtils()->getStringFromFile("myxml.xml");
Any ideas how to parse this in rapidxml, or any other suggestions how to solve this?
I'm new to c++ so any help is very much appreciated.
I have used PugiXML and it is relatively simple to use. A file is read as:
xml_document* doc = new xml_document;
unsigned char* pBuffer = NULL;
unsigned long bufferSize = 0;
std::string fullPath = CCFileUtils::sharedFileUtils()->fullPathFromRelativePath(filename);
pBuffer = CCFileUtils::sharedFileUtils()->getFileData(fullPath.c_str(), "r", &bufferSize);
if(bufferSize == 0)
{
CCLog("file failed to open");
delete doc;
}
xml_parse_result results = doc->load_buffer(pBuffer, bufferSize);
CCLog("filename: %s | result: %s | status: %d",filename, results.description(), results.status);
if(pBuffer)
{
delete [] pBuffer;
pBuffer = NULL;
}
xml_node mainNode = doc->child("main");
mainNode then has contents of the parsed file.
From the examples I got the basic idea of this code.
However I am not sure, what I am missing, as muxing.c demuxing.c and decoding_encoding.c
all use different approaches.
The process of converting an audio file to another file should go roughly like this:
inputfile -demux-> audiostream -read-> inPackets -decode2frames->
frames
-encode2packets-> outPackets -write-> audiostream -mux-> outputfile
However I found the following comment in demuxing.c:
/* Write the raw audio data samples of the first plane. This works
* fine for packed formats (e.g. AV_SAMPLE_FMT_S16). However,
* most audio decoders output planar audio, which uses a separate
* plane of audio samples for each channel (e.g. AV_SAMPLE_FMT_S16P).
* In other words, this code will write only the first audio channel
* in these cases.
* You should use libswresample or libavfilter to convert the frame
* to packed data. */
My questions about this are:
Can I expect a frame that was retrieved by calling one of the decoder functions, f.e.
avcodec_decode_audio4 to hold suitable values to directly put it into an encoder or is
the resampling step mentioned in the comment mandatory?
Am I taking the right approach? ffmpeg is very asymmetric, i.e. if there is a function
open_file_for_input there might not be a function open_file_for_output. Also there are different versions of many functions (avcodec_decode_audio[1-4]) and different naming
schemes, so it's very hard to tell, if the general approach is right, or actually an
ugly mixture of techniques that where used at different version bumps of ffmpeg.
ffmpeg uses a lot of specific terms, like 'planar sampling' or 'packed format' and I am having a hard time, finding definitions for these terms. Is it possible to write working code, without deep knowledge of audio?
Here is my code so far that right now crashes at avcodec_encode_audio2
and I don't know why.
int Java_com_fscz_ffmpeg_Audio_convert(JNIEnv * env, jobject this, jstring jformat, jstring jcodec, jstring jsource, jstring jdest) {
jboolean isCopy;
jclass configClass = (*env)->FindClass(env, "com.fscz.ffmpeg.Config");
jfieldID fid = (*env)->GetStaticFieldID(env, configClass, "ffmpeg_logging", "I");
logging = (*env)->GetStaticIntField(env, configClass, fid);
/// open input
const char* sourceFile = (*env)->GetStringUTFChars(env, jsource, &isCopy);
AVFormatContext* pInputCtx;
AVStream* pInputStream;
open_input(sourceFile, &pInputCtx, &pInputStream);
// open output
const char* destFile = (*env)->GetStringUTFChars(env, jdest, &isCopy);
const char* cformat = (*env)->GetStringUTFChars(env, jformat, &isCopy);
const char* ccodec = (*env)->GetStringUTFChars(env, jcodec, &isCopy);
AVFormatContext* pOutputCtx;
AVOutputFormat* pOutputFmt;
AVStream* pOutputStream;
open_output(cformat, ccodec, destFile, &pOutputCtx, &pOutputFmt, &pOutputStream);
/// decode/encode
error = avformat_write_header(pOutputCtx, NULL);
DIE_IF_LESS_ZERO(error, "error writing output stream header to file: %s, error: %s", destFile, e2s(error));
AVFrame* frame = avcodec_alloc_frame();
DIE_IF_UNDEFINED(frame, "Could not allocate audio frame");
frame->pts = 0;
LOGI("allocate packet");
AVPacket pktIn;
AVPacket pktOut;
LOGI("done");
int got_frame, got_packet, len, frame_count = 0;
int64_t processed_time = 0, duration = pInputStream->duration;
while (av_read_frame(pInputCtx, &pktIn) >= 0) {
do {
len = avcodec_decode_audio4(pInputStream->codec, frame, &got_frame, &pktIn);
DIE_IF_LESS_ZERO(len, "Error decoding frame: %s", e2s(len));
if (len < 0) break;
len = FFMIN(len, pktIn.size);
size_t unpadded_linesize = frame->nb_samples * av_get_bytes_per_sample(frame->format);
LOGI("audio_frame n:%d nb_samples:%d pts:%s\n", frame_count++, frame->nb_samples, av_ts2timestr(frame->pts, &(pInputStream->codec->time_base)));
if (got_frame) {
do {
av_init_packet(&pktOut);
pktOut.data = NULL;
pktOut.size = 0;
LOGI("encode frame");
DIE_IF_UNDEFINED(pOutputStream->codec, "no output codec");
DIE_IF_UNDEFINED(frame->nb_samples, "no nb samples");
DIE_IF_UNDEFINED(pOutputStream->codec->internal, "no internal");
LOGI("tests done");
len = avcodec_encode_audio2(pOutputStream->codec, &pktOut, frame, &got_packet);
LOGI("encode done");
DIE_IF_LESS_ZERO(len, "Error (re)encoding frame: %s", e2s(len));
} while (!got_packet);
// write packet;
LOGI("write packet");
/* Write the compressed frame to the media file. */
error = av_interleaved_write_frame(pOutputCtx, &pktOut);
DIE_IF_LESS_ZERO(error, "Error while writing audio frame: %s", e2s(error));
av_free_packet(&pktOut);
}
pktIn.data += len;
pktIn.size -= len;
} while (pktIn.size > 0);
av_free_packet(&pktIn);
}
LOGI("write trailer");
av_write_trailer(pOutputCtx);
LOGI("end");
/// close resources
avcodec_free_frame(&frame);
avcodec_close(pInputStream->codec);
av_free(pInputStream->codec);
avcodec_close(pOutputStream->codec);
av_free(pOutputStream->codec);
avformat_close_input(&pInputCtx);
avformat_free_context(pOutputCtx);
return 0;
}
Meanwhile I have figured this out and written an Android Library Project that does this
(for audio files). https://github.com/fscz/FFmpeg-Android
See the file /jni/audiodecoder.c for details
I have only been able to find solutions dated 2010 and earlier. So I wanted to see if there was a more up-to-date stance on this.
I'd like to avoid using Java and purely use C++, to access files (some less-or-more than 1MB) stored away in the APK. Using AssetManager means I can't access files like every other file on every other operating system (including iOS).
If not, is there a method in C++ where I could somehow map fopen/fread to the AssetManager APIs?
I actually found pretty elegant answer to the problem and blogged about it here.
The summary is:
The AAssetManager API has NDK bindings. This lets you load assets from the APK.
It is possible to combine a set of functions that know how to read/write/seek against anything and disguise them as a file pointer (FILE*).
If we create a function that takes an asset name, uses AssetManager to open it, and then disguises the result as a FILE* then we have something that's very similar to fopen.
If we define a macro named fopen we can replace all uses of that function with ours instead.
My blog has a full write up and all the code you need to implement in pure C. I use this to build lua and libogg for Android.
Short answer
No. AFAIK mapping fread/fopen in C++ to AAssetManager is not possible. And if were it would probably limit you to files in the assets folder. There is however a workaround, but it's not straightforward.
Long Answer
It IS possible to access any file anywhere in the APK using zlib and libzip in C++.
Requirements : some java, zlib and/or libzip (for ease of use, so that's what I settled for). You can get libzip here: http://www.nih.at/libzip/
libzip may need some tinkering to get it to work on android, but nothing serious.
Step 1 : retrieve APK location in Java and pass to JNI/C++
String PathToAPK;
ApplicationInfo appInfo = null;
PackageManager packMgmr = parent.getPackageManager();
try {
appInfo = packMgmr.getApplicationInfo("com.your.application", 0);
} catch (NameNotFoundException e) {
e.printStackTrace();
throw new RuntimeException("Unable to locate APK...");
}
PathToAPK = appInfo.sourceDir;
Passing PathToAPK to C++/JNI
JNIEXPORT jlong JNICALL Java_com_your_app(JNIEnv *env, jobject obj, jstring PathToAPK)
{
// convert strings
const char *apk_location = env->GetStringUTFChars(PathToAPK, 0);
// Do some assigning, data init, whatever...
// insert code here
//release strings
env->ReleaseStringUTFChars(PathToAPK, apk_location);
return 0;
}
Assuming that you now have a std::string with your APK location and you have zlib on libzip working you can do something like this:
if(apk_open == false)
{
apk_file = zip_open(apk_location.c_str(), 0, NULL);
if(apk_file == NULL)
{
LOGE("Error opening APK!");
result = ASSET_APK_NOT_FOUND_ERROR;
}else
{
apk_open = true;
result = ASSET_NO_ERROR;
}
}
And to read a file from the APK:
if(apk_file != NULL){
// file you wish to read; **any** file from the APK, you're not limited to regular assets
const char *file_name = "path/to/file.png";
int file_index;
zip_file *file;
struct zip_stat file_stat;
file_index = zip_name_locate(apk_file, file_name, 0);
if(file_index == -1)
{
zip_close(apk_file);
apk_open = false;
return;
}
file = zip_fopen_index(apk_file, file_index, 0);
if(file == NULL)
{
zip_close(apk_file);
apk_open = false;
return;
}
// get the file stats
zip_stat_init(&file_stat);
zip_stat(apk_file, file_name, 0, &file_stat);
char *buffer = new char[file_stat.size];
// read the file
int result = zip_fread(file, buffer, file_stat.size);
if(result == -1)
{
delete[] buffer;
zip_fclose(file);
zip_close(apk_file);
apk_open = false;
return;
}
// do something with the file
// code goes here
// delete the buffer, close the file and apk
delete[] buffer;
zip_fclose(file);
zip_close(apk_file);
apk_open = false;
Not exactly fopen/fread but it gets the job done. It should be pretty easy to wrap this to your own file reading function to abstract the zip layer.
I am trying to load a TGA file in Android NDK.
I open the file using AssetManager, read in the entire contents of the TGA file into a memory buffer, and then I try to extract the pixel data from it.
I can read the TGA header part of the file without any problems, but when I try to advance the memory pointer past the TGA header, the app crashes. If I don't try to advance the memory pointer, it does not crash.
Is there some sort of limitation in Android NDK for pointer arithmetic?
Here is the code:
This function opens the asset file:
char* GEAndroid::OpenAssetFile( const char* pFileName )
{
char* pBuffer = NULL;
AAssetManager* assetManager = m_pState->activity->assetManager;
AAsset* assetFile = AAssetManager_open(assetManager, pFileName, AASSET_MODE_UNKNOWN);
if (!assetFile) {
// Log error as 'error in opening the input file from apk'
LOGD( "Error opening file %s", pFileName );
}
else
{
LOGD( "File opened successfully %s", pFileName );
const void* pData = AAsset_getBuffer(assetFile);
off_t fileLength = AAsset_getLength(assetFile);
LOGD("fileLength=%d", fileLength);
pBuffer = new char[fileLength];
memcpy( pBuffer, pData, fileLength * sizeof( char ) );
}
return pBuffer;
}
And down here in my texture class I try to load it:
char* pBuffer = g_pGEAndroid->OpenAssetFile( fileNameWithPath );
TGA_HEADER textureHeader;
char *pImageData = NULL;
unsigned int bytesPerPixel = 4;
textureHeader = *reinterpret_cast<TGA_HEADER*>(pBuffer);
// I double check that the textureHeader is valid and it is.
bytesPerPixel = textureHeader.bits/8; // Divide By 8 To Get The Bytes Per Pixel
m_imageSize = textureHeader.width*textureHeader.height*bytesPerPixel; // Calculate The Memory Required For The TGA Data
pImageData = new char[m_imageSize];
// the line below causes the crash
pImageData = reinterpret_cast<char*>(pBuffer + sizeof( TGA_HEADER)); // <-- causes a crash
If I replace the line above with the following line (even though it is incorrect), the app runs, although obviously the texture is messed up.
pImageData = reinterpret_cast<char*>(pBuffer); // <-- does not crash, but obviously texture is messed up.
Anyone have any ideas?
Thanks.
Why reinterpret_cast? You're adding an integer to a char*; that operation produces a char*. No typecast necessary.
One caveat for pointer juggling on Android (and on ARM devices in general): ARM cannot read/write unaligned data from memory. If you read/write an int-sized variable, it needs to be at an address that's a multiple of 4; for short, a multiple of 2. Bytes can be at any address. This does not, as far as I can see, apply to the presented snippet. But do keep in mind. It does throw off binary format parsing occasionally, especially when ported from Intel PCs.
Simply assigning an unaligned value to a pointer does not crash. Dereferencing it might.
Sigh, I just realized the mistake. I allocate memory for pImageData, then set the point to the buffer. This does not sit well when I try to create an OpenGL texture with the pixel data. Modifying it so I memcpy the pixel data from (pBuffer + sizeof( TGA_HEADER) ) to pImageData fixes the problem.