I have a locally stored file, around 2.3MB in size, about 500 000 lines altogether and I would like to store it into a HashSet into memory. Since the file is large, and reading is so slow, I have split the file into 5 smaller ones, less than 100 000 lines each.
My idea is to instantiate 5 separate threads from the Application class. Each thread would read its own file and store data in its own set. Upon completion, it would return the obtained subset to the main thread, ie. to the Application class, which would then store in the main set.
Thread code is as follows:
private class LoadFileThread extends Thread {
private String filename;
private Set<String> subSet;
private MyApplication application;
public LoadFileThread(String filename, MyApplication ctx) {
this.filename = filename;
this.application = ctx;
this.subSet = new HashSet<String>();
}
#Override
public void run() {
AssetManager am = application.getAssets();
BufferedReader reader = null;
try {
InputStream is = am.open(filename);
reader = new BufferedReader(new InputStreamReader(
is));
String line = null;
while ((line = reader.readLine()) != null) {
subSet.add(line.toUpperCase());
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {reader.close();}catch (IOException ignorable) {}
}
application.setSubSet(subSet, this.getName());
}
}
Method setSubSet in the Application class:
public synchronized void setSubSet(Set<String> subSet, String name) {
myMainSet.addAll(subSet);
Log.d("Thread finished", name);
}
Two problems occur:
Reading is still waaaaay to slow.
I get an out of memory error when calling addAll on the main set.
Is there a better way to do this? How?
With 500,000 lines and readLine () you are doing 500,000 reads.
Create a 64k buffer and read into that.
Process each full line you can then read another 64k.
That should cut your reads into a fraction of 500,000
Related
How can i read a large text file into my Application?
This is my code but it does not work. My code must read a file called list.txt. The code worked only with a file with only 10.000 lines.
can someone helps me?
Thanks!
My code:(Worked with small files, but not with large files)
private void largefile(){
String strLine2="";
wwwdf2 = new StringBuffer();
InputStream fis2 = this.getResources().openRawResource(R.raw.list);
BufferedReader br2 = new BufferedReader(new InputStreamReader(fis2));
if(fis2 != null) {
try {
LineNumberReader lnr = new LineNumberReader(br2);
String linenumber = String.valueOf(lnr);
while ((strLine2 = br2.readLine()) != null) {
wwwdf2.append(strLine2 + "\n");
}
// Toast.makeText(getApplicationContext(), linenumber, Toast.LENGTH_LONG).show();
Toast.makeText(getApplicationContext(), wwwdf2, Toast.LENGTH_LONG).show();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Since you are processing a large file, you should process the data in chunks . Here your file reading is fine but then you keep adding all rows in string buffer and finally passing to Toast.makeText(). It creates a big foot-print in memory. Instead you can read 100-100 lines and call Toast.makeText() to process in chunks. One more thing, use string builder instead of string buffer go avoid unwanted overhead of synchronization. You initializing wwwdf2 variable inside the method but looks it is a instance variable which I think is not required. Declare it inside method to make it's scope shorter.
I am basically trying to read a long list of numbers(doubles)from a text file and save them into an array. I have these lines of code but it doesn't work when I load into my android smartphone. The readfile() does work completely when I use debug mode to check if my code reads the ExamScore, it does read and store the values as expected in my laptop. When it loads into smartphone, it just doesn't work. I save my ExamScore.txt in the root directory of android studio, for example, Users->AndroidStudioProjects->Project A. The main concern I have is that:
How do I know if this ExamScore.txt is saved into my smartphone as well when I build the app? Do I have to save the text file into my smartphone separately or something?The error I get is
java.io.FileNotFoundException: ExamScore.txt: open failed: ENOENT (No such file or directory)
static double[] readfile() throws FileNotFoundException{
Scanner scorefile = new Scanner(new File("ExamScore.txt"));
int count = -1;
double[] score = new double[8641];
while (scorefile.hasNext()) {
count = count + 1;
score[count] = Double.parseDouble(scorefile.nextLine());
}
scorefile.close();
return score;
}
In my main code,
double []score=readfile();
I save my ExamScore.txt in the root directory of android studio, for example, Users->AndroidStudioProjects->Project A... How do I know if this ExamScore.txt is saved into my smartphone as well when I build the app?
It isn't.
You need to create an assets folder.
Refer: Where do I place the 'assets' folder in Android Studio?
And you would use getAssets() to read from that folder.
public class MainActivity extends Activity {
private double[] readfile() throws FileNotFoundException{
InputStream fileStream = getAssets().open("ExamScore.txt");
// TODO: read an InputStream
}
}
Note: that is a read-only location of your app.
Or you can use the internal SD card.
How do I read the file content from the Internal storage - Android App
EDIT With refactored code in other answer
public static List<Double> readScore(Context context, String filename) {
List<Double> scores = new ArrayList<>();
AssetManager mgr = context.getAssets();
try (
BufferedReader reader = new BufferedReader(
new InputStreamReader(mgr.open(fileName)));
) {
String mLine;
while ((mLine = reader.readLine()) != null) {
scores.add(Double.parseDouble(mLine));
}
} catch (NumberFormatException e) {
Log.e("ERROR: readScore", e.getMessage());
}
return scores;
}
And then
List<Double> scores = readScore(MainActivity.this, "score.txt");
For those who are wondering, this is my solution! Thank you all for your help!!!! The issue I had was I didn't write it in the main activity but wrote the code in other java file. After writing this in the main activity file and putting my text file inside the assets folder. The issue is resolved :
public static LinkedList<Double> score=new LinkedList<Double>();
public void readScore() throws java.io.IOException {
BufferedReader reader = new BufferedReader(
new InputStreamReader(getAssets().open("score.txt")));
String mLine;
while ((mLine = reader.readLine()) != null) {
score.add(Double.parseDouble(mLine));
}
reader.close();
}
How can I parse /proc/cpuinfo virtual file of my Android tablet to get information of the processor's core and clockspeed?
I don’t need all information provided by the above file; just these two bits.
Can someone please help?
It is not clear if you want this information inside your app, or just for your own use.
you can get this information on with adb:
adb shell cat /proc/cpuinfo
If you want to use this information in your app, create a simple function to return a Map<String,String>, for example,
public static Map<String, String> getCpuInfoMap() {
Map<String, String> map = new HashMap<String, String>();
try {
Scanner s = new Scanner(new File("/proc/cpuinfo"));
while (s.hasNextLine()) {
String[] vals = s.nextLine().split(": ");
if (vals.length > 1) map.put(vals[0].trim(), vals[1].trim());
}
} catch (Exception e) {Log.e("getCpuInfoMap",Log.getStackTraceString(e));}
return map;
}
Note, this will not get multiple cpus information, overwrites. Most of the values are similar anyways. or Modify to create List of CpuInfoMaps.
try,
Log.d("getCpuInfoMap test", getCpuInfoMap().toString());
I hope its not too late for an answer but, this is how i get the current frequency for a specific cpu core:
public class MainActivity extends Activity{
private static final int INSERTION_POINT = 27;
private static String getCurFrequencyFilePath(int whichCpuCore){
StringBuilder filePath = new StringBuilder("/sys/devices/system/cpu/cpu/cpufreq/scaling_cur_freq");
filePath.insert(INSERTION_POINT, whichCpuCore);
return filePath.toString();
}
public static int getCurrentFrequency(int whichCpuCore){
int curFrequency = -1;
String cpuCoreCurFreqFilePath = getCurFrequencyFilePath(whichCpuCore);
if(new File(cpuCoreCurFreqFilePath).exists()){
try {
BufferedReader br = new BufferedReader(new FileReader(new File(cpuCoreCurFreqFilePath)));
String aLine;
while ((aLine = br.readLine()) != null) {
try{
curFrequency = Integer.parseInt(aLine);
}
catch(NumberFormatException e){
Log.e(getPackageName(), e.toString());
}
}
if (br != null) {
br.close();
}
}
catch (IOException e) {
Log.e(getPackageName(), e.toString());
}
}
return curFrequency;
}
}
From here its a piece of cake, you simply call the method :-D
int core1CurrentFreq = getCurrentFrequency(1, this);
Sometimes the cores go offline, in which case the file path will not exist and -1 will be returned
NOTE. the returned value is in KHz
MHz value is core1CurrentFreq / 1e3
GHz value is core1CurrentFreq / 1e6
Some explainations on the getCurFrequencyFilePath() method since it is not all that clear.
Current frequency is usually stored in the file: scaling_cur_freq
The file path is:
"/sys/devices/system/cpu/cpu(XX)/cpufreq/scaling_cur_freq"
where (XX) is substituted for the cpu core number eg:
"/sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq"
The INSERTION_POINT variable is nothing more than the index of (XX), the point at which we want to place the number corresponding to the cpu core
I suggest you take a look at some of the other files in the cpufreq folder, you can use them to get other information like maximum and minimum frequency, list of availables frequencies etc.
Click this
Link
and scroll down to heading 3
In an Android application, I have a list of image URLs like this:
List<String> urls = new ArrayList<String>(100);
urls.add("http://www.example.org/1.jpg");
urls.add("http://www.example.org/2.png");
urls.add("http://www.example.org/3.jpg");
urls.add("http://www.example.org/4.jpg");
urls.add("http://www.example.org/5.png");
urls.add("http://www.example.org/6.png");
urls.add("http://www.example.org/7.png");
urls.add("http://www.example.org/8.jpg");
urls.add("http://www.example.org/9.jpg");
urls.add("http://www.example.org/10.gif");
...
urls.add("http://www.example.org/100.jpg");
Now I have to get the filesize and MIME type for all of these URLs, and this should be done as fast as possible, of course.
What I did is the following:
for (String url : urls) {
int fileSize;
try {
URLConnection urlConnection;
urlConnection = new URL(url).openConnection();
urlConnection.connect();
final String mimeType = urlConnection.getContentType();
final int fileSize = urlConnection.getContentLength();
// do something with those two pieces of information
}
catch (MalformedURLException e) {
continue;
}
catch (IOException e) {
// some special handling
}
}
But this is terribly slow. This is because it is using a single thread and requesting the URLs one by one, while a web browser would always access multiple files at a time, isn't it?
So how can I make it faster?
For HttpClient, I've read that you should re-use instances and there are some ways to use them in a multi-threaded environment.
But how would I do this with URLConnection or any other class that gives you filesize and MIME type?
Edit:
The images are not all on the same host, but spread across only a few servers, say 100 images spread across 5 host names.
Can you use a few threads or run several AsynTasks at once that do the job? Is there anything you have to pay attention to, such as recycling URLConnection objects or so?
I'm not quite sure how to use multiple threads to share task list (100 image files) and merge the results (MIME types and file sizes) afterwards. Can you help?
Split your work up into smaller peaces and let a worker Thread handle it:
The worker Thread:
public class Downloader extends Thread {
private final String mUrl;
private String mMimeType;
private int mFileSize;
public Downloader(String url) {
mUrl = url;
}
#Override
public void run() {
try {
URLConnection urlConnection;
urlConnection = new URL(mUrl).openConnection();
urlConnection.connect();
mMimeType = urlConnection.getContentType();
mFileSize = urlConnection.getContentLength();
} catch (IOException e) {
e.printStackTrace();
}
}
public String getMimeType() {
return mMimeType;
}
public int getFileSize() {
return mFileSize;
}
}
Instantiate, run and wait for the worker:
ArrayList<String> urls = new ArrayList<String>(10);
// ...
ArrayList<Thread> threads = new ArrayList<Thread>(10);
for (String url : urls) {
Thread t = new Downloader(url);
threads.add(t);
t.start();
}
for (Thread t : threads) {
try {
// do not wait for other threads in main UI thread!
t.join();
//((Downloader) t).getMimeType();
//((Downloader) t).getFileSize();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Make sure to note wait for the worker Thread in your UI Thread.
The answer above should only be used for a small set of URLs. A ThreadPool may not be necessary because the Threads will wait for IO most of the time.
Here is your requested answer with a ThreadPool.
It's using the same Downloader class as the above example with only one change:
Spawning Threads is done by the ThreadPool and the single tasks don't need to be a real Thread anymore. So let the Downloader implement a Runnable instead of extending a Thread:
public class Downloader implements Runnable {
Hopefully it's what you are looking for.
public class ThreadedDownloader {
private static final int KEEP_ALIVE_TIME = 1;
private static final TimeUnit KEEP_ALIVE_TIME_UNIT = TimeUnit.SECONDS;
private static int NUMBER_OF_CORES = Runtime.getRuntime().availableProcessors();
private LinkedBlockingQueue<Runnable> mDecodeWorkQueue = new LinkedBlockingQueue<Runnable>();
private ThreadPoolExecutor mDecodeThreadPool = new ThreadPoolExecutor(NUMBER_OF_CORES,
NUMBER_OF_CORES, KEEP_ALIVE_TIME, KEEP_ALIVE_TIME_UNIT, mDecodeWorkQueue) {
#Override
protected void afterExecute(Runnable r, Throwable t) {
super.afterExecute(r, t);
Downloader d = (Downloader) r;
// do something with finished Downloader d
// like saving it's result to some sort of list
// d.getMimeType();
// d.getFileSize();
if (mDecodeWorkQueue.isEmpty()) {
onAllDownloadsFinised();
}
}
};
/** Download a list of urls and check it's mime time and file size. */
public void download(List<String> urls) {
for (String url : urls) {
mDecodeThreadPool.execute(new Downloader(url));
}
}
/** Calles when all downloads have finished. */
private void onAllDownloadsFinised() {
// do whatever you want here
// update UI or something
}
}
I don't have example Code, but the HTTP verb HEAD is what you're looking for. It retrieves the headers (including mime and content-length) without transferring the content body.
This answer goes into more detail about what HEAD does.
I think you have most, if not all of the pieces to your solution in the answers already submitted.
Here's what I'd do:
1) make the head requests using Executors.newCachedThreadPool();
2) store the responses in a Map
3) create a payload Map
4) load the real images using AsyncTask instances
5) when each bitmap load completes, store the results in the payload map
Just my thoughts.
I implemented a JSON interface for getting model data over http in one of my android projects.
this works so far and I would like to write some tests. I created a test project as suggested in the android documentation. for testing the JSON interface I need some test data which I would like to put in a file.
my research showed up that it's best to put these files in the assets folder of the android test project. to access files in the assets folder one should extend the test class by InstrumentationTestCase. then it should be possible to access the files by calling getAssets().open() on a resources object. so I came up with the following code:
public class ModelTest extends InstrumentationTestCase {
public void testModel() throws Exception {
String fileName = "models.json";
Resources res = getInstrumentation().getContext().getResources();
InputStream in = res.getAssets().open(fileName);
...
}
}
unfortunately I'm getting an "no such file or directory (2)" error when trying to access "models.json" file. (/assets/models.json)
when getting a list of the available files by
String[] list = res.getAssets().list("");
"models.json" is listed in there.
I'm running these tests on Android 4.2.2 api level 17.
public static String readFileFromAssets(String fileName, Context c) {
try {
InputStream is = c.getAssets().open(fileName);
int size = is.available();
byte[] buffer = new byte[size];
is.read(buffer);
is.close();
String text = new String(buffer);
return text;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Then use the following code:
JSONObject json = new JSONObject(Util.readFileFromAssets("abc.txt", getApplicationContext()));
please use below code:
AssetManager assetManager = getResources().getAssets();
InputStream inputStream = null;
try {
inputStream = assetManager.open("foo.txt");
if ( inputStream != null)
Log.d(TAG, "It worked!");
} catch (IOException e) {
e.printStackTrace();
}