I am now trying to import csv files from a certain directory in sd card from an android device. Recently, I can successfully import a single csv files. However, I have no ideas on how to get the list of all csv files and then using a loop to import the csv file one by one.
This is the my code for importing single csv:
button_import_csv.setOnClickListener(new View.OnClickListener(){
public void onClick(View v){
DatabaseHelper helper = new DatabaseHelper(getApplicationContext());
SQLiteDatabase db = helper.getWritableDatabase();
try{
FileReader file = new FileReader("/sdcard/downloadedfolder/A1/adv_sales_order.csv");
BufferedReader buffer = new BufferedReader(file);
ContentValues contentValues=new ContentValues();
String line = "";
String tableName ="adv_sales_order";
db.beginTransaction();
while ((line = buffer.readLine()) != null) {
String[] str = line.split("\t");
contentValues.put("order_date", str[0]);
contentValues.put("cust_code", str[1]);
contentValues.put("customer_ref_no", str[2]);
contentValues.put("line_no", str[3]);
contentValues.put("item_code", str[4]);
contentValues.put("tran_code", str[5]);
contentValues.put("order_qty", str[6]);
db.insert(tableName, null, contentValues);
}
db.setTransactionSuccessful();
db.endTransaction();
}catch (IOException e){
}
}
});
The columns for different csv fileS are not the same.(For example,some may has 4 columns named A,B,C,D and the other one may has columns named as C,D,E,F) Besides hard coding all columns for each csv file, are there any possible ways?
Can anyone tell me any solution???Thank you.
There are two possibilities I can think of...
First: If you are in control of the filenames then give them names with a sequential numeric aspect, e.g., file1.csv, file2.csv etc You can then simply use a for loop to build the filenames and process them. Example...
// Lets say you have 5 files named file1.csv thru file5.csv
for(int i = 1; i < 6; i++) {
String filename = "file" + i + ".csv";
// Process the file which has the above filename
}
Second: Get all of the files in the directory using the listFiles() method. Example...
// This code assumes you have a File object for the directory called dir
File[] files = dir.listFiles();
for(int i = 0; i < files.length; i++) {
String filename = files[i].getAbsolutePath();
if (filename.endsWith(".csv")) {
// Process the file which has the above filename
}
}
I'm not sure if either of the code blocks above are perfect but basically they both simply use a for loop. There are other ways but those are the most straight-forward.
EDIT:
Some csv files use the first line to describe the column names. In some ways this is a bit like a schema of a dataset. Example (using comma-separated values)...
A,B,C,D
valueA,valueB,valueC,valueD
...
Using this approach means you can get access to the column names by reading the first line and splitting it to make an array. You can then use a for loop to put the ContentValues. Try the following...
// Read the first line separately and split to get the column names
line = buffer.readLine();
String[] cols = line.split("\t");
db.beginTransaction();
while ((line = buffer.readLine()) != null) {
String[] str = line.split("\t");
for (int i = 0; i < cols.length; i++) {
contentValues.put(cols[i], str[i]);
}
db.insert(tableName, null, contentValues);
}
db.setTransactionSuccessful();
db.endTransaction();
BTW I notice you're splitting on "\t" so make sure your column names on the first line are tab-delimited (obviously).
Related
How to read filenames from the download directory on android 10 and higher?
After filtering the names I want to let users select from this list to open
the file.
On Android versions 8 (API 28) the FILE api is quite simple to use.
Filenames are simple to read with the method DirListOld. With these names
I can read the content of the files.
I tried to make a method to do the same on Android 10 (API 30) and higher.
But documentation is not very clear. I did some experimenting with
MediaStore methods, but I could not get the filenames only got directory
names on external storage.
How to filter the results is not very well documented and examples of the
MediaStore.Downlaods are totally absent.
My experiment is shown in method DirListNew.
Also I had to ask for a permission for MANAGE_EXTERNAL_STORAGE. Without this
permission even DirListNew results in an empty string. As I read in several
comments Google-Play is not generous in giving this permission. Why not
special permission for only downloaded files. I don't have to read all external
files.
I don't understand why Google-Android developers made such a mess for retrieving
simple downloaded files.
public String DirListOld()
{
String sName;
File oDownloadDir;
String sDownloadDir;
StringBuilder dirContent = new StringBuilder();
oDownloadDir = this.getApplicationContext().getExternalFilesDir(Environment.DIRECTORY_DOWNLOADS);
try {
sDownloadDir = oDownloadDir.getName();
if (!sDownloadDir.equals("") )
{
for (File f : Objects.requireNonNull(oDownloadDir.listFiles()))
{
if (f.isFile())
{
sName = f.getName();
dirContent.append(sName);
dirContent.append("\n");
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
return dirContent.toString();
} // DirListOld
#RequiresApi(api = Build.VERSION_CODES.Q)
public String DirListNew()
{
EditText editText = findViewById(R.id.editTextMultiLine2);
StringBuilder dirContent = new StringBuilder();
String[] projection = new String[] {
MediaStore.Downloads.DATA
};
String selection = null;
String[] selectionArgs = null;
String sortOrder = null;
Cursor cursor = getApplicationContext().getContentResolver().query(
MediaStore.Files.getContentUri("external"),
projection,
selection,
selectionArgs,
sortOrder
);
if (cursor != null) {
cursor.moveToFirst();
//iterate over rows
for (int i = 0; i < cursor.getCount(); i++) {
//iterate over the columns
for(int j = 0; j < cursor.getColumnNames().length; j++){
//append the column value to the string builder and delimit by \n
dirContent.append(cursor.getString(j));
dirContent.append("\n");
}
//add a new line carriage return
dirContent.append("\n");
//move to the next row
cursor.moveToNext();
}
//close the cursor
cursor.close();
}
return dirContent.toString();
} // DirListNew
I have a method which reads data from file line by line and takes value between coma, then puts this value into INSERT query. Data in file saved in this way:
–,08:10,–,20:20,08:15,08:16,20:26,20:27,08:20,08:21,20:31,20:32,08:30,08:31,20:40,20:41,08:37,08:38,20:46
20:47,08:48,08:50,20:56,20:57,09:00,09:01,21:07,21:08
08:53,–,17:43,09:01,09:03,09:13,09:15,18:02,18:04,–,–,09:19,09:25
Here is actual my code:
public void insertTime(SQLiteDatabase database, String table) throws FileNotFoundException {
BufferedReader br = null;
String line;
try {
int j = 0;
br = new BufferedReader(new InputStreamReader(context.getAssets().open("time.txt")));
database.beginTransaction();
while ((line = br.readLine()) != null) {
j++;
String query = "INSERT INTO "+table+""+j+" (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
// use comma as separator
String[] time = line.split(",");
for(int i = 1; i < time.length; i+=2) {
statement.bindString(1,time[i-1]);//arrival
statement.bindString(2,time[i]);//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The problem is that data insert very slow, despite I use SQLiteStatement and transactions. For example, when I insert 69000 rows it takes about 65,929 seconds.
What have I to change in my code to improve speed of insertion ?
UPDATE
OK, I have simplified my code, I got rid of BufferedReader and now it looks like this
public void insertTime(SQLiteDatabase database) throws FileNotFoundException {
database.beginTransaction();
int r = 0;
while (r < 122) {
r++;
String query = "INSERT INTO table_1 (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
for(int i = 1; i < 1100; i++) {
statement.bindString(1,i+"");//arrival
statement.bindString(2,i+"");//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
}
But it still so long inserts data, more than 2 min. Do you have any ideas how to increase speed of my second example ?
Here is a very very detailed post on every method of increasing SQL insertion speed.
Move beginTransaction() and setTransactionSuccessful() outside of while loop and it will be way faster.
A new transaction is started for each item in the while() loop.
It might go a bit faster if you only have 1 transaction to do all your insertions.
Also, when your data is corrupt and String.split doesn't give you at least 2 items, then your transaction will not be ended properly due to an Exception being thrown.
Every time you insert a row in a table with indexes, the indexes have to be adjusted. That operation can be costly. Indexes are kept as b-trees and if you hit the rebalance point, you're bound to have a slowdown. One thing you can do to test this is to remove your indexes. You could also drop the indexes, insert, then re-create the indexes.
For those using JDBC (Java): to be sure, do you first set the autoCommit to FALSE?
I guess so, because you work with explicit transactions.
The performace gain I got by explicitly setting the autocommit off was over 1000 times!
So:
Class.forName("org.sqlite.JDBC");
String urlInput = "jdbc:sqlite:" + databaseFile;
databaseConnection = DriverManager.getConnection(urlInput);
databaseConnection.setAutoCommit( false);
And:
String sql = "INSERT INTO " + TABLE_NAME + " ( type, bi, ci, fvi, tvi, content_type) VALUES ('V',?,?,?,?,'rtf')";
PreparedStatement psi = databaseConnection.prepareStatement(sql);
for( Item item : items) {
psi.setInt(1, item.property1);
// ....
count = psi.executeUpdate();
}
databaseConnection.commit();
databaseConnection.setAutoCommit( true);
So, when somebody forgets this, this may have a huge effect.
i'm creating a csv file with the data from the database. All the data of the selected consumer Id's must append into the csv file. In my program the csv file was attached on email, let's say I have 3 selected consumer ID, the email sent, but when I checked the csv file, the first selected ID was did not show up and it leaves the first row blank, while the 2nd and 3rd append on the csv. I don't what happen why I cannot see the data of the first ID i selected. help me please..
here's my code:
private String getConsumersDetails(ArrayList<String> arraylistConsumerId)
{
for (int j = 0; j < arraylistConsumerId.size(); j++)
{
//Cursor cursorConsumerDetails = null;
Cursor cursorConsumerDetails = databaseAdapter.getCursorRegisteredConsumer(Integer.parseInt(arraylistConsumerId.get(j)));
//if (cursorConsumerDetails.getCount() == 0)
//cursorConsumerDetails = databaseAdapter.getCursorRegisteredConsumer(arraylistConsumerId.get(j));
// consumerData.append("\n"+lastname+","+firstname+","+middleinitial+","+cellphone+","+emailadd+","+carefriend+","+company+","+regdate);
if (cursorConsumerDetails.getCount() > 0)
{
lastname = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_LASTNAME));
firstname = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_FIRSTNAME));
middleinitial = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_MIDDLEINITIAL));
cellphone = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_CELLPHONENO));
emailadd = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_EMAIL));
carefriend = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_CAREFRIEND));
company = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_COMPANY));
regdate = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_REGISTRATIONDATE));
code = cursorConsumerDetails.getString(cursorConsumerDetails.getColumnIndex(Constants.CONSUMER_COMPANYCODE));
}
consumerData.append("\n"+lastname+","+firstname+","+middleinitial+","+cellphone+","+emailadd+","+carefriend+","+company+","+regdate);
}
return consumerData.toString();
}
This one calls the code above..
....
consumerData = new StringBuilder();
File personFile;
time = String.valueOf(calendar.get(Calendar.HOUR_OF_DAY))+String.valueOf(calendar.get(Calendar.MINUTE))+String.valueOf(calendar.get(Calendar.SECOND));
personFile = createCSV("Person_Table_"+date+"_"+time, Constants.COLUMN_STRING_PERSON_TABLE, getConsumersDetails(arraylistConsumerId), fileCounter, false);
createEmailWithAttachments(personFile);
....
I don't know how its done in android but in general to
obtain a csv file from sqlite3 database.
.mode csv
.output filename.csv
select * from table;
csv file will be present directory where the sqlite database file is stored.
I have this piece of code that allows me to read the contents of the gallery of the phone and scroll randomly its contents.
public static Uri getRandomImage(ContentResolver resolver) {
String[] projection = new String[] {
BaseColumns._ID
};
Random rand = new Random();
int p = 2 + rand.nextInt(8-2+1);
Uri uri = p == 0 ? Media.EXTERNAL_CONTENT_URI : Media.EXTERNAL_CONTENT_URI;
Cursor cursor = Media.query(resolver, uri, projection, null, MediaColumns._ID);
if (cursor == null || cursor.getCount() <= 0) {
return null;
}
cursor.moveToPosition(new Random().nextInt(cursor.getCount()));
return Uri.withAppendedPath(uri, cursor.getString(0));
}
But I would like to modify this code to read the contents of a folder set by me. How can I do this?
thank you very much
Probably one of the easiest way to do: name each file in your folder numerically i.e. (supposing they are all jpg) name them 1.jpg, 2.jpg, 3.jpg and so on. Now all you need to do is generate an array of distinct random numbers between 1 and number of files in your folder and generate a uri by concatenating this random numbers with .jpg and then display them.
Maybe you should set the URI to a different value?
Random rand = new Random();
int p = 2 + rand.nextInt(8-2+1);
Uri uri = p == 0 ? Media.EXTERNAL_CONTENT_URI : Media.EXTERNAL_CONTENT_URI;
This code always sets the URI to Media.EXTERNAL_CONTENT_URI ... no matter what the value of the integer p is...
Personally I actually can't even see what you are trying to do there...
Getting random files out or selected directory is quite easy.
You can use this snippet:
private File getRandomFile(File root) {
File[] files = root.listFiles();
Random randomizer = new Random();
return files[randomizer.nextInt(files.length - 1)];
}
If you need the method to return Uri, you can change the return and method signature:
private Uri getRandomFile(File root) {
File[] files = root.listFiles();
Random randomizer = new Random();
File f = files[randomizer.nextInt(files.length - 1)];
return Uri.fromFile(f);
}
As for cursor requirement, you should be able to wrap this method to another method and bridge request accordingly.
I currently have a CSV file that I parse and am trying to insert the data into the android database. The problem I am having is that it is taking way too long to insert all of the data. It's a good amount of data but I feel like it shouldn't take 20min or so to complete.
Basically, I create my database, then begin the parsing. While parsing through each individual CSV row, I grab the required data and insert it into the database. In total there are around 40000 rows.
Is there any way I can speed up this process? I have tried batch inserts but it never really helped (unless I did it wrong).
Code down below.
Thanks.
DatabaseHelper (i have two insert commands based on the amount of data in each csv row):
// add zipcode
public void add9Zipcode(String zip, String city, String state, String lat,
String longi, String decom) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
db.close();
}
public void add12Zipcode(String zip, String city, String state, String lat,
String longi, String decom, String tax, String pop, String wages) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
values.put(KEY_TAX, tax);
values.put(KEY_POP, pop);
values.put(KEY_WAGES, wages);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally{
db.endTransaction();
}
db.close();
}
Parse File:
public void parse(ArrayList<String> theArray, DatabaseHandler db) {
String[] data = null;
// while loop to get split the data into new lines
// for loop to split each string in the array list of zipcodes
for (int x = 0; x < theArray.size(); x++) {
if(x == 10000 || x == 20000 || x == 30000 || x == 40000){
Log.d(TAG, "x is 10k, 20k, 30k, 40k");
}
// split string first into an array
data = theArray.get(x).split(",");
// separate based on the size of the array: 9 or 12
if (data.length == 9) {
db.add9Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8]);
} else if (data.length == 12) {
db.add12Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8], data[9], data[10], data[11]);
/*
* theZip.zip = data[0]; theZip.city = data[2]; theZip.state =
* data[3]; theZip.lat = data[5]; theZip.longi = data[6];
* theZip.decom = data[8]; theZip. = data[9]; theZip.population
* = data[10]; theZip.wages = data[11];
*/
}
}
Refer to this answer I made previously: Inserting 1000000 rows in sqlite3 database
In short, use an InsertHelper and do more than one insert per transaction - unless you did something wonky, the speed increase should be noticeable.
Edit:
In short:
Your SQLiteOpenHelper should be a singleton used across your entire application.
Don't go around calling close() on your SQLiteDatabase instance - it's cached in the SQLiteOpenHelper and every time you close you force the helper to reopen it.
Batch your inserts, start a transaction outside the call to the addZipCode methods and mark it as successful after you've done all the inserts - then commit the transaction.
Use an InsertHelper - it will format the insert properly as a prepared statement and is nice and reusable.
Be mindful of synchronizing access to the database - unless you intend to do all your database work on the UI-thread (which is not recommended) - you either need to enable locking or guard access to the database to avoid concurrent access.