To sync a mysql database with sqlite database using a PHP web app, I am going to
create a "view" mirror of SQLITE tables in MySQL
select changed data on mirror "view" in MySQL
use mysqldump to dump the sql for sqlite and save it in a file on server
download that file from mobile
import data directly to sqlite
another approach would be not using a middle "view" table and dump data using "SELECT" or mysqldump and parse it on the device or server, so it would be readable by sqlite
is there any disadvantages in creating a mirror "view" for each table that SQLITE needs on client devices? or is there another 'standard 'way to sync these 2 which i did not find on the internet?
Edit: after searching and coding some more finally i decided to go like this:
build "insert into ..." statement for sqlite from PHP on the server and save it to a zipped file
download the file on mobile and insert extracted sql directly to SQLITE
here is the php code for generating mysqldump like output:
/** key [SQLITE column name] => value [MySQL select column] */
$table_structure = array(
'id' => 'id',
'title' => 'title',
'type' => 'COALESCE(type, "")', /** u need to wrap any column that is nullable like this **/
'flags' => 'COALESCE(type, "")', /** u need to wrap any column that is nullable like this **/
'view_field' => 'view_field',
'view_year' => 'view_year',
....
....
'updated_at' => 'UNIX_TIMESTAMP(updated_at)');
$into_query = '';
$into_values = '';
$indexer = 0;
foreach ($table_structure as $sqlite => $mysql) {
$into_query .= (($indexer !== 0)? ',':'').$sqlite;
$into_values .= (($indexer !== 0)? ',",",':'').'"\'",'.$mysql.',"\'"';
$indexer++;
}
$output_filepath = "some where nice";
$query = 'SELECT CONCAT("INSERT INTO '.$table.'('.$into_query.') VALUES(",'.$into_values.',");") FROM '.$table.' ' ... "INTO OUTFILE $output_filepath CHARACTER SET utf8";
I hope it's a good approach and may help someone some day :)
Approach 1: using sqlite db file
whatever step you have mentioned in question is correct. this approach used when you have to send lots of data(many tables) from device to web/php vice versa.
its takes time in slower internet connection as well in 2G. in 3G its fast(its also depends on whatever code you have written at mobile/PHP to upload & download file).
make sure database file size is small if it large then make zip file which include database file.
Approach 2: make using json (SELECT *) parse in device
this one is very long process when you have lost of table which have lots of fields. you have to get data from table one by one, send it to server/php its send back data after process. then you have to again insert data in device table.
this is also depends on INTERNET as well you code. its too slow for devices and take more time.
common things in both approach send/receive data only when its needed or on those which require for devices/PHP. do not send unused data to work fast.
Related
I am using SQLite in my RN apps with help of react-native-sqlite-storage. Then, one of my table has a column which contain a very long string. When I try to insert the string, no error occured. But, when I select the row, it return null. This open issue is similar with my problem.
Then, I found this documentation. It said that
During part of SQLite's INSERT and SELECT processing, the complete
content of each row in the database is encoded as a single BLOB. So
the SQLITE_MAX_LENGTH parameter also determines the maximum number of
bytes in a row.
The maximum string or BLOB length can be lowered at run-time using the
sqlite3_limit(db,SQLITE_LIMIT_LENGTH,size) interface.
My question is, if I have this piece of working code to select data from a table:
const db = await SQLite.openDatabase('databasename', 'default');
db.executeSql('SELECT * FROM tablename', [], (res) => {
console.log(res.rows.item(0));
}, err => console.log(err));
How to implement sqlite3_limit(db, SQLITE_LIMIT_LENGTH, size) to above code in order to increase the SQLITE_LIMIT_LENGTH?
You can't easily. Android's default sqlite implementation doesn't provide access to it. You would need to import your own version of the sqlite Java library (which is really a C library accessed via JNI), then provide your own RN module to access it in Java, then use only that module to access the db at all. Its a lot of work. How large is this string? You may want to consider storing it in a file and putting the filename in the db.
The issue you linked particularly mentioned images. Sqlite is not really a good solution for storing images, for a variety of reasons. Use a file and store the filename for that.
Aim -
I am trying to store longitude and latitude data from the geolocation API using React native. I created a .db file put it into the /assets/ folder in android. First, instead of the location data, I am trying to insert or retrieve any kind of data.
ERROR -
It shows that there is no table named Test when I try to console.log() it.
Problem -
Whenever I try to insert data into the database I don't see any new data added and also When I try to read an already populated database I see nothing. Where am I wrong? I am fairly new to this.
What is the response returned by the SQL transaction. I console.log it but couldn't understand.
Where should be the transaction function be located?
I am using DBbrowser for SQLite
OS - MacOS
CODE -
I am using this function to send or receive data.
sendData = () => {
// var SQLite = require('react-native-sqlite-storage')
var db = SQLite.openDatabase({name: 'a', createFromLocation : "~test.sqlite"}, this.openCB, this.errorCB);
db.transaction(txn => {
txn.executeSql(
// "INSERT INTO Test (latitude,longitude) VALUES(?,?)",
'SELECT * FROM Test',
[], //Argument to pass for the prepared statement
(res) => {
// let row = res.rows.(1);
console.log(res);
} //Callback function to handle the result response
);
});
};
// I mapped this button to send data.
<TouchableOpacity onPress={this.sendData} style={styles.button}>
<Text> SEND DATA</Text
</TouchableOpacity>
You won't be able to modify a database stored in the assets folder. The typical method is before ever trying to open the database to insert records, you first check to see if the database exists in your apps data folder (not sure if React has the same folder structures or not). If the database doesn't exist in the data folder, then you copy your clean database from assets to the data folder, then open the database from the data folder. As long as the app isn't deleted off the device, the database in the data folder will persist, along with all of the data you have inserted into it. Once the app has been deleted and reinstalled, the database will have to be copied from assets once again and will be clean, with no new data in it.
I don't know if the same data folder structure applies to React, but in native Android I use:
"/data/data/com.mycompany.myapp/databases/"
OR
DB_PATH = myContext.getFilesDir().getParentFile().getPath() + "/databases/";
I would like to know which apache/httpd parameters a relevant to set, if I expect a high number of connections from different IPs, getting same ammount of data in the same time. As example: I would have 10000 mobile devices that try to get data from server in the same time. The dataset is about 100k big and is generated by a php script. Each call make a query to the database. How apache(and mysql) should be configured to stay alive during this requests?
<?php
$con = mysql_connect( 'localhost', 'root', 'pass') or die('Could not connect' );
mysql_select_db('mydb', $con) or die('Could not select database.');
$result = mysql_query("SELECT * FROM results");
mysql_close($con);
$items = array();
while($row = mysql_fetch_array($result)){
$items[] = array(
's' => $row['sport'],
....
);
echo json_encode($items);
}
Regards!
Dimitri.
First of all use mysqli API and not mysql (http://it2.php.net/mysqli).
Then you MUST use the bind param technique (http://it2.php.net/manual/en/mysqli-stmt.bind-param.php) beacuse the generated query have so much better performance because mysql deosn't have to rebuild the eexecution plan of the query but only reexecute it.
Server side pay attention to doesn't slow MyISAM driver but InnoDB. If you can manage the table structure adopt well known normalization practice : http://en.wikipedia.org/wiki/Database_normalization.
Also do the code inside your while in async way so u can "parallelize" the execution of the code.
I have a question for you guys.
I have been working on a project application that in one part uses an SQLite database loaded from a txt file (it has about 100k-200k rows of 5 strings separated by the ^ sign).
Now my question is, since this is my first time working with databases, how does .txt import for modifiable databases work? If I understand right, it pulls all data from the txt file once and creates a database that it keeps to work on, so when I modify the database I modify the newly created one and not the txt? Does the code try to pull info again from the txt whenever the app loads, and would loading 200k 10char words every time be too much? :)
The database consists of music bands in this format: name/genre/popular[yes/no]/selected
The selected column is the only one being modified by the user (and the app for that matter). If I use the regular approach to databases with added implementation from a txt file will the selected column reset every time (do not want that)?
Don't distribute your app with a huge txt-file and import it on the users device. This takes time and is annoying.
Rather distribute your app with a pre-populated database and copy it over from the res-folder. You can use android-sqlite-asset-helper to automate this.
Also, yes. The Database is always stored on the internal memory and you can't access it on a non-rooted device (unless you're using the AVD).
To import your txt-contents into a database, create a script or something that parses the contents and executes the corresponding SQL-queries. Again, your App should ship with the database, not the raw-file!
I was a little bored and hacked together a short Python-Script to read all entries from your txt-file and insert them into a SQLite Database:
import sqlite3
import re
counter = 0;
pattern = re.compile('^([^\^]+)\^([\w\s]+)\^(yes|no)\^\w+$');
conn = sqlite3.connect("imported.db");
cursor = conn.cursor();
# Create the Table:
conn.execute('''
CREATE TABLE Bands (
name TEXT,
genre TEXT,
popular INTEGER,
selected INTEGER
);''');
# Now, insert:
with open('bands.txt', 'r') as f:
for line in f:
match = pattern.search(line);
if match:
cursor.execute('''
INSERT INTO Bands (name, genre, popular, selected)
VALUES (?,?,?,0)''',
(
match.group(1), match.group(2),
(1 if match.group(3) == 'yes' else 0)
)
);
counter+=1;
conn.commit();
conn.close();
print "Imported ", counter, " bands!";
This will assume that the txt-file is named bands.txt, each value is separated by a / and each entry will be on it's own line. The resulting database-file is imported.db.
Also, I use INTEGER for all True|False-fields (popular, selected). These will then hold a 0 for false and a 1 for true.
Last but not least, the RegEx only allows "yes" and "no" for the popular-value.
I'm developing with SDK 1.6.2
I execute a query using the following SQL
var dbrows = db.execute('select item_logo , item_id,item_title,item_content from item');
I successfully get the values for item_id , item_title & item_content. However Ti.API.Info(getFieldByName('item_logo') returns a null value. I have the blob data in SQLLite ITEM table and it is a valid jpeg picture.
Is there any specific processing to be done at Titanium End (Code) to read the Blob data in a loop? Can't find any specific info # Titanium API docs.
Thanks
Vishnoo
you can do this, but the recommended approach is to store the blob as a file and keep the native file path in the database.
See this response from Appcelerator
http://developer.appcelerator.com/question/97041/imageview-sqlite-blob-example-not-working