I am using appengine and objectify as a backend for my app. And when i query in the datastore i get a Entity object which has the data of required row. But, using objectify how will i query entities and get oly selected fileds from it? because querying the whole entity will be heavy and it needs more data bandwidth.
Eg : In a entity with 4 columns, --> Id,name,description,age. I should query oly Id,name,age. I dont want description to be queried.
The GAE datastore does not work like an RDBMS; you can't arbitrarily pick and choose which fields to query out of an entity. The standard behavior of a datastore query is to follow an index (which maps attribute value to entity key), then fetch-by-key all the entities found.
There is a feature called "projection queries" (which Objectify supports; look for the project() method on the query command object), however it is not a general purpose SELECT statement like you get in SQL. Projection queries capitalize on the fact that the index itself contains the index values, so if you only want data that's in the index, you don't need to perform a subsequent fetch of the whole Entity. However, this comes with some restrictions:
You must maintain a multiproperty index with all the data you wish to project.
You must maintain single-property indexes for each of the fields in the multiproperty index.
You can only project on queries that follow that particular index.
Queries bypass Objectify's memcache-based entity cache.
Be aware of the cost of using projection queries. In your example, you will need single-property indexes on Name and Age plus a multiproperty index on {__key__, Name, Age}. Instead of 3 write operations per entity written, your new entity will cost 8 write ops. On the other hand, the cost of a projection query is a constant 1 read op.
On the other other hand, the cost of a batch get from memcache is 0, and the worst-case cost is 1 read op. Unless your description field is known to be causing you problems, this is a massive premature optimization.
If you are looking for projection queries, they are not yet implemented in Objectify.
https://groups.google.com/forum/#!topic/objectify-appengine/uvLIHhHMEM0
Related
I would like to perform wildcard queries on a Firebase database,
and do not know whether the libraries support this. - (my guess it does not.)
I want an optimal solution where the network traffic will be as little as possible, assume the dataset is millions of records.
Dataset (using prefix wildcard):
user_id:
id_001123:
name: "java"
id_002124:
name: "objective-c"
id_003125:
name: "swift"
How would the code look like to retrieve the record id_002124 name field using a wildcard, you only have a portion of the id. eg. "%124".
The expected result would be id_002124, and the name being "objective-c"
Any mobile language example code would great.
(Objective-C/Swift/Java)
Firebase Database queries can order/filter data based on the object's keys (e.g. id_001123), their values (doesn't apply in your case), or the value of a child property (e.g. the fact that name = "java").
For each of these Firebase can match items that have a specific value you specify, items starting at a value you specify, or items ending at a value you specify.
So you can create a query matching the item named objective-c with:
ref.child("user_id").orderByChild("name").equalTo("objective-c")
Or you can create a query matching id_001124 and id_001125 with:
ref.child("user_id").orderByKey().startAt("id_001125")
Or simply getting all items starting with id_0011:
ref.child("user_id").orderByKey().startAt("id_0011")
Or you can create a query matching id_001123 and id_001124 with:
ref.child("user_id").orderByKey().endAt("id_001124")
Firebase Database cannot filter based on the end of a value. So there is no way in your current data structure to get all items whose key ends in a 4.
Please read more about Firebase Database queries here: https://firebase.google.com/docs/database/android/lists-of-data#sorting_and_filtering_data
And see some of the many previous questions about searching for FirebaseL
How to perform sql "LIKE" operation on firebase?
Firebase query - Find item with child that contains string
Firebase "like" search on string
I'm implementing a Content Provider, which is backed by a fairly complex SQLite DB schema. The database has a few junction tables and I'm unsure whether they should be visible to the user of the Content Provider or not.
The parent tables are exposed via the Contract, each one has its own content URI, etc. Now, when inserting data via ContentResolver#applyBatch() method, I create ContentProviderOperation per each table's content URI. So far everything is clear. But my problem is, how should the junction tables be populated, as they don't have their own content URIs?
To illustrate this, here's an example. I have 2 "parent" tables, Movies and Actors. The relationship between them is many-to-many and therefore I have a junction table called MoviesActors.
To insert at one batch I do the following:
List<ContentProviderOperation> operations = new ArrayList<>;
// movie
operations.add(ContentProviderOperation.newInsert(Contract.Movie.ContentUri).withValue("movie_id", "23asd2kwe0231123sa").build());
// actor
operations.add(ContentProviderOperation.newInsert(Contract.Actor.ContentUri).withValue("actor_id", "89asd02kjlwe081231a").build());
getContentResolver().applyBatch(authority, operations);
The junction table MoviesActors should be inserted with a row containing movie_id and actor_id. How do I take care of the junction table in this situation?
The only thing, which comes to my mind is extend the Contract to have content URI pointing to the junction tables and add another ContentProviderOperation, since otherwise, how do you communicate movie_id and actor_id to ContentProvider#applyBatch()?
I rather not expose the junction table to the user of the ContentProvider, but I might be wrong here... perhaps that's how it should be done on Android?
I've searched this topic for days already and haven't found an answer to that.
Any help would be greatly appreciated.
Bonus question:
Is it necessary to expose every single table via the Contract? For instance, when having child tables in one-to-many relationship. I'm specifically referring to Insert/Update/Delete since I know with Query I can simply do a join, but maybe I'm wrong also here.
Thanks a lot!
NOTE: I'm not interested in 3rd party library solutions.
I think you're tackling the problem from the wrong end. You're trying to design an interface to match your database structure, but the interface should come first.
In the first place, the interface should meet all the requirements of your ContentProvider client. If your ContentProvider client needs access to the junction table you'll have to expose it (in some way, see below), otherwise you don't have to. A good interface hides the actual implementation details, so the ContentProvider client doesn't need to care about whether the ContentProvider is backed by an SQLite database, by a bunch of in-memory maps or even a web-service.
Also, you should not think of a ContentProvider just as an interface to a database and the Contract as the database schema. A ContentProvider is much more versatile and powerful than that. The major difference is that ContentProviders are addressed by URIs whereas in SQL you just have table names. In contrast to a table name, a URI has a structure. URIs have a path that identifies the object (or directory of objects) that you want to operate on. Also you can add query parameters to a URI to modify the behavior of an operation. In this respect a ContentProvider can be designed much like a RESTful service.
See below for a concrete (but incomplete) example of a Contract of a simple movie database. This is basically how one would design a RESTful web-service, except for one thing: Just like in your code, movie-id and actor-id are provided by the caller. A real RESTful service would create and assign these automatically and return them to the caller. A ContentProvider can only return long IDs when inserting new objects.
Insert a new movie
insert on /movies/
Values: {"movie_id": <movie-id>, "title": <movie-title>, "year": ...}
Insert a new actor
insert on /actors/
Values: {"actor_id": <actor-id>, "name": <actor-name>, "gender": ...}
Add an existing actor to a movie
insert on /movies/movie-id/actors/
Values: {"actor_id": <actor-id>}
Add an existing movie to an actor:
insert on /actors/actor-id/movies/
Values: {"movie_id": <movie-id>}
Optional: add a new actor directly to a movie:
insert on /movies/movie-id/actors/
Values: {"actor_id": <actor-id>, "name": <actor-name>, "gender": ... }
If no actor with the given id exists, this operation will create the new actor and link it to the movie in a single step. If an actor with this ID already exists an exception would be thrown.
The same could be done the other way round, adding a new movie to an actor.
Delete an actor from a movie
delete on /movies/movie-id/actors/actor-id
or
delete on /actors/actors-id/movies/movie-id
Get all movies
query on /movies/
Get a specific movie
query on /movies/movie-id
Get all actors playing in a specific movie
query on /movies/movie-id/actors/
Get all movies a specific actor has played in
query on /actors/actor-id/movies/
The optional query selection statement can be used to filter the result. To get movies from the last 10 years a specific actor has played in, you would add the selection movies_year>=2005 to the last query.
By using a contract like this you wouldn't expose the junction table, instead you provide a REST-like interface to your database.
The job of the ContentProvider is to map these operations onto the database or any other back-end.
Editing my previous question, which boils down to this:
Given the following data structure (object composition):
is it bad design, or is it acceptable, that instead of two related tables, I store the sets of Reminders into a string column of Event table in some serialized way (e.g. JSON):
given that it would simplify my application significantly?
The reasons why I am considering this are:
I don't need the flexibility of accessing Reminders separately;
I always need to query all Reminders per Event and process them in code (e.g. choose the next three reminders based on time and weekday_flags — which is not trivial to do with a where clause);
In the end I need to display a list of Events with associated Reminders as stated above, so ideally I already have a dataset (cursor) with size matching the number of Events that I can feed to the adapter, and not the large join result that needs further processing and re-indexing;
Reminder entity is small and is not going to have more fields in the future, and there usually won't be more than 3-4 reminders per event, therefore storing them as a serialized string is not a size or migration concern.
As pointed out by Christophe Beyls here, the system's serialization/parceling mechanisms cannot be used to persist data because they might change eventually. So it's either JSON or some custom lightweight delimited format.
For my Android app, I want to save data using sqlite with this format:
name, date, attr1, attr2, attr3,...
These are the requirements:
each date can only contain each name once
there can be a variable number of attributes(numbers) for each name
each specific name has the same number of attributes
The app will be used to track events throughout the day. Events can have zero or more numeric properties.
The questions are: is sqlite the best way to store things here? If so how do I design my database? What other ways are there to store this kind of data?
is sqlite the best way to store things here?
This will depend on a number of other factors, such as how the data will be queried and used, the volume of transactions, data growth and retention, etc. From what you've described, though, SQLite is a great option, offering functionality out-of-the-box that supports some of your requirements directly, and is commonly used in such cases.
If you don't have much experience with relational databases, implementing this functionality may seem difficult at first, but like learning a new language or framework, it will get easier with time.
If so how do I design my database?
Let's step through each of your enumerated requirements...
each date can only contain each name once
SQLite supports the UNIQUE constraint. For example, if your columns were named name and date, you could add the following to your CREATE TABLE statement:
UNIQUE(name, date)
(A more complete CREATE TABLE statement is in the next example below, and it includes this constraint.)
This constraint prevents the insertion of rows with name/date pairs that already exist. Using android.database.sqlite.SQLiteDatabase, if you attempt to insert a row into the table with a duplicate name/date pair, a SQLiteConstraintException will be thrown at runtime. You will need to handle this exception in your java code.
there can be a variable number of attributes(numbers) for each name
This is a textbook case for normalizing the database, putting your data into multiple tables. For example:
CREATE TABLE names (
name_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
name TEXT NOT NULL,
date DATETIME,
UNIQUE(name, date));
CREATE TABLE attrs (
name_id INTEGER NOT NULL,
attr_value INTEGER NOT NULL,
FOREIGN KEY(attr_value) REFERENCES names(name_id));
Your queries that retrieve attribute data would then JOIN the two tables. Since you indicated that "Events can have zero or more numeric properties", a LEFT OUTER JOIN might be most appropriate, as it will return names and dates even if there are no attributes.
Here's an example query, by name:
SELECT n.name, n.date, a.attr_value
FROM names AS n
LEFT OUTER JOIN attrs AS a
ON n.name_id = a.name_id
WHERE n.name = 'SMITH'
ORDER BY n.name, n.date, a.attr_value;
This query would return results like the following:
name date attr_value
--------------- ---------- ------------
SMITH 2015-02-13 1027
SMITH 2015-02-13 4426
SMITH 2015-02-13 8390
SMITH 2015-02-20 4426
SMITH 2015-02-20 8152
SMITH 2015-02-20 9328
You can then iterate through and process these results in java. If your results include multiple names and/or dates, then in your loop you should keep track of the last used name and date. If the name/date in the current record is the same, the attribute belongs to the current one. If the name/date is different, then this is a new one.
Note that this approach to your database design is flexible, allowing you to query on the attributes, for instance, to see what name/date pairs are associated.
Also note that there is a FOREIGN KEY constraint on the attrs table, meaning that if you attempt to insert a record into that table with a name_id that does not exist in the names table, a SQLiteConstraintException will be thrown at runtime. You will need to handle this exception in your java code.
each specific name has the same number of attributes
You will need to accommodate this requirement in your java code, probably doing some checks in the database prior to performing an INSERT.
What other ways are there to store this kind of data?
Flat files, JSON, XML, third-party data stores (with their own libraries), to name a few.
I'm not sure but I think the best way to achieve your requirement is to use sqlite and to solve your problem you can have 3 columns only. One for the name and one for the date and the other contains a JSON array that represents the rest of the attributes.
I have a database full of values and one of the columns in an origin (where the particular item comes from). I am trying to populate a spinner with all the origins listed in the database, but obviously I only want them repeated once. I am trying to use the sql statement:
String sql = "SELECT _id, origin FROM items GROUP BY origins";
But I am also using a custom SimpleCursorAdapter to populate the spinner and because of that I also need to query the _id column. When I add the _id column to the query it produces a query with all the repeated origins because the id makes it to where no row is a duplicate. What do I need to do to pass both columns, but remove the duplicates? Being able to organize them alphabetically would also be great!
You get a view with duplicate origin values, since every origin value can "belong" to several ids.
I don't think that the following is a "clean" solution but it gives you no duplicate values along with the highest id corresponding to each origin value:
Select origin, max(_id) from items group by origin
You might also consider normalizing your database. Rather than repeating the origins multiple times in the table, you could create a separate origins reference table. Then, in your original table, you could reference the origin by it's primary key. For example:
origins
***********************
id name
-----------------------
1 origin_1
2 origin_2
3 origin_3
***********************
detail_data
**************************
id origin_id data
1 1 ...
2 1 ...
3 3 ...
4 3 ...
**************************
Then if you only wanted origins for which data existed, you could do something like:
SELECT DISTINCT o.*
FROM origins AS o
JOIN detail_data AS dd ON (dd.origin_id = o.id)
ORDER BY o.name ASC;
I generally prefer a normalized database because normalization can offer advantages such as a reduced footprint size, cleaner data, and a more flexible structure. I will admit, however, that normalization seems less beneficial with Android compared to traditional database systems (i.e., desktops, servers). And normalization has disadvantages, too, such as more complex queries (as we can see above). For maximizing flexibility, maintainability and robustness for the long-term, though, normalization may be the better approach.
To add to Preli's answer, if you want them in alphabetical order, just change it to this:
SELECT origin, MAX(_id) FROM items GROUP BY origin ORDER BY origin