Develop Reference

JSoup web-scraping: How to select each value

I'm currently trying to get some values from a website with Jsoup for my android application.
Basically, I wanted to extract these values:
so that in the code, the model name should go to gpuModel variable, prices goes to gpuPrice and so on for every single Graphic card. and then I would store them in firebase using this code:
public void addToDatabase(String model, int price, int rating, double value,
double bench) {
Map<String, Object> docData = new HashMap<>();
docData.put("model", model);
docData.put("rating", rating);
docData.put("bench", bench);
docData.put("value", value);
docData.put("price", price);
db.collection("gpu").document(model).set(docData);
}
This is the HTML code for each GPU:
i am currently trying to extract only the gpu model name to keep it simple (in the end i want to extract all the other values too), this is my current web scraping code:
public void webScrape(){
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
Document doc = Jsoup.connect("https://www.videocardbenchmark.net/GPU_mega_page.html").get();
Elements gpus = doc.select("[id^=gpu]");
for (Element i : gpus) {
Elements gpuModel = i.select("tr td:nth-child(2)");
String gpuName = gpuModel.text().replace("/", "");
addToDatabase(gpuName, 12,12,12,12);
}
} catch (IOException e){
e.printStackTrace();
}
runOnUiThread(new Runnable() {
#Override
public void run() {
}
});
}
}).start();
going by my logic, this should select all the GPU, and then in the for loop, it would extract only the card's name. set the gpuName to whatever was extracted and push it to the database.
But instead, I got this in my database:
It extracted the GPU name, but also all the other unwanted fields (including the prices, value, test date, etc).
so my question is, how do i select each value separately? all the examples i found on the internet was done with a very simple website where each value has their own ID so I can't really learn from them.
If there's anything i should provide, please do tell me.
simple code representation of what i want:
Document doc = Jsoup.connect(the url).get();
Elements gpus = doc.select(all gpu);
for (each gpu in all gpus) {
gpuName = gpu.select(name);
gpuPrice = gpu.select(price);
gpuValue = gpu.select(value);
gpuPower = gpu.select(power);
addToDatabase(gpuName, gpuPrice, gpuPower, gpuValue);
}

You should use a:nth-child(2) as the model name selector inside your run method:
Elements gpuModel = i.select("a:nth-child(2)");

Android - XML or SQLite for static data

I am making Android app for practicing driving licence theory tests. I will have about 3000 questions. Question object would have several atributes (text, category, subcategory, answers, group). I will create them and put in app, so data won't ever change. When user chooses category, app would go througt data, look which question meets requirements (that user selected) and put it in list for displaying. What should I use to store data/questions, XML or SQLite? Thanks in advance.
Edit:
I forgot to mentiont that app won't use internet connection. Also, I planned to make simple java app for entering data. I would copy text from government's website (I don't have access to their database and I have to create mine), so I thought to just put question's image url to java program and it would download it and name it automaticaly. Also, when entering new question's text it would tell me if that question already exist before I enter other data. That would save me time, I wouldn't have to save every picture and name it my self. That is what I thought if using XML. Can I do this for JSON or SQLite?

If you do not have to perform complex queries, I would recommend to store your datas in json since very well integrated in android apps using a lib such as GSON or Jackson.
If you don't want to rebuild your app / redeploy on every question changes. You can imagine to have a small webserver (apache, nginx, tomcat) that serves the json file that you will request on loading of the app. So that you will download the questions when your app is online or use the cached one.
XML is a verbose format for such an usage, and does not bring much functions....
To respond to your last question, you can organise your code like that :
/**
* SOF POST http://stackoverflow.com/posts/37078005
* #author Jean-Emmanuel
* #company RIZZE
*/
public class SOF_37078005 {
#Test
public void test() {
QuestionsBean questions = new QuestionsBean();
//fill you questions
QuestionBean b=buildQuestionExemple();
questions.add(b); // success
questions.add(b); //skipped
System.out.println(questions.toJson()); //toJson
}
private QuestionBean buildQuestionExemple() {
QuestionBean b= new QuestionBean();
b.title="What is the size of your boat?";
b.pictures.add("/res/images/boatSize.jpg");
b.order= 1;
return b;
}
public class QuestionsBean{
private List<QuestionBean> list = new ArrayList<QuestionBean>();
public QuestionsBean add(QuestionBean b ){
if(b!=null && b.title!=null){
for(QuestionBean i : list){
if(i.title.compareToIgnoreCase(b.title)==0){
System.out.println("Question "+b.title+" already exists - skipped & not added");
return this;
}
}
System.out.println("Question "+b.title+" added");
list.add(b);
}
else{
System.out.println("Question was null / not added");
}
return this;
}
public String toJson() {
ObjectMapper m = new ObjectMapper();
m.configure(Feature.ALLOW_SINGLE_QUOTES, true);
String j = null;
try {
j= m.writeValueAsString(list);
} catch (JsonProcessingException e) {
e.printStackTrace();
System.out.println("JSON Format error:"+ e.getMessage());
}
return j;
}
}
public class QuestionBean{
private int order;
private String title;
private List<String> pictures= new ArrayList<String>(); //path to picture
private List<String> responseChoice = new ArrayList<String>(); //list of possible choices
public int getOrder() {
return order;
}
public void setOrder(int order) {
this.order = order;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public List<String> getPictures() {
return pictures;
}
public void setPictures(List<String> pictures) {
this.pictures = pictures;
}
public List<String> getResponseChoice() {
return responseChoice;
}
public void setResponseChoice(List<String> responseChoice) {
this.responseChoice = responseChoice;
}
}
}
CONSOLE OUTPUT
Question What is the size of your boat? added
Question What is the size of your boat? already exists - skipped & not added
[{"order":1,"title":"What is the size of your boat?","pictures":["/res/images/boatSize.jpg"],"responseChoice":[]}]
GIST :
provides you the complete working code I've made for you
https://gist.github.com/jeorfevre/5d8cbf352784042c7a7b4975fc321466
To conclude, what is a good practice to work with JSON is :
1) create a bean in order to build your json (see my example here)
2) build your json and store it in a file for example
3) Using android load your json from the file to the bean (you have it in andrdoid)
4) use the bean to build your form...etc (and not the json text file) :D

I would recommend a database (SQLite) as it provides superior filtering functionality over xml.

Create the db using DB Browser for SQLite
And then use the library SQLiteAssetHelper in the link-
https://github.com/jgilfelt/android-sqlite-asset-helper
Tutorial on how to use -
http://www.javahelps.com/2015/04/import-and-use-external-database-in.html

You can use Paper https://github.com/pilgr/Paper its a fast NoSQL data storage for Android.

SQLite is the best for your system. because you will have to maintain (text, category, subcategory, answers, group) etc. So if you create db and create table for them. That will be easy to manage and you can relationship with each other which is not possible to XML.

Why my ArrayList<String> is NOT sorted correctly?

I've been searching online and trying, but I didn't find the solution.
I have the following ArrayList:
{ Cate1, Cate3, Cate6, Cate2, ...., thru Cate10 }
I gave tried the following solutions:
public ArrayList<String> GetAllCategories_ByAscOrder() {
db = getWritableDatabase();
ArrayList<String> Category_ArrayList = new ArrayList<String>();
Cursor cursor = db.query(Category_TABLE, null, null, null, null, null, Category_List + " ASC");
if(cursor != null)
{
while(cursor.moveToNext())
{
String CATEGORY = cursor.getString(cursor.getColumnIndex(Category_List));
Category_ArrayList.add(CATEGORY);
}
}
cursor.close();
return Category_ArrayList;
}
And these:
Collections.sort(CATEGORY_LIST, new Comparator<String>(){
public int compare(String obj1, String obj2)
{
return obj1.compareToIgnoreCase(obj2);
}
});
}
//OR THIS:
Collections.sort(CATEGORY_LIST, new Comparator<String>(){
public int compare(String obj1, String obj2)
{
return obj1.compareTo(obj2);
}
});
}
//OR THIS:
Collections.sort(CATEGORY_LIST, String.CASE_INSENSITIVE_ORDER);
But ALL of them gave me the same sorting results:
Cate1, Cart10, Cate2, Cate3,, etc.... Cate9
I want the sorted list to be like this:
Cate1 thru Cate10
Can someone please guide me on how I can achieve this?
Thank you very much
Edit:
I forgot to mention that I let the users freely name their Category Names.

do like this :
Collections.sort(list , new Comparator<String>(){
public int compare( String a, String b ){
// if contains number
if( a.substring(4).matches("\\d+") && b.substring(4).matches("\\d+")) {
return new Integer( a.substring(4) ) - new Integer( b.substring(4) );
}
// else, compare normally.
return a.compareTo( b );
}
});

Its sorted by lexicographical order.
If you want it sorted like that, you should switch to using two digits,
eg. Cate01, Cate02, ...
Note that this happens in Windows/Linux filesystems too (if you have numbered files in a folder).

Technically, the results you got are correct - Cart10 alphabetically comes before Cart2 (since 1 comes before 2).
Try adding leading 0s to your numbers before sorting: Cart01, Cart02, etc - though you'll need to pad with leading zeros to make sure you cover the largest you expect your list to get (if it'll be over 100 elements, you'll need more zeros).
Alternatively, why not just store it as ArrayList<Integer> and prepend with "Cart" when you go to output the results? How you ultimately solve this depends on what you want to use the values for...

Android - Using ORMLite DAO as a ContentProvider

I have seen pattern C in the Google IO presentation and I am very anxious to implement this pattern. However, I do really like the ORMLite library and would like to use this library in my application as well.
When I say Google IO presentation I mean this one: https://www.youtube.com/watch?v=xHXn3Kg2IQE by Virgil Dobjanschi.
Now I have been searching a lot for an implementation that shows me how to use ORMLite in conjunction with Contentproviders.
Now my problem here is that the ORMLite DAO is conflicting with the Contentprovider. They essentially do the same and are a pain to integrate into each other. (Using Ormlite in Conjunction with Android's Content Provider others discussing this and agreeing upon this claim.)
A few libraries have implemented ORMLite into the contentprovider API pattern, one example is: https://github.com/blandware/android-atleap
However, underwater they still revert the model to ContentValues (simple types).
Android - Using Dao Pattern with contentProvider
This question is similair to my situation but 3 years ago and I'm suggesting an alternate solution below.
#jcwenger's answer is very useful, but I was wondering if anything has changed in the past 3 years. I'm facing the same issue and perhaps now since ORMLite has matured, it's more rewarding to use ORMLite?
My colleague next to me really, really wants to use ORMLite since he doesn't want to have to write any mapping himself. I know of the existance of the atleap and Android-OrmLiteContentProvider projects. These only provide a cursor to the activity and my colleague want to have lists of models or a single model. Can this be achieved?
My colleague suggests writing my own implementation of the Cursor, SyncAdapter? and Contentprovider (has to be done regardless) to work with models. However can the same functionality still be achieved with lists etc? Passing events to the activity to contentobservers etc?
Is this viable?
Edit
We'll most likely use the contentproviders privately. We do not need to expose these contentproviders. However the advantages that contentproviders provide are great. How else could I notify my GUI to update when the data has changed?
I also have to display data from multiple tables (joins and other data, not contained in the same table) in one activity and download images etc.

So since I couldn't find a proper answer, this is how I solved it after a while of trying:
public class CardProvider extends ContentProvider {
private InternalDatabase dbhelper;
private RuntimeExceptionDao<Card, UUID> cardDao;
/**
* Content authority for this provider.
*/
private static final String AUTHORITY = CardUris.CONTENT_AUTHORITY;
// The constants below represent individual URI routes, as IDs. Every URI pattern recognized by
// this ContentProvider is defined using sUriMatcher.addURI(), and associated with one of these
// IDs.
//
// When a incoming URI is run through sUriMatcher, it will be tested against the defined
// URI patterns, and the corresponding route ID will be returned.
/**
* URI ID for route: /cards
*/
public static final int ROUTE_CARDS = 1;
/**
* URI ID for route: /cards/{ID}
*/
public static final int ROUTE_CARDS_ID = 2;
/**
* UriMatcher, used to decode incoming URIs.
*/
private static final UriMatcher sUriMatcher = new UriMatcher(UriMatcher.NO_MATCH);
static {
sUriMatcher.addURI(AUTHORITY, "cards", ROUTE_CARDS);
sUriMatcher.addURI(AUTHORITY, "cards/*", ROUTE_CARDS_ID);
}
#Override
public int delete(Uri arg0, String arg1, String[] arg2) {
// TODO Auto-generated method stub
return 0;
}
#Override
public String getType(Uri uri) {
final int match = sUriMatcher.match(uri);
switch (match) {
case ROUTE_CARDS:
return CardUris.CONTENT_CARDS;
case ROUTE_CARDS_ID:
return CardUris.CONTENT_ITEM_CARD;
default:
throw new UnsupportedOperationException("Unknown uri: " + uri);
}
}
#Override
public Uri insert(Uri arg0, ContentValues arg1) {
// TODO Auto-generated method stub
return null;
}
#Override
public boolean onCreate() {
dbhelper = OpenHelperManager.getHelper(getContext(), InternalDatabase.class);
cardDao = dbhelper.getRuntimeExceptionDao(Card.class);
return true;
}
#Override
public Cursor query(Uri uri, String[] arg1, String arg2, String[] arg3,
String arg4) {
int uriMatch = sUriMatcher.match(uri);
switch (uriMatch) {
case ROUTE_CARDS_ID:
/*String id = uri.getLastPathSegment();
Card card = null;
try {
card = cardDao.queryBuilder().where().eq(Entry.ID_FIELD_NAME, id).queryForFirst();
} catch (SQLException e) {
e.printStackTrace();
}*/
//return null;
case ROUTE_CARDS:
// Return all known entries.
// Note: Notification URI must be manually set here for loaders to correctly
// register ContentObservers.
// build your query
QueryBuilder<Card, UUID> qb = cardDao.queryBuilder();
// when you are done, prepare your query and build an iterator
CloseableIterator<Card> iterator = null;
Cursor cursor = null;
try {
//qb.query();
iterator = cardDao.iterator(qb.where().eq("relevant", 1).and().eq("removed", false).prepare());
// get the raw results which can be cast under Android
AndroidDatabaseResults results =
(AndroidDatabaseResults)iterator.getRawResults();
cursor = results.getRawCursor();
} catch (SQLException e) {
e.printStackTrace();
} finally {
//iterator.closeQuietly();
}
cursor.setNotificationUri(this.getContext().getContentResolver(), uri);
return cursor;
default:
throw new UnsupportedOperationException("Unknown uri: " + uri);
}
}
#Override
public int update(Uri arg0, ContentValues arg1, String arg2, String[] arg3) {
// TODO Auto-generated method stub
return 0;
}
}
You could probably give purpose to the insert, update and delete methods, but the dao does this too and is what I am using.

Best way you can format a string in an Android TextView that is populated by SimpleCursorAdapter

I have two TextViews in a layout using a row layout for a SimpleCursorAdapter. The TextViews have their values populated from the SimpleCursorAdapter.
The first view displays the name of a computer host, that's fine but the second displays the mac address of the host, at current in the database I've got I'm storing it as a string in Hex format.
What I want to do is not have the formatting for the mac address in the database but format it programatically using a custom TextView or other else that will work. So basically AABBCCEEFF00 will become AA:BB:CC:EE:FF:00 when the SimpleCursorAdapter populates that TextView extened class.
What's the best way forward with this and where should I plug in the code if I do use custom TextView?
I've also seen reference to View Binding? but unsure if that's the route to go, I'm very much about making code modular and rather not override complex code when extending a simple class is possible.

As far as I could understand your question,You want to convert AABBCCDDEEFF into AA:BB:CC:DD:EE:FF programmatically.let me know if I didn't get your question.
look at following code,
from here,you are going to call a function to get formatted mac
String formattedMac = new String();
formattedMac=getFormattedMac("AABBCCDDEEFF");
Log.e("formatted Mac", formattedMac);//it will be look like, AA:BB:CC:DD:EE:FF
and here is the function returning formatted mac,
private String getFormattedMac(String mac) {
String result = new String();
result="";
int start=0;
int end=2;
int max=mac.length()+2;
while(end<max)
{
result = result+mac.substring(start, end);
if(end<mac.length())
{
result = result+":";
}
Log.e("result", result);
start=start+2;
end=end+2;
}
return result;
}

Turns out the best ways was with a ViewBinder setup, I followed this answer in the end.
Android, using SimpleCursorAdapter to set colour not just strings
Once I knew this I also changed the column to be a Blob instead of VarChar to make it a bit nicer as well.
SimpleCursorAdapter.ViewBinder binder = new SimpleCursorAdapter.ViewBinder() {
#Override
public boolean setViewValue(View view, Cursor cursor, int columnIndex) {
String name = cursor.getColumnName(columnIndex);
if ("mac_address".equals(name)) {
byte[] macAddress = cursor.getBlob(columnIndex);
TextView textView = (TextView)view;
textView.setText(Target.getMacAddress(macAddress));
return true;
}
return false;
}
};