Develop Reference

Jsoup attribute selector returning empty

I am trying to get images from google
String url = "https://www.google.com/search?site=imghp&tbm=isch&source=hp&q=audi&gws_rd=cr";
org.jsoup.nodes.Document doc = Jsoup.connect(url).get();
Elements elements = doc.select("div.isv-r.PNCib.MSM1fd.BUooTd");
ImageData is encoded in base64 so in order to get actual image url I first get the data id which is set as an attribute , this works
for (Element element : elements) {
String id = element.attr("data-id")).get();
I need to make new connection with url+"#imgrc="+id ,
org.jsoup.nodes.Document imgdoc = Jsoup.connect(url+"#"+id).get();
Now in the browser when I inspect my required data is present inside <div jsname="CGzTgf"> , so I also do the same in Jsoup
Elements images = imgdoc.select("div[jsname='CGzTgf']");
//futher steps
But images always return empty , I am unable to find the error , I do this inside new thread in android , any help will be appreciated

Turns out the way you're doing it you'll be looking in the wrong place entirely. The urls are contained within some javascript <script> tag included in the response.
I've extracted and filtered fro the relevant <script> tag (one containing attribute nonce.
I then filter those tags for one containing a specific function name used AND a generic search string I'm expecting to find (something that won't be in the other <script> tags).
Next, the value obtained needs to be stripped to get the JSON object containing about a hundred thousand arrays. I've then navigated this (manually), to pull out a subset of nodes containing relevant URL nodes. I then filter this again to get a List<String> to get the full URLs.
Finally I've reused some code from an earlier solution here: https://stackoverflow.com/a/63135249/7619034 with something similar to download images.
You'll then also get some console output detailing which URL ended up in which file id. Files are labeled image_[x].jpg regardless of actual format (so you may need to rework it a little - Hint: take file extension from url if provided).
import com.jayway.jsonpath.JsonPath;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.List;
public class GoogleImageDownloader {
private static int TIMEOUT = 30000;
private static final int BUFFER_SIZE = 4096;
public static final String RELEVANT_JSON_START = "AF_initDataCallback(";
public static final String PARTIAL_GENERIC_SEARCH_QUERY = "/search?q";
public static void main(String[] args) throws IOException {
String url = "https://www.google.com/search?site=imghp&tbm=isch&source=hp&q=audi&gws_rd=cr";
Document doc = Jsoup.connect(url).get();
// Response with relevant data is in a <script> tag
Elements elements = doc.select("script[nonce]");
String jsonDataElement = getRelevantScriptTagContainingUrlDataAsJson(elements);
String jsonData = getJsonData(jsonDataElement);
List<String> imageUrls = getImageUrls(jsonData);
int fileId = 1;
for (String urlEntry : imageUrls) {
try {
writeToFile(fileId, makeImageRequest(urlEntry));
System.out.println(urlEntry + " : " + fileId);
fileId++;
} catch (IOException e) {
e.printStackTrace();
}
}
}
private static String getRelevantScriptTagContainingUrlDataAsJson(Elements elements) {
String jsonDataElement = "";
int count = 0;
for (Element element : elements) {
String jsonData = element.data();
if (jsonData.startsWith(RELEVANT_JSON_START) && jsonData.contains(PARTIAL_GENERIC_SEARCH_QUERY)) {
jsonDataElement = jsonData;
// IF there are two items in the list, take the 2nd, rather than the first.
if (count == 1) {
break;
}
count++;
}
}
return jsonDataElement;
}
private static String getJsonData(String jsonDataElement) {
String jsonData = jsonDataElement.substring(RELEVANT_JSON_START.length(), jsonDataElement.length() - 2);
return jsonData;
}
private static List<String> getImageUrls(String jsonData) {
// Reason for doing this in two steps is debugging is much faster on the smaller subset of json data
String urlArraysList = JsonPath.read(jsonData, "$.data[31][*][12][2][*]").toString();
List<String> imageUrls = JsonPath.read(urlArraysList, "$.[*][*][3][0]");
return imageUrls;
};
private static void writeToFile(int i, HttpURLConnection response) throws IOException {
// opens input stream from the HTTP connection
InputStream inputStream = response.getInputStream();
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream("image_" + i + ".jpg");
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
}
// Could use JSoup here but I'm re-using this from an earlier answer
private static HttpURLConnection makeImageRequest(String imageUrlString) throws IOException {
URL imageUrl = new URL(imageUrlString);
HttpURLConnection response = (HttpURLConnection) imageUrl.openConnection();
response.setRequestMethod("GET");
response.setConnectTimeout(TIMEOUT);
response.setReadTimeout(TIMEOUT);
response.connect();
return response;
}
}
Partial Result I tested with:
I've used JsonPath for filtering the relevant nodes which is good when you only care about a small portion of the JSON and don't want to deserialise the whole object. It follows a similar navigation style to DOM/XPath/jQuery navigation.
Apart from this one library and Jsoup, the libraries used are very bog standard.
Good Luck!

How to prevent escaping during serialization with Retrofit2 and Gson Converter

I am lost in thoughts and tried my best to search and to come up with a possible solution. I have tried most of the answers that could be related to what I am trying to achieve but still no luck.
So to elaborate my question, I am receiving this response from an API
{
"profileList": [
{
"id": "7mmfHGLc0MGtZeQNno/WFqDjlAPj26CS",
"name": "Alexandria Victoria Maxene Kluber van de Gr\\\"oot\""
}
]
}
What I wanted to achieve is to get the "name" field value without the value being escaped. Because on my current setup, what I get from that response after deserialization process is Alexandria Victoria Maxene Kluber van de Gr\"oot""
I let my Retrofit handle the API request and response so that's what I am getting, I currently don't want to tear off my handlers for specific reason, but I am hoping someone could point me in the right direction.
Here is my Retrofit Builder code:
Gson gson = new GsonBuilder()
.disableHtmlEscaping()
.create();
m_builder = new Retrofit.Builder()
.baseUrl(url)
.addConverterFactory(GsonConverterFactory.create(gson));
Thank you in advance.

You should try to use JsonPrimitive in your POJO. See below example:
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonPrimitive;
import java.io.File;
import java.io.FileReader;
import java.util.List;
public class GsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
Gson gson = new GsonBuilder().create();
System.out.println(gson.fromJson(new FileReader(jsonFile), Profiles.class));
}
}
class Profiles {
private List<Profile> profileList;
// getters, setters, toString
}
class Profile {
private String id;
private JsonPrimitive name;
// getters, setters, toString
}
Above example for your JSON payload prints:
Profiles{profileList=[Profile{id='7mmfHGLc0MGtZeQNno/WFqDjlAPj26CS', name='"Alexandria Victoria Maxene Kluber van de Gr\\\"oot\""'}]}

MainActivity puts valid String into Extras for an Intent but the Intent finds null String. What to do?

I am baffled by what's happening.
I put the file name "Help" into the Extras of the Intent that I then immediately start. (There can't be any problems with this first code block. With breakpoint after putExtra, Watch expression helpIntent.getStringExtra(PACKAGE_NAME + ".filename") returned "Help".)
import static com.dslomer64.sqhell.BuildConfig.APPLICATION_ID;
...
public class MainActivity extends Activity
{
public final static String PACKAGE_NAME = APPLICATION_ID;
...
#Override public boolean onOptionsItemSelected(MenuItem item)
{
if(item.toString().equals("Help") || item.toString().equals("Acknowledgements"))
{
Intent
helpIntent;
helpIntent = new Intent(this, HelpActivity.class);
helpIntent.putExtra(PACKAGE_NAME + ".filename", item.getTitle().toString());
startActivity(helpIntent);
return true;
}
...
}
The fun begins in the spawned Intent:
public class HelpActivity extends Activity
{
...
public void showHelp(AssetManager assets, String filename)
{
try
{
////////////////////////////////////////////
// filename = null ********* #1
stream = assets.open(filename);
//////////////////////////////////////////// }
catch (IOException e) { e.printStackTrace(); }
...
}
#Override
public void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
Log.w("`````Help", "onCreate--bundle info: " + bundle.toString()) ;
Log.w("`````Help","onCreate--bundle key: " + bundle.getString("filename") );
Bundle bundle = getIntent().getExtras();
setContentView(R.layout.activity_help);
////////////////////////////////////////////
String fn = bundle.getString((PACKAGE_NAME + ".filename"));
// fn = null ***************************** #2
////////////////////////////////////////////
showHelp(getAssets(), fn );
}
The first sign of trouble (#1) was "filename is empty". Tracing back was easy. Now we're at #2, where--just compare--I use the same basic syntax to try to get the filename "Help" out of the Extras of the Intent. But it's null.
I'm not an expert with Intent but I've successfully used one a few times in the past.
I guess if I supplied a compilable program that shows the error it would be easier. But ...
If nobody spots anything right off, I'll get on that.
But it's straightforward!
But not.
================
EDIT
I added 2 Log statements (see above) and got this output:
W/`````Help: onCreate--bundle info:
Bundle[{com.dslomer64.sqhell.filename=Help}]
//////////////////////////////////////^^^^
W/`````Help: onCreate--bundle key: null
Not sure how this helps since it looks like "Help" is where it should be. And I assume key is null is equivalent to the original problem about filename being null.
============================
EDIT #2: Here's the WHOLE HelpActivity class. If this doesn't help, I'll get on the compilable thing:
package com.dslomer64.sqhell;
import android.app.Activity;
import android.content.res.AssetManager;
import android.os.Bundle;
import android.util.Log;
import android.view.WindowManager;
import android.widget.EditText;
import java.io.IOException;
import java.io.InputStream;
import java.util.Scanner;
import static android.provider.ContactsContract.Directory.PACKAGE_NAME;
public class HelpActivity extends Activity
{
Scanner scDict;
InputStream stream;
EditText help;
public void showHelp(AssetManager assets, String filename)
{
help = /*(EditText)*/ findViewById(R.id.txaHelp);
try
{
stream = assets.open(filename);
}
catch (IOException e) { e.printStackTrace(); }
scDict = new Scanner(stream).useDelimiter("\n\r\t ,.:;-( )"); // added (SPACE) recently
help.getText().clear();
while(scDict.hasNext())
help.append(scDict.next());
hideKeyboard();
help.setSelection(0);
}
#Override
public void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
Bundle bundle = getIntent().getExtras();
setContentView(R.layout.activity_help);
Log.w("`````Help", "onCreate--bundle info: " + bundle.toString()) ;
Log.w("`````Help", "onCreate--bundle key: " + bundle.getString(PACKAGE_NAME + ".filename"));
String fn = bundle.getString(PACKAGE_NAME + ".filename");
showHelp(getAssets(), fn );
}
}
=========================
In following cricket's advice, I noticed a long line in the importarea:
import static android.provider.ContactsContract.Directory.PACKAGE_NAME;
I don't remember ever seeing such before. So I took it out and got a few errors, wherever PACKAGE_NAME was used. Since each line had the "red underline" indicating use keypress 'Alt-Enter' for an easy import construction, I found that I HAD TWO CHOICES: the one above and MainActivity.PACKAGE_NAME.
I assume the previous time I did this, I chose the wrong item to import.
All is well.
Spent... too long on this.
Thanks to all who helped and Cricket for steering me in the right direction.
=====================
FINAL EDIT:
All I needed to do all along was delete the long import statement. And also pay attention to what I selected after doing it.
Criminey.
I thought I was doing what Cricket suggested all along. Didn't get the funny import in other classes. Why here????

it looks like "Help" is where it should be.
It does
... key is null
Its actually the value, but you're getting the value of the key "filename", not PACKAGE_NAME + ".filename", which is what you used in the putExtra.
The keys don't match, so the value is null
Suggestion: define your own constant like
public static final String FILENAME_EXTRA = APPLICATION_ID + ".filename";
and use that in all classes
Additionally your Scanner and everything requiring the file to be opened should be inside the try block.

you put data in the Intent, not the Bundle.
So, try String fn = getIntent().getStringExtra(PACKAGE_NAME + ".filename").Then you will find what you want.

you are using a wrong key when retrieving the value.
Log.w("`````Help","onCreate--bundle key: " + bundle.getString(PACKAGE_NAME + ".filename") );

Parsing specific texts using Jsoup

I have a cities.txt file placed in my res/raw folder. Inside it contains the following.
<div class="state">Alabama</div>
<ul><li>auburn</li>
<li>birmingham</li> </ul>
<div class="state">Alaska</div>
<ul><li>anchorage</li>
<li>fairbanks</li></ul>
<div class="state">Arizona</div>
<ul><li>flagstaff</li>
<li>mohave county</li></ul>
I want to grab the cities for the state Alabama and display it on a ListView. The ouput should be like this.
auburn
birmingham
My current code grabs all the six cities and displays them on the ListView instead. This is my code.
package com.example.readfile;
import java.io.InputStream;
import java.util.ArrayList;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import android.app.Activity;
import android.content.res.Resources;
import android.os.AsyncTask;
import android.os.Bundle;
import android.widget.ArrayAdapter;
import android.widget.ListView;
public class Cities extends Activity {
ListView listUSCities;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.city_layout);
listUSCities = (ListView) findViewById(R.id.listcities);
new MyTask().execute();
}
class MyTask extends AsyncTask<Void, Void, ArrayList<String>> {
ArrayList<String> arr_linkText = new ArrayList<String>();
#Override
protected ArrayList<String> doInBackground(Void... params) {
Document doc;
try {
Resources res = getResources();
InputStream in_s = res.openRawResource(R.raw.cities);
byte[] b = new byte[in_s.available()];
in_s.read(b);
// Parsing using Jsoup starts here
doc = Jsoup.parse(new String(b));
// Parsing the states
Elements links = doc.select("div");
for (Element link : links) {
if (link.text().contains("Alabama")) {
// Extracting the cities
Elements cities = doc.select("a");
for (Element city : cities) {
arr_linkText.add(city.text());
}
}
}
} catch (Exception e) {
// e.printStackTrace();
}
return arr_linkText; // << retrun ArrayList from here
}
#Override
protected void onPostExecute(ArrayList<String> result) {
ArrayAdapter<String> adapter = new ArrayAdapter<String>(
Cities.this, android.R.layout.simple_list_item_1,
android.R.id.text1);
for (String temp_result : result) {
adapter.add(temp_result);
}
listUSCities.setAdapter(adapter);
}
}
}
How can I extract the cities only for that specified state?
How do I stop parsing the file after I extracted the cities to optimize speed?
The actual cities.txtcontains more information, I only provided a sample. I will appreciate your help. Thank you!

// Parsing the states
Elements links = doc.select("div");
for (Element link : links) {
if (link.text().contains("Alabama")) {
// Extracting the cities
Elements cities = link.select("a");//<- 'doc' is the whole doc, link is your state.
for (Element city : cities) {
arr_linkText.add(city.text());
}
break;//breaks off the loop, since you have found what you want.
}
}

That is an odd structure for an HTML document. The <div> is used just for the header, and the list is off by itself. Seeing as you trimmed the actual document, this may or may not work. The elements you are after are in the ul element following your div, so you need to go to the next sibling and search there. This will only work if there are no other elements between your div and ul elements.
Elements links = doc.select("div");
for (Element link : links) {
if (link.text().contains("Alabama")) {
// Extracting the cities in the list that is next in the DOM
Elements cities = link.nextElementSibling().select("a");
for (Element city : cities) {
arr_linkText.add(city.text());
}
}
}

Error when fetching a second XML from the web

im new to android development and im trying to build my first app which looks for a online generated xml file to display information. In the first activity i created a ListView with all the entries from an XML file, as soon as i click on an entry it passes the id and goes to the 2nd activity which should access another XML file with the details. However i keep getting this error when trying to fetch the XML for the details:
java.lang.ClassCastException:
org.apache.harmony.xml.dom.ElementImpl
Any ideas whats wrong? Here is the source for the "details" activity:
package en.android.itleaked.com;
import java.io.InputStream;
import java.net.URL;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import android.app.Activity;
import android.graphics.drawable.Drawable;
import android.os.Bundle;
import android.sax.Element;
import android.widget.ImageView;
import android.widget.TextView;
public class showReleases extends Activity {
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.releasedetails);
getFeed();
}
public void getFeed() {
Bundle extras = getIntent().getExtras();
try {
URL url2 = new URL("http://www.it-leaked.com/app/details.php?id=" + extras.getString("id"));
DocumentBuilderFactory dbf2 = DocumentBuilderFactory.newInstance();
DocumentBuilder db2 = dbf2.newDocumentBuilder();
Document doc2 = db2.parse(new InputSource(url2.openStream()));
doc2.getDocumentElement().normalize();
NodeList nodeList2 = doc2.getElementsByTagName("item");
String relTitle[] = new String[nodeList2.getLength()];
String relCover[] = new String[nodeList2.getLength()];
for (int i = 0; i < nodeList2.getLength(); i++) {
Node node2 = nodeList2.item(i);
Element fstElmnt2 = (Element) node2;
NodeList nameList2 = ((Document) fstElmnt2).getElementsByTagName("title");
Element nameElement2 = (Element) nameList2.item(0);
nameList2 = ((Node) nameElement2).getChildNodes();
relTitle[i] = ((Node) nameList2.item(0)).getNodeValue();
NodeList coverList2 = ((Document) fstElmnt2).getElementsByTagName("cover");
Element coverElement2 = (Element) coverList2.item(0);
coverList2 = ((Node) coverElement2).getChildNodes();
relCover[i] = ((Node) coverList2.item(0)).getNodeValue();
}
TextView txtView = (TextView)findViewById(R.id.TextView01);
txtView.setText(relTitle[0]);
ImageView imgView =(ImageView)findViewById(R.id.ImageView01);
Drawable drawable = LoadImageFromWebOperations(relCover[0]);
imgView.setImageDrawable(drawable);
}
catch (Exception e) {
TextView txtView2 = (TextView)findViewById(R.id.TextView02);
txtView2.setText("Error: " + e);
}
}
private Drawable LoadImageFromWebOperations(String url)
{
try
{
InputStream is = (InputStream) new URL(url).getContent();
Drawable d = Drawable.createFromStream(is, "src name");
return d;
}catch (Exception e) {
System.out.println("Exc="+e);
return null;
}
}
}
Here is the URL for the XML with an attached id so you can see what it looks like:
http://www.it-leaked.com/app/details.php?id=50969
Any ideas whats going on? By the way i added the number 2 to every variable which has something to do with the XML parsing / fetching just to make sure theres no conflict with the other activity, but im still getting the same error..
I hope you can help me out.
Thanks in advance

This question is a little old but I believe the ClassCastException is due to attempting to cast the elements in the nodelist to android.sax.Element type rather than org.w3c.dom.Element; check the imports.

Looking at your exception (which is java.lang.ClassCastException) the problem is in casting some class to another.
In your code i didn't understand the reason casting Element to Document - Element has getElementsByTagName method which you are using. Look here: http://developer.android.com/reference/org/w3c/dom/Element.html
It is the root of all evil. Element and Document both implementing Node interface, but Document isn't implements Element - that's why Element can't be cast to Document.
Anyway, expcetion line number can determine the exact error position.