I am trying to get images from google
String url = "https://www.google.com/search?site=imghp&tbm=isch&source=hp&q=audi&gws_rd=cr";
org.jsoup.nodes.Document doc = Jsoup.connect(url).get();
Elements elements = doc.select("div.isv-r.PNCib.MSM1fd.BUooTd");
ImageData is encoded in base64 so in order to get actual image url I first get the data id which is set as an attribute , this works
for (Element element : elements) {
String id = element.attr("data-id")).get();
I need to make new connection with url+"#imgrc="+id ,
org.jsoup.nodes.Document imgdoc = Jsoup.connect(url+"#"+id).get();
Now in the browser when I inspect my required data is present inside <div jsname="CGzTgf"> , so I also do the same in Jsoup
Elements images = imgdoc.select("div[jsname='CGzTgf']");
//futher steps
But images always return empty , I am unable to find the error , I do this inside new thread in android , any help will be appreciated
Turns out the way you're doing it you'll be looking in the wrong place entirely. The urls are contained within some javascript <script> tag included in the response.
I've extracted and filtered fro the relevant <script> tag (one containing attribute nonce.
I then filter those tags for one containing a specific function name used AND a generic search string I'm expecting to find (something that won't be in the other <script> tags).
Next, the value obtained needs to be stripped to get the JSON object containing about a hundred thousand arrays. I've then navigated this (manually), to pull out a subset of nodes containing relevant URL nodes. I then filter this again to get a List<String> to get the full URLs.
Finally I've reused some code from an earlier solution here: https://stackoverflow.com/a/63135249/7619034 with something similar to download images.
You'll then also get some console output detailing which URL ended up in which file id. Files are labeled image_[x].jpg regardless of actual format (so you may need to rework it a little - Hint: take file extension from url if provided).
import com.jayway.jsonpath.JsonPath;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.List;
public class GoogleImageDownloader {
private static int TIMEOUT = 30000;
private static final int BUFFER_SIZE = 4096;
public static final String RELEVANT_JSON_START = "AF_initDataCallback(";
public static final String PARTIAL_GENERIC_SEARCH_QUERY = "/search?q";
public static void main(String[] args) throws IOException {
String url = "https://www.google.com/search?site=imghp&tbm=isch&source=hp&q=audi&gws_rd=cr";
Document doc = Jsoup.connect(url).get();
// Response with relevant data is in a <script> tag
Elements elements = doc.select("script[nonce]");
String jsonDataElement = getRelevantScriptTagContainingUrlDataAsJson(elements);
String jsonData = getJsonData(jsonDataElement);
List<String> imageUrls = getImageUrls(jsonData);
int fileId = 1;
for (String urlEntry : imageUrls) {
try {
writeToFile(fileId, makeImageRequest(urlEntry));
System.out.println(urlEntry + " : " + fileId);
fileId++;
} catch (IOException e) {
e.printStackTrace();
}
}
}
private static String getRelevantScriptTagContainingUrlDataAsJson(Elements elements) {
String jsonDataElement = "";
int count = 0;
for (Element element : elements) {
String jsonData = element.data();
if (jsonData.startsWith(RELEVANT_JSON_START) && jsonData.contains(PARTIAL_GENERIC_SEARCH_QUERY)) {
jsonDataElement = jsonData;
// IF there are two items in the list, take the 2nd, rather than the first.
if (count == 1) {
break;
}
count++;
}
}
return jsonDataElement;
}
private static String getJsonData(String jsonDataElement) {
String jsonData = jsonDataElement.substring(RELEVANT_JSON_START.length(), jsonDataElement.length() - 2);
return jsonData;
}
private static List<String> getImageUrls(String jsonData) {
// Reason for doing this in two steps is debugging is much faster on the smaller subset of json data
String urlArraysList = JsonPath.read(jsonData, "$.data[31][*][12][2][*]").toString();
List<String> imageUrls = JsonPath.read(urlArraysList, "$.[*][*][3][0]");
return imageUrls;
};
private static void writeToFile(int i, HttpURLConnection response) throws IOException {
// opens input stream from the HTTP connection
InputStream inputStream = response.getInputStream();
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream("image_" + i + ".jpg");
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
}
// Could use JSoup here but I'm re-using this from an earlier answer
private static HttpURLConnection makeImageRequest(String imageUrlString) throws IOException {
URL imageUrl = new URL(imageUrlString);
HttpURLConnection response = (HttpURLConnection) imageUrl.openConnection();
response.setRequestMethod("GET");
response.setConnectTimeout(TIMEOUT);
response.setReadTimeout(TIMEOUT);
response.connect();
return response;
}
}
Partial Result I tested with:
I've used JsonPath for filtering the relevant nodes which is good when you only care about a small portion of the JSON and don't want to deserialise the whole object. It follows a similar navigation style to DOM/XPath/jQuery navigation.
Apart from this one library and Jsoup, the libraries used are very bog standard.
Good Luck!
Trying to get data from website through HTML parsing.
java.lang.ArrayIndexOutOfBoundsException: length=0; index=0
Why is the array empty?
I included only the class that contains Jsoup because im sure the error is in it. And also i included the HTML part of the website in case you guys wanna take a look at it.
Jsoup Class
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
public class contentExcluding {
public String[] createData(final String [] text, final String [] pictures)
{
Runnable rMain = new Runnable() {
#Override
public void run() {
try {
Document doc = Jsoup.connect("http://egyptianstreets.com/").get();
Element content = doc.getElementById("featured-multi-main-img");
Elements headlines = content.getElementsByTag("img");
Elements img = content.getElementsByAttribute("src");
String x = img.toString();
pictures[0] = x;
text[0] = x;
} catch (IOException e) {
e.printStackTrace();
}
}
};
Thread t = new Thread(rMain);
t.start();
return pictures;
}
}
HTML
<div id="head-wrap" class="left relative">
<div class="head-wrap-out">
<div class="head-wrap-in">
<div id="featured-multi-wrap" class="left relative">
<div id="featured-multi-main" class="left relative">
<a href="http://egyptianstreets.com/2016/08/06/egyptian-rowers-nadia-negm-and-abdel-khalek-elbana-reach-quarter-finals-at-olympics/" rel="bookmark">
<div id="featured-multi-main-img" class="left relative">
<img width="1000" height="512" src="http://egyptianstreets.com/wp-content/uploads/2016/08/pablo-2-1000x512.png" class="unlazy reg-img wp-post-image" alt="pablo (2)" /> <img width="400" height="240" src="http://egyptianstreets.com/wp-content/uploads/2016/08/pablo-2-400x240.png" class="unlazy mob-img wp-post-image" alt="pablo (2)" />
It looks like pictures or text are 0 length arrays.
Check that you initialized them (before call createData method) with a higher size of 0.
Example:
String[] pictures = contentExcluding.createData(new String[1], new String[1]);
I'm working on an app that simply has to go out to UrbanDictionary and return the results of a word search in JSON format...then display it in an Android project.
Here is my code in its entirety:
package org.twodee.mitchemc.webapi;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import android.app.ListActivity;
import android.os.AsyncTask;
import android.os.Bundle;
import android.util.Log;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.ArrayAdapter;
import android.widget.Button;
import android.widget.EditText;
public class MainActivity extends ListActivity {
private ArrayList<String> definitionsList;
private ArrayAdapter<String> adapter;
private EditText wordText;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
wordText = (EditText) findViewById(R.id.wordText);
Button defineButton = (Button) findViewById(R.id.defineButton);
definitionsList = new ArrayList<String>();
adapter = new ArrayAdapter<String>(this, android.R.layout.simple_list_item_1, definitionsList);
setListAdapter(adapter);
defineButton.setOnClickListener(new OnClickListener() {
#Override
public void onClick(View v) {
String word = wordText.getText().toString();
new DownloadTask().execute(word);
}
});
}
private ArrayList<String> getDefs(String word) {
ArrayList<String> definitions = new ArrayList<String>();
try {
URL url = new URL("http://api.urbandictionary.com/v0/define?term="
+ word);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
InputStream in = connection.getInputStream();
String json = Utilities.slurp(in);
Log.d("FOO", json);
JSONArray array = new JSONArray(json);
for (int i = 0; i < array.length(); ++i) {
JSONObject object = array.getJSONObject(i);
definitions.add(object.getString("definition"));
}
in.close();
} catch (MalformedURLException e) {
} catch (IOException e) {
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
}
return definitions;
}
class DownloadTask extends AsyncTask<String, Void, ArrayList<String>> {
#Override
protected ArrayList<String> doInBackground(String... params) {
String word = params[0];
return getDefs(word);
}
#Override
protected void onPostExecute(ArrayList<String> defs) {
super.onPostExecute(defs);
definitionsList.clear();
definitionsList.addAll(defs);
adapter.notifyDataSetChanged();
}
}
}
Pay no attention to the Utilities.slurp behind the curtain.
So, if I run this, it doesn't crash, and it actually does give me results, but it throws them in LogCat...not my app. I previously found the following exception:
of type org.json.jsonarray cannot be converted to jsonobject
I looked over this forum and found the notion that I've got objects within an array (or something like that) but I can't figure out how to reflect that in code.
What I'd like to have happen is you type, say, "Hey" into the app, hit "Define" and then there's a list of definitions below it. Only the definitions...so the JSON part of it should be "definition"
Here's some JSON output that I get in LogCat. I hope this is readable:
12-06 19:00:57.647: W/System.err(1400): org.json.JSONException: Value {"total":32,"result_type":"exact","list":[{"thumbs_down":428,"author":"D.B. Echo","definition":"A contraction for \"Hello, I find you attractive and would like to dance with you, share some drinks with you, and then perhaps have sex with you. Nothing serious, of course, and I doubt that this will result in a long term relationship, but I would appeciate you considering my proposal.\"","permalink":"http:\/\/hey.urbanup.com\/1099714","thumbs_up":1530,"word":"hey","current_vote":"","example":"Guy at bar to attractive girl: \"Hey.\"\r\n\r\nAttractive girl to guy: \"Go to hell, creep.\" (Moves to other side of bar.)","defid":1099714},{"thumbs_down":250,"author":"Erica","definition":"hey is a way to say \"hi\" or \"what's up\"\r\nthe type of 'hey' horses eat is hAy you morons!!!","permalink":"http:\/\/hey.urbanup.com\/574637","thumbs_up":541,"word":"Hey","current_vote":"","example":"hey what's up!?!\r\nhorses eat hay!","defid":574637},{"thumbs_down":144,"author":"SoMe RaNdOm PeRsOn","definition":"I word to use to get someone's attention. Even if you don't know the persons name by simply saying \"hey\" you can easily get their attention","permalink":"http:\/\/hey.urbanup.com\/1106278","thumbs_up":294,"word":"hey","current_vote":"","example":"guy 1:Hey Alex what's that guys name over there?\r\nAlex: How the hell should I know?\r\nGuy 1: Hey!!\r\nMystery guy: ::looks around confused::\r\nGuy 1: O hi.","defid":1106278},{"thumbs_down":120,"author":"Jamesrob92","definition":"1. The most informal form of greeting. Usually used in a friendly manner or to sound cool and relaxed.\r\n2. An exclamation used to get someone's attention","permalink":"http:\/\/hey.urbanup.com\/2153666","thumbs_up":249,"word":"hey","current_vote":"","example":"1. Hey, what's up?\r\n2. HEY! Wait for me!\r\n3. HEY YOU! Get away from my wife!","defid":2153666},{"thumbs_down":133,"author":"marshmallow","definition":"a slang term used by many people in place of hello","permalink":"http:\/\/hey.urbanup.com\/1129122","thumbs_up":219,"word":"hey","current_vote":"","example":"Hey Ma...whats crackin?","defid":1129122},{"thumbs_down":79,"author":"hey malasadas","definition":"an expression used to get someone's attention","permalink":"http:\/\/hey.urbanup.com\/198491","thumbs_up":128,"word":"hey","current_vote":"","example":"hey man, what the hell are you doing?!","defid":198491},{"thumbs_down":54,"author":"HappyGirl1993","definition":"Considered to be a lot more flirtatious than 'hello' or 'hi.'","permalink":"http:\/\/hey.urbanup.com\/5302185","thumbs_up":68,"word":"Hey","current_vote":"","example":"Example...\n\nGuy: Hey.\r\nGirl: Hi.\r\nGuy: *Whoa, total FAIL!*","defid":5302185},{"thumbs_down":37,"author":"Eric Klein...","definition":"1. Interjection, an informal greeting. \r\n\r\n2. Verb, to 'hey' someone. The act of driving down public roads in a lane near a sidewalk, coming upon an unsuspecting pedestrian, and hollering \"HEY!\" as loudly as possible in their general direction. Most effective if done with all windows rolled down, and all members of the vehicle participating. (Originating in Sioux Falls, SD)","permalink":"http:\/\/hey.urbanup.com\/2705914","thumbs_up":42,"word":"Hey","current_vote":"","example":"1. \"Hey buddy, how's the syphilis?\"\r\n\r\n2. \"Jimmy and I heyed a kid so bad the other day, he fell off his bike!\"","defid":2705914},{"thumbs_down":80,"author":"J-Lib","definition":"a slang interjection or prompt often used in place of \"eh\" or \"right\" in order to make a sentence into a question; used in parts of Canada","permalink":"http:\/\/hey.urbanup.com\/1262450","thumbs_up":67,"word":"hey","current_vote":"","example":"It's supposed to snow tomorrow, hey?\r\nThat guy was pretty attractive, hey?","defid":1262450},{"thumbs_down":7,"author":"Bam Bam Branson","definition":"Why would you even search this?","permalink":"http:\/\/hey.urbanup.com\/5850424","thumbs_up":8,"word":"Hey","current_vote":"","example":"Hey is Idk","defid":5850424}],"pages":4,"sounds":["http:\/\/media.urbandictiona
Any help would be greatly appreciated! Thanks in advance!
UPDATE: Here's the code that works:
String json = Utilities.slurp(in);
JSONObject jsonObject = new JSONObject(json);
JSONArray array = jsonObject.getJSONArray("list");
for (int i = 0; i < array.length(); ++i) {
JSONObject object = array.getJSONObject(i);
definitions.add(object.getString("definition"));
}
Based on the json you get back from that request, you are setting up your JSONObject wrong. Here is an example of the json that comes back:
{
"has_related_words": true,
"result_type": "exact",
"list": [
{
"defid": 2957653,
"word": "test",
"author": "sm1g",
"permalink": "http://test.urbanup.com/2957653",
"definition": "To check if something coresponds the promised result or what effect does it have at all.",
"example": "By typing in the word \"test\" you prolly tried to search if there was a definition for this word.",
"thumbs_up": 123,
"thumbs_down": 46,
"current_vote": ""
}
],
"sounds": [
"http://media.urbandictionary.com/sound/test-8076.mp3"
],
"total": 20,
"pages": 2
}
In your code, to grab an array like list, you would need to do:
String json = Utilities.slurp(in);
JSONObject jsonObject = new JSONObject(json);
//now get the list array
JSONArray list = new JSONArray(jsonObject.getArray("list"));
The method names may not be exact.
hi i want to display my html image on android emulator, please help
package com.Htmlview;
import ja va.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.text.MessageFormat;
import android.app.Activity;
import android.content.res.AssetManager;
import android.os.Bundle;
import android.webkit.WebView;
public class Htmlview extends Activity {
/** Called when the activity is first created. */
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
//setContentView(R.layout.main);
WebView webview = new WebView(this);
setContentView(webview);
try {
String imageString="<html><body>" +
"<h2>Norwegian Mountain Trip</h2>" +
"<img src=\"C:/Users/Public/Pictures/Sample Pictures/pulpit.jpg\" alt=\"Pulpit rock\" width=\"304\" height=\"228\" /></body></html>";
AssetManager mgr = this.getAssets();
InputStream is = mgr.open("index3.html");
BufferedInputStream in = new BufferedInputStream(is); // read the contents of the file
webview.loadData(MessageFormat.format(imageString,arguments),"text/html", "UTF-8");
} catch (IOException e) {
e.printStackTrace();
}
}
}
You're never going to be able to access something on your workstation's hard drive from within Android. What you want to do is put the image into the assets folder in your Android app (which will then get bundled into the app) and link to it appropriately with file:///android_assets/. There are other ways of dealing with the issue, but that is the simplest.
im new to android development and im trying to build my first app which looks for a online generated xml file to display information. In the first activity i created a ListView with all the entries from an XML file, as soon as i click on an entry it passes the id and goes to the 2nd activity which should access another XML file with the details. However i keep getting this error when trying to fetch the XML for the details:
java.lang.ClassCastException:
org.apache.harmony.xml.dom.ElementImpl
Any ideas whats wrong? Here is the source for the "details" activity:
package en.android.itleaked.com;
import java.io.InputStream;
import java.net.URL;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import android.app.Activity;
import android.graphics.drawable.Drawable;
import android.os.Bundle;
import android.sax.Element;
import android.widget.ImageView;
import android.widget.TextView;
public class showReleases extends Activity {
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.releasedetails);
getFeed();
}
public void getFeed() {
Bundle extras = getIntent().getExtras();
try {
URL url2 = new URL("http://www.it-leaked.com/app/details.php?id=" + extras.getString("id"));
DocumentBuilderFactory dbf2 = DocumentBuilderFactory.newInstance();
DocumentBuilder db2 = dbf2.newDocumentBuilder();
Document doc2 = db2.parse(new InputSource(url2.openStream()));
doc2.getDocumentElement().normalize();
NodeList nodeList2 = doc2.getElementsByTagName("item");
String relTitle[] = new String[nodeList2.getLength()];
String relCover[] = new String[nodeList2.getLength()];
for (int i = 0; i < nodeList2.getLength(); i++) {
Node node2 = nodeList2.item(i);
Element fstElmnt2 = (Element) node2;
NodeList nameList2 = ((Document) fstElmnt2).getElementsByTagName("title");
Element nameElement2 = (Element) nameList2.item(0);
nameList2 = ((Node) nameElement2).getChildNodes();
relTitle[i] = ((Node) nameList2.item(0)).getNodeValue();
NodeList coverList2 = ((Document) fstElmnt2).getElementsByTagName("cover");
Element coverElement2 = (Element) coverList2.item(0);
coverList2 = ((Node) coverElement2).getChildNodes();
relCover[i] = ((Node) coverList2.item(0)).getNodeValue();
}
TextView txtView = (TextView)findViewById(R.id.TextView01);
txtView.setText(relTitle[0]);
ImageView imgView =(ImageView)findViewById(R.id.ImageView01);
Drawable drawable = LoadImageFromWebOperations(relCover[0]);
imgView.setImageDrawable(drawable);
}
catch (Exception e) {
TextView txtView2 = (TextView)findViewById(R.id.TextView02);
txtView2.setText("Error: " + e);
}
}
private Drawable LoadImageFromWebOperations(String url)
{
try
{
InputStream is = (InputStream) new URL(url).getContent();
Drawable d = Drawable.createFromStream(is, "src name");
return d;
}catch (Exception e) {
System.out.println("Exc="+e);
return null;
}
}
}
Here is the URL for the XML with an attached id so you can see what it looks like:
http://www.it-leaked.com/app/details.php?id=50969
Any ideas whats going on? By the way i added the number 2 to every variable which has something to do with the XML parsing / fetching just to make sure theres no conflict with the other activity, but im still getting the same error..
I hope you can help me out.
Thanks in advance
This question is a little old but I believe the ClassCastException is due to attempting to cast the elements in the nodelist to android.sax.Element type rather than org.w3c.dom.Element; check the imports.
Looking at your exception (which is java.lang.ClassCastException) the problem is in casting some class to another.
In your code i didn't understand the reason casting Element to Document - Element has getElementsByTagName method which you are using. Look here: http://developer.android.com/reference/org/w3c/dom/Element.html
It is the root of all evil. Element and Document both implementing Node interface, but Document isn't implements Element - that's why Element can't be cast to Document.
Anyway, expcetion line number can determine the exact error position.