I am trying to parse HTML in android from a webpage, and since the webpage it not well formed, I get SAXException.
Is there a way to parse HTML in Android?
I just encountered this problem. I tried a few things, but settled on using JSoup. The jar is about 132k, which is a bit big, but if you download the source and take out some of the methods you will not be using, then it is not as big.
=> Good thing about it is that it will handle badly formed HTML
Here's a good example from their site.
File input = new File("/tmp/input.html");
Document doc = Jsoup.parse(input, "UTF-8", "http://example.com/");
//http://jsoup.org/cookbook/input/load-document-from-url
//Document doc = Jsoup.connect("http://example.com/").get();
Element content = doc.getElementById("content");
Elements links = content.getElementsByTag("a");
for (Element link : links) {
String linkHref = link.attr("href");
String linkText = link.text();
}
Have you tried using Html.fromHtml(source)?
I think that class is pretty liberal with respect to source quality (it uses TagSoup internally, which was designed with real-life, bad HTML in mind). It doesn't support all HTML tags though, but it does come with a handler you can implement to react on tags it doesn't understand.
String tmpHtml = "<html>a whole bunch of html stuff</html>";
String htmlTextStr = Html.fromHtml(tmpHtml).toString();
We all know that programming have endless possibilities.There are numbers of solutions available for a single problem so i think all of the above solutions are perfect and may be helpful for someone but for me this one save my day..
So Code goes like this
private void getWebsite() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
Document doc = Jsoup.connect("http://www.ssaurel.com/blog").get();
String title = doc.title();
Elements links = doc.select("a[href]");
builder.append(title).append("\n");
for (Element link : links) {
builder.append("\n").append("Link : ").append(link.attr("href"))
.append("\n").append("Text : ").append(link.text());
}
} catch (IOException e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
result.setText(builder.toString());
}
});
}
}).start();
}
You just have to call the above function in onCreate Method of your MainActivity
I hope this one is also helpful for you guys.
Also read the original blog at Medium
Maybe you can use WebView, but as you can see in the doc WebView doesn't support javascript and other stuff like widgets by default.
http://developer.android.com/reference/android/webkit/WebView.html
I think that you can enable javascript if you need it.
Related
I am trying to convert iOS application into android. But I just start learning Java a few days ago. I'm trying to get a value from a tag inside html.
Here is my swift code:
if let url = NSURL(string: "http://www.example.com/") {
let htmlData: NSData = NSData(contentsOfURL: url)!
let htmlParser = TFHpple(HTMLData: htmlData)
//the value which i want to parse
let nPrice = htmlParser.searchWithXPathQuery("//div[#class='round-border']/div[1]/div[2]") as NSArray
let rPrice = NSMutableString()
//Appending
for element in nPrice {
rPrice.appendString("\n\(element.raw)")
}
let raw = String(NSString(string: rPrice))
//the value without trimming
let stringPrice = raw.stringByReplacingOccurrencesOfString("<[^>]+>", withString: "", options: .RegularExpressionSearch, range: nil)
//result
let trimPrice = stringPrice.stringByReplacingOccurrencesOfString("^\\n*", withString: "", options: .RegularExpressionSearch)
}
Here is my Java code using Jsoup
public class Quote extends Activity {
TextView price;
String tmp;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_quote);
price = (TextView) findViewById(R.id.textView3);
try {
doc = Jsoup.connect("http://example.com/").get();
Element content = doc.getElementsByTag("//div[#class='round-border']/div[1]/div[2]");
} catch (IOException e) {
//e.printStackTrace();
}
}
}
My problems are as following:
I got NetworkOnMainThreatException whenever i tried any codes.
I'm not sure that using getElementByTag with this structure is correct.
Please help,
Thanks.
I got NetworkOnMainThreatException whenever i tried any codes.
You should use Volley instead of Jsoup. It will be a faster and more efficient alternative. See this answer for some sample code.
I'm not sure that using getElementByTag with this structure is correct.
Element content = doc.getElementsByTag("//div[#class='round-border']/div[1]/div[2]");
Jsoup doesn't understand xPath. It works with CSS selectors instead.
The above line of code can be corrected like this:
Elements divs = doc.select("div.round-border > div:nth-child(1) > div:nth-child(2)");
for(Element div : divs) {
// Process each div here...
}
I wanna save all web page including .css .js on android by programmatically.
So far I tried html get method and jsoup , webview content but all of them I could not save all page with css and js. These methods just save html parts of WEB Page. When I save the all page ,I want to open it offline.
Thanks in advance
You have to take the html, parse it and get the urls of the resources and then make requests for those urls too.
public class Stack {
private static final String USER_AGENT = "";
private static final String INITIAL_URL = "";
public static void main(String args[]) throws Exception {
Document doc = Jsoup
.connect(INITIAL_URL)
.userAgent(USER_AGENT)
.get();
Elements scripts = doc.getElementsByTag("script");
Elements css = doc.getElementsByTag("link");
for(Element s : scripts) {
String url = s.absUrl("src");
if(!url.isEmpty()) {
System.out.println(url);
Document docScript = Jsoup
.connect(url)
.userAgent(USER_AGENT)
.ignoreContentType(true)
.get();
System.out.println(docScript);
System.out.println("--------------------------------------------");
}
}
for(Element c : css) {
String url = c.absUrl("href");
String rel = c.attr("rel") == null ? "" : c.attr("rel");
if(!url.isEmpty() && rel.equals("stylesheet")) {
System.out.println(url);
Document docScript = Jsoup
.connect(url)
.userAgent(USER_AGENT)
.ignoreContentType(true)
.get();
System.out.println(docScript);
System.out.println("--------------------------------------------");
}
}
}
}
I have similar problem...
Using this code we can get images,.css,.js. However some html contents are still missing.
For instance when we save a web page via chrome,there are 2 options.
Complete html
html only
Out of .css,.js,.php..."Complete html" consists of more elements than "only html". The requirement is to download the html as complete like chrome does in the first option.
I need to get information from this website: http://rowans.diekantankys.nl/bonnen/index.php?id=4 (It's in dutch)
From line 36 and on is a table in which you can see the debt op people on this website:
<td>Marc</td>
<td>16.75</td>
</tr> <tr>
<td>Marlieke</td>
<td>7.27</td>
</tr> <tr>
<td>Anne Ruth</td>
<td>4.70</td>
but all the functions and methods that I found that should download an HTML file from a website/web-server to a string/array somehow fail, can anyone give me a method on which I can give my full error report?
My apology's if this is considered: "Not a real question", I don't know how to formulate this
Thanks in advance
I would recommend to use (http://jsoup.org "JSoup") to download the parse the HTML from URL
You can get the HTML as document and read the text on the elements
Document doc = Jsoup.connect("http://rowans.diekantankys.nl/bonnen/index.php?id=4").get();
In you case on this website , you need to get the text in the table body i.e., tbody
String table = doc.body().getElementsByTag("tbody").text()
So, you need to download the content first in an background thread and then update the TextView on UIThread.
new AsyncTask<Void, Integer, Long>(){
#Override
protected Long doInBackground(Void... params) {
try {
final Document doc = Jsoup.connect("http://rowans.diekantankys.nl/bonnen/index.php?id=4").get();
runOnUiThread(new Runnable() {
#Override
public void run() {
String tableContent = doc.body().getElementsByTag("tbody").text();
// you can split the text and read it, as required.
textView.setText(tableContent);
}
});
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}.execute();
Hope it helps you. Let me know if any issue.
You have to download all the HTML content, put it into a string and parse manually the code. If you are interested in the values into that table, I would suggest you to search into the string that contains the downloaded HTML for a uniqui piece of string that identifies the beginning of the table (eg. search for '' tag and you put the HTML section of code of the table into a different variable. Then you proceed to parse manually that string searching for and and extracting the values with a loop. Another way would be use regular expressions but it becames more difficult if you're not faimiliar with them. It's not a relevant resource but in this android app, I did exactly what just explained https://github.com/rexromae/mytotem_android/blob/master/com.torvergata.mytotem/src/com/torvergata/mytotem/student/StudentLogin.java
I'm making an Android app for my board community. The board provider gives me RSS feeds from general categories but don't generate feeds from topics. So I retreive topics URLs from these feeds and want to parse HTML with Jsoup and give it to a WebView.
It works nice except with the select() function which returns nothing.
The "HTML RETREIVED" log gives me : <html><head><title>The topic title</title></head><body></body></html>
h1 tags are in the code on test purpose : it displays well on WebView and the title of the parsed webpage too.
I also putted the log line right after the select() line. It returns nothing too.
I've tried in a pure Java project to parse with Jsoup only and it goes well.
So I assumed something's wrong with Android.
PS : Internet permission is active in the manifest.
Did I miss something ?
Here is the code :
String html;
Bundle param = this.getIntent().getExtras();
String url = param.getString("url");
try {
Document doc = Jsoup.connect(url).get();
doc.select(".topic .clear").remove();
String title = doc.title().toString();
html = doc.select(".username strong, .entry-content").toString();
html = "<html><head><title>"+title+"</title></head><body><h1>"+title+"</h1>"+html+"</body></html>";
WebView webview = new WebView(this);
getWindow().requestFeature(Window.FEATURE_PROGRESS);
setContentView(webview);
webview.getSettings().setJavaScriptEnabled(true);
final Activity activity = this;
webview.setWebChromeClient(new WebChromeClient() {
public void onProgressChanged(WebView view, int progress) {
activity.setProgress(progress * 1000);
Log.d("LOADING",""+ progress);
}
});
webview.loadData(html, "text/html", "UTF-8");
//webview.loadUrl(url);
Log.i("HTML RETREIVED", ""+html);
} catch (IOException e) {
Log.e("ERROR", "Error while generate topic");
}
Ok I've found out something interesting.
The class I wanted to select was not here because I'm getting the mobile version of the webpage. It appears Android App use a mobile user-agent, which is quite normal but not said anywhere.
Anyway I know what thinking about now.
I don't know if you already tested the Google IO application, but there is a cool feature displaying all the tweets including Google IO hashtags.
I really would like to offer the same feature to my users.
I can do something similar using the API, but I would have to create a custom listview, parsing XML/JSON feeds and that's quite overcomplicated! and of course this list will not update automatically and be a livefeed.
In the application, I have just seen that when I turn off wifi, This is indeed a webview with this url:
http://www.google.com/search?%20tbs=mbl%3A1&hl=en&source=hp&biw=1170&bih=668&q=%23io2011&btnG=Search
Here is a screenshot of the app and the same url in the browser
High resolution picture: http://cl.ly/3q1r0c2J3H163E3G2p2X
But using this url in a webview display only a google search, and does not offer same feature.
I know this app will certainly be opensources, but i am so negative about "within next days" that google promise.
We are still waiting for the Twitter app source code!
If you wait 'til after the conference is over, you'll find the source code for the app here. You'll also find last year's application's source code there.
Update:
Just viewed the source code, and you're almost right. It's a webview with this URL: http://www.google.com/search?tbs=mbl%3A1&hl=en&source=hp&biw=1170&bih=668&q=%23io2011&btnG=Search so it just seems you put %20 in there by accident maybe.
Code:
public static final String EXTRA_QUERY = "com.google.android.iosched.extra.QUERY";
public static final String CONFERENCE_HASHTAG = "#io2011";
private String mSearchString;
//onCreate()
final Intent intent = BaseActivity.fragmentArgumentsToIntent(getArguments());
mSearchString = intent.getStringExtra(EXTRA_QUERY);
if (TextUtils.isEmpty(mSearchString)) {
mSearchString = CONFERENCE_HASHTAG;
}
if (!mSearchString.startsWith("#")) {
mSearchString = "#" + mSearchString;
}
//onCreateView
mWebView = (WebView) root.findViewById(R.id.webview);
mWebView.post(new Runnable() {
public void run() {
mWebView.getSettings().setJavaScriptEnabled(true);
mWebView.getSettings().setJavaScriptCanOpenWindowsAutomatically(false);
try {
mWebView.loadUrl(
"http://www.google.com/search?tbs="
+ "mbl%3A1&hl=en&source=hp&biw=1170&bih=668&q="
+ URLEncoder.encode(mSearchString, "UTF-8")
+ "&btnG=Search");
} catch (UnsupportedEncodingException e) {
Log.e(TAG, "Could not construct the realtime search URL", e);
}
}
});
Probably implemented with Loaders API with throttling.
Impatiently waiting for source code as well.