I wanna save all web page including .css .js on android by programmatically.
So far I tried html get method and jsoup , webview content but all of them I could not save all page with css and js. These methods just save html parts of WEB Page. When I save the all page ,I want to open it offline.
Thanks in advance
You have to take the html, parse it and get the urls of the resources and then make requests for those urls too.
public class Stack {
private static final String USER_AGENT = "";
private static final String INITIAL_URL = "";
public static void main(String args[]) throws Exception {
Document doc = Jsoup
.connect(INITIAL_URL)
.userAgent(USER_AGENT)
.get();
Elements scripts = doc.getElementsByTag("script");
Elements css = doc.getElementsByTag("link");
for(Element s : scripts) {
String url = s.absUrl("src");
if(!url.isEmpty()) {
System.out.println(url);
Document docScript = Jsoup
.connect(url)
.userAgent(USER_AGENT)
.ignoreContentType(true)
.get();
System.out.println(docScript);
System.out.println("--------------------------------------------");
}
}
for(Element c : css) {
String url = c.absUrl("href");
String rel = c.attr("rel") == null ? "" : c.attr("rel");
if(!url.isEmpty() && rel.equals("stylesheet")) {
System.out.println(url);
Document docScript = Jsoup
.connect(url)
.userAgent(USER_AGENT)
.ignoreContentType(true)
.get();
System.out.println(docScript);
System.out.println("--------------------------------------------");
}
}
}
}
I have similar problem...
Using this code we can get images,.css,.js. However some html contents are still missing.
For instance when we save a web page via chrome,there are 2 options.
Complete html
html only
Out of .css,.js,.php..."Complete html" consists of more elements than "only html". The requirement is to download the html as complete like chrome does in the first option.
Related
Below is the HTML tag for the audio link url,
And below is the logic I used to get the URL,
String url = "https://www.dictionary.com/browse/happy?s=t";
Document doc = Jsoup.connect(url).get();
Elements wordBlock = doc.getElementsByClass("e16867sm0");
Element e = wordBlock.get(0);
Elements audioSection = e.getElementsByClass("e1rg2mtf7");
String audioUrl = audioSection.get(0).attr("audio");
But still I am unable to get the URL,
How can we get the URL of audio by using class id.
You can use either doc.select("source[type=audio/ogg]" for the first file or doc.select("source[type=audio/mpeg]" for the second one.
You can also use source[type^=audio] - it will give you both.
I am trying to convert iOS application into android. But I just start learning Java a few days ago. I'm trying to get a value from a tag inside html.
Here is my swift code:
if let url = NSURL(string: "http://www.example.com/") {
let htmlData: NSData = NSData(contentsOfURL: url)!
let htmlParser = TFHpple(HTMLData: htmlData)
//the value which i want to parse
let nPrice = htmlParser.searchWithXPathQuery("//div[#class='round-border']/div[1]/div[2]") as NSArray
let rPrice = NSMutableString()
//Appending
for element in nPrice {
rPrice.appendString("\n\(element.raw)")
}
let raw = String(NSString(string: rPrice))
//the value without trimming
let stringPrice = raw.stringByReplacingOccurrencesOfString("<[^>]+>", withString: "", options: .RegularExpressionSearch, range: nil)
//result
let trimPrice = stringPrice.stringByReplacingOccurrencesOfString("^\\n*", withString: "", options: .RegularExpressionSearch)
}
Here is my Java code using Jsoup
public class Quote extends Activity {
TextView price;
String tmp;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_quote);
price = (TextView) findViewById(R.id.textView3);
try {
doc = Jsoup.connect("http://example.com/").get();
Element content = doc.getElementsByTag("//div[#class='round-border']/div[1]/div[2]");
} catch (IOException e) {
//e.printStackTrace();
}
}
}
My problems are as following:
I got NetworkOnMainThreatException whenever i tried any codes.
I'm not sure that using getElementByTag with this structure is correct.
Please help,
Thanks.
I got NetworkOnMainThreatException whenever i tried any codes.
You should use Volley instead of Jsoup. It will be a faster and more efficient alternative. See this answer for some sample code.
I'm not sure that using getElementByTag with this structure is correct.
Element content = doc.getElementsByTag("//div[#class='round-border']/div[1]/div[2]");
Jsoup doesn't understand xPath. It works with CSS selectors instead.
The above line of code can be corrected like this:
Elements divs = doc.select("div.round-border > div:nth-child(1) > div:nth-child(2)");
for(Element div : divs) {
// Process each div here...
}
it's needs to insert loaded from url html data into webview.
my code
private String getHtmlFromURLJsoup(String url) throws IOException{
Document doc = Jsoup.connect(url)
.userAgent("Mozilla")
.cookie("auth", "token")
.timeout(10000)
.get();
return doc.html();
}
mWebView.loadUrl("javascript:(function () { " +
"setMainContent('" + getHtmlFromURLJsoup(s).replaceAll("\n", "").replaceAll("'", "") + "');" +
"})()");
Content loaded fine, but images not displayed. How can I load html from url with data (images, styles, scripts...)?
Try this method from WebView:
loadDataWithBaseURL(null, htmlBody, "text/html", "UTF-8", null);
htmlBody - your html downloaded data.
Also try to change String to CharSequence in your method:
private CharSequence getHtmlFromURLJsoup(String url) throws IOException{
Are you asking how to implement your own webbrowser?
The html will have links to those other things (images, styles, scripts...) - you need to locate the links in the html and go and fetch them separately - this is what a web browser does.
I'm making an Android app for my board community. The board provider gives me RSS feeds from general categories but don't generate feeds from topics. So I retreive topics URLs from these feeds and want to parse HTML with Jsoup and give it to a WebView.
It works nice except with the select() function which returns nothing.
The "HTML RETREIVED" log gives me : <html><head><title>The topic title</title></head><body></body></html>
h1 tags are in the code on test purpose : it displays well on WebView and the title of the parsed webpage too.
I also putted the log line right after the select() line. It returns nothing too.
I've tried in a pure Java project to parse with Jsoup only and it goes well.
So I assumed something's wrong with Android.
PS : Internet permission is active in the manifest.
Did I miss something ?
Here is the code :
String html;
Bundle param = this.getIntent().getExtras();
String url = param.getString("url");
try {
Document doc = Jsoup.connect(url).get();
doc.select(".topic .clear").remove();
String title = doc.title().toString();
html = doc.select(".username strong, .entry-content").toString();
html = "<html><head><title>"+title+"</title></head><body><h1>"+title+"</h1>"+html+"</body></html>";
WebView webview = new WebView(this);
getWindow().requestFeature(Window.FEATURE_PROGRESS);
setContentView(webview);
webview.getSettings().setJavaScriptEnabled(true);
final Activity activity = this;
webview.setWebChromeClient(new WebChromeClient() {
public void onProgressChanged(WebView view, int progress) {
activity.setProgress(progress * 1000);
Log.d("LOADING",""+ progress);
}
});
webview.loadData(html, "text/html", "UTF-8");
//webview.loadUrl(url);
Log.i("HTML RETREIVED", ""+html);
} catch (IOException e) {
Log.e("ERROR", "Error while generate topic");
}
Ok I've found out something interesting.
The class I wanted to select was not here because I'm getting the mobile version of the webpage. It appears Android App use a mobile user-agent, which is quite normal but not said anywhere.
Anyway I know what thinking about now.
I am trying to parse HTML in android from a webpage, and since the webpage it not well formed, I get SAXException.
Is there a way to parse HTML in Android?
I just encountered this problem. I tried a few things, but settled on using JSoup. The jar is about 132k, which is a bit big, but if you download the source and take out some of the methods you will not be using, then it is not as big.
=> Good thing about it is that it will handle badly formed HTML
Here's a good example from their site.
File input = new File("/tmp/input.html");
Document doc = Jsoup.parse(input, "UTF-8", "http://example.com/");
//http://jsoup.org/cookbook/input/load-document-from-url
//Document doc = Jsoup.connect("http://example.com/").get();
Element content = doc.getElementById("content");
Elements links = content.getElementsByTag("a");
for (Element link : links) {
String linkHref = link.attr("href");
String linkText = link.text();
}
Have you tried using Html.fromHtml(source)?
I think that class is pretty liberal with respect to source quality (it uses TagSoup internally, which was designed with real-life, bad HTML in mind). It doesn't support all HTML tags though, but it does come with a handler you can implement to react on tags it doesn't understand.
String tmpHtml = "<html>a whole bunch of html stuff</html>";
String htmlTextStr = Html.fromHtml(tmpHtml).toString();
We all know that programming have endless possibilities.There are numbers of solutions available for a single problem so i think all of the above solutions are perfect and may be helpful for someone but for me this one save my day..
So Code goes like this
private void getWebsite() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
Document doc = Jsoup.connect("http://www.ssaurel.com/blog").get();
String title = doc.title();
Elements links = doc.select("a[href]");
builder.append(title).append("\n");
for (Element link : links) {
builder.append("\n").append("Link : ").append(link.attr("href"))
.append("\n").append("Text : ").append(link.text());
}
} catch (IOException e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
result.setText(builder.toString());
}
});
}
}).start();
}
You just have to call the above function in onCreate Method of your MainActivity
I hope this one is also helpful for you guys.
Also read the original blog at Medium
Maybe you can use WebView, but as you can see in the doc WebView doesn't support javascript and other stuff like widgets by default.
http://developer.android.com/reference/android/webkit/WebView.html
I think that you can enable javascript if you need it.