I'm developing a simple Android application that has to scrape content of one website and then add this to the URL of another website that I have to scrape. The first scraping is okay, but the second gives me an error: "HTTP error fetching URL". The second URL is fine because when I give it to the first scraper everything is okay (but the URL is changing, so I cannot use it as a constraint). I think that I have to finish the thread of the first scraper somehow, but I don't know how.
My current code:
private void getBodyText() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
String url="http://example.com";
Document doc = Jsoup.connect(url).get();
Element body = doc.body();
builder.append(body.text());
} catch (Exception e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
textView.setText(builder.toString());
token1=builder.toString();
checkPairing();
}
});
}
}).start();
}
private void checkPairing() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
String url="http:example.com/"+token1;
Document doc = Jsoup.connect(url).get();
Element body = doc.body();
builder.append(body.text());
} catch (Exception e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
if (builder.toString()!="Not Found") {
tokenPar = builder.toString();
}
}
});
}
}).start();
}
I've also tried to execute the second function directly from onCreate() after the first scraper is executed but nothing changed.
Related
I just want to read (-1) value which is the only value in website but I am not able to fetch this value Can any buddy guide me how to do that. I am getting the value of API url when I am pasting it in web browser and android is not fetching it. Link of API is given below.
Link OF API
Code I am trying is:
private void getWebsite() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
Document doc = Jsoup.connect("https://api.thingspeak.com/channels/411172/feeds.json?results=1").get();
String title = doc.title();
Elements links = doc.select("[pre]");
builder.append(title).append("\n");
for (Element link : links) {
builder.append("\n").append("Link : ").append(link.attr("pre"))
.append("\n").append("Text : ").append(link.text());
}
} catch (IOException e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
result.setText(builder.toString());
}
});
}
}).start();
}
Screen shot of web API call in browser
Error I am getting is Error: HTTP error fetching URL
As I see response from your API is JSON formatted you have to modify your code as follow
private void getWebsite() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
Document doc = Jsoup.connect("https://api.thingspeak.com/channels/411172/feeds.json?results=1")
.ignoreContentType(true)
.get();
String json = doc.body().text();
builder.append(json);
} catch (IOException e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
result.setText(builder.toString());
}
});
}
}).start();
}
I load my articles from my wordpress webpage. And want to display them in my app.
In order to be allowed to wait for a internet response I had to create a Thread.
Using this method:
private void loadArticles(final String url) {
Thread thread = new Thread(new Runnable(){
#Override
public void run() {
try {
//Get my data and run createArticles();
} catch (Exception e) {
e.printStackTrace();
}
}
});
thread.start();
}
Then in the createArticles function I try to use my function addViewToParent(...) and I get an error that says:
Only the original thread that created a view hierarchy can touch its
views.
How can i work around this problem?
EDIT: I use these networking functions if it matters...
//Connect to the url
URL pageURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) pageURL.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(connection.getInputStream())
);
If you want to modify your Views, e.g. show your article, you have to run the code on the UI/main thread. To achieve this, you could call run runOnUiThread() in your Activity:
private void loadArticles(final String url) {
Thread thread = new Thread(new Runnable(){
#Override
public void run() {
try {
//Get my data ...
runOnUiThread(new Runnable() {
public void run() {
// modify Views here
createArticles();
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
});
thread.start();
}
You need to use function runOnUiThread. This is function for user interface thread.
private void loadArticles(final String url) {
//Get my data ...
runOnUiThread(new Runnable() {
public void run() {
// modify Views here
createArticles();
}
});
});
Just use a Handler instead, so you'll have access to the UI Thread:
new Handler().post(new Runnable(){
#Override
public void run() {
try {
//Get my data and run createArticles();
} catch (Exception e) {
e.printStackTrace();
}
}
});
I have list of web pages(over 100) with I have to vistit and collect data from.
I decided to save the html from all of them to one file, and then use Jsoup to find the interesting data.
But problem is to I do not know how to run 100 threads, and save the responses into one file, any ideas?
maybe it's not a masterpiece, but it works, and I wanted to make it as simple as possible.
ArrayList<String> links = new ArrayList<>();
Elements myDiv;
private void saveDetails() throws IOException {
if(repeat < links.size()){
repeat++;
textView.setText(String.valueOf(repeat));
saveFile(myDiv.toString());
myDiv = null;
getDetails(links.get(repeat));
}else {
textView.setText("finished");
}
}
private void getDetails(String urlStr) {
final String detailsUrl = urlStr;
new Thread() {
#Override
public void run() {
Message msg = Message.obtain();
try {
Document doc = Jsoup.connect(detailsUrl).get();
myDiv = doc.select(".exhibitor-contact");
} catch (IOException e1) {
e1.printStackTrace();
}
detailsHandler.sendMessage(msg);
}
}.start();
}
private Handler detailsHandler = new Handler() {
public void handleMessage(Message msg) {
super.handleMessage(msg);
try {
saveDetails();
} catch (IOException e) {
e.printStackTrace();
}
}
};
You don't need to save all of them in a file and then process them. You can gather information one by one. It is my suggestion:
arrayList urls = {100 site-url}; //in correct syntax
Document doc = null;
for (String url : urls) {
doc = Jsoup.connect(url).get();
//now proccess doc.toString as you want(in regular expression for example)
//save your desired information
}
I'm creating a subroutine to get an image using a url. Here is a similar subroutine that I used to parse text from an XML into my UI:
public void loading(int index) {
final Handler handler = new Handler();
new Thread(new Runnable() {
#Override
public void run() {
try {
arrayList = XMLPullParser.parse("http://localhost/news.xml", "news", 1, 2, 3);
//URL url = new URL("http://localhost/"+arrayList2.get(1).toString());
//final Bitmap bmp = BitmapFactory.decodeStream(url.openConnection().getInputStream());
handler.post(new Runnable() {
public void run() {
chk1.setText(arrayList.get(0).toString());
chk2.setText(arrayList.get(1).toString());
chk3.setText(arrayList.get(2).toString());
try {
img1.setImageBitmap(bitmap("http://localhost/1.jpg"));
img2.setImageBitmap(bitmap("http://localhost/2.jpg"));
img3.setImageBitmap(bitmap("http://localhost/3.jpg"));
} catch (IOException e) {
e.printStackTrace();
}
}
});
} catch (URISyntaxException e) {
e.printStackTrace();
}
}
}).start();
}
Here is the change I made to load the images:
static Bitmap bmp;
public Bitmap bitmap(String url_input) throws IOException {
final String get_url = url_input;
new Thread(new Runnable() {
#Override
public void run() {
try {
final URL url = new URL(get_url);
bmp = BitmapFactory.decodeStream(url.openConnection().getInputStream());
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
return bmp;
}
The subroutine works when parsing text but doesn't work properly when loading the bitmaps. I need to set images to 3 image buttons and want to use a subroutine to do it.
I know it should not use static for my bmp variable, it's just a test to let me know if it works.
Your bitmaps are not displayed because your threading setup is incorrect. Your bitmap method returns before thread it started has finished. Also you are using static variable used by all threads.
I recommend to use AsyncTask and do all your background work there - there is no need to spawn separate threads for every image. Also pay attention to assign values to your checkboxes and images in onPostExecute method because these can be assigned only in main (gui) thread.
You could do it something like this. This asserts that you have the code in an activity. But the thread handling should probably be changes to use an AsyncTask instead.
bitmap(img1, "http://localhost/1.jpg");
.
.
.
public void bitmap(final ImageView imageView, String url_input) throws IOException {
final String get_url = url_input;
new Thread(new Runnable() {
#Override
public void run() {
try {
final URL url = new URL(get_url);
final Bitmap bmp = BitmapFactory.decodeStream(url.openConnection().getInputStream());
if (bmp != null) {
runOnUiThread(new Runnable() {
#Override
public void run() {
imageView.setImageBitmap(bmp);
}
});
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
}
I need to use this api,
"http://sconysoft.com/api/md5.php?q="
for doing something like this app :
https://play.google.com/store/apps/details?id=com.sconysoft.md5encoderdecoder&hl=en
Actually, i have one button with Two EditText in Android app, just i need send first string to this api, and if this hash encoded(in url) exist, show that in Edit Text2.
and this is above app Codec class :
public class Codec {
public String decode(String md5) throws Exception {
try {
HttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(new HttpGet("http://sconysoft.com/api/md5.php?q=" + md5));
BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
JSONObject jo = new JSONObject(br.readLine());
if(jo.getString("md5").equals(""))
throw new Exception();
return jo.getString("word");
} catch (Exception e) {
}
throw new Exception("Can't decode given md5");
}
}
main :
public boolean isNetworkAvailable() {
ConnectivityManager cm = (ConnectivityManager)
getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo networkInfo = cm.getActiveNetworkInfo();
if (networkInfo != null && networkInfo.isConnected()) {
return true;
}
return false;
}
public void decode(View view) {
final EditText etWord = (EditText)findViewById(R.id.edit1);
final EditText etMd5 = (EditText)findViewById(R.id.edit2);
if(!isNetworkAvailable()) {
etWord.setText("Network unavailable");
return;
}
final ProgressDialog pd = ProgressDialog.show(view.getContext(),"Waiting for Server", "It should take a few seconds");
Thread th = new Thread() {
#Override
public void run() {
try {
Codec cd = new Codec();
String md5 = etMd5.getText().toString();
try {
final String word = cd.decode(md5);
runOnUiThread(new Runnable() {
public void run() {
etWord.setText(word);
pd.dismiss();
}
});
} catch (final Exception e) {
runOnUiThread(new Runnable() {
public void run() {
etWord.setText(e.getMessage());
pd.dismiss();
}
});
}
} catch(Exception e) {
runOnUiThread(new Runnable() {
public void run() {
pd.dismiss();
}
});
}
}
};
th.start();
}
}
i also try this codes with same situation in developing, but unfortunately, this codes doesn't work and i cannot see this buttons codes or other something on this app codes.
Can someone help me, how to doing this with one button and Two EditTexts?
Need some help about doing this with above API.
I am assuming
public void decode(View view)
gets called from the button onClick?
I suggest doing the network call (and managing the progress dialog) in an AsyncTask. Examples:
Using AsyncTask for android network connection
How to use AsyncTask to show a ProgressDialog while doing background work in Android?
You update the second EditText (you could use TextView) in the AsyncTask's onPostExecute.