I have list of web pages(over 100) with I have to vistit and collect data from.
I decided to save the html from all of them to one file, and then use Jsoup to find the interesting data.
But problem is to I do not know how to run 100 threads, and save the responses into one file, any ideas?
maybe it's not a masterpiece, but it works, and I wanted to make it as simple as possible.
ArrayList<String> links = new ArrayList<>();
Elements myDiv;
private void saveDetails() throws IOException {
if(repeat < links.size()){
repeat++;
textView.setText(String.valueOf(repeat));
saveFile(myDiv.toString());
myDiv = null;
getDetails(links.get(repeat));
}else {
textView.setText("finished");
}
}
private void getDetails(String urlStr) {
final String detailsUrl = urlStr;
new Thread() {
#Override
public void run() {
Message msg = Message.obtain();
try {
Document doc = Jsoup.connect(detailsUrl).get();
myDiv = doc.select(".exhibitor-contact");
} catch (IOException e1) {
e1.printStackTrace();
}
detailsHandler.sendMessage(msg);
}
}.start();
}
private Handler detailsHandler = new Handler() {
public void handleMessage(Message msg) {
super.handleMessage(msg);
try {
saveDetails();
} catch (IOException e) {
e.printStackTrace();
}
}
};
You don't need to save all of them in a file and then process them. You can gather information one by one. It is my suggestion:
arrayList urls = {100 site-url}; //in correct syntax
Document doc = null;
for (String url : urls) {
doc = Jsoup.connect(url).get();
//now proccess doc.toString as you want(in regular expression for example)
//save your desired information
}
Related
I'm developing a simple Android application that has to scrape content of one website and then add this to the URL of another website that I have to scrape. The first scraping is okay, but the second gives me an error: "HTTP error fetching URL". The second URL is fine because when I give it to the first scraper everything is okay (but the URL is changing, so I cannot use it as a constraint). I think that I have to finish the thread of the first scraper somehow, but I don't know how.
My current code:
private void getBodyText() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
String url="http://example.com";
Document doc = Jsoup.connect(url).get();
Element body = doc.body();
builder.append(body.text());
} catch (Exception e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
textView.setText(builder.toString());
token1=builder.toString();
checkPairing();
}
});
}
}).start();
}
private void checkPairing() {
new Thread(new Runnable() {
#Override
public void run() {
final StringBuilder builder = new StringBuilder();
try {
String url="http:example.com/"+token1;
Document doc = Jsoup.connect(url).get();
Element body = doc.body();
builder.append(body.text());
} catch (Exception e) {
builder.append("Error : ").append(e.getMessage()).append("\n");
}
runOnUiThread(new Runnable() {
#Override
public void run() {
if (builder.toString()!="Not Found") {
tokenPar = builder.toString();
}
}
});
}
}).start();
}
I've also tried to execute the second function directly from onCreate() after the first scraper is executed but nothing changed.
I'm trying to send a variable string to Runnable, but find myself having to create two variables for this. One normal string content and one final string finalcontent that is assigned the content of the other string.
String content = "";
try {
content = response.body().string();
} catch (IOException e) {
// Handle exception
}
final String finalcontent = content;
runOnUiThread(new Runnable() {
#Override
public void run() {
TextView text = (TextView)findViewById(R.id.textView);
text.setText(finalcontent);
}
});
Is this the normal way to do this or is there a better way to avoid creating two variables?
it's limitation of Java language design but you should be able to write
final String content;
try {
content = response.body().string();
} catch (IOException e) {
content = "";
// Handle exception
}
runOnUiThread(new Runnable() {
#Override
public void run() {
TextView text = (TextView)findViewById(R.id.textView);
text.setText(content);
}
});
You may create a custom implementation of Runnable, here is an example
void foo(){
String content = "";
try {
content = response.body().string();
} catch (IOException e) {
// Handle exception
}
runOnUiThread(new MyCustomRunnable(content));
}
private class MyCustomRunnable implements Runnable {
private String content;
public MyCustomRunnable(String content) {
this.content = content;
}
#Override
public void run() {
TextView text = (TextView)findViewById(R.id.textView);
if (text != null) {
text.setText(content);
}
}
}
Why don't use RxJava?I think you just want to update UI after you get data from internet.
Observable observable = Observable.create(OnSubscribe()...)
.subscribeOn(Shceduler.io).observeOn(AndroidSchedulers.mainthread)
I want when server sends some response in form of WebView then immediately my activity gets refreshed and so WebView in form of banner ad.
I write code for display banner ad but ad is showing only when my activity recreated i.e. when I rotate my screen then banner is showing but when it is in same static mode then banner is not showing.
So, please let me know what I will do so that when server gave some response immediately it will be shown on my activity.
void startDemo() {
//Set Http Client Options
final OptimusHTTP client = new OptimusHTTP();
client.enableDebugging();
client.setMethod(OptimusHTTP.METHOD_POST);
client.setMode(OptimusHTTP.MODE_SEQ);
FreqDetector_Goertzel.getInstance().startRecording(new FreqDetector_Goertzel.RecordTaskListener() {
private String urlRedirect = "";
private String imgSmallBanner = "";
#Override
public void onSuccess(int val)
{
String pSet = pVal.getPatternSet(val, 5);
if (pSet != null) {
FreqDetector_Goertzel.getInstance().stopRecording();
EasyDeviceInfo deviceInfo = new EasyDeviceInfo(MainActivity.this);
final HashMap<String, String> device_params = new HashMap<>();
device_params.put("aid", deviceInfo.getAndroidID());
device_params.put("pattern", pSet);
if (isNetworkAvailable(MainActivity.this)) {
try {
client.makeRequest(MainActivity.this, new HttpReq(), Defaults.MATCHINGSERVER, device_params, new OptimusHTTP.ResponseListener() {
#Override
public void onSuccess(String s) {
try {
if (s != null && !s.contains("No Match Found"))
{
JSONObject jsonObject = null;
jsonObject = new JSONObject(s);
imgSmallBanner = Uri.decode(jsonObject.optString("smallImgUrl", "NA"));
urlRedirect = Uri.decode(jsonObject.optString("redirectUrl", "NA"));
loadAdvertisement(urlRedirect, imgSmallBanner);
} else {
//Did not match
startDemo();
}
} catch (JSONException e) {
e.printStackTrace();
}
}
#Override
public void onFailure(String s) {
}
});
} catch (Exception e) {
e.printStackTrace();
}
} else {
//Internet not available. Do not do anything.
}
}
}
#Override
public void onFailure(String s) {
}
});
}
void loadAdvertisement(String clickUrl, String imgSmallName) {
String click_url;
String img_small_url;
stopDemo();
click_url = Uri.decode(Uri.encode(clickUrl));
img_small_url = imgSmallName;
StringBuilder htmlData2 = new StringBuilder();
htmlData2.append("<html><body style='margin:0;padding:0;background-color:black;'><a href='").append(click_url).append("' ><img src='").append(img_small_url).append("' height=50 style='margin:0 auto;display:block;' /></a></body></html>");
webView_img_small.loadDataWithBaseURL("file:///android_asset/", htmlData2.toString(), "text/html", "utf-8", null);
webView_img_small.setVisibility(View.VISIBLE);
/* What I will do here so when server sends response it will immediately being refreshed and shown on activity without recreating it.*/ }
here you can find some response: http://developer.android.com/guide/topics/ui/how-android-draws.html
for me a call to invalidate() only refresh the view and a call to requestLayout() refresh the view and compute the size of the view in the screen.
You can try to use Activity.recreate(). This method will destroy your current Activity and create a new Activity same way when you rotate device.
Hope this helps.
I am working on android application. In my app I got the xml data response from server and stored it in a string. Now I need to get each value of that xml and display in a dropdown. How can I do that. Please help me with this. Will be really thankful.
My xml data:
<?xml version="1.0" encoding="utf-8"?>
<root>
<status>first<status>
<description>very good</description>
<Firstnames>
<name>CoderzHeaven</name>
<name>Android</name>
<name>iphone</name>
</Firstnames>
<SecondNames>
<name>Google</name>
<name>Android</name>
</SecondNames>
</root>
I am getting the above mentioned xml data from server. Now I need to display that in listview. How can I get those values using xmlparser. I tried with different examples but it didnt work for me.
You will need to create an extra class and parametrize your adapter with objects of this class, an example data model would look like:
public class DataClass {
private String status, description;
private ArrayList<String> fnames, lnames;
public DataClass() {
fnames = new ArrayList<String>();
lnames = new ArrayList<String>();
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
public String getDescription() {
return description;
}
public void setDescription(String description) {
this.description = description;
}
public ArrayList<String> getFnames() {
return fnames;
}
public ArrayList<String> getLnames() {
return lnames;
}
}
As for the XML parser, there are literally tons of examples, you're definitely in advantage if you can use search. Just to give you a staring point, tutorials one, two, three, four.
If you experience problems, post your efforts and the code that didn't work, what have you tried and so on. Then you'll get help, otherwise nobody on SO is going to write code for you. https://stackoverflow.com/help/how-to-ask
Here's how you can do it if the xml is inside of your apps assets folder.
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
InputStream input = null;
try {
input = getApplicationContext().getAssets().open("data.xml");
} catch (IOException e) {
e.printStackTrace();
}
DocumentBuilder builder = null;
try {
builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
} catch (ParserConfigurationException e) {
e.printStackTrace();
}
Document doc = null;
if (builder == null) {
Log.e("TAG", "Builder is empty.");
return;
}
try {
doc = builder.parse(input);
} catch (SAXException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
if (doc == null) {
Log.e("TAG", "Document is empty.");
return;
}
// Get Firstnames element
Element firstNames = (Element) doc.getElementsByTagName("Firstnames").item(0);
// Get name nodes from Firstnames
NodeList nameNodes = firstNames.getElementsByTagName("name");
// Get count of names inside of Firstnames
int cChildren = nameNodes.getLength();
List<String> names = new ArrayList<String>(cChildren);
for (int i=0; i<cChildren; i++) {
names.add(nameNodes.item(i).getTextContent());
Log.d("TAG","Name: "+names.get(i));
}
// Do same with SecondNames
}
When trying to add words to a database via php, only the first word of both strings gets added.
I send the text via this code:
public void sendTextToDB() {
valcom = editText1.getText().toString();
valnm = editText2.getText().toString();
t = new Thread() {
public void run() {
try {
url = new URL("http://10.0.2.2/HB/hikebuddy.php?function=setcomm&comment="+valcom+"&name="+valnm);
h = (HttpURLConnection)url.openConnection();
if( h.getResponseCode() == HttpURLConnection.HTTP_OK){
is = h.getInputStream();
}else{
is = h.getErrorStream();
}
h.disconnect();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
Log.d("Test", "CONNECTION FAILED 1");
}
}
};
t.start();
}
When tested with spaces and commas etc. in a browser, the php function adds all text.
The strings also return the full value when inserted into a dialog.
How do I fix this?
Thank you.
You need to URL-encode valcom and valnm when putting them into the URL.
See java.net.URLEncoder.encode: http://developer.android.com/reference/java/net/URLEncoder.html
url = new URL("http://10.0.2.2/HB/hikebuddy.php?function=setcomm&comment="
+ URLEncoder.encode(valcom)
+ "&name="+ URLEncoder.encode(valnm));