Develop Reference

Android: extracting website using BufferedReader: Information extracted are outdated

I am reading html source code of a public website using the following code:
Code:
#Override
protected Void doInBackground(Void... params)
{
try
{
URL url = new URL(""+URL);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String inputLine;
PageCode = "";
OriginalPageCode = "";
while ((inputLine = in.readLine()) != null)
{
PageCode += inputLine;
}
OriginalPageCode = PageCode;
try
{
extract_website_and_save(); // extracting data from PageCode
}
catch (Exception e1)
{
}
in.close();
}
Background:
The above code sometimes can fetch the most updated website properly. But occasionally it linked to an outdated version of the website and hence unable to obtain the most updated information for the website.
I am curious why the above will occur, does it related to extracting from cache instead of the real updated website??
I therefore used Chrome to browse the same link, and discovered that Chrome also fetched the outdated website.
I have tried restarting the device, but the problem continues.
After 30 minutes to an hour, I requested the app to fetch again and it then can extract the most updated information. I at the same time browse the website using Chrome, Chrome can now obtain the most updated website.
Question:
The above BufferedReader should have no relationship with Chrome? But they follow the same logic and hence extracting from cache instead of from the most updated website?

I strongly suspect the end point is being cached by URL
Try something like this
urlSrt = urlSrt + "?x=" + new Random().nextInt(100000);
// If your URL already is passing parameters i.e. example.com?x=1&p=pass - then modify
// the urlSrt line to to use an "&" and not "?"
// i.e. urlSrt = urlSrt + "&x=" + new Random().nextInt(100000);
URL url = new URL(urlSrt);
URLConnection con = url.openConnection();
con.setUseCaches(false); //This will stop caching!
So if you modify your code to something like this.
URLConnection con = url.openConnection();
con.setUseCaches(false);
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));

want to build lotus notes leave application for android

I want to build lotus notes leave application on android. For that purpose I need some lotus script files which will provide me data for showing in my app. But first thing what I need is to get server login
But after trying to login I am not getting proper response. I need advice how can I proceed to build the app leave application for ibm lotus notes.
protected static void tryLogin()
{ ``
HttpURLConnection connection;
OutputStreamWriter request = null;
URL url = null;
String response = null;
String parameters = "username="+"ABCD"+"password="+"!!!!!!!!";
try
{
url = new URL("http://10.194.5.33/dvlp/wdcidmanage.nsf/hwlsp?wsdl");
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
// connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestMethod("POST");
request = new OutputStreamWriter(connection.getOutputStream());
request.write(parameters);
request.flush();
request.close();
String line = "";
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
BufferedReader reader = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
// Response from server after login process will be stored in response variable.
response = sb.toString();
System.out.println("response--------------------------"+response);
// You can perform UI operations here
// Toast.makeText(this,"Message from Server: \n"+ response, 0).show();
isr.close();
reader.close();
}
catch(IOException e)
{
// Error
System.out.println("error"+"----------------error is there------------");
}
}
this is my code snippet for login. in the server side what i need to do for login ?

If I understood, your need to consume a Domino WS: http://xx.xxx.x.xx/dvlp/wdcidmanage.nsf/hwlsp?wsdl
Ask the Domino administrator to add Anonymous in the ACL of the wdcidmanage.nsf
Consume the WS in androide: http://www.c-sharpcorner.com/UploadFile/88b6e5/how-to-call-web-service-in-android-using-soap/

For an overview of Domino web server authentication see this article. I wrote the article with Domino REST services in mind, but a lot of it applies to SOAP-based services too. This is because authentication is normally done in a layer that's common to REST and SOAP.
You probably want to start with basic authentication. That means sending an Authorization header with each web service request. The value of the Authorization header is just the base64 encoded user name and password as described in this Wikipedia article.
In your comment you said, "when i am trying to establish connection with it it returns me a html page." That sounds like the server is set up for session authentication. As the first article says, you can set up a web site rule to override session authentication for your web service. Then you will get back an HTTP 401 response when the request isn't properly authenticated.

Android - Google Sign-in on Web API - How to send Google Sign-in POST-request?

I've got an Android project that allows me to log-in with Google-accounts. Everything works as it's supposed to and I'm able to retrieve the Person's (com.google.android.gsm.model.people.Person) data, like Google email, username, profile picture url, etc.
I also have a Web API hosted online. In this Web API I can get a JSON-list of products by using the following: mywebhost/api/products.
Obviously these GET-requests are Authorized by OAuth 2.0, meaning I have to log-in with a Google Account on the mywebhost/Account/Login page to be authorized to get the JSON-list from the Web API. (When I'm not logged in, I receive the following JSON-list: {"$id":"1","Message":"Authorization has been denied for this request."})
I know how to send POST-requests in Android. For example with the code:
public class TaskPostAPI extends AsyncTask<String, Void, String>
{
GoogleApiClient googleAPI;
public TaskPostAPI(GoogleApiClient googleAPI){
this.googleAPI = googleAPI;
}
#Override
protected String doInBackground(String... urls){
String response = "";
for(String url : urls){
DefaultHttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url);
try{
List<NameValuePair> nvPairs = new ArrayList<NameValuePair>(3);
//nvPairs.add(new BasicNameValuePair("personName", Plus.PeopleApi.getCurrentPerson(googleAPI).getDisplayName()));
//nvPairs.add(new BasicNameValuePair("personGooglePlusProfile", Plus.PeopleApi.getCurrentPerson(googleAPI).getUrl()));
//nvPairs.add(new BasicNameValuePair("personEmail", Plus.AccountApi.getAccountName(googleAPI)));
// TODO: Use the correct nvPairs to be able to Log-in with the Google-account
post.setEntity(new UrlEncodedFormEntity(nvPairs));
HttpResponse execute = client.execute(post);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(new InputStreamReader(content));
String s = "";
while((s = buffer.readLine()) != null)
response += s;
}
catch(Exception ex){
ex.printStackTrace();
}
}
return response;
}
#Override
protected void onPostExecute(String result){
// Do nothing yet
}
}
So, now the question:
In the code sample above, what should the nvPairs be to be able to successfully log-in with a Google-account. Or should I use something completely different than normal HttpPost to log-in with a Google-account on my Web API?
Also: The url provided is the default Log-in page. On this page I have three different options to log-in. Since we only want to use the Google log-in, how can I retrieve the url of the Google Log-in button from the Web API Login-page to use for the POST-request?
2b. With the second question: What Plug-in I could use in FireFox to see the links I'm redirected to? So I know which Login-url I should use instead of the default Login-page.
Thanks in advance for the responses.
Edit 1: I've tried a different approach, but I'm not sure it will work for the HttpGet-requests. What I've tried is opening a WebView with the Log-in page, and after I reach the page where I come when I successfully logged-in, I close the WebView.
However, when I use the HttpGet-requests, I still get the Unauthorized JSON back, so how can I use this WebView Log-in to make the HttpGet-request "believe" I'm logged-in and Authorized?
If someone still has an idea using the first approach (HttpPost-request), or if someone has a completely different approach, let me know.

Ok, I've found the POST I can use in the C# web project (/POST/ExternalLogin). There I also see what I should send:
In the header:
Content-Type application/x-www-form-urlencoded
Cookie with __RequestVerificationToken
In the body:
provider ("Google")
returnUrl
__RequestVerificationToken
The second __RequestVerificationToken needs to be a different one than the one used in the Cookie, but after decrypting it in the C# Web API it should be the same. This is the only problem I have right now, but I've got a different stackoverflow question for that.
For the full code:
public class TaskPostAPI extends AsyncTask<String, Void, String>
{
private String TOKEN = "__RequestVerificationToken";
#Override
protected String doInBackground(String... urls){
String response = "";
for(String url : urls){
HttpPost post = new HttpPost(url);
try{
// Add the default Content-type to the Header
post.addHeader("Content-type", "application/x-www-form-urlencoded");
// Get the baseUrl from the given url
URL u = new URL(url);
String baseUrl = u.getProtocol() + "://" + u.getHost();
// POST-request requires anti-forgery Cookie
// Get all Cookies
CookieManager cookieManager = CookieManager.getInstance();
String cookie = cookieManager.getCookie(baseUrl);
String[] cookies = cookie.split(";");
// Put all Cookies in a HashMap with cookieKey & cookieToken
HashMap<String, String> cookieStrings = new HashMap<String, String>();
for(String cook : cookies){
String[] cs = cook.split("=");
cookieStrings.put(cs[0], cs[1]);
}
// Add the Cookie to the Header
post.addHeader("Cookie", TOKEN + "=" + cookieStrings.get(TOKEN) + "");
// POST-request requires cookieToken, provider and returnUrl
List<NameValuePair> nvPairs = new ArrayList<NameValuePair>(3);
nvPairs.add(new BasicNameValuePair(TOKEN, cookieStrings.get(TOKEN)));
nvPairs.add(new BasicNameValuePair("provider", "Google"));
nvPairs.add(new BasicNameValuePair("returnUrl", baseUrl));
post.setEntity(new UrlEncodedFormEntity(nvPairs));
Log.i("COOKIE OUTPUT", TOKEN + "=" + cookieStrings.get(TOKEN) + "");
// Send the POST-request
HttpResponse execute = MainActivity.HttpClient.execute(post);
// Get the response of the POST-request
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(new InputStreamReader(content));
String s = "";
while((s = buffer.readLine()) != null)
response += s;
}
catch(Exception ex){
ex.printStackTrace();
}
}
return response;
}
In this piece of code the following line is incorrect and still needs fixing:
nvPairs.add(new BasicNameValuePair(TOKEN, cookieStrings.get(TOKEN)));
cookieStrings.get(TOKEN) need to be replaced with the correct token to send.

Prestashop and web service(Restful), HttpURLConnection, BufferedReader

I want to access the resources of a site created by prestashop via restful web services, which I enter the URL you must enter a key (that is generated by prestashop when we create a restful web service) in the field of username.
so I am trying to read a xml string:
<?xml version="1.0" encoding="UTF-8"?>
<prestashop>
<manufacturers>
<manufacturer id="1" xlink:href="http://127.0.0.1/test/api/manufacturers/1" />
<manufacturer id="2" xlink:href="http://127.0.0.1/test/api/manufacturers/2" />
</manufacturers>
</prestashop>
over HTTP:
I have the following code:
public class MainTest
{
public static String readUrl(HttpURLConnection conn) throws Exception
{
BufferedReader reader = null;
try
{
reader = new BufferedReader(new InputStreamReader((conn.getInputStream())));
StringBuffer buffer = new StringBuffer();
int read;
char[] chars = new char[1024];
while ((read = reader.read(chars)) != -1)
buffer.append(chars, 0, read);
return buffer.toString();
} finally
{
if (reader != null)
reader.close();
}
}
public static void main(String[] args) throws Exception
{
URL url = new URL("http://127.0.0.1/test/api/manufacturers");
HttpURLConnection conn = null;
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setReadTimeout(30000);
conn.setRequestProperty("Accept", "application/XML");
conn.setRequestProperty("Authentication-Key", "ALHKUNM0J6JJAQC21E4TVWHBM6FAKACF");
System.out.println("true2");
String xml="";
xml = readUrl(conn);
System.out.println(xml);
}
}
but it give me this error
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 401 for URL: http://127.0.0.1/test/api/manufacturers
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at com.test.services.URLReader.main(URLReader.java:28)
i think the problem is in this ligne
reader = new BufferedReader(new InputStreamReader((conn.getInputStream())));
please help me if you have any solution
regards

You request property for authentication is wrong!
First, Prestashop REST API is using basic authentication.
Then you will need to encrypt your credentials based on base64 encryption.
So download commons-codec-1.5.jar from http://commons.apache.org/proper/commons-codec/.
Here is the way I did it.
import org.apache.commons.codec.binary.Base64
//.....
String username = "Your Prestashop webservice key";
String password = "";// leave it empty
String authToBytes = username + ":" + password;
//....
byte authBytes = Base64.encodeBase64(authToBytes.getBytes())// I keep it generic
String authBytesString = new String(authBytes);
//then your code
conn.setRequestProperty("Authorization", "Basic " + authBytesString);
//...
It should work now.
Find a small Prestashop java API at http://www.onasus.com/2012/10/3712/prestashop-web-services-java-api/
I found many many other ways to consume the web services. One of them uses the class java.net.Authenticator which handles the HTTP Basic authentication for you automatically. Find out more at http://examples.javacodegeeks.com/core-java/net/authenticator/access-password-protected-url-with-authenticator/ .

From HTTP Status Codes:
401 Unauthorized
Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided.[2] The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource. See Basic access authentication and Digest access authentication.
Try to find out what kind of authentication your service demands. Often it's easier to play around with curl before you write a program with HttpURLConnection.
curl -v \
-G \
-H 'Accept: application/xml' \
-H 'WWW-Authenticate: Basic 938akjsdfh==' \
http://127.0.0.1/test/api/manufacturers
Another thing: you should always check the response code before accessing getInputStream(). If you encounter something like 4xx, getInputStream() will throw an exception. In this case, maybe you can grab some information of the error if you read from getErrorStream().

Bizarre URL issue When connecting to Guardian API

I am connecting to the Guardian's News feed to download JSON. Every now and then I get get back JavaScript and it causes an error. The URL is stored in a static final string so is always the same. The url that I see in the debugger is correct for JSON. In fact, if I copy the url out of the debugger expressions window and paste it into a browser, I get back JSON. I don't think the problem is with the Guardian. I think something in my connection code might be corrupting the url somehow, and the Guardian is sending me JavaScript as a default. But I really don't know.
Here is the URL that is used in all cases. Notice format-json:
static final String GuardianUrl = "http://content.guardianapis.com/search?format=json&show-fields=headline%2Cbody%2Cthumbnail%2CtrailText%2ClastModified&date-id=date%2Ftoday&api-key=mykey";
Here is the top of their JSON file when it comes back correctly:
{
"response":{
"status":"ok",
"userTier":"approved",
"total":203,
"startIndex":1,
"pageSize":10,
"currentPage":1,
"pages":21,
"orderBy":"newest",
"results":[{
etc.
And this is what I get back when it comes as JavaScript:
09-21 15:00:18.853: E/JSON Parser(22101): Error converting string to json <html><head> <script language='javascript' type='text/javascript'>function init(_frm) { if (_frm.sent.value == 0) { _frm.sent.value=1; _frm.submit(); } }</script></head><body onload=init(auth)><form name=auth action='http://192.168.3.1:10080/ui/dynamic/guest-login.html' METHOD=GET><input type=hidden name='mac_addr' value='e0:75:7d:d3:15:0a'><input type=hidden name='url' value='http://content.guardianapis.com/search?format=json&#38show-fields=headline%2Cbody%2Cthumbnail%2CtrailText%2ClastModified&#38date-id=date%2Ftoday&#38api-key=pdva9u6ac2rqsx9a7hexzrv3'><input type=hidden name='ip_addr' value='192.168.3.142'><input type=hidden id=sent value='0'><noscript><input type=submit value='continue'></noscript></form></body></html> ... etc
And this is the code that connects and downloads the file:
public String getJSONFromUrl(String _url) throws IOException {
URL url = new URL(_url);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
// Making HTTP request
try {
//ByteArrayOutputStream out = new ByteArrayOutputStream();
//InputStream in = connection.getInputStream();
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
return null;
}
String line;
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while((line = reader.readLine()) != null) {
builder.append(line);
}
if (builder.toString().length()>0)
json = builder.toString();
return json;
} finally {
connection.disconnect();
}
}

It looks like a problem with your router, not your code. When it doesn't work, the response you are seeing is coming from your router.
Are you using the guest network feature on a Linksys E4200 by any chance? The response you are getting is the guest login page.
This may be useful: http://homecommunity.cisco.com/t5/Social-Support/Amazon-com-EA6500-Guest-network-became-inaccessible-after-a/td-p/583866