Android: extracting website using BufferedReader: Information extracted are outdated - android

I am reading html source code of a public website using the following code:
Code:
#Override
protected Void doInBackground(Void... params)
{
try
{
URL url = new URL(""+URL);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String inputLine;
PageCode = "";
OriginalPageCode = "";
while ((inputLine = in.readLine()) != null)
{
PageCode += inputLine;
}
OriginalPageCode = PageCode;
try
{
extract_website_and_save(); // extracting data from PageCode
}
catch (Exception e1)
{
}
in.close();
}
Background:
The above code sometimes can fetch the most updated website properly. But occasionally it linked to an outdated version of the website and hence unable to obtain the most updated information for the website.
I am curious why the above will occur, does it related to extracting from cache instead of the real updated website??
I therefore used Chrome to browse the same link, and discovered that Chrome also fetched the outdated website.
I have tried restarting the device, but the problem continues.
After 30 minutes to an hour, I requested the app to fetch again and it then can extract the most updated information. I at the same time browse the website using Chrome, Chrome can now obtain the most updated website.
Question:
The above BufferedReader should have no relationship with Chrome? But they follow the same logic and hence extracting from cache instead of from the most updated website?

I strongly suspect the end point is being cached by URL
Try something like this
urlSrt = urlSrt + "?x=" + new Random().nextInt(100000);
// If your URL already is passing parameters i.e. example.com?x=1&p=pass - then modify
// the urlSrt line to to use an "&" and not "?"
// i.e. urlSrt = urlSrt + "&x=" + new Random().nextInt(100000);
URL url = new URL(urlSrt);
URLConnection con = url.openConnection();
con.setUseCaches(false); //This will stop caching!
So if you modify your code to something like this.
URLConnection con = url.openConnection();
con.setUseCaches(false);
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));

Related

How to prevent denial of services(DOS) attacks in android?

I have used map in my android application. I passed origin and destination latlon and get data from map url then parse the response.
But while auditing below code as marked for DOS attack stating that "This code might allow an attacker to crash the program or otherwise make it unavailable to legitimate users."
Concern : What if attacker push too large file then it will go on line by line and loop will be run for too long.
Proposed solution : Do not allow to read more than specific file size, so that it won't read file beyond some limit
Here is my code :
String url = "https://maps.googleapis.com/maps/api/directions/json"+ "?" + str_origin + "&" + str_dest + "&" + "sensor=false";
private String downloadDataFromUrl(String strUrl) throws IOException {
String data = "";
InputStream iStream = null;
HttpsURLConnection urlConnection = null;
try {
URL url = new URL(strUrl);
urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.connect();
iStream = urlConnection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(iStream),1024);
StringBuffer sb = new StringBuffer();
String line = "";
while ((line = br.readLine()) != null) {
sb.append(line);
}
data = sb.toString();
br.close();
} catch (Exception e) {
Log.d("Exception", e.toString());
} finally {
iStream.close();
urlConnection.disconnect();
}
return data;
}
Please provide solution. Thanks in advance.
Edit 1:by calling append() it appends Untrusted data to a StringBuilder instance initialized with the default backing-array size (16). This can cause the JVM to over-consume heap memory space.
If you download from an unknown URL, the data can indeed be arbitrary and BufferedReader.readLine() can encounter a line so long the program cannot handle it. This question indicates that limiting BufferedReader line length may not be trivial.
Number of lines can be too big as well, in which case line count check instead of simple null check in the while loop seems to be enough.
Question is why would you allow the user to input an arbitrary URL and download it without checking. The URL can easily be a several GB binary file. Your first line indicates that you intend to use the Google Maps API, which AFAIK does not return excessively large lines, rendering the DOS concern moot (except in some ultrasecure applications, which I do not think Android is suitable to use for).

HttpUrlConnection fetched different content from website content

i have tried to fetch the website content from android.
in .NET, it can be done by
WebRequest request = HttpWebRequest.Create(threeROllrl);
WebResponse response = request.GetResponse();
StreamReader vr = new StreamReader(response.GetResponseStream());
string result = vr.ReadToEnd();
however, in android, i tried to use
URL url = new URL(urlstr);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setReadTimeout(8000 /* milliseconds */);
urlConnection.setConnectTimeout(8000 /* milliseconds */);
urlConnection.setRequestMethod("GET");
urlConnection.connect();
BufferedReader reader = null;
reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
StringBuilder stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line + "\n");
}
but it returned different content (in android httpurlconnection, no html content is fetched at all, but the metadata, style and scripts only), i wanna if it is because the response are the mobile version, i want to how retrieve the desktop content of the website in the android httpurlconnection.
after checked, i have found that the android httpurlconnection is actually fetching the header of the website and some bottom content, but it cannot fetch middle body. or it is stopped for some unknown reason (but no error is found).
thanks
Change the User-agent header in your http request to be as a desktop
here is a list of user agent that you can use

Android HttpsURLConnection Authentication

I'm relatively new to java and have been trying to get the source of a webpage after logging in (website is https://stem7.maxnet.co.nz/ispcentre/home.php)
The code I'm using is me trying to expand upon a tutorial. (possibly too far)
The code below returns the source of the login page and stops there.
private String getSource(URL url) throws IOException {
HttpsURLConnection spoof = (HttpsURLConnection)url.openConnection();
StringBuffer sb = new StringBuffer();
String basicAuth = "Basic " + Base64.encodeToString("username:password".getBytes(), Base64.DEFAULT);
spoof.setRequestProperty ("Authorization", basicAuth);
spoof.setRequestMethod("POST");
spoof.setUseCaches(false);
spoof.setDoInput(true);
spoof.setDoOutput(true);
spoof.setRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)");
BufferedReader in = new BufferedReader(new InputStreamReader(
spoof.getInputStream()));
String strLine = "";
while ((strLine = in.readLine()) != null) {
sb.append(strLine);
}
return sb.toString();
}
Any help would be much appreciated. Even a point in the right direction. Cheers
That page uses form authentication which means when you login, the backend expects the request to contain 2 parameters , one for the username, and one for the password, but your code is using basic authentication which fails everytime, that is why you keep getting the source for the login page (the server cannot authenticate you so it redirects the client back to the login page, hence the HTML source for the login page is returned to you. So you could try doing this
connection.setRequestProperty("username", "username");
connection.setRequestProperty("password", "password");

google calculator results for android

I am making an android application which requires to send a mathematical question like 1+1 to google's calculator and I need to get that result which is displayed on the web. How can I achieve this on android?
One possibility is to create a URL for the equation you are trying to calculate and then use a URLConnection to open the URL and read the webpage source code to find the answer to the equation.
For example if you have the equation:
2+2
Then the URL to calculate the result with the Google Chrome calculator would be:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=2%2B2
You will have to construct the proper query in the URL for the equation you are solving. In this URL the query at the end has the equation 2+2:
q=2%2B2 (where the %2B represents the + sign)
After constructing the URL open it with a URLConnection and read the source. The answer to the equation will be in this element:
<span class="cwcot" id="cwos">4</span>
So you can parse the source in order to find that particular span element and retrieve the result of your equation.
This is probably more work than you expected but it is the only solution I can think of to accomplish what you asked. Also, this approach may be error prone and may break easily. I would consider using a different approach altogether such as launching an intent to use the calculator app on the mobile device (even though this approach has issues as well).
EDIT:
This worked for me (it will output: 2 + 2 = 4):
public static void test() {
try {
String source = getUrlSource();
String span = "<span class=\"nobr\"><h2 class=\"r\" style=\"display:inline;font-size:138%\">";
int length = span.length();
int index = source.indexOf(span) + length;
String equation = source.substring(index, source.indexOf("<", index));
System.out.println( "equation: " + equation);
} catch (IOException e) {
e.printStackTrace();
}
}
private static String getUrlSource() throws IOException {
String url = "https://www.google.com/search";
String charset = "UTF-8";
String param1 = "2+2";
String query = String.format("?q=%s", URLEncoder.encode(param1, charset));
HttpsURLConnection urlConn = (HttpsURLConnection)new URL(url + query).openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla/5.0");
urlConn.setRequestProperty("Accept-Charset", charset);
BufferedReader in = new BufferedReader(new InputStreamReader(urlConn.getInputStream()));
String inputLine;
StringBuilder a = new StringBuilder();
while ((inputLine = in.readLine()) != null)
a.append(inputLine);
in.close();
return a.toString();
}

Creating a playlist with Youtube DATA API on Android

I am trying to use the Youtube GDATA API in order to add a new playlist to a youtube account.
I base my code on the documentation: https://developers.google.com/youtube/2.0/developers_guide_protocol_playlists#Adding_a_playlist
I first get an access token and use my developer key appropriately.
The post seems to work just fine, but when trying to get back the response, I get a file not found exception while calling getInputStream.
Does anyone has an idea?
Thanks
Here is the connection code (an updated cleaner version):
#Override
protected Boolean doInBackground(Void... arg0) {
BufferedReader input = null;
InputStreamReader inputStreamReader = null;
StringBuilder postContentXml = new StringBuilder();
postContentXml.append("<?xml version='1.0' encoding='UTF-8'?>").
append("<entry xmlns='http://www.w3.org/2005/Atom'") .
append(" xmlns:yt='http://gdata.youtube.com/schemas/2007'>").
append("<title type='text'>Sports Highlights Playlist</title>").
append("<summary>A selection of sports highlights</summary>").
append("</entry>");
byte[] buffer = postContentXml.toString().getBytes();
StringBuilder response = new StringBuilder();
try {
URL url = new URL("https://gdata.youtube.com/feeds/api/users/default/playlists");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
// Initialize connection parameters
urlConnection.setConnectTimeout(30000);
urlConnection.setReadTimeout(30000);
urlConnection.setDoOutput(true);
urlConnection.setRequestMethod("POST");
// Headers initialization
urlConnection.setRequestProperty("Content-Type", "application/atom+xml");
urlConnection.setRequestProperty("Content-Length", String.valueOf(buffer.length));
urlConnection.setRequestProperty("Authorization", mAuthToken);
urlConnection.setRequestProperty("GData-Version", "2");
urlConnection.setRequestProperty("X-GData-Key", YoutubeUtils.getDevKey());
OutputStream os = urlConnection.getOutputStream();
os.write(buffer, 0, buffer.length);
os.flush();
os.close();
InputStream inputStream = urlConnection.getInputStream();
inputStreamReader = new InputStreamReader(inputStream, "UTF-8");
input = new BufferedReader(inputStreamReader, 4096);
String strLine = null;
while ((strLine = input.readLine()) != null) {
response.append(strLine);
}
input.close();
inputStreamReader.close();
inputStream.close();
urlConnection.disconnect();
Log.d("CreatePlaylistTask", "Response: " + response);
}
catch(Exception e) {
Log.d("CreatePlaylistTask", "Error occured: " + e.getMessage());
}
return true;
}
I'm assuming that the POST wasn't actually successful.
If I had to guess from looking at your code, I'd think that the problem might be the Authorization header value. What does myAuthToken look like, and what type of token is it? If it's an OAuth 2 token, for instance, then the value needs to be Bearer TOKEN_VALUE, not just TOKEN_VALUE.
Also, please note that v3 of the YouTube Data API will be released in the near future, and it will offer better support on Android using the new Google APIs Client Library for Java.
I have put together a sample Android application which uses the YouTube Data v3 API to demonstrate how you can load a playlist into a ListView.
https://github.com/akoscz/YouTubePlaylist
Note that you MUST have a valid API key for this sample application to work. You can register your application with the Google Developer Console and enable the YouTube Data API. You need to Register a Web Application NOT an Android application, because the API key that this sample app uses is the "Browser Key".

Categories

Resources