I am developing an Android app in which I need to get the information from a website that requires login credentials.
After login it will display the home page, then clicking a button will direct it to the data page.
I am getting the HTML of the login page and sending the user ID and password to it. It is successful and I am getting the HTML code of the home page, and now I am making another request for the useful data.
I am passing the cookie information along with the URL but it is redirecting to login page. Am I missing any other vital Information that I need to pass?
// Login code
Connection connection = Jsoup.connect(url).method(Method.GET);
Response response = connection.execute();
cookies = response.cookies();
doc = response.parse();
Element viewstateNode = doc.getElementById("__VIEWSTATE");
Element eventvalidationNode = doc.getElementById("__EVENTVALIDATION");
String viewStateValue = viewstateNode.attr("value");
String eventValidationValue = eventvalidationNode.attr("value");
connection = Jsoup.connect(url)
.data("__VIEWSTATE", viewStateValue)
.data("__EVENTVALIDATION", eventValidationValue)
.data("txtLoginID", username)
.data("txtPWD", password)
.data("btnLogin", "Go%21")
.followRedirects(true)
.userAgent("Google")
.cookies(cookies)
.maxBodySize(100000)
.timeout(60000);
response = connection.method(Method.POST).execute();
doc = response.parse();
// Doc contains the HTML code of next page after successful login. Now I need to get the HTML code of next page with useful data.
// I am using the following code to make the second request.
Connection connection = Jsoup.connect(url_datapage)
.cookies(cookies)
.userAgent("Google")
.maxBodySize(100000)
.timeout(60000)
.method(Method.GET);
Response response = connection.execute();
doc = response.parse();
But doc contain the HTML code of login page. It is redirecting to home page again.
Related
I am working around it for whole day and night and it just unable to login, I am giving up so I take the last chance here.
Here is the website that I wish to login: https://unitreg.utar.edu.my/portal/courseRegStu/login.jsp
And this is my POST code
Connection.Response oc = Jsoup.connect("https://unitreg.utar.edu.my/portal/courseRegStu/loginPro.jsp")
.data("preKap", mpreCap,"reqFregkey", mEmail,"reqPassword", mPassword,"kaptchafield", mCaptcha)
.cookie("JSESSIONID", SessID)
.method(Connection.Method.POST)
.execute();
To verify, when you logged in into it, and you open a new tab and paste the login link on it, it will prompt HTML header "The website cannot display the page"
Document document1 = Jsoup.connect("https://unitreg.utar.edu.my/portal/courseRegStu/login.jsp")
.cookie("JSESSIONID", SessID)
.get();
Log.d("WTF", document1.toString());
if (document1.title().contains("Welcome to Course Registration System")){
Log.d("HERE","Fail!");
Please help if you can, I am exhausted...
I am trying to parse data from a site based on the user's input but the site uses dynamic urls (URLs that contain "?") and so once the user inputs a number and submits, the url changes. The issue with this is that my app connects to the initial url where the user inputs the data but once the user clicks "submit" and the url changes, the next time I try to retrieve the data it give me null since the url is different. Is there a way to work around this issue using Jsoup on android?
You can get current URL by using url() method from Connection.Response class. You can get this instance by using execute() on created Connection.
So your code can look more or less like
String loginPage = "http://www.domain.com/login.php";
Connection.Response response = Jsoup
.connect(loginPage)
.data("username", "XXX", "password", "YYY")
.followRedirects(true)
.method(Method.POST)
.execute();
String url = response.url().toString();//<-- here you should get new url
Map<String, String> loginCookies = response.cookies();
Document doc = Jsoup.connect(url)
.cookies(loginCookies)
.get();
I'm trying to do a test request to a server to determine whether if my session id is still valid or not:
Connection.Response testRequest = Jsoup.connect(URL)
.method(Method.GET)
.cookie(SESSION_KEY, sessionId)
.execute();
If it's still valid, I will access the page that I'm after. However, if the sessionId isn't valid, I will get redirected to a sign in page instead.
My problem is that if I check the testRequest.statusCode(), it'll say 200 a.k.a. "OK", regardless if it was a valid session id or not.
In case I use an invalid session, I get redirected (status code 302) to the login page. But jsoup only seems to capture the page/status after I'm redirected.
Is there any way to get the "first" status code? Or should I use HttpClient for this instead of jsoup?
Here is my solution in case someone's interested:
Connection.Response testRequest = Jsoup.connect(URL)
.method(Method.GET)
.followRedirects(false) // <-- did the trick
.cookie(SESSION_KEY, sessionId)
.execute();
I am posting to user's wall using following code using restFB.
String myAccessToken = null;
String appId="my app id ......";
String appSecret="my app secret";
// facebook user id of the user on whose wall to post
String userId="user id ...";
DefaultFacebookClient dfc = new DefaultFacebookClient();
AccessToken accessToken = dfc.obtainAppAccessToken(appId,appSecret);
myAccessToken = accessToken.getAccessToken();
FacebookClient facebookClient = new DefaultFacebookClient(myAccessToken);
FacebookType publishMessageResponse = facebookClient.publish(userId+"/feed", FacebookType.class, Parameter.with("message", "test"));
System.out.println("Published message ID: " + publishMessageResponse.getId());
All is working fine.
But now I want to include user's facebook first name in the post. Is there any way to do so? I dont want to generate yet another request to facebook for getting the username, as I am already generating 2 requests for getting user id(from redirect url). Is there any mechanism that graph api itself replaces some kind of tag in the post text with user name.
E.g I send the following text to post on user wall:
#username# is doing something...
And it is replaced by facebook graph api
Mark is doing something....
I dont want to generate yet another request to facebook for getting the username, as I am already generating 2 requests for getting user id(from redirect url).
No you can't. You have to make the request to fetch the name. /me?fields=name
But why does it matter, it'll take very less time to query this.
I have an android application using a webview on which the user has to log in with username and password before being redirected to the page i would like to scrape data off with jsoup. Since the jsoup thread would be a different session the user would have to login again.
Now i would like to use the cookie received from the webview to send with the jsoup request to be able to scrape my data.
The cookie is being synced with cookiesyncmanager with following code. This is basically where I am stuck cause i dont know how to read out the cookie nor how to attach it to the jsoup request. Please help? :)
public void onPageFinished(WebView view, String url) {
CookieSyncManager.getInstance().sync();
The jsoup scrape I am doing after the user has logged in with something like this:
doc = Jsoup.connect("https://need.authentication.com").get();
Elements elements = doc.select("span.tabCount");
Element count = elements.first();
Log.d(TAG, "test"+(count));
I'm not an android developer but maybe you can try something like this:
final String url = "https://need.authentication.com";
// -- Android Cookie part here --
CookieSyncManager.getInstance().sync();
CookieManager cm = CookieManager.getInstance();
String cookie = cm.getCookie(url); // returns cookie for url
// ...
// -- JSoup part here --
// Jsoup uses cookies as "name/value pairs"
doc = Jsoup.connect("https://need.authentication.com").header("Cookie", cookie).get();
// ...
I hope this helps a bit, but as i said before: im no android developer (and code isn't tested!)
Here's some documentation:
CookieManager
CookieSyncManager
Jsoup Connection