I want to parse an external webpage with Jsoup in an android app. Actually I manage to do it except it's the mobile version. So... I tried to set User Agent to something else than a "mobile" one.
The problem is that with .userAgent() option and .header(), IOException returns 403 error.
Document doc = Jsoup.connect(url).header("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2").get();
And the Http Response :
Date Mon, 12 Dec 2011 04:08:38 GMT
Content-Type text/html; charset=utf-8
Transfer-Encoding chunked
Connection keep-alive
P3P CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires Mon, 12 Dec 2011 00:00:00 GMT
Last-Modified Mon, 12 Dec 2011 04:08:38 GMT
Pragma no-cache
X-Content-Type-Options nosniff
X-XSS-Protection 1; mode=block
Content-Encoding gzip
Is there something here preventing the connection ?
Thanks
Your request is most likely getting blocked by your mobile data carrier (they only allow mobile User-Agents).
Related
I have developed ReactNative mobile application, When I run security tests using [ImmuniWeb] report shows me that my application makes post request to 'https://www.googleapis.com/experimentsandconfigs/v1/getExperimentsAndConfigs'1 I have not used any of the google analytical tools or any other google cloud services in my application.
One of the Example request is traced as below.
Date/Time: 16.05.2020 17:27 CEST
Requested URL: https://www.googleapis.com/experimentsandconfigs/v1/getExperimentsAndConfigs
Requested HTTP Headers:
User-Agent: Android/com.google.android.gms/16089022 (generic_x86 OSM1.180201.031); gzip
content-length: 1069
content-type: application/x-protobuf
Host: www.googleapis.com
Connection: Keep-Alive
Accept-Encoding: gzip
Response HTTP Headers:
Content-Type: application/octet-stream
Vary: Accept-Encoding
Date: Sat, 16 May 2020 15:27:24 GMT
Expires: Sat, 16 May 2020 15:27:24 GMT
Cache-Control: public, max-age=0
X-Content-Type-Options: nosniff
Server: sffe
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Alt-Svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Transfer-Encoding: chunked
content-length: 49581
I have also created dummy app to check the same it is reporting the same issue in security report.
you can see the full report here. Security Report
Can anyone please help me solving this? Thanks in advance.
Just launched a new AngluarJs mobile site that passed developer and QA testing and now live a small percentage of users are getting errors(could not find expected value).
Tracked this down to Android Chrome users that have the "Reduce Data Usage" setting toggled ON which then routes requests through Google's data compression proxy.
The reason this is generating an error is that the server does not receive the custom http header that is passed from the client. We can clearly see that the Google proxy is stripping it.
How can we stop Google's proxy stripping custom headers and is this expected behaviour? I could not find anything related in the developer Docs.
Google's Compression Proxy doesn't strip response headers, at least those beginning with X-. See below the response headers of this web page using Compression Proxy:
cache-control: public, max-age=60
content-encoding: gzip
content-length: 15283
content-type: text/html; charset=utf-8
date: Sat, 31 Jan 2015 20:17:30 GMT
expires: Sat, 31 Jan 2015 20:18:30 GMT
last-modified: Sat, 31 Jan 2015 20:17:30 GMT
server: GFE/2.0
status: 200 OK
vary: *
version: HTTP/1.1
via: 1.1 Chrome-Compression-Proxy
x-frame-options: SAMEORIGIN
x-original-content-length: 15283
x-request-guid: 719ab2fb-7f3b-4296-b6fd-497997ffe761
So the possible solution is to use X-SiteHost header name.
I'm trying to get an access token to use the Google Web Play API, but it won't work, and it won't give a helpful error message, and Google doesn't provide a lick of support.
https://developers.google.com/android-publisher/authorization
Client ID for web applications
Client ID: 37625491467222.apps.googleusercontent.com
Email address: 37625491467222#developer.gserviceaccount.com
Client secret: XB5gwFt8w29Fs34540gr9urEGEmUHCLI
Redirect URIs: http://www.kizbit.com/oauth2callback
JavaScript origins: http://www.kizbit.com
Refresh token:
http://www.kizbit.com/oauth2callback?code=4/IAoP7JxYfe82KsuUwQRWQw2na_vreOM.MtjvGJqCG3IYshQV0ieZDArkfr1icAI
[root#308321 ~]# cat authcode.txt
grant_type=authorization_code&code=4%2FIAoP7JxYfe82KsuUwQRWQw2na_vreOM.MtjvGJqCG3IYshQV0ieZDArkfr1icAI&client_id=37625491467222.apps.googleusercontent.com&client_secret=XB5gwFt8w29Fs34540gr9urEGEmUHCLI&redirect_uri=http%3A%2F%2Fwww.kizbit.com%2Foauth2callback
[root#308321 ~]# wget -S --post-file=authcode.txt https://accounts.google.com/o/oauth2/token
--2012-06-28 01:08:37-- https://accounts.google.com/o/oauth2/token
Resolving accounts.google.com... 2001:4860:b007::54, 74.125.142.84
Connecting to accounts.google.com|2001:4860:b007::54|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.0 400 Bad Request
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Thu, 28 Jun 2012 05:08:40 GMT
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
2012-06-28 01:08:
37 ERROR 400: Bad Request.
I know you're not supposed to give out keys but it doesn't matter because it doesn't work!
The refresh token doesn't work either.
Not encoding the file doesn't work either:
[root#308321 ~]# cat authcode2.txt
grant_type=authorization_code
code=4/IAoP7JxYfe82KsuUwQRWQw2na_vreOM.MtjvGJqCG3IYshQV0ieZDArkfr1icAI
client_id=37625491467222.apps.googleusercontent.com
client_secret=XB5gwFt8w29Fs34540gr9urEGEmUHCLI
redirect_uri=http://www.kizbit.com/oauth2callback
[root#308321 ~]# wget -S --post-file=authcode2.txt https://accounts.google.com/o/oauth2/token
--2012-06-28 01:33:44-- https://accounts.google.com/o/oauth2/token
Resolving accounts.google.com... 2001:4860:b007::54, 74.125.142.84
Connecting to accounts.google.com|2001:4860:b007::54|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.0 400 Bad Request
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Thu, 28 Jun 2012 05:33:47 GMT
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
2012-06-28 01:33:44 ERROR 400: Bad Request.
I don't know how I solved this. Some possible things I may have done:
Make the access token request from the same IP as the web page authorization request to get the code.
Use wget -dSO- --post-file to debug.
Make the access token request very quickly after the web page authorization. Maybe the code only lasts 10m?
Add more IPs to the Javascript origins in the Google API Console.
Re-URL encode the POST file and strip off any trailing newlines.
Anyways, once I got the access token and the all-so-important refresh token, I'm was good.
I've created an android app using twitter4j.
None of the api calls can authenticate when on 3g, but work perfectly when on wi-fi.
What could be the problem?
I've tested this extensively and can confirm this is only a problem on Android 4.0 (ICS) devices. Why, I'm unsure yet, though some parts of HttpURLConnection have changed in 4.0, so some of these changes have made twitter4j incompatible.
I thought it may have been this:
Android 4.0 ICS turning HttpURLConnection GET requests into POST requests
but after analysing the requests with tcpdump it seems like it is in fact making a valid GET, but for some reason, the twitter api still says it's a bad request..
GET /1/users/show.json?include_entities=true&user_id=6842472 HTTP/1.1
Authorization: OAuth oauth_consumer_key="<redacted>",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1331150787",oauth_nonce="2157367237",oauth_version="1.0",oauth_token="<redacted>",oauth_signature="<redacted>"
Accept-Encoding: gzip
X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-2.2.5.xml
User-Agent: twitter4j http://twitter4j.org/ /2.2.5
X-Twitter-Client-Version: 2.2.5
X-Twitter-Client: Twitter4J
Host: api.twitter.com
Connection: Keep-Alive
HTTP/1.1 400 Bad Request
Date: Wed, 07 Mar 2012 20:06:30 GMT
Status: 400 Bad Request
X-RateLimit-Class: api
X-RateLimit-Reset: 1331153824
X-Revision: DEV
Last-Modified: Wed, 07 Mar 2012 20:06:30 GMT
X-Frame-Options: SAMEORIGIN
X-Transaction: a0bf805e990a5c79
Content-Type: application/json; charset=utf-8
X-RateLimit-Remaining: 0
X-RateLimit-Limit: 150
Pragma: no-cache
Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
X-Runtime: 0.00742
X-MID: 2295d07237fcb4763f7f54d2bf46e27dd40e022f
Expires: Tue, 31 Mar 1981 05:00:00 GMT
Vary: Accept-Encoding
Server: tfe
Transfer-Encoding: chunked
Connection: close
Content-Encoding: gzip
Set-Cookie:
...
Unless someone can find what it is in ICS/HttpURLConnection that is breaking this the only workaround I can see is to write an alternate backend for twitter4j that uses HttpClient instead of HttpURLConnection, which is exactly what I plan to do.
EDIT: So I wrote the a client backend for twitter4j that uses the apache httpclient and it suffers from the exact same problem!! Some more sleepless hours and wireshark has brought me to the following very handy solution, that actually works:
Initialise your twitter4j instance like this:
ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
configurationBuilder.setOAuthConsumerKey(Const.CONSUMER_KEY);
configurationBuilder.setOAuthConsumerSecret(Const.CONSUMER_SECRET);
configurationBuilder.setUseSSL(true);
Configuration configuration = configurationBuilder.build();
twitter = new TwitterFactory(configuration).getInstance();
The setUseSSL(true) is the magic line.
I have a jQueryMobile application with JSF 2.0. (JBoss 7)
The issue is: index.jsf returns html containing jQM List which does not render on Android (Just plain links instead of the nice jQM list). This works as expected in other browsers such as Chrome, Firefox, Safari etc.
The strange thing is if I save the same html output (returned by index.jsf) as index.html and access html file, it works as expected.
Puzzled and frustrated why the same html will work when served as html file but NOT when served as jsf page.
index.jsf response header:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Powered-By: JSF/2.0
Set-Cookie: JSESSIONID=5ZLbSNpM3cumhlXupMMFEbiE; Path=/itranzit
Vary: Accept-Encoding
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 05 Dec 2011 12:03:13 GMT
index.html response header
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"2905-1323086637000"-gzip
Last-Modified: Mon, 05 Dec 2011 12:03:57 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Type: text/html
Transfer-Encoding: chunked
Date: Mon, 05 Dec 2011 12:04:00 GMT
Appreciate any help. Thanks heaps!