I was trying to write an android app to help people do easier searches on multiple websites like craiglist, ebay ..etc
I am not sure where really to start from, or the name of the technology I would be using. To make it more accurate question, these websites are all forms based that you either post stuffs or retreive stuffs through searching. How can I make my android app actually in which it will ask the user for a item name (that part I know) and then send item name to say craiglist so it gets the results back?
I was thinking of copying the url that is displayed on my broswer when I click search on any of the website and using the same link with string manipulation to include whats needed.
Is this correct? Any other ideas are I know? I am kinda all over the place, but I need a starting point.
Some sites might expose a public API for you to connect with, but that's site specific.
Sites that do not expose such a system can be "scraped", which means you're using string manipulation (as you mentioned) to parse our the data you're after.
Both ways require basic knowledge on how to perform HTTP operations, there are several ways to do this but the easiest way it to use the HttpURLConnection class as described in various tutorials on the web.
Googling for "android http GET POST tutorial" should point you in the right way to get started with your adventure :)
Related
let me ask my question with an example:
Assume that I have some PHP files for my android app that I put it on 000webhost[dot]com; so, my question is how can I use parse (parseplatform[dot]org) in this situation?
I'm new in backend, so I'm sorry if this question is stupid!
For some reason I can't use back4app, aws, etc.
Don't worry about php language, if it's necessary I can write it in another language. (my backend logic is not complex)
If you need a simple web-site with simple backend logic, I'd do that using Express.js. Since Parse Server is essentially an Express.js middleware, you can use the same process to serve both Parse Server and the web-site.
If you prefer to use a different technology, Parse Platform provides many different sdks (https://parseplatform.org/#sdks) that you can use for many different backend technologies (including PHP), or you can connect directly to the REST (https://docs.parseplatform.org/rest/guide/) or GraphQL (https://docs.parseplatform.org/graphql/guide/) apis using any client.
When I asked this question, I didn’t know what exactly should I search until I found this amazing article. This article opened my mind, the most helpful thing that I found there, was this:
Parse is more flexible. Given the freedom to host Parse on the
server of your choice.
You can choose the self-hosting option and deploy your Parse project
on your selected server, such as Digital Ocean.
And then I found this awesome video on youtube.
Now I can run parse on my own server (vps or cloud).
I know the title of this question may not be clear, but I don’t want to change it, because someone like me doesn’t know the right keyword for research.
Hope this help someone else too.
I am a new guy to full stack web application development. I want to design a web application which has data stored in say back end databases. Now I want to design a desktop web client as well as android application which will be able to fetch data from back end. So how do I need to start? What APIs can be used or how can I expose data from same back end to multiple clients?
Also I want to handle massive amount of request. How to design such a system? What to use in back end to store data and handle requests efficiently.
Any video / document / reference containing useful information will be much appreciated.
Wow, you have a whole forest of questions to settle. You are going to need to go do your own research on such things as algorithms and data flow for your application before you can make any reasonable choice of platform. Here are a couple of basic ideas to get you going: 1) look at Java and Node.js. There are lots of other possible platforms but chances are you will end up using one of those two. Try to think about what the actual code you will generate in each of those will look like. A little or a lot? 2) Just store your data in files, most probably using JSON. Maybe you will end up doing something more fancy after you figure out where you are going with your project, but you will be surprised how well the simple file-based solution will scale.
When you have done a bunch more research, and maybe even coded up a few ideas on your platform of choice, then come back and massively edit your question. Only then will specific suggestions for tool choices be possible.
I just finished a tutorial on how to develop an android application that retrieves updated posts from a Blog using JSON data.
The link to the JSON data used to retrieve the posts, was the blog name ending with "/api/get_recent_summary"
How can I find the link of the JSON data for different websites?
For example the website for The Time Magazine
http://time.com
The quickest and easiest way is to use google developer tools in Google Chrome.
1st go to google developer tools. F12 or Ctrl+Shift+I or top right Ellipsis->More Tools->Developer Tools
2nd Click on the "Network" tab.
3rd click on the "XHR" sub-tab. XHR(XMLHttpRequest)
if a site uses json it will be listed under the XHR sub-tab. You can search through the different return objects, select one and use the "preview" sub-sub-tab to view it.
View JSON
View JSON URL
Although the above way is the easiest it is not the most stable way of getting the info you need. Many sites make changes to the return data without any notice. This will probably break your app...
I think what you are looking for is an API(Application programming interface). Most web APIs return json or xml. You should start by searching for the api documentation for the specific site that you want to get json data from. Example documentation for sites that have public api feeds are github api or youtub api. Many of these will require authentication in order to get the desired json response but the documentation should show you how to do it.
Using a legitimate web api is the most stable way to go. Meaning your app has less chance of randomly breaking all of the time due to feed changes, url changes... I hope this helps!
I know this is an older question, but I felt compelled to chime in. If you goal is to simply determine if a site uses JSON for data exchange, then the solution proposed by Tim is a very good solution. However, if you are looking to scrape data from an arbitrary site, there is no guarantee that the site uses JSON as data exchange, as stated by #KenanZahirovic. There are numerous sites that do not do this. Instead they embed the data into the HTML, or use XML, or some other method for getting the content to the client. There is no standard way of doing it, which is why data scraping is so difficult. It requires figuring out how data is transferred and building a client for that.
For scenarios where you need to gather data from multiple sources, you may end up with multiple clients due to the nuances between the sources. This site explains some best practices for data scraping. However, this would likely require a server-side application. Having a server-side application that gathers data and stores it in a database would make the most sense for this scenario. This way you can have a consistent API that the client hits to access the data.
An algorithm has been proposed that can scrape many sites fairly confidently. If you only want to have a client, this may or may not be the best way to go. It all depends on how much processing you want the client to do. If at all possible, try and off load processing to a server.
This answer might be way more than is required, but ,again, I felt compelled to chime in. I am sure the previous answer was sufficient. I do recommend though that you mark an answer as the accepted answer.
Best of luck!
I've come across this problem when thinking about creating different apps for Android. I see these apps that show you the weather cast or sport results, and I wonder which is the source they get all data from.
I assume they don't have their own database which they fill with data manually and live. I suppose they use some kind of web service, data source or something like that. I don't know if web service is the correct term to use in this case (english is not even my native language).
So I would like to know the correct term for what I'm looking for, plus any guidance on finding this kind of services, in order to be able to develop apps using this kind of data sources.
I think this question will help many programming students like me.
It depends what you trying to develope- Some apps use simple GET/POST Requests and get it's data back formatted as JSON or XML (which is the common case).
Some uses SOAP for getting their data.
In some cases they are using plain Sockets or CouchDB.
In some cases they are using Push Mechanism (specially for android) like GCM to send their data to the client.
It always depends what you want to do.
This one is puzzling me...
For my Android application I want to be able to return the current time in any location the user requests (right down to city level). The request itself comes from a voice command, so I cannot use alternative methods.
The most obvious way (so I thought) was to query Google, but I need your brains to help me solve how to extract the data I want please!
In Google, query:
"What's the time in China"
You should see a 'widget' at the top displaying the answer nicely. View the page source and search on 'China' and it's there - I thought great, this is going to be easy!
But...
"What's the time in London"
Although the display in Google appears the same, there is no result for 'London' when searching the source HTML...?
Using Google Chrome, I inspect the element of the 'widget' and can then view the contents I wish to extract in all cases.
So, my questions is firstly, am I missing something obvious (Google don't want us to be able to do this for example)? Or is there a way to 'force' the result to be in the HTML by structuring the query perhaps?
Secondly, is there a way to access the specific 'element container' and return the results to my Android Application?
I thank you in advance...
You should really use an API instead of parsing html.
I guess there are many many API which will give you the time in a given city.
http://www.worldweatheronline.com/time-zone-api.aspx for example (havnt tried it myself)
If you still want to extract data from the google result (which is html), check for html parsers like in this post : Parse HTML in Android