Not sure if this is the right forum and struggling to articulate this questions but here goes:
We have a process in place where we regularly release testflight app builds in order to test new functionality and fixes in our app. We always point these builds at our production environment (API’s, databases, etc). This causes us some issues like:
We are regularly trying to test stuff which has not yet been released to production (i.e. backend stuff may have not been updated yet)
When we do move code from our backend test environment to our production environment it has potentially not been tested as the app never points there.
We dont see what our real production users see as we are always on more advanced builds.
I am sure there are a number of ways we could do this a lot better but a key requirement we have going forward is to be able to release versions of our app which point to our test environment so we can correctly test things end-to-end to help alleviate the above issues. At the same time we also need versions of our app pointing at production as we use the app ourselves day to day.
The two obvious choices that I can come up with are:
Option 1 - Have separate test and production apps
By this I mean we have two apps released with two separate bundle id’s and names (like app_test and app_prod). You can then load both of these apps on your phone at the same time.
Option 2 - Same app built to either point at test or prod
This is basically where I would release the same app pointing at either production or test. You can only have one or the other on your phone at any one time.
Option 1 feels like the right thing to do but I have a number of issues with it:
The configuration of push and provisioning certificates will be more complicated
Inbound push notifications will be hard to understand and debug on phones if you have the two apps running
We would need to either submit a second app to app stores or we will need to use a third party release tool like hockeyapp or whatever.
Our semi automated build process would need reworking to support multiple bundle ids and stuff
Option 2 on the other hand presents itself with problems too:
You can only have test or production apps on a single device at any one time. This implies we would need to get more devices to test from
There is a risk of releasing an app pointing to test environments to our production users.
I guess a 3rd option would be to have the ability to reconfigure the app to use either environment (by way of a hidden dev menu or something). This concerns us in regards to caching settings but maybe that could be worked around. I guess it also means we could only run one version of the app on a device at any one time.
Anyway, this is quite an open question for advice. I am really keen to hear what other people are doing when it comes to testing apps. I have read up a bit on this and spoken to a number of people to get input (which is what leads me to the above two options) but would appreciate more input and advice. How do you solve this problem? Maybe I am just approaching this from the completely wrong angle!
Related
I'm thinking of changing the name of my application id by appending "-DEBUG" which allows me to install both a release and a debug version of it on the same device (and also helps with stuff like Crashlytics filtering though there are other ways to deal with that) but it poses a problem with the google app engine backend since all the security features are tied to the application name as they should be.
I'm thinking of two possible solutions but wanted to see if anyone else has hit this issue and may have a more elegant approach.
1) instead, I can change the app version via gradle versionNameSuffix though this doesn't allow the two version of the app to co-exist
2) register both app ids with the app engine backend
Am I thinking about this the wrong way? In general, how do people deal with Release vs QA instances of their apps and setting up corresponding versions of their backends? Also, do the Release vs QA backend instances share the same data store? Thanks.
Personally I've always used the suffix approach you're considering, except with a more systematic set of suffixes (-dev if there's a single "in development" version, -dev-<developersusername> if each developer in the team gets a separate one, -staging, -qa, -canary, -prod and the like, depending on the exact deployment workflow for a given app).
And, any external system I need to use from the various versions, such as an app engine backend in your case, I register all relevant versions on. Typically a small, simple script makes light work of that!-)
I would consider it very risky to share a datastore or other persistent bunch of data between a production version (whose data are presumably precious and absolutely to be preserved) and in-development versions that are quite possibly buggy -- couldn't one of the somewhat-expected bugs in the new versions wipe out precious to-be-preserved production data?
Is there any service that can be used to monitor how a beta-user engages in an android application I made?
I've looked into different analytic solutions, but they all come with the disclaimer that I should not send "unique identification information about the users" - fair enough, and I can appreciate the privacy concerns. But I need to dig that information during my beta testing.
Currently, I'm emailing the apk files to a few people to install the app and test on their phones. They give me feedback, but not all of them are good at describing exactly what they are doing. I need more detailed information - like how they opened the app (was it a fresh open, or did they relaunch it from the running app list?), what exactly they did in the app and if possible, to get some debugging information too, since some issues are unique to the specific model of phone they use.
In a nutshell, it means that I need to dig into my beta-testers devices - and they all agree to it too, so its not like I'm spying on them or some such. (At the very least, I want to record their behavior in my app with permission)
Is anything like this available? If not, are there any other approaches I can use to solve/debug issues that generate from end-user behavior? (NOTE: I'm not talking about app crashing/hanging. The app is stable - its just not working correctly)
Stuff that doesn't seem to work:
http://acra.ch/
This looks more like a crash reporting tool than a usage/monitoring tool. :(
http://try.crashlytics.com/
Similar issues to above
Paid levels of BugSense come with a feature called Bread Crumbs, which I've never used but which sound like what you need.
I am going to have a alpha release of my app, which is not yet on the market. I want to have the app APK link sent out to friends via email so they can download it from my site CDN.
One question here: if I want to give them updates, what will be a good way? Can I download the new APK within the app, and somehow install the APK to replace the old one without anything to do with the market? So my friends can have the app upgraded while it is still in alpha release?
When I did it, I used Zubhium -- they were a web service with a small API that you could install into your app, giving you a mini "app store"-style backend and handling distribution for you. It would host and distribute your APK, connect up to their server when the app launched, check for updates, invalidate old versions, gather crash logs for you, etc. It was very good.
Zubhium are now https://www.vessel.io -- I presume they still have the above features as part of their now-much-bigger service, but when I checked you had to give them a credit card number even to sign up for the free plan, so I've not played with it.
A friend of mine uses http://testflightapp.com for iOS, and it looks like they have an Android version now, so that's certainly worth checking out. A quick search also shows up http://applover.me. #Janusz recommends http://hockeyapp.net/features in his comment.
As #Nanne points out in his answer, the Play Store itself now lets you distribute to limited alpha- and beta-test groups. That looks like it has fairly minimal features compared to the third-party services (no A/B testing, etc.), but will be familiar and free. And it doesn't need an extra SDK rolled into your app.
So, my general answer is that there's more than one professional beta-testing API/service that you could use, that they're generally very useful, quite easy to roll into your app, solve all the problems you're anticipating and more, and often have a free plan to get started. I'd recommend picking one of them rather than trying to roll your own solution.
If you want this only to be able to release your app in Alpha, and maybe later in beta, take a look at the android market again.
Check out this link: https://support.google.com/googleplay/android-developer/answer/3131213?hl=en
It boils down to the fact that you can have an Alpha-test, and a beta test, each with selected users. You can upload your app as normal, so you'll have updates via the market, but not everyone can download your app.
For the beta at least, you can select a community that is the source of your users, so all that are in that community could be testers.
This is the best method for testing I believe.
We have an app, that is synchronizing with remote DB, and all users can modify this db. More over, each user has some data private to it, which is synchronized only between this users devices and not with other devices.
So, what are the best practices for automation testing of such app?
How can I see it:
We need at least 2 devices, to make changes on first devices and finding this changes become visible on second devices. At least, this the most high level tests that i do by hands. There are also two different cases, when those 2 "devices" belongs to one user (I use users Google account to differentiate between users) and to different users...
How this multiuser environment can be automated for tests?
You are going to have to probably do something less than ideal to test this, either create a special test build that allows you to override the account you are in (letting you jump between two) or build a way to have different databases for different tests that are in the state you need for any given user (e.g. by having a different endpoint for your application to hit for each test).
Without seeing your code these are the most general ideas, it might be a way pops up if i knew more but sadly I do not have access to that information.
I am nearly done with my first android app and there's still a bit more work to do but I want to get feedback from users about what they like/hate/bugs to fix, etc. I plan on making the app free with ads. Now I have been testing my app on my phone (HTC Magic) and plan on doing some simple testing on the emulator using different configurations. Would it be a good idea to release the app as being beta as it is now? And then fix up any issues and implement the full features I want in it later on and re-release it? I don't want the beta ratings to hurt the final version so would I'm guessing I simply release it as a new app instead of an update to the beta?
Also If I was to release the beta should I be releasing it with or without ads?
I would love to hear your experience with your apps!
Thanks
In this kind of situations, that's exactly what you can try to do: release a beta version of your app so that the bugs don't hurt your rating on the Market. What I prefer to do in this kind of cases:
Don't add Ads. After all, it's just to be a testing experiment. Not too many people is going to download your app.
Offer a bounty for people who discover bugs. For example, you can give a free fully functional copy of your app to the people who helped you find bugs.
Use a Log system for Android that automatically send you reports of your app.
Try to make sure your app will be tested on different devices with different screen sizes, configurations, etc.
Once you have fixed bugs and done the appropriate modifications you can go ahead and share your app with the whole world.
Don't push half baked product in android market. To get feedback and track crashes do a beta distribution. Try Zubhium , simple yet powerful platform for android dev's where you can upload apk, distribute to users. Optionally, you can enable crash reporting to get realtime crash reports and enable in app support using ZubhiumSDK to your market version. That way you are giving your users a outlet to express their feedback, comments, feature request etc.