Preface: I'm very new to graphics.
I'm trying to learn a bit more about AR and have a pet project that should allow me to explore my location for AR objects. effectively I want to stick a flat image in the real world and use my phone's sensors to move it such that it is only visible if I'm looking at it.
I followed the examples on the Android Developers page and I have both a rotation matrix and orientation. Both seem to make sense. I have my image which for now I just want to always fix at North but will eventually randomize it to be X degrees off North as well as some up/down etc.
I'm lost as to how to place my image in 3D space and how to use the rotation vector to determine the image's position on the screen.
Any help or links to articles would be greately appreciated.
I'm using the sample code from Android's developer's page:
#Override
public void onSensorChanged(SensorEvent event) {
if (event.sensor.getType() == Sensor.TYPE_ACCELEROMETER) {
mGravity = event.values;
} else if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD) {
mGeomagnetic = event.values;
}
if (mGravity == null || mGeomagnetic == null) {
return;
}
float R[] = new float[16];
float I[] = new float[9];
if (SensorManager.getRotationMatrix(R, I, mGravity, mGeomagnetic)) {
Log.d(TAG, ...);
}
}
My gut instinct was to take the Matrix.invertM of R and then use Matrix.multiplyMV multiplying the inverted rotation matrix by the world position of object I want to render (for now lets assume the object is 5m north of me putting it at (0, 5, 0, 0)). I used a compass and a level to make sure my surface is flat and my phone is oriented N. The result of the print statement above is the following rotation matrix:
-0.37, 0.93, 0.02, 0.00
-0.92, -0.37, 0.07, 0.00
0.07, 0.01, 1.00, 0.00
0.00, 0.00, 0.00, 1.00
This seems wrong as the matrix should be the identity matrix:
1, 0, 0, 0
0, 1, 0, 0
0, 0, 1, 0
0, 0, 0, 1
Rather then trying to see where you are wrong, I would highly recommend you to try and get into it yourself. There are samples and answered questions on the web that show exactly what you want, so obviously you can try them and learn from them (one example: Android sensors for OpenGL, there are also some on github).
Also, since you have said you are very new to graphics, I would suggest to first do some heavy reading on matrices and transformations and understanding of the 3D space.
Examples: http://www.songho.ca/opengl/gl_transform.html
https://www.raywenderlich.com/3664/opengl-tutorial-for-ios-opengl-es-2-0
I wish to get my phone's current orientation by the following method:
Get the initial orientation (azimuth) first via the getRotationMatrix() and getOrientation().
Add the integration of gyroscope reading over time to it to get the current orientation.
Phone Orientation:
The phone's x-y plane is fixed parallel with the ground plane. i.e., is in a "texting-while-walking" orientation.
"getOrientation()" Returnings:
Android API allows me to easily get the orientation, i.e., azimuth, pitch, roll, from getOrientation().
Please note that this method always returns its value within the range: [0, -PI] and [o, PI].
My Problem:
Since the integration of the gyroscope reading, denoted by dR, may be quite big, so when I do CurrentOrientation += dR, the CurrentOrientation may exceed the [0, -PI] and [o, PI] ranges.
What manipulations are needed so that I can ALWAYS get the current orientation within the the [0, -PI] and [o, PI] ranges?
I have tried the following in Python, but I highly doubt its correctness.
rotation = scipy.integrate.trapz(gyroSeries, timeSeries) # integration
if (headingDirection - rotation) < -np.pi:
headingDirection += 2 * np.pi
elif (headingDirection - rotation) > np.pi:
headingDirection -= 2 * np.pi
# Complementary Filter
headingDirection = ALPHA * (headingDirection - rotation) + (1 - ALPHA) * np.mean(azimuth[np.array(stepNo.tolist()) == i])
if headingDirection < -np.pi:
headingDirection += 2 * np.pi
elif headingDirection > np.pi:
headingDirection -= 2 * np.pi
Remarks
This is NOT that simple, because it involves the following trouble-makers:
The orientation sensor reading goes from 0 to -PI, and then DIRECTLY JUMPS to +PI and gradually gets back to 0 via +PI/2.
The integration of the gyrocope reading also leads to some trouble. Should I add dR to the orientation or subtract dR.
Do please refer to the Android Documentations first, before giving a confirmed answer.
Estimated answers will not help.
The orientation sensor actually derives its readings from the real magnetometer and the accelerometer.
I guess maybe this is the source of the confusion. Where is this stated in the documentation? More importantly, does the documentation somewhere explicitly state that the gyro readings are ignored? As far as I know the method described in this video is implemented:
Sensor Fusion on Android Devices: A Revolution in Motion Processing
This method uses the gyros and integrates their readings. This pretty much renders the rest of the question moot; nevertheless I will try to answer it.
The orientation sensor is already integrating the gyro readings for you, that is how you get the orientation. I don't understand why you are doing it yourself.
You are not doing the integration of the gyro readings properly, it is more complicated than CurrentOrientation += dR (which is incorrect). If you need to integrate the gyro readings (I don't see why, the SensorManager is already doing it for you) please read Direction Cosine Matrix IMU: Theory how to do it properly (Equation 17).
Don't try integrating with Euler angles (aka azimuth, pitch, roll), nothing good will come out.
Please use either quaternions or rotation matrices in your computations instead of Euler angles. If you work with rotation matrices, you can always convert them to Euler angles, see
Computing Euler angles from a rotation matrix by Gregory G. Slabaugh
(The same is true for quaternions.) There are (in the non-degenrate case) two ways to represent a rotation, that is, you will get two Euler angles. Pick the one that is in the range you need. (In case of gimbal lock, there are infinitely many Euler angles, see the PDF above). Just promise you won't start using Euler angles again in your computations after the rotation matrix to Euler angles conversion.
It is unclear what you are doing with the complementary filter. You can implement a pretty damn good sensor fusion based on the Direction Cosine Matrix IMU: Theory manuscript, which is basically a tutorial. It's not trivial to do it but I don't think you will find a better, more understandable tutorial than this manuscript.
One thing that I had to discover myself when I implemented sensor fusion based on this manuscript was that the so-called integral windup can occur. I took care of it by bounding the TotalCorrection (page 27). You will understand what I am talking about if you implement this sensor fusion.
UPDATE: Here I answer your questions that you posted in comments after accepting the answer.
I think the compass gives me my current orientation by using gravity and magnetic field, right? Is gyroscope used in the compass?
Yes, if the phone is more or less stationary for at least half a second, you can get a good orientation estimate by using gravity and the compass only. Here is how to do it: Can anyone tell me whether gravity sensor is as a tilt sensor to improve heading accuracy?
No, the gyroscopes are not used in the compass.
Could you please kindly explain why the integration done by me is wrong? I understand that if my phone's pitch points up, euler angle fails. But any other things wrong with my integration?
There are two unrelated things: (i) the integration should be done differently, (ii) Euler angles are trouble because of the Gimbal lock. I repeat, these two are unrelated.
As for the integration: here is a simple example how you can actually see what is wrong with your integration. Let x and y be the axes of the horizontal plane in the room. Get a phone in your hands. Rotate the phone around the x axis (of the room) by 45 degrees, then around the y axis (of the room) by 45 degrees. Then, repeat these steps from the beginning but now rotate around the y axis first, and then around the x axis. The phone ends up in a totally different orientation. If you do the integration according to CurrentOrientation += dR you will see no difference! Please read the above linked Direction Cosine Matrix IMU: Theory manuscript if you want to do the integration properly.
As for the Euler angles: they screw up the stability of the application and it is enough for me not to use them for arbitrary rotations in 3D.
I still don't understand why you are trying to do it yourself, why you don't want to use the orientation estimate provided by the platform. Chances are, you cannot do better than that.
I think you should avoid the depreciated "Orientation Sensor", and use sensor fusion methods like getRotationVector, getRotationMatrix that already implement fusion algorithms specially of Invensense, which already use gyroscope data.
If you want a simple sensor fusion algorithm called a balance filter
(refer http://www.filedump.net/dumped/filter1285099462.pdf) can be used. Approach is as in
http://postimg.org/image/9cu9dwn8z/
This integrates the gyroscope to get angle, then high-pass filters the result to remove
drift, and adds it to the smoothed accelerometer and compass results. The integrated, high-pass-fil-tered gyro data and the accelerometer/compass data are added in such a way that the two parts add
to one, so that the output is an accurate estimate in units that make sense.
For the balance filter, the time constant may be tweaked to tune the response. The shorter the time
constant, the better the response but the more acceleration noise will be allowed to pass through.
To see how this works, imagine you have the newest gyro data point (in rad/s) stored in gyro, the
newest angle measurement from the accelerometer is stored in angle_acc, and dtis the time from
the last gyro data until now. Then your new angle would be calculated using
angle = b * (angle + gyro*dt) + (1 - b) *(angle_acc);
You may start by trying b = 0.98 for instance. You will also probably want to use a fast gyroscope measurement time dt so the gyro doesn’t drift more than a couple of degrees before the next measurement is taken. The balance filter is useful and simple to implement, but is not the ideal sensor fusion approach.
Invensense’s approach involves some clever algorithms and probably some form of Kalman filter.
Source: Professional Android Sensor Programming, Adam Stroud.
If the azimuth value is inaccurate due to magnetic interference, there is nothing that you can do to eliminate it as far as I know. To get a stable reading of the azimuth you need to filter the accelerometer values if TYPE_GRAVITY is not available. If TYPE_GRAVITY is not available, then I am pretty sure that the device does not have a gyro, so the only filter that you can use is low pass filter. The following code is an implementation of a stable compass using TYPE_GRAVITY and TYPE_MAGNETIC_FIELD.
public class Compass implements SensorEventListener
{
public static final float TWENTY_FIVE_DEGREE_IN_RADIAN = 0.436332313f;
public static final float ONE_FIFTY_FIVE_DEGREE_IN_RADIAN = 2.7052603f;
private SensorManager mSensorManager;
private float[] mGravity;
private float[] mMagnetic;
// If the device is flat mOrientation[0] = azimuth, mOrientation[1] = pitch
// and mOrientation[2] = roll, otherwise mOrientation[0] is equal to Float.NAN
private float[] mOrientation = new float[3];
private LinkedList<Float> mCompassHist = new LinkedList<Float>();
private float[] mCompassHistSum = new float[]{0.0f, 0.0f};
private int mHistoryMaxLength;
public Compass(Context context)
{
mSensorManager = (SensorManager) context.getSystemService(Context.SENSOR_SERVICE);
// Adjust the history length to fit your need, the faster the sensor rate
// the larger value is needed for stable result.
mHistoryMaxLength = 20;
}
public void registerListener(int sensorRate)
{
Sensor magneticSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
if (magneticSensor != null)
{
mSensorManager.registerListener(this, magneticSensor, sensorRate);
}
Sensor gravitySensor = mSensorManager.getDefaultSensor(Sensor.TYPE_GRAVITY);
if (gravitySensor != null)
{
mSensorManager.registerListener(this, gravitySensor, sensorRate);
}
}
public void unregisterListener()
{
mSensorManager.unregisterListener(this);
}
#Override
public void onAccuracyChanged(Sensor sensor, int accuracy)
{
}
#Override
public void onSensorChanged(SensorEvent event)
{
if (event.sensor.getType() == Sensor.TYPE_GRAVITY)
{
mGravity = event.values.clone();
}
else if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD)
{
mMagnetic = event.values.clone();
}
if (!(mGravity == null || mMagnetic == null))
{
mOrientation = getOrientation();
}
}
private void getOrientation()
{
float[] rotMatrix = new float[9];
if (SensorManager.getRotationMatrix(rotMatrix, null,
mGravity, mMagnetic))
{
float inclination = (float) Math.acos(rotMatrix[8]);
// device is flat
if (inclination < TWENTY_FIVE_DEGREE_IN_RADIAN
|| inclination > ONE_FIFTY_FIVE_DEGREE_IN_RADIAN)
{
float[] orientation = sensorManager.getOrientation(rotMatrix, mOrientation);
mCompassHist.add(orientation[0]);
mOrientation[0] = averageAngle();
}
else
{
mOrientation[0] = Float.NAN;
clearCompassHist();
}
}
}
private void clearCompassHist()
{
mCompassHistSum[0] = 0;
mCompassHistSum[1] = 0;
mCompassHist.clear();
}
public float averageAngle()
{
int totalTerms = mCompassHist.size();
if (totalTerms > mHistoryMaxLength)
{
float firstTerm = mCompassHist.removeFirst();
mCompassHistSum[0] -= Math.sin(firstTerm);
mCompassHistSum[1] -= Math.cos(firstTerm);
totalTerms -= 1;
}
float lastTerm = mCompassHist.getLast();
mCompassHistSum[0] += Math.sin(lastTerm);
mCompassHistSum[1] += Math.cos(lastTerm);
float angle = (float) Math.atan2(mCompassHistSum[0] / totalTerms, mCompassHistSum[1] / totalTerms);
return angle;
}
}
In your activity instantiate a Compass object say in onCreate, registerListener in onResume and unregisterListener in onPause
private Compass mCompass;
#Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
mCompass = new Compass(this);
}
#Override
protected void onPause()
{
super.onPause();
mCompass.unregisterListener();
}
#Override
protected void onResume()
{
super.onResume();
mCompass.registerListener(SensorManager.SENSOR_DELAY_NORMAL);
}
Its better to let android's implementation of Orientation detection handle it. Now, yes values you get are from -PI to PI, and you can convert them to degrees (0-360).Some Relevant parts:
Saving data to be processed:
#Override
public void onSensorChanged(SensorEvent sensorEvent) {
switch (sensorEvent.sensor.getType()) {
case Sensor.TYPE_ACCELEROMETER:
mAccValues[0] = sensorEvent.values[0];
mAccValues[1] = sensorEvent.values[1];
mAccValues[2] = sensorEvent.values[2];
break;
case Sensor.TYPE_MAGNETIC_FIELD:
mMagValues[0] = sensorEvent.values[0];
mMagValues[1] = sensorEvent.values[1];
mMagValues[2] = sensorEvent.values[2];
break;
}
}
Calculating roll, pitch and yaw (azimuth).mR and mI are arrys to hold rotation and inclination matrices, mO is a temporary array. The array mResults has the values in degrees, at the end:
private void updateData() {
SensorManager.getRotationMatrix(mR, mI, mAccValues, mMagValues);
/**
* arg 2: what world(according to app) axis , device's x axis aligns with
* arg 3: what world(according to app) axis , device's y axis aligns with
* world x = app's x = app's east
* world y = app's y = app's north
* device x = device's left side = device's east
* device y = device's top side = device's north
*/
switch (mDispRotation) {
case Surface.ROTATION_90:
SensorManager.remapCoordinateSystem(mR, SensorManager.AXIS_Y, SensorManager.AXIS_MINUS_X, mR2);
break;
case Surface.ROTATION_270:
SensorManager.remapCoordinateSystem(mR, SensorManager.AXIS_MINUS_Y, SensorManager.AXIS_X, mR2);
break;
case Surface.ROTATION_180:
SensorManager.remapCoordinateSystem(mR, SensorManager.AXIS_MINUS_X, SensorManager.AXIS_MINUS_Y, mR2);
break;
case Surface.ROTATION_0:
default:
mR2 = mR;
}
SensorManager.getOrientation(mR2, mO);
//--upside down when abs roll > 90--
if (Math.abs(mO[2]) > PI_BY_TWO) {
//--fix, azimuth always to true north, even when device upside down, realistic --
mO[0] = -mO[0];
//--fix, roll never upside down, even when device upside down, unrealistic --
//mO[2] = mO[2] > 0 ? PI - mO[2] : - (PI - Math.abs(mO[2]));
//--fix, pitch comes from opposite , when device goes upside down, realistic --
mO[1] = -mO[1];
}
CircleUtils.convertRadToDegrees(mO, mOut);
CircleUtils.normalize(mOut);
//--write--
mResults[0] = mOut[0];
mResults[1] = mOut[1];
mResults[2] = mOut[2];
}
I am getting various sensor readings from my device (programing for android) and i am looking to get the roll (which seems to be a number 1-100) converted into an angle in degrees and also convert the magnetometer heading into degrees..
any simple equations would be appreciated.. my geometry is a fleeting memory..
public void onSensorChanged(SensorEvent evt) {
int type=evt.sensor.getType();
if(type == Sensor.TYPE_ORIENTATION){
azimuth = evt.values[0]; // azimuth rotation around the z-axis
pitch = evt.values[1]; // pitch rotation around the x-axis
roll = evt.values[2]; // roll rotation around the y-axis
}
//Smoothing the sensor data a bit seems like a good idea.
if (type == Sensor.TYPE_MAGNETIC_FIELD) {
orientation[0]=(orientation[0]*1+evt.values[0])*0.5f;
orientation[1]=(orientation[1]*1+evt.values[1])*0.5f;
orientation[2]=(orientation[2]*1+evt.values[2])*0.5f;
} else if (type == Sensor.TYPE_ACCELEROMETER) {
acceleration[0]=(acceleration[0]*2+evt.values[0])*0.33334f;
acceleration[1]=(acceleration[1]*2+evt.values[1])*0.33334f;
acceleration[2]=(acceleration[2]*2+evt.values[2])*0.33334f;
}
if ((type==Sensor.TYPE_MAGNETIC_FIELD) || (type==Sensor.TYPE_ACCELEROMETER)) {
float newMat[]=new float[16];
//Toast toast = Toast.makeText(ctx.getApplicationContext(), "accel", Toast.LENGTH_SHORT);
//toast.show();
SensorManager.getRotationMatrix(newMat, null, acceleration, orientation);
if(displayOri==0||displayOri==2){
SensorManager.remapCoordinateSystem(newMat,SensorManager.AXIS_X*-1, SensorManager.AXIS_MINUS_Y*-1,newMat);
}else{
SensorManager.remapCoordinateSystem(newMat,SensorManager.AXIS_Y, SensorManager.AXIS_MINUS_X,newMat);
}
matrix=newMat;
}
}
I should add that i am not sure i just want roll.. my app locks in landscape mode but obviously the user can turn their phone to any angle on any access.. so i probably need all three of the above to get the angle im looking for.
the angle in question is as if the user is looking through their phone, no matter how they are holding it, at the real world and i want the angle they are looking off the horizon..
for instance if they are looking at the horizon i want 90degrees returned, if they are looking straight up in the sky i should get 180, straight down -180.. then i will also want the degrees from magnetic north that they are looking.. using the magnetometer
values[2] should allready contain degree-value, that's mentioned in a reference:
values[2]: Roll, rotation around Y axis (-90<=roll<=90), with positive
values when the z-axis moves toward the x-axis.
Update
Take a look at this picture: http://developer.android.com/images/axis_device.png
Here you see blue axis - Y axis. When your phone turns around it, it's called "rolling". The angle of the rotation will be contained in values[2].
I'm currently experimenting with some sensors of Android phones. For testing, I'm using a Samsung Galaxy S. As it does not have a gyroscope, I'm using accelerometer and sensor for magnetic field.
What I basically want to do, is to get a certain angle, when moving the device. I try to explain: consider you are holding the phone in landscape mode in front of your face and then you turn yourself by 90 degrees to the right.
I use the following code to get the current rotation matrix:
#Override
public void onSensorChanged(SensorEvent event) {
if (event.sensor.getType() == Sensor.TYPE_ACCELEROMETER) {
mGravity = event.values.clone();
}
if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD) {
mGeomagnetic = event.values.clone();
}
if (mGravity != null && mGeomagnetic != null) {
float R[] = new float[9];
float I[] = new float[9];
boolean success = SensorManager.getRotationMatrix(R, I, mGravity,
mGeomagnetic);
}
}
This works well and then I use SensorManager.getAngleChange(angleChange, R, lastR); to achieve the angle change.
I then get (roughly) 65° in angleChange[1] if I turn myself as described above and do not tilt the phone or change anything else...
But if I also tilt the phone by 90° when turning myself (so that display is looking to the ceiling afterwards) I get (roughly) 90° in angleChange[1].
I'm very confused now, why such a rotation affects the value in angleChange[1] and on the other hand why it is needed to get the expected 90°.
What I want to achieve is to get the angle when moving the phone as described above (not in 90° degree steps but this sort of orientation change) no matter which other orientation changes (along the two other axes) are made.
Is there any possibility for this?
I want to detect the correct rotations around X axis with Android sensors. After googling, I find this code:
public void onSensorChanged(SensorEvent event) {
Sensor sensor = event.sensor;
switch(sensor.getType()) {
case Sensor.TYPE_ACCELEROMETER:
mAcc = event.values.clone();
break;
case Sensor.TYPE_MAGNETIC_FIELD:
mMag = event.values.clone();
break;
}
if (mAcc == null || mMag == null) return;
float R[] = new float[9];
if (SensorManager.getRotationMatrix(R, null, mAcc, mMag)) {
SensorManager.getOrientation(R, mOrientation);
}
}
mOrientation[1] represents the radians around the X axis. However, the value is very odd.
When the phone lies flat top up on the table, it's 0.
When the head of the phone pointing to the ground, it's PI/2.
When the phone lies flat bottom up on the table, it's 0 again.
When the head of the phone pointing to the sky, it -PI/2.
The states between 1,2 have the same rotation values of those between 2,3. How could I tell which state my phone is in?
Please verify your readings.
The signs & range you report are completely out of sync with
what is expected on an android device.
developer.android.com/reference/android/hardware/Sensor.html#TYPE_ORIENTATION
Note: This sensor type exists for legacy reasons, please use getRotationMatrix() in conjunction with remapCoordinateSystem() and getOrientation() to compute these values instead.
Regards
CVS#2600Hertz