I'm trying to understand what is the best way to go about doing this:
Basically I will have a bunch of Android/iOS users who will be logging sensor data on their phone, except each user will have its own timestamp depending on when people start the app etc. So say I have two users:
User 1: (5,45), (6,34), (8,32)
User 2: (5,35), (7,32), (9,32)
The format is (time t, and some arbitrary value).
What would be the best way to synchronize the two datasets? Do I have to write an algorithm which is then going to go back and say on User 2, insert the following entry (6,35). So eventually the new data looks like:
User 1: (5,45), (6,34), (7,34), (8,32), (9,32)
User 2: (5,35), (6,35), (7,32), (8,32), (9,32)
This could be very data intensive though, because I am expecting to have about 300 users, and each will have about 36,000 data entries. Any advice would be appreciated - also something I could do on the app side that could help that would be appreciated.
I believe I am going to have to do something like this when I have all my data - but since this is an actively developing project I thought I'd get some advice first.
You could create a DataFrame from each user's data. Then store them all in a Panel. Pandas will line them all up based on the timestamp and you can use the forward fill method, ffill, to propagate the values:
In [62]: df1 = DataFrame([45,34,32], index=[5,6,8], columns=['value'])
In [63]: df2 = DataFrame([35,32,32], index=[5,7,9], columns=['value'])
In [64]: p = Panel({'user1': df1, 'user2': df2})
In [75]: p.ffill().to_frame().unstack()
Out[75]:
user1 user2
minor value value
major
5 45 35
6 34 35
7 34 32
8 32 32
9 32 32
Or, you could do the same thing using just Series and DataFrames. I guess it depends on what you want to do with it:
In [78]: s1 = Series([45,34,32], index=[5,6,8])
In [79]: s2 = Series([35,32,32], index=[5,7,9])
In [80]: df = DataFrame([s1,s2])
In [81]: df
Out[81]:
5 6 7 8 9
0 45 34 NaN 32 NaN
1 35 NaN 32 NaN 32
In [82]: df = DataFrame([s1,s2]).T
In [83]: df
Out[83]:
0 1
5 45 35
6 34 NaN
7 NaN 32
8 32 NaN
9 NaN 32
In [84]: df.ffill()
Out[84]:
0 1
5 45 35
6 34 35
7 34 32
8 32 32
9 32 32
Related
I am getting raw data from OBD adapter, It is getting parsed for single line but for multiline response I am not able to parse.
I used below code to parse
String hexaData = getResult(); // getResult is here the raw data("4904014A4D422A490402333637364904033135303049040400000000") after parsing I am getting parsed value(JMB*36761500) which is correct and only working for single line
Log.i("CalID_hexa", hexaData);
final String result = getResult();
String workingData;
if (result.contains(":")) {//CAN(ISO-15765) protocol.
workingData = result.replaceAll(".:", "").substring(9);//9 is xxx49040701, xxx is bytes of information to follow.
Matcher m = Pattern.compile("[^a-z0-9 ]", Pattern.CASE_INSENSITIVE).matcher(convertHexToString(workingData));
if(m.find()) workingData = result.replaceAll("0:49", "").replaceAll(".:", "");
} else {//ISO9141-2, KWP2000 Fast and KWP2000 5Kbps (ISO15031) protocols.
workingData = result.replaceAll("49040.", "");
}
calId = convertHexToString(workingData).replaceAll("[\u0000-\u001f]", "");
L.stringStringHashMap.put("CAL_ID",calId);
public String convertHexToString(String hex) {
StringBuilder sb = new StringBuilder();
//49204c6f7665204a617661 split into two characters 49, 20, 4c...
for (int i = 0; i < hex.length() - 1; i += 2) {
//grab the hex in pairs
String output = hex.substring(i, (i + 2));
//convert hex to decimal
int decimal = Integer.parseInt(output, 16);
//convert the decimal to character
sb.append((char) decimal);
}
return sb.toString();
}
How to parse multiline response like below response on PID 09 04 from OBD Adapter:
18 DA F1 11 10 13 49 04 01 33 37 38
18 DA F1 1D 10 13 49 04 01 33 37 38
18 DA F1 11 21 30 35 2D 52 59 45 2D
18 DA F1 11 22 41 37 34 30 00 00 55
18 DA F1 1D 21 30 36 2D 52 59 45 2D
18 DA F1 1D 22 41 37 37 30 00 00 55
How can I parse the above?
If you want to parse multiple line responses as single line responses then you can first form the array from the multiple line response separating the lines by character \n. After forming the array of responses, you can join the strings in one single line.
You have two issues here. The first is that you sent your request using a broadcast address, that's why two ECUs (namely, 18 DA F1 11 and 18 DA F1 1D) answered to your request. You either have to sort these out or ask only one ECU directly.
The second is you should learn about ISO 15765-2 (also named ISO-TP), which is a transport protocol, much like TCP (or rather IP) is for Ethernet. https://en.wikipedia.org/wiki/ISO_15765-2 contains a rough outline.
A general notice: If you're serious about implementing OBD2 communication, write a layered architecture, in which the communication layer (BLE, TTY, TCP, etc.) sits below the vehicle protocol which ISO-TP sits on top of. On the actual application layer you can then decode the individual PIDs and their measurements.
I have an Android App where I get Heart Rate Measurements from a Polar H10 Device.
I'm totally lost on how to interpret the heart rate. Various links to the bluetooth.com site are resulting in 404 errors unfortunately.
The characteristics value is i.e.
[16, 59, 83, 4]
From what I understood the second byte (59) is the heart rate in BPM. But this does not seem to be decimal as the value goes up to 127 and then goes on -127, -126, -125, ... It is not hex either.
I tried (in kotlin)
characteristic.value[1].toUInt()
characteristic.value[1].toInt()
characteristic.value[1].toShort()
characteristic.value[1].toULong()
characteristic.value[1].toDouble()
All values freak out as soon as the -127 appears.
Do I have to convert the 59 to binary (59=111011) and see it in there? Please give me some insight.
### Edit (12th April 2021) ###
What I do to get those values is a BluetoothDevice.connectGatt().
Then hold the GATT.
In order to get heart rate values I look for
Service 0x180d and its
characteristic 0x2a37 and its only
descriptor 0x2902.
Then I enable notifications by setting 0x01 on the descriptor. I then get ongoing events in the GattClientCallback.onCharacteristicChanged() callback. I will add a screenshot below with all data.
From what I understood the response should be 6 bytes long instead of 4, right? What am I doing wrong?
On the picture you see the characteristic on the very top. It is linked to the service 180d and the characteristic holds the value with 4 bytes on the bottom.
See Heart Rate Value in BLE for the links to the documents. As in that answer, here's the decode:
Byte 0 - Flags: 16 (0001 0000)
Bits are numbered from LSB (0) to MSB (7).
Bit 0 - Heart Rate Value Format: 0 => UINT8 beats per minute
Bit 1-2 - Sensor Contact Status: 00 => Not supported or detected
Bit 3 - Energy Expended Status: 0 => No Present
Bit 4 - RR-Interval: 1 => One or more values are present
So the first byte is a heart rate in UInt8 format, and the next two bytes are an RR interval.
To read this in Kotlin:
characteristic.getIntValue(FORMAT_UINT8, 1)
This return a heart rate of 56 bpm.
And ignore the other two bytes unless you want the RR.
It seems I found a way by retrieving the value as follows
val hearRateDecimal = characteristic.getIntValue(BluetoothGattCharacteristic.FORMAT_UINT8, 1)
2 things are important
first - the format of UINT8 (although I don't know when to use UINT8 and when UINT16. Actually I thought I need to use UINT16 as the first byte is actually 16 (see the question above)
second - the offset parameter 1
What I now get is an Integer even beyond 127 -> 127, 128, 129, 130, ...
I'm making a project on android studio.
I must convert a number into an hour, minutes and seconds date.
The number is: 0,18865386
The result must be: 04:31:40
I've obtained this date whit excel (used for test calc) just changing the cell format to HOURS but i don't understand how to calculate it.
Any solutions?
1 corresponds to 24 hours.
In hours: 0,18865386 means 24 x 0,18865386 ~= 4,5 hours which corresponds to your 04:31:40.
If you need more precision, e.g. in seconds:
The number of seconds in a day = 24 x 60 x 60 = 86400.
Then 0,18865386 means 86400 x 0,18865386 = 16300 seconds = 04:31:40.
When I perform the following request: POST https://signer.periscope.tv/sign HTTP/1.1 in Periscope app it generates 3 timestamps.
Below are 3 examples of those timestamps, but I can't figure out what they mean.
I know that number 3 is always the Linux timestamp, but what are 1 & 2 and how are they related to the Linux timestamp?
What is "tpForBroadcasterFrame" and how is related to timestamp?
1 ntpForBroadcasterFrame: 15704244410975025152
2 ntpForLiveFrame : 15704244125303865344
3 timestamp : 1447440545
#
1 ntpForBroadcasterFrame: 15704244443861590016
2 ntpForLiveFrame : 15704244125303865344
3 timestamp : 1447440553
#
1 ntpForBroadcasterFrame: 15704244474141110272
2 ntpForLiveFrame : 15704244125303865344
3 timestamp : 1447440560
The first two look to be Network Time Protocol time stamps. The third one is more correctly called a POSIX timestamp.
NTP stores data in an unsigned 64 bit integer that represents 32 bits of seconds since 1900 and 32 bits of fractional seconds, so...
Masking off the first 32 bits of 15704244410975025152 gives 3656429334 seconds since 1900. The other 32 bits have no mapping to a POSIX timestamp as its minimum resolution is 1 second.
Subtracting 2208988800, the number of seconds between 1900 and 1970, from 3656429334 gives 1447440534 seconds since the Posix epoch, or Fri, 13 Nov 2015 18:48:54 GMT
Quick hack code:
#include <iostream>
constexpr uint64_t epochdelta = 2208988800L; // number of seconds between 1900 and 1970
int main()
{
uint64_t num= 15704244410975025152ULL;
uint32_t seconds = (uint32_t)(num >> 32);
std::cout << seconds << " seconds since 1900" << std::endl;
std::cout << seconds - epochdelta << " seconds since 1970" << std::endl;
return 0;
}
I am a bit puzzled by the sensor reading rates in Android. The code below reports delays of ~53 ms (ZTE Blade, rate sensor events is set to SENSOR_DELAY_FASTEST).
public void onSensorChanged(SensorEvent event) {
synchronized (this) {
switch (event.sensor.getType()) {
case Sensor.TYPE_MAGNETIC_FIELD:
TimeNew = event.timestamp;
delay = (long)((TimeNew - TimeOld)/1000000);
TimeOld = TimeNew;
Log.d("Test", delay + " ms");
break;
}
}
}
The log:
DEBUG/Test(23024): 52 ms
DEBUG/Test(23024): 53 ms
DEBUG/Test(23024): 54 ms
DEBUG/Test(23024): 56 ms
DEBUG/Test(23024): 52 ms
DEBUG/Test(23024): 52 ms
DEBUG/Test(23024): 55 ms
DEBUG/Test(23024): 52 ms
If we want to average say 100 samples, and then save the data, the time between each 100th sample will vary significantly. That is presumably because the sensor value is not changing in regular time periods.
But am I missing something ? Is there a way to get (more) regular measurements (e.g. 100 ms) ? Or perhaps I should go for averaging over a specific period of time instead of no.of samples ?
Also 50ms seems to be a bit long time. Could that be hardware limitation of the device ? Will this number vary on different platforms ?
Any advice is appreciated.
I would average over a period of time rather than a number of samples. I would expect different devices with different device capabilities would generate substantially different results if you were to go with the number-based approach. If you want more regular management of the sampling I would probably disconnect the test from the event and just poll the device at whatever frequency you wish.