Integrating im2txt model in Android phone - android

I am new to TensorFlow and cannot find out the solution of these questions.
How can I retrain the im2txt model for my new dataset such that the dataset on which the im2txt model was trained does not get lost and my new dataset is added to the MSCOCO dataset to caption the new images (i.e training dataset= MSCOCO dataset + My new dataset). Someone, please share the detailed procedure and the problems that I can face while retraining.
I have found out the TensorFlow tutorial for running inception V3 model in android on real-time datasets, Can this method be applied as well to the im2txt model i.e. can this be made to caption an image taken from a mobile in real time. Someone, please share the detailed steps how to do this.

After the weeks of struggle can be able to run and execute the im2txt model on Android.
Since I found the solutions from different blogs and different questions and answers, Felt that it might be useful if the all(maximum) solution is at one place.So, Sharing the steps followed.
You need to clone the tensorflow project https://github.com/tensorflow/tensorflow/releases/tag/v1.5.0 in order to freeze the graph and for some more utils.
Downloaded the im2txt model form https://github.com/KranthiGV/Pretrained-Show-and-Tell-model
Followed the steps described in the above link Can able to Run the inference to generate captions on the Linux desktop Successfully after renaming some variable in the graph(to overcome NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/bias not found in checkpoint types of errors)
Now we need to freeze the existing model to obtain the frozen graph in order to use in android/ios
from cloned tensorflow project using freeze_graph.py( tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py) one can freeze the graph from any model by giving the following command
An example of command-line usage is:
bazel build tensorflow/python/tools:freeze_graph && \
bazel-bin/tensorflow/python/tools/freeze_graph \
--input_graph=some_graph_def.pb \
--input_checkpoint=model.ckpt-8361242 \
--output_graph=/tmp/frozen_graph.pb --output_node_names=softmax
--input_binary=true
we need to supply the all the output_node_names which we required to run the model, from "Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_wrapper.py" we can list out the output node names as 'softmax', 'lstm/initial_state' and 'lstm/state'
when I run the freeze graph command by supplying the output node names as 'softmax', 'lstm/initial_state' and 'lstm/state' got the error "AssertionError: softmax is not in the graph"
From the answers for
How to freeze an im2txt model?(How to freeze an im2txt model?)
by Steph and Jeff Tang
The current model ckpt.data, ckpt.index and ckpt.meta files and a graph.pbtxt should be loaded in inference mode (see InferenceWrapper in im2txt). It builds a graph with the correct names 'softmax', 'lstm/initial_state' and 'lstm/state'. You save this graph (with the same ckpt format) and then you can apply the freeze_graph script to obtain the frozen model.
to do this in Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_utils\inference_wrapper.base.py, just add something like saver.save(sess, "model/ckpt4") after saver.restore(sess, checkpoint_path) in def _restore_fn(sess):. Then rebuild and run_inference and you'll get a model that can be frozen, transformed, and optionally memmapped, to be loaded by iOS and Android apps
Now I run the command as below
python tensorflow/python/tools/freeze_graph.py \
--input_meta_graph=/tmp/ckpt4.meta \
--input_checkpoint=/tmp/ckpt4 \
--output_graph=/tmp/ckpt4_frozen.pb \
--output_node_names="softmax,lstm/initial_state,lstm/state" \
--input_binary=true
and loaded the obtained ckpt4_frozen.pb file into the Android application and got the error
"java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'DecodeJpeg' with these attrs. Registered devices: [CPU], Registered kernels:
[[Node: decode/DecodeJpeg = DecodeJpegacceptable_fraction=1, channels=3, dct_method="", fancy_upscaling=true, ratio=1, try_recover_truncated=false]]"
From https://github.com/tensorflow/tensorflow/issues/2883
Since DecodeJpeg isn't supported as part of the Android tensorflow core, you'll need to strip it out of the graph first
bazel build tensorflow/python/tools:strip_unused && \
bazel-bin/tensorflow/python/tools/strip_unused \
--input_graph=ckpt4_frozen.pb \
--output_graph=ckpt4_frozen_stripped_graph.pb \
--input_node_names=convert_image/Cast,input_feed,lstm/state_feed\
--output_node_names=softmax,lstm/initial_state,lstm/state\
--input_binary=true
When I try to load ckpt4_frozen_stripped_graph.pb in android i faced errors so i followed Jeff Tang's answer (Error using Model after using optimize_for_inference.py on frozen graph)
Instead of tools:strip_unused I used graph transformation tool
bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=/tmp/ckpt4_frozen.pb \
--out_graph=/tmp/ckpt4_frozen_transformed.pb \
--inputs="convert_image/Cast,input_feed,lstm/state_feed" \
--outputs="softmax,lstm/initial_state,lstm/state" \
--transforms='
strip_unused_nodes(type=float, shape="1,299,299,3")
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms'
I can able load the obtained ckpt4_frozen_transformed.pb on android successfully.
when I supply the input as float array of RGB image pixels for input node "convert_image/Cast" and fetch the output from "lstm/initail_state" node successfully.
Now the challenge is to understand the beam search in "Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_utils\caption_generator.py" and same should be implemented on the Android side.
If you observe the python script caption_generator.py at
softmax, new_states, metadata = self.model.inference_step(sess,input_feed,state_feed)
input_feed is an int32 bit array and state_feed is a multidimensional float array
on the android side, I tried feeding int32 bit array for "input_feed", since there is no Java API to feed multidimensional array so I fed float array to lstm/state_feed as it is which fetched previously from "lstm/initail_state" node.
Got two errors one is for the input_fedd is expecting int 64bit and
"java.lang.IllegalArgumentException: -input rank(-1) <= split_dim < input rank (1), but got 1" at lstm/state_feed.
For the first error, I changed the input_feed feed data type from int32 to int 64.
About second error it is expecting rank two tensor.
If you see tensorflow java sources the data type float array which we are feeding is converted to a rank one tensor, we should feed data type in such a way that rank two tensor should be created but at present, I didn't find any API to feed multidimensional float array there
When I browsing tensorflow java source I found the API which was not exposed as Android API where we can create a rank two tensor. so I rebuilt and both libtensorflow_inference.so and libandroid_tensorflow_inference_java.jar by enabling the rank two tensor creation call.(for building process refer https://blog.mindorks.com/android-tensorflow-machine-learning-example-ff0e9b2654cc)
Now I can able to run the inference on Android and get the one caption for the image.But the accuracy very low.
Reason limiting for the one caption is I didn't find a way to fetch the outputs as a multidimensional array which is required for generating more number of cations for a single image.
String actualFilename = labelFilename.split("file:///android_asset/")[1];
vocab = new Vocabulary(assetManager.open(actualFilename));
inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFilename);
final Graph g = c.inferenceInterface.graph();
final Operation inputOperation = g.operation(inputName);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + inputName + "'");
}
final Operation outPutOperation = g.operation(outputName);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + outputName + "'");
}
// The shape of the output is [N, NUM_CLASSES], where N is the batch size.
int numClasses = (int) inferenceInterface.graph().operation(outputName)
.output(0).shape().size(1);
Log.i(TAG, "Read " + vocab.totalWords() + " labels, output layer size is " + numClasses);
// Ideally, inputSize could have been retrieved from the shape of the input operation. Alas,
// the placeholder node for input in the graphdef typically used does not specify a shape, so it
// must be passed in as a parameter.
inputSize = inputSize;
// Pre-allocate buffers.
outputNames = new String[]{outputName + ":0"};
outputs = new float[numClasses];
inferenceInterface.feed(inputName + ":0", pixels, inputSize, inputSize, 3);
inferenceInterface.run(outputNames, runStats);
inferenceInterface.fetch(outputName + ":0", outputs);
startIm2txtBeamSearch(outputs);
//Implemented Beam search in JAVA
private void startIm2txtBeamSearch(float[] outputs) {
int beam_size = 1;
//TODO:Prepare vocab ids from file
ArrayList<Integer> vocab_ids = new ArrayList<>();
vocab_ids.add(1);
int vocab_end_id = 2;
float lenth_normalization_factor = 0;
int maxCaptionLength = 20;
Graph g = inferenceInterface.graph();
//node input feed
String input_feed_node_name = "input_feed";
Operation inputOperation = g.operation(input_feed_node_name);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + input_feed_node_name + "'");
}
String output_feed_node_name = "softmax";
Operation outPutOperation = g.operation(output_feed_node_name);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + output_feed_node_name + "'");
}
int output_feed_node_numClasses = (int) outPutOperation.output(0).shape().size(1);
Log.i(TAG, "Output layer " + output_feed_node_name + ", output layer size is " + output_feed_node_numClasses);
FloatBuffer output_feed_output = FloatBuffer.allocate(output_feed_node_numClasses);
//float [][] output_feed_output = new float[numClasses][];
//node state feed
String input_state_feed_node_name = "lstm/state_feed";
inputOperation = g.operation(input_state_feed_node_name);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + input_state_feed_node_name + "'");
}
String output_state_feed_node_name = "lstm/state";
outPutOperation = g.operation(output_state_feed_node_name);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + output_state_feed_node_name + "'");
}
int output_state_feed_node_numClasses = (int) outPutOperation.output(0).shape().size(1);
Log.i(TAG, "Output layer " + output_state_feed_node_name + ", output layer size is " + output_state_feed_node_numClasses);
FloatBuffer output_state_output = FloatBuffer.allocate(output_state_feed_node_numClasses);
//float[][] output_state_output= new float[numClasses][];
String[] output_nodes = new String[]{output_feed_node_name, output_state_feed_node_name};
Caption initialBean = new Caption(vocab_ids, outputs, (float) 0.0, (float) 0.0);
TopN partialCaptions = new TopN(beam_size);
partialCaptions.push(initialBean);
TopN completeCaption = new TopN(beam_size);
captionLengthLoop:
for (int i = maxCaptionLength; i >= 0; i--) {
List<Caption> partialCaptionsList = new LinkedList<>(partialCaptions.extract(false));
partialCaptions.reset();
long[] input_feed = new long[partialCaptionsList.size()];
float[][] state_feed = new float[partialCaptionsList.size()][];
for (int j = 0; j < partialCaptionsList.size(); j++) {
Caption curCaption = partialCaptionsList.get(j);
ArrayList<Integer> senArray = curCaption.getSentence();
input_feed[j] = senArray.get(senArray.size() - 1);
state_feed[j] = curCaption.getState();
}
//feeding
inferenceInterface.feed(input_feed_node_name, input_feed, new long[]{input_feed.length});
inferenceInterface.feed(input_state_feed_node_name, state_feed, new long[]{state_feed.length});
//run
inferenceInterface.run(output_nodes, runStats);
//fetching
inferenceInterface.fetch(output_feed_node_name, output_feed_output);
inferenceInterface.fetch(output_state_feed_node_name, output_state_output);
float[] word_probabilities = new float[partialCaptionsList.size()];
float[] new_state = new float[partialCaptionsList.size()];
for (int k = 0; k < partialCaptionsList.size(); k++) {
word_probabilities = output_feed_output.array();
//output_feed_output.get(word_probabilities[k]);
new_state = output_state_output.array();
//output_feed_output.get(state[k]);
// For this partial caption, get the beam_size most probable next words.
Map<Integer, Float> word_and_probs = new LinkedHashMap<>();
//key is index of probability; value is index = word
for (int l = 0; l < word_probabilities.length; l++) {
word_and_probs.put(l, word_probabilities[l]);
}
//sorting
// word_and_probs = word_and_probs.entrySet().stream()
// .sorted(Map.Entry.comparingByValue())
// .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,(e1, e2) -> e1, LinkedHashMap::new));
word_and_probs = MapUtil.sortByValue(word_and_probs);
//considering first (beam size probabilities)
LinkedHashMap<Integer, Float> final_word_and_probs = new LinkedHashMap<>();
for (int key : word_and_probs.keySet()) {
final_word_and_probs.put(key, word_and_probs.get(key));
if (final_word_and_probs.size() == beam_size)
break;
}
for (int w : final_word_and_probs.keySet()) {
float p = final_word_and_probs.get(w);
if (p < 1e-12) {//# Avoid log(0).
Log.d(TAG, "p is < 1e-12");
continue;
}
Caption partialCaption = partialCaptionsList.get(k);
ArrayList<Integer> sentence = new ArrayList<>(partialCaption.getSentence());
sentence.add(w);
float logprob = (float) (partialCaption.getPorb() + Math.log(p));
float scroe = logprob;
Caption beam = new Caption(sentence, new_state, logprob, scroe);
if (w == vocab_end_id) {
completeCaption.push(beam);
} else {
partialCaptions.push(beam);
}
}
if (partialCaptions.getSize() == 0)//run out of partial candidates; happens when beam_size = 1.
break captionLengthLoop;
}
//clear buffer retrieve sub sequent output
output_feed_output.clear();
output_state_output.clear();
output_feed_output = null;
output_state_output = null;
output_feed_output = FloatBuffer.allocate(output_feed_node_numClasses);
output_state_output = FloatBuffer.allocate(output_state_feed_node_numClasses);
Log.d(TAG, "----" + i + " Iteration completed----");
}
Log.d(TAG, "----Total Iterations completed----");
LinkedList<Caption> completeCaptions = completeCaption.extract(true);
for (Caption cap : completeCaptions) {
ArrayList<Integer> wordids = cap.getSentence();
StringBuffer caption = new StringBuffer();
boolean isFirst = true;
for (int word : wordids) {
if (!isFirst)
caption.append(" ");
caption.append(vocab.getWord(word));
isFirst = false;
}
Log.d(TAG, "Cap score = " + Math.exp(cap.getScore()) + " and Caption is " + caption);
}
}
//Vocab
public class Vocabulary {
String TAG = Vocabulary.class.getSimpleName();
String start_word = "<S>", end_word = "</S>", unk_word = "<UNK>";
ArrayList<String> words;
public Vocabulary(File vocab_file) {
loadVocabsFromFile(vocab_file);
}
public Vocabulary(InputStream vocab_file_stream) {
words = readLinesFromFileAndLoadWords(new InputStreamReader(vocab_file_stream));
}
public Vocabulary(String vocab_file_path) {
File vocabFile = new File(vocab_file_path);
loadVocabsFromFile(vocabFile);
}
private void loadVocabsFromFile(File vocabFile) {
try {
this.words = readLinesFromFileAndLoadWords(new FileReader(vocabFile));
//Log.d(TAG, "Words read from file = " + words.size());
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
private ArrayList<String> readLinesFromFileAndLoadWords(InputStreamReader file_reader) {
ArrayList<String> words = new ArrayList<>();
try (BufferedReader br = new BufferedReader(file_reader)) {
String line;
while ((line = br.readLine()) != null) {
// process the line.
words.add(line.split(" ")[0].trim());
}
br.close();
if (!words.contains(unk_word))
words.add(unk_word);
} catch (IOException e) {
e.printStackTrace();
}
return words;
}
public String getWord(int word_id) {
if (words != null)
if (word_id >= 0 && word_id < words.size())
return words.get(word_id);
return "No word found, Maybe Vocab File not loaded";
}
public int totalWords() {
if (words != null)
return words.size();
return 0;
}
}
//MapUtil
public class MapUtil {
public static <K, V extends Comparable<? super V>> Map<K, V> sortByValue(Map<K, V> map) {
List<Map.Entry<K, V>> list = new ArrayList<>(map.entrySet());
list.sort(new Comparator<Map.Entry<K, V>>() {
#Override
public int compare(Map.Entry<K, V> o1, Map.Entry<K, V> o2) {
if (o1.getValue() instanceof Float && o2.getValue() instanceof Float) {
Float o1Float = (Float) o1.getValue();
Float o2Float = (Float) o2.getValue();
return o1Float >= o2Float ? -1 : 1;
}
return 0;
}
});
Map<K, V> result = new LinkedHashMap<>();
for (Map.Entry<K, V> entry : list) {
result.put(entry.getKey(), entry.getValue());
}
return result;
}
}
//Caption
public class Caption implements Comparable<Caption> {
private ArrayList<Integer> sentence;
private float[] state;
private float porb;
private float score;
public Caption(ArrayList<Integer> sentence, float[] state, float porb, float score) {
this.sentence = sentence;
this.state = state;
this.porb = porb;
this.score = score;
}
public ArrayList<Integer> getSentence() {
return sentence;
}
public void setSentence(ArrayList<Integer> sentence) {
this.sentence = sentence;
}
public float[] getState() {
return state;
}
public void setState(float[] state) {
this.state = state;
}
public float getPorb() {
return porb;
}
public void setPorb(float porb) {
this.porb = porb;
}
public float getScore() {
return score;
}
public void setScore(float score) {
this.score = score;
}
#Override
public int compareTo(#NonNull Caption oc) {
if (score == oc.score)
return 0;
if (score < oc.score)
return -1;
else
return 1;
}
}
//TopN
public class TopN {
//Maintains the top n elements of an incrementally provided set.
int n;
LinkedList<Caption> data;
public TopN(int n) {
this.n = n;
this.data = new LinkedList<>();
}
public int getSize() {
if (data != null)
return data.size();
return 0;
}
//Pushes a new element
public void push(Caption x) {
if (data != null) {
if (getSize() < n) {
data.add(x);
} else {
data.removeLast();
data.add(x);
}
}
}
//Extracts all elements from the TopN. This is a destructive operation.
//The only method that can be called immediately after extract() is reset().
//Args:
//sort: Whether to return the elements in descending sorted order.
//Returns: A list of data; the top n elements provided to the set.
public LinkedList<Caption> extract(boolean sort) {
if (sort) {
Collections.sort(data);
}
return data;
}
//Returns the TopN to an empty state.
public void reset() {
if (data != null) data.clear();
}
}
Even though accuracy is very low I am sharing this because it might be useful for some to load the show and tell models in android.

Related

Android + ObjectBox Search Query Issue

I am stuck with the ObjectBox Like Query. I have done as below when I search for something.
QueryBuilder<MItemDetail> builder = mItemDetailListBox.query();
builder.contains(MItemDetail_.productName, search);
itemList = builder.build().find();
For example, My data is:
paracetamol
paracetamol potest
paracetamol_new
Problem:
Now as you know the contains works simply as that returns a list of items that contain a given search string.
What I Want:
If I search para new, I want the result paracetamol_new
If I search para p, I want the result paracetamol potest
If I search para e e, I want the result paracetamol potest and paracetamol_new
Is there any function or utility available in ObjectBox that can help me to achieve this?
Do let me know If you have any questions.
Edited:
The given links in a comment, My question is different. I know all the methods contains(), startsWith, and endsWith but my problem not getting solved using that.
With Reference to this answer I have done some changes as given and I got a perfect solution as I wanted.
QueryBuilder<MItemDetail> builder = mItemDetailListBox.query();
// builder.contains(MItemDetail_.productName, search);
builder.filter(new QueryFilter<MItemDetail>() {
#Override
public boolean keep(#NonNull MItemDetail entity) {
return like(entity.getProductName(), "%"+ search + "%");
}
}).order(MItemDetail_.productName);
businessModels = builder.build().find();
In the following methods, I have added one more replace statement .replace(" ",".*?")
private static boolean like(final String str, final String expr) {
String safeString = (str == null) ? "" : str;
String regex = quoteMeta(expr);
regex = regex.replace("_", ".").replace(" ",".*?").replace("%", ".*?");
Pattern p = Pattern.compile(regex,
Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
return p.matcher(safeString).matches();
}
private static String quoteMeta(String s) {
if (s == null) {
throw new IllegalArgumentException("String cannot be null");
}
int len = s.length();
if (len == 0) {
return "";
}
StringBuilder sb = new StringBuilder(len * 2);
for (int i = 0; i < len; i++) {
char c = s.charAt(i);
if ("[](){}.*+?$^|#\\".indexOf(c) != -1) {
sb.append("\\");
}
sb.append(c);
}
return sb.toString();
}
Thank you.

How to change edittext string sum to int [duplicate]

I'm trying to write a Java routine to evaluate math expressions from String values like:
"5+3"
"10-4*5"
"(1+10)*3"
I want to avoid a lot of if-then-else statements.
How can I do this?
With JDK1.6, you can use the built-in Javascript engine.
import javax.script.ScriptEngineManager;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
public class Test {
public static void main(String[] args) throws ScriptException {
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine engine = mgr.getEngineByName("JavaScript");
String foo = "40+2";
System.out.println(engine.eval(foo));
}
}
I've written this eval method for arithmetic expressions to answer this question. It does addition, subtraction, multiplication, division, exponentiation (using the ^ symbol), and a few basic functions like sqrt. It supports grouping using (...), and it gets the operator precedence and associativity rules correct.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)` | number
// | functionName `(` expression `)` | functionName factor
// | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else return x;
}
}
double parseFactor() {
if (eat('+')) return +parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')'");
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
if (eat('(')) {
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')' after argument to " + func);
} else {
x = parseFactor();
}
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation
return x;
}
}.parse();
}
Example:
System.out.println(eval("((4 - 2^3 + 1) * -sqrt(3*3+4*4)) / 2"));
Output: 7.5 (which is correct)
The parser is a recursive descent parser, so internally uses separate parse methods for each level of operator precedence in its grammar. I deliberately kept it short, but here are some ideas you might want to expand it with:
Variables:
The bit of the parser that reads the names for functions can easily be changed to handle custom variables too, by looking up names in a variable table passed to the eval method, such as a Map<String,Double> variables.
Separate compilation and evaluation:
What if, having added support for variables, you wanted to evaluate the same expression millions of times with changed variables, without parsing it every time? It's possible. First define an interface to use to evaluate the precompiled expression:
#FunctionalInterface
interface Expression {
double eval();
}
Now to rework the original "eval" function into a "parse" function, change all the methods that return doubles, so instead they return an instance of that interface. Java 8's lambda syntax works well for this. Example of one of the changed methods:
Expression parseExpression() {
Expression x = parseTerm();
for (;;) {
if (eat('+')) { // addition
Expression a = x, b = parseTerm();
x = (() -> a.eval() + b.eval());
} else if (eat('-')) { // subtraction
Expression a = x, b = parseTerm();
x = (() -> a.eval() - b.eval());
} else {
return x;
}
}
}
That builds a recursive tree of Expression objects representing the compiled expression (an abstract syntax tree). Then you can compile it once and evaluate it repeatedly with different values:
public static void main(String[] args) {
Map<String,Double> variables = new HashMap<>();
Expression exp = parse("x^2 - x + 2", variables);
for (double x = -20; x <= +20; x++) {
variables.put("x", x);
System.out.println(x + " => " + exp.eval());
}
}
Different datatypes:
Instead of double, you could change the evaluator to use something more powerful like BigDecimal, or a class that implements complex numbers, or rational numbers (fractions). You could even use Object, allowing some mix of datatypes in expressions, just like a real programming language. :)
All code in this answer released to the public domain. Have fun!
For my university project, I was looking for a parser / evaluator supporting both basic formulas and more complicated equations (especially iterated operators). I found very nice open source library for JAVA and .NET called mXparser. I will give a few examples to make some feeling on the syntax, for further instructions please visit project website (especially tutorial section).
https://mathparser.org/
https://mathparser.org/mxparser-tutorial/
https://mathparser.org/api/
And few examples
1 - Simple furmula
Expression e = new Expression("( 2 + 3/4 + sin(pi) )/2");
double v = e.calculate()
2 - User defined arguments and constants
Argument x = new Argument("x = 10");
Constant a = new Constant("a = pi^2");
Expression e = new Expression("cos(a*x)", x, a);
double v = e.calculate()
3 - User defined functions
Function f = new Function("f(x, y, z) = sin(x) + cos(y*z)");
Expression e = new Expression("f(3,2,5)", f);
double v = e.calculate()
4 - Iteration
Expression e = new Expression("sum( i, 1, 100, sin(i) )");
double v = e.calculate()
Found recently - in case you would like to try the syntax (and see the advanced use case) you can download the Scalar Calculator app that is powered by mXparser.
The correct way to solve this is with a lexer and a parser. You can write simple versions of these yourself, or those pages also have links to Java lexers and parsers.
Creating a recursive descent parser is a really good learning exercise.
HERE is another open source library on GitHub named EvalEx.
Unlike the JavaScript engine this library is focused in evaluating mathematical expressions only. Moreover, the library is extensible and supports use of boolean operators as well as parentheses.
You can also try the BeanShell interpreter:
Interpreter interpreter = new Interpreter();
interpreter.eval("result = (7+21*6)/(32-27)");
System.out.println(interpreter.get("result"));
You can evaluate expressions easily if your Java application already accesses a database, without using any other JARs.
Some databases require you to use a dummy table (eg, Oracle's "dual" table) and others will allow you to evaluate expressions without "selecting" from any table.
For example, in Sql Server or Sqlite
select (((12.10 +12.0))/ 233.0) amount
and in Oracle
select (((12.10 +12.0))/ 233.0) amount from dual;
The advantage of using a DB is that you can evaluate many expressions at the same time. Also most DB's will allow you to use highly complex expressions and will also have a number of extra functions that can be called as necessary.
However performance may suffer if many single expressions need to be evaluated individually, particularly when the DB is located on a network server.
The following addresses the performance problem to some extent, by using a Sqlite in-memory database.
Here's a full working example in Java
Class. forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite::memory:");
Statement stat = conn.createStatement();
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount");
rs.next();
System.out.println(rs.getBigDecimal(1));
stat.close();
conn.close();
Of course you could extend the above code to handle multiple calculations at the same time.
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount, (1+100)/20.0 amount2");
Another way is to use the Spring Expression Language or SpEL which does a whole lot more along with evaluating mathematical expressions, therefore maybe slightly overkill. You do not have to be using Spring framework to use this expression library as it is stand-alone. Copying examples from SpEL's documentation:
ExpressionParser parser = new SpelExpressionParser();
int two = parser.parseExpression("1 + 1").getValue(Integer.class); // 2
double twentyFour = parser.parseExpression("2.0 * 3e0 * 4").getValue(Double.class); //24.0
This article discusses various approaches. Here are the 2 key approaches mentioned in the article:
JEXL from Apache
Allows for scripts that include references to java objects.
// Create or retrieve a JexlEngine
JexlEngine jexl = new JexlEngine();
// Create an expression object
String jexlExp = "foo.innerFoo.bar()";
Expression e = jexl.createExpression( jexlExp );
// Create a context and add data
JexlContext jctx = new MapContext();
jctx.set("foo", new Foo() );
// Now evaluate the expression, getting the result
Object o = e.evaluate(jctx);
Use the javascript engine embedded in the JDK:
private static void jsEvalWithVariable()
{
List<String> namesList = new ArrayList<String>();
namesList.add("Jill");
namesList.add("Bob");
namesList.add("Laureen");
namesList.add("Ed");
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine jsEngine = mgr.getEngineByName("JavaScript");
jsEngine.put("namesListKey", namesList);
System.out.println("Executing in script environment...");
try
{
jsEngine.eval("var x;" +
"var names = namesListKey.toArray();" +
"for(x in names) {" +
" println(names[x]);" +
"}" +
"namesListKey.add(\"Dana\");");
}
catch (ScriptException ex)
{
ex.printStackTrace();
}
}
if we are going to implement it then we can can use the below algorithm :--
While there are still tokens to be read in,
1.1 Get the next token.
1.2 If the token is:
1.2.1 A number: push it onto the value stack.
1.2.2 A variable: get its value, and push onto the value stack.
1.2.3 A left parenthesis: push it onto the operator stack.
1.2.4 A right parenthesis:
1 While the thing on top of the operator stack is not a
left parenthesis,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Pop the left parenthesis from the operator stack, and discard it.
1.2.5 An operator (call it thisOp):
1 While the operator stack is not empty, and the top thing on the
operator stack has the same or greater precedence as thisOp,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Push thisOp onto the operator stack.
While the operator stack is not empty,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
At this point the operator stack should be empty, and the value
stack should have only one value in it, which is the final result.
This is another interesting alternative
https://github.com/Shy-Ta/expression-evaluator-demo
The usage is very simple and gets the job done, for example:
ExpressionsEvaluator evalExpr = ExpressionsFactory.create("2+3*4-6/2");
assertEquals(BigDecimal.valueOf(11), evalExpr.eval());
It seems like JEP should do the job
It's too late to answer but I came across same situation to evaluate expression in java, it might help someone
MVEL does runtime evaluation of expressions, we can write a java code in String to get it evaluated in this.
String expressionStr = "x+y";
Map<String, Object> vars = new HashMap<String, Object>();
vars.put("x", 10);
vars.put("y", 20);
ExecutableStatement statement = (ExecutableStatement) MVEL.compileExpression(expressionStr);
Object result = MVEL.executeExpression(statement, vars);
Try the following sample code using JDK1.6's Javascript engine with code injection handling.
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
public class EvalUtil {
private static ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");
public static void main(String[] args) {
try {
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || 5 >3 "));
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || true"));
} catch (Exception e) {
e.printStackTrace();
}
}
public Object eval(String input) throws Exception{
try {
if(input.matches(".*[a-zA-Z;~`#$_{}\\[\\]:\\\\;\"',\\.\\?]+.*")) {
throw new Exception("Invalid expression : " + input );
}
return engine.eval(input);
} catch (Exception e) {
e.printStackTrace();
throw e;
}
}
}
This is actually complementing the answer given by #Boann. It has a slight bug which causes "-2 ^ 2" to give an erroneous result of -4.0. The problem for that is the point at which the exponentiation is evaluated in his. Just move the exponentiation to the block of parseTerm(), and you'll be all fine. Have a look at the below, which is #Boann's answer slightly modified. Modification is in the comments.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)`
// | number | functionName factor | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else if (eat('^')) x = Math.pow(x, parseFactor()); //exponentiation -> Moved in to here. So the problem is fixed
else return x;
}
}
double parseFactor() {
if (eat('+')) return parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
eat(')');
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
x = parseFactor();
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
//if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation -> This is causing a bit of problem
return x;
}
}.parse();
}
import java.util.*;
public class check {
int ans;
String str="7 + 5";
StringTokenizer st=new StringTokenizer(str);
int v1=Integer.parseInt(st.nextToken());
String op=st.nextToken();
int v2=Integer.parseInt(st.nextToken());
if(op.equals("+")) { ans= v1 + v2; }
if(op.equals("-")) { ans= v1 - v2; }
//.........
}
I think what ever way you do this it's going to involve a lot of conditional statements. But for single operations like in your examples you could limit it to 4 if statements with something like
String math = "1+4";
if (math.split("+").length == 2) {
//do calculation
} else if (math.split("-").length == 2) {
//do calculation
} ...
It gets a whole lot more complicated when you want to deal with multiple operations like "4+5*6".
If you are trying to build a calculator then I'd surgest passing each section of the calculation separatly (each number or operator) rather than as a single string.
You might have a look at the Symja framework:
ExprEvaluator util = new ExprEvaluator();
IExpr result = util.evaluate("10-40");
System.out.println(result.toString()); // -> "-30"
Take note that definitively more complex expressions can be evaluated:
// D(...) gives the derivative of the function Sin(x)*Cos(x)
IAST function = D(Times(Sin(x), Cos(x)), x);
IExpr result = util.evaluate(function);
// print: Cos(x)^2-Sin(x)^2
package ExpressionCalculator.expressioncalculator;
import java.text.DecimalFormat;
import java.util.Scanner;
public class ExpressionCalculator {
private static String addSpaces(String exp){
//Add space padding to operands.
//https://regex101.com/r/sJ9gM7/73
exp = exp.replaceAll("(?<=[0-9()])[\\/]", " / ");
exp = exp.replaceAll("(?<=[0-9()])[\\^]", " ^ ");
exp = exp.replaceAll("(?<=[0-9()])[\\*]", " * ");
exp = exp.replaceAll("(?<=[0-9()])[+]", " + ");
exp = exp.replaceAll("(?<=[0-9()])[-]", " - ");
//Keep replacing double spaces with single spaces until your string is properly formatted
/*while(exp.indexOf(" ") != -1){
exp = exp.replace(" ", " ");
}*/
exp = exp.replaceAll(" {2,}", " ");
return exp;
}
public static Double evaluate(String expr){
DecimalFormat df = new DecimalFormat("#.####");
//Format the expression properly before performing operations
String expression = addSpaces(expr);
try {
//We will evaluate using rule BDMAS, i.e. brackets, division, power, multiplication, addition and
//subtraction will be processed in following order
int indexClose = expression.indexOf(")");
int indexOpen = -1;
if (indexClose != -1) {
String substring = expression.substring(0, indexClose);
indexOpen = substring.lastIndexOf("(");
substring = substring.substring(indexOpen + 1).trim();
if(indexOpen != -1 && indexClose != -1) {
Double result = evaluate(substring);
expression = expression.substring(0, indexOpen).trim() + " " + result + " " + expression.substring(indexClose + 1).trim();
return evaluate(expression.trim());
}
}
String operation = "";
if(expression.indexOf(" / ") != -1){
operation = "/";
}else if(expression.indexOf(" ^ ") != -1){
operation = "^";
} else if(expression.indexOf(" * ") != -1){
operation = "*";
} else if(expression.indexOf(" + ") != -1){
operation = "+";
} else if(expression.indexOf(" - ") != -1){ //Avoid negative numbers
operation = "-";
} else{
return Double.parseDouble(expression);
}
int index = expression.indexOf(operation);
if(index != -1){
indexOpen = expression.lastIndexOf(" ", index - 2);
indexOpen = (indexOpen == -1)?0:indexOpen;
indexClose = expression.indexOf(" ", index + 2);
indexClose = (indexClose == -1)?expression.length():indexClose;
if(indexOpen != -1 && indexClose != -1) {
Double lhs = Double.parseDouble(expression.substring(indexOpen, index));
Double rhs = Double.parseDouble(expression.substring(index + 2, indexClose));
Double result = null;
switch (operation){
case "/":
//Prevent divide by 0 exception.
if(rhs == 0){
return null;
}
result = lhs / rhs;
break;
case "^":
result = Math.pow(lhs, rhs);
break;
case "*":
result = lhs * rhs;
break;
case "-":
result = lhs - rhs;
break;
case "+":
result = lhs + rhs;
break;
default:
break;
}
if(indexClose == expression.length()){
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose);
}else{
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose + 1);
}
return Double.valueOf(df.format(evaluate(expression.trim())));
}
}
}catch(Exception exp){
exp.printStackTrace();
}
return 0.0;
}
public static void main(String args[]){
Scanner scanner = new Scanner(System.in);
System.out.print("Enter an Mathematical Expression to Evaluate: ");
String input = scanner.nextLine();
System.out.println(evaluate(input));
}
}
A Java class that can evaluate mathematical expressions:
package test;
public class Calculator {
public static Double calculate(String expression){
if (expression == null || expression.length() == 0) {
return null;
}
return calc(expression.replace(" ", ""));
}
public static Double calc(String expression) {
String[] containerArr = new String[]{expression};
double leftVal = getNextOperand(containerArr);
expression = containerArr[0];
if (expression.length() == 0) {
return leftVal;
}
char operator = expression.charAt(0);
expression = expression.substring(1);
while (operator == '*' || operator == '/') {
containerArr[0] = expression;
double rightVal = getNextOperand(containerArr);
expression = containerArr[0];
if (operator == '*') {
leftVal = leftVal * rightVal;
} else {
leftVal = leftVal / rightVal;
}
if (expression.length() > 0) {
operator = expression.charAt(0);
expression = expression.substring(1);
} else {
return leftVal;
}
}
if (operator == '+') {
return leftVal + calc(expression);
} else {
return leftVal - calc(expression);
}
}
private static double getNextOperand(String[] exp){
double res;
if (exp[0].startsWith("(")) {
int open = 1;
int i = 1;
while (open != 0) {
if (exp[0].charAt(i) == '(') {
open++;
} else if (exp[0].charAt(i) == ')') {
open--;
}
i++;
}
res = calc(exp[0].substring(1, i - 1));
exp[0] = exp[0].substring(i);
} else {
int i = 1;
if (exp[0].charAt(0) == '-') {
i++;
}
while (exp[0].length() > i && isNumber((int) exp[0].charAt(i))) {
i++;
}
res = Double.parseDouble(exp[0].substring(0, i));
exp[0] = exp[0].substring(i);
}
return res;
}
private static boolean isNumber(int c) {
int zero = (int) '0';
int nine = (int) '9';
return (c >= zero && c <= nine) || c =='.';
}
public static void main(String[] args) {
System.out.println(calculate("(((( -6 )))) * 9 * -1"));
System.out.println(calc("(-5.2+-5*-5*((5/4+2)))"));
}
}
How about something like this:
String st = "10+3";
int result;
for(int i=0;i<st.length();i++)
{
if(st.charAt(i)=='+')
{
result=Integer.parseInt(st.substring(0, i))+Integer.parseInt(st.substring(i+1, st.length()));
System.out.print(result);
}
}
and do the similar thing for every other mathematical operator accordingly ..
It is possible to convert any expression string in infix notation to a postfix notation using Djikstra's shunting-yard algorithm. The result of the algorithm can then serve as input to the postfix algorithm with returns the result of the expression.
I wrote an article about it here, with an implementation in java
Yet another option: https://github.com/stefanhaustein/expressionparser
I have implemented this to have a simple but flexible option to permit both:
Immediate processing (Calculator.java, SetDemo.java)
Building and processing a parse tree (TreeBuilder.java)
The TreeBuilder linked above is part of a CAS demo package that does symbolic derivation. There is also a BASIC interpreter example and I have started to build a TypeScript interpreter using it.
External library like RHINO or NASHORN can be used to run javascript. And javascript can evaluate simple formula without parcing the string. No performance impact as well if code is written well.
Below is an example with RHINO -
public class RhinoApp {
private String simpleAdd = "(12+13+2-2)*2+(12+13+2-2)*2";
public void runJavaScript() {
Context jsCx = Context.enter();
Context.getCurrentContext().setOptimizationLevel(-1);
ScriptableObject scope = jsCx.initStandardObjects();
Object result = jsCx.evaluateString(scope, simpleAdd , "formula", 0, null);
Context.exit();
System.out.println(result);
}
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class test2 {
public static void main(String[] args) throws ScriptException {
String s = "10+2";
ScriptEngineManager mn = new ScriptEngineManager();
ScriptEngine en = mn.getEngineByName("js");
Object result = en.eval(s);
System.out.println(result);
}
}
I have done using iterative parsing and shunting Yard algorithm and i have really enjoyed developing the expression evaluator ,you can find all the code here
https://github.com/nagaraj200788/JavaExpressionEvaluator
Has 73 test cases and even works for Bigintegers,Bigdecimals
supports all relational, arithmetic expression and also combination of both .
even supports ternary operator .
Added enhancement to support signed numbers like -100+89 it was intresting, for details check TokenReader.isUnaryOperator() method and i have updated code in above Link

Realm used like a LinkedList in Android

I have a Realm database of sensor values with the following constructor:
public DataEntry(float x, float y, float z, long timestamp, int xIndex, String xValue) {
this.x = x;
this.y = y;
this.z = z;
this.xIndex = xIndex;
this.timestamp = timestamp;
this.xValue = xValue;
}
I would like this database to be a fixed size of a 100 points. I have this working in a LinkedList approach where I delete the first entry and add one to the end resulting in a fixed size then I use MPChartLib to plot. This is what I would like to mimic using a Realm database.
My approach is to first delete the first entry using the code below:
private void removeFirst(){
RealmResults<DataEntry> result1 = mRealm.where(DataEntry.class).findAll();
mRealm.beginTransaction();
Log.i(TAG, "remove the first element of the database ");
DataEntry first =result1.first();
first.removeFromRealm();
Log.i(TAG, "shift the indexes of the database by one " + result1.size());
for (int i = 0; i < result1.size(); i++) {
DataEntry u = result1.get(i);
int index = u.getxIndex();
u.setxIndex(index - 1);
String xValue = "" +index;
u.setxValue(xValue);
}
mRealm.commitTransaction();
}
Then I add a new data point from acquired data
#Subscribe
public void onSensorUpdatedEvent(SensorUpdatedEvent event) {
if (event.getSensor().getId() == this.sensor.getId()) {
// Log.i(TAG, "remove the first element of the database ");
removeFirst();
mRealm.beginTransaction();
DataEntry entry = mRealm.createObject(DataEntry.class);
// Log.i(TAG, "in database update event with index = " + nextIndex);
entry.setxIndex(nextIndex);
entry.setxValue("" + nextIndex);
nextIndex++;
entry.setAndroidDevice(mAndroidId);
entry.setTimestamp(event.getDataPoint().getTimestamp());
currT.setText(precsion.format(event.getDataPoint().getTimestamp()));
if (event.getDataPoint().getValues().length > 0) {
entry.setX(event.getDataPoint().getValues()[0]);
currX.setText(precsion.format(event.getDataPoint().getValues()[0]));
} else {
entry.setX(0.0f);
}
if (event.getDataPoint().getValues().length > 1) {
entry.setY(event.getDataPoint().getValues()[1]);
currY.setText(precsion.format(event.getDataPoint().getValues()[1]));
} else {
entry.setY(0.0f);
}
if (event.getDataPoint().getValues().length > 2) {
entry.setZ(event.getDataPoint().getValues()[2]);
currZ.setText(precsion.format(event.getDataPoint().getValues()[2]));
} else {
entry.setZ(0.0f);
}
deltaT.setText(precsion.format((event.getDataPoint().getTimestamp() - lastT) / scaleT));
lastT = event.getDataPoint().getTimestamp();
entry.setAccuracy(event.getDataPoint().getAccuracy());
mRealm.commitTransaction();
}
}
MPChartLib uses the xIndex to establish position along the x axis so I change these values as part of the removeFirst class. My results in MpChartLib do not give me the desired result. I may be using it beyond its application or I may not be using the Realm in the the correct manner.
What sort of makes sense is to start with 100 blank database entries then fill them one at a time until I've reached 100 then "shift" entries such that entry 1 becomes entry 0, entry 99 becomes 98 and I update entry 99 with the new data.
While I think this might work it seems very inefficient to shift values of 100 fixed database points.
I would rather prefer to delete the first one and add a new one at the end like I can do with a LinkedList.
Kind of lost as to how to proceed. Any assistance is greatly appreciated.
RealmQuery.findAll() won't give you a stable order of the results. You need to sort the results based on some fields to achieve that.
I think what you can do is combining using #PrimaryKey and RealmQuery.max().
public class DataEntry {
#PrimaryKey
private long id;
// ...
}
private void addEntry(DataEntry entry){
realm.beginTransaction();
RealmResults<DataEntry> results = realm.where(DataEntry.class).findAllSorted("id");
if (results.count.size() >= 100) {
// Remove the first entry
results.get(0).removeFromRealm();
}
// NOTE: Consider the integer overflow and the empty results here!!!!
entry.setId(restuls.max("id").longValue() + 1;
realm.copyToRealm(entry);
realm.commitTransaction();
}
The id won't be in range [1-100] but i believe you can do some math to achieve that.
Using the Primary key and sorting as beeender recommends works. I modified the timestamp as the primary key and included a string version of the timestamp which MPChartLib seems to require.
So my DataEntry Realm database
#RealmClass
public class DataEntry extends RealmObject {
private String sTimestamp;
private String androidDevice;
private float x;
private float y;
private float z;
private int accuracy;
#PrimaryKey
private long timestamp;
// no arguments constructor for Realm database
public DataEntry() {
}
public DataEntry(float x, float y, float z, long timestamp, String sTimestamp) {
this.x = x;
this.y = y;
this.z = z;
this.timestamp = timestamp;
this.sTimestamp = sTimestamp;
}
And the code for updating the database:
#Subscribe
public void onSensorUpdatedEvent(SensorUpdatedEvent event) {
if (event.getSensor().getId() == this.sensor.getId()) {
mRealm.beginTransaction();
RealmResults<DataEntry> results = mRealm.
where(DataEntry.class).findAllSorted("timestamp");
if (results.size() == 1) {
startTime = results.get(0).getTimestamp();
// Log.i(TAG, "first data received: startTime " + startTime);
}
// Log.i(TAG, "check size of the database " +results.size());
if (results.size() >= 100) {
// Log.i(TAG, "remove the first element of the database ");
results.get(0).removeFromRealm();
}
// Log.i(TAG, "create element for the database ");
DataEntry entry = mRealm.createObject(DataEntry.class);
entry.setAndroidDevice(mAndroidId);
long mTime = (long) (event.getDataPoint().getTimestamp()-startTime );
// Log.i(TAG, "time set at = " +mTime);
entry.setTimestamp(mTime);
// currT.setText(precsion.format(mTime));
entry.setsTimestamp(String.valueOf(mTime));
if (event.getDataPoint().getValues().length > 0) {
entry.setX(event.getDataPoint().getValues()[0]);
currX.setText(precsion.format(event.getDataPoint().getValues()[0]));
} else {
entry.setX(0.0f);
}
if (event.getDataPoint().getValues().length > 1) {
entry.setY(event.getDataPoint().getValues()[1]);
currY.setText(precsion.format(event.getDataPoint().getValues()[1]));
} else {
entry.setY(0.0f);
}
if (event.getDataPoint().getValues().length > 2) {
entry.setZ(event.getDataPoint().getValues()[2]);
currZ.setText(precsion.format(event.getDataPoint().getValues()[2]));
} else {
entry.setZ(0.0f);
}
deltaT.setText(precsion.format((event.getDataPoint().getTimestamp() - lastT) / scaleT));
totalT = ((event.getDataPoint().getTimestamp() - lastT ) / scaleT) * results.size()/1000;
totalTText.setText(String.valueOf(totalT));
lastT = event.getDataPoint().getTimestamp();
entry.setAccuracy(event.getDataPoint().getAccuracy());
// Log.i(TAG, "copy element to the database ");
mRealm.copyToRealm(entry);
// Log.i(TAG, " database has size = " + results.size());
mRealm.commitTransaction();
}
}
Then set the data into MPChartLib format
private void setData() {
// Log.i(TAG, "In setData perform database query , sort by timestamp");
RealmResults<DataEntry> result1 = mRealm.where(DataEntry.class).findAllSorted("timestamp");
// Log.i(TAG, "set Database size is = " + result1.size());
// Log.i(TAG, "Extract x data ");
RealmLineDataSet<DataEntry> set1 = new RealmLineDataSet<>(result1, "x");
set1.setDrawCubic(false);
set1.setLabel("Realm X");
set1.setDrawCircleHole(false);
set1.setColor(ColorTemplate.rgb("#FF5722"));
set1.setCircleColor(ColorTemplate.rgb("#FF5722"));
set1.setLineWidth(1.8f);
set1.setCircleSize(3.6f);
// Log.i(TAG, "Extract y data ");
RealmLineDataSet<DataEntry> set2 = new RealmLineDataSet<>(result1, "y");
set1.setDrawCubic(false);
set1.setLabel("Realm Y");
set1.setDrawCircleHole(false);
set1.setColor(ColorTemplate.rgb("#FF5722"));
set1.setCircleColor(ColorTemplate.rgb("#FF5722"));
set1.setLineWidth(1.8f);
set1.setCircleSize(3.6f);
// Log.i(TAG, "Extract z data ");
RealmLineDataSet<DataEntry> set3 = new RealmLineDataSet<>(result1, "z");
set1.setDrawCubic(false);
set1.setLabel("Realm Z");
set1.setDrawCircleHole(false);
set1.setColor(ColorTemplate.rgb("#FF5722"));
set1.setCircleColor(ColorTemplate.rgb("#FF5722"));
set1.setLineWidth(1.8f);
set1.setCircleSize(3.6f);
// Log.i(TAG, "add datasets ");
ArrayList<ILineDataSet> dataSets = new ArrayList<>();
dataSets.add(set1);
dataSets.add(set2);
dataSets.add(set3);
// Log.i(TAG, "Create Line Data for graphing " );
RealmLineData lineData = new RealmLineData(result1, "sTimestamp", dataSets);
// Log.i(TAG, "Set Data on Chart ");
mChart.setData(lineData);
// Log.i(TAG, "Re-draw ");
mChart.invalidate();
}
I'm ok with not having x-axis labels as they are huge numbers. my plan is to shut off the axis labels.

MPAndroidChart with null values

I'm using the MPAndroidChart and am really enjoying it.
A 'little' need I have is that I can put null values to the 'entrys'. I'm monitoring the apache conections on servers of my system, and I would to see if they is down (where I put the null value) or if they just no conections (0).
I tried, but the Entry class don't accept 'null' as value showing the message: 'The constructor Entry(null, int) is undefined'
Thanks!
A possible solution for you could be to check weather the object you received is null, or not. If the object is null, you don't even create an Entry object instead of just setting it's value to null.
Example:
// array that contains the information you want to display
ConnectionHolder[] connectionHolders = ...;
ArrayList<Entry> entries = new ArrayList<Entry>();
int cnt = 0;
for(ConnectionHolder ch : connectionHolders) {
if(ch != null) entries.add(new Entry(ch.getNrOfConnections(), cnt));
else {
// do nothing
}
cnt++; // always increment
}
This would create e.g. a LineChart where no circles are drawn on indices where the ConnectionHolder object was null.
For a future release of the library, I will try to add the feature so that null values are supported.
My solution is to draw another DataSet with TRANSPARENT (or arbitrary) color:
- chart with fixed number of X values
- Y values are updated periodically
- boolean flag indicate transparent part (or another color)
private static final int SERIES_SIZE = 360;
int xIndex = -1;
float xIndexVal;
private LineChart chart;
private boolean currentFlag;
public void createChart(LineDataSet dataSet) {
LineData chartData = new LineData();
prepareDataSet(dataSet);
chartData.addDataSet(dataSet);
for (int i = 0; i < SERIES_SIZE; i++) {
chartData.addXValue("" /*+ i*/);
}
chart.setData(chartData);
}
private void prepareDataSet(LineDataSet dataSet, YAxis axis, int color) {
// configure set
}
public void update(Float val, boolean flag) {
List<ILineDataSet> dsl = chart.getData().getDataSets();
Log.d("chart", String.format("%s --- %d sets, index %d", descr, dsl.size(), xIndex));
if (xIndex == SERIES_SIZE - 1) {
// remove all entries at X index 0
for (int i = 0; i < chart.getData().getDataSetCount(); i++) {
Entry entry0 = chart.getData().getDataSetByIndex(i).getEntryForIndex(0);
if (entry0 != null && entry0.getXIndex() == 0) {
chart.getData().removeEntry(entry0, i);
Log.d("chart", String.format("entry 0 removed from dataset %d, %d entries in the set", i, chart.getData().getDataSetByIndex(i).getEntryCount()));
}
else {
Log.d("chart", String.format("all %d entries in the set kept", chart.getData().getDataSetByIndex(i).getEntryCount()));
}
}
// remove empty set, if any
for (Iterator<ILineDataSet> mit = dsl.iterator(); mit.hasNext(); ) {
if (mit.next().getEntryCount() == 0) {
mit.remove();
Log.d("chart", String.format("set removed, %d sets", dsl.size()));
}
}
// move all entries by -1
for (ILineDataSet ds : dsl) {
for (Entry entry : ((LineDataSet)ds).getYVals()) {
entry.setXIndex(entry.getXIndex() - 1);
}
}
}
else {
xIndex++;
}
if (currentFlag != flag) {
currentFlag = !currentFlag;
LineDataSet set = new LineDataSet(null, "");
prepareDataSet(set, chart.getAxisLeft(), currentFlag ? Color.TRANSPARENT : Color.BLUE);
chart.getData().addDataSet(set);
if (xIndex != 0) {
chart.getData().addEntry(new Entry(xIndexVal, xIndex - 1), dsl.size() - 1);
}
}
xIndexVal = val;
chart.getData().addEntry(new Entry(val, xIndex), dsl.size() - 1);
chart.notifyDataSetChanged();
chart.invalidate();
}

Indexing Android

My problem is I have around 1000+ records in an Android App
string field1;
string field2;
string field3;
string field4;
//...
I want to search in this set of records and get the best results on two fields (field1 and field2).
Currently I read each record and compare() (string compare) with the text i want to search so that takes a long time.
What is the best method to perform search?
Store each records in SQLite DB and do "select query where like"
Hash-Mapped
? any other suggestions?
Or may be create an Index of the records and do search.
If you want to search for not exact matches, I would try to make an ArrayList of MyAppRecord where
public class MyAppRecord {
private String record;
private int deviance;
}
and get for each record the deviance of the String you want to find with:
public static int getLevenshteinDistance (String s, String t) {
if (s == null || t == null) {
throw new IllegalArgumentException("Strings must not be null");
}
int n = s.length(); // length of s
int m = t.length(); // length of t
if (n == 0) {
return m;
} else if (m == 0) {
return n;
}
int p[] = new int[n+1]; //'previous' cost array, horizontally
int d[] = new int[n+1]; // cost array, horizontally
int _d[]; //placeholder to assist in swapping p and d
// indexes into strings s and t
int i; // iterates through s
int j; // iterates through t
char t_j; // jth character of t
int cost; // cost
for (i = 0; i<=n; i++) {
p[i] = i;
}
for (j = 1; j<=m; j++) {
t_j = t.charAt(j-1);
d[0] = j;
for (i=1; i<=n; i++) {
cost = s.charAt(i-1)==t_j ? 0 : 1;
// minimum of cell to the left+1, to the top+1, diagonally left and up +cost
d[i] = Math.min(Math.min(d[i-1]+1, p[i]+1), p[i-1]+cost);
}
// copy current distance counts to 'previous row' distance counts
_d = p;
p = d;
d = _d;
}
// our last action in the above loop was to switch d and p, so p now
// actually has the most recent cost counts
return p[n];
}
}
save it to your MyAppRecord-object and finally sort your ArrayList by the deviance of its MyAppRecord-objects.
Note that this could take some time, depending on your set of records. And NOTE that there is no way of telling wether dogA or dogB is on a certain position in your list by searching for dog.
Read up on the Levensthein distance to get a feeling on how it works. You may get the idea of sorting out strings that are possibly to long/short to get a distance that is okay for a threshold you may have.
It is also possible to copy "good enough" results to a different ArrayList.

Categories

Resources