I'm trying to write a Java routine to evaluate math expressions from String values like:
"5+3"
"10-4*5"
"(1+10)*3"
I want to avoid a lot of if-then-else statements.
How can I do this?
With JDK1.6, you can use the built-in Javascript engine.
import javax.script.ScriptEngineManager;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
public class Test {
public static void main(String[] args) throws ScriptException {
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine engine = mgr.getEngineByName("JavaScript");
String foo = "40+2";
System.out.println(engine.eval(foo));
}
}
I've written this eval method for arithmetic expressions to answer this question. It does addition, subtraction, multiplication, division, exponentiation (using the ^ symbol), and a few basic functions like sqrt. It supports grouping using (...), and it gets the operator precedence and associativity rules correct.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)` | number
// | functionName `(` expression `)` | functionName factor
// | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else return x;
}
}
double parseFactor() {
if (eat('+')) return +parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')'");
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
if (eat('(')) {
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')' after argument to " + func);
} else {
x = parseFactor();
}
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation
return x;
}
}.parse();
}
Example:
System.out.println(eval("((4 - 2^3 + 1) * -sqrt(3*3+4*4)) / 2"));
Output: 7.5 (which is correct)
The parser is a recursive descent parser, so internally uses separate parse methods for each level of operator precedence in its grammar. I deliberately kept it short, but here are some ideas you might want to expand it with:
Variables:
The bit of the parser that reads the names for functions can easily be changed to handle custom variables too, by looking up names in a variable table passed to the eval method, such as a Map<String,Double> variables.
Separate compilation and evaluation:
What if, having added support for variables, you wanted to evaluate the same expression millions of times with changed variables, without parsing it every time? It's possible. First define an interface to use to evaluate the precompiled expression:
#FunctionalInterface
interface Expression {
double eval();
}
Now to rework the original "eval" function into a "parse" function, change all the methods that return doubles, so instead they return an instance of that interface. Java 8's lambda syntax works well for this. Example of one of the changed methods:
Expression parseExpression() {
Expression x = parseTerm();
for (;;) {
if (eat('+')) { // addition
Expression a = x, b = parseTerm();
x = (() -> a.eval() + b.eval());
} else if (eat('-')) { // subtraction
Expression a = x, b = parseTerm();
x = (() -> a.eval() - b.eval());
} else {
return x;
}
}
}
That builds a recursive tree of Expression objects representing the compiled expression (an abstract syntax tree). Then you can compile it once and evaluate it repeatedly with different values:
public static void main(String[] args) {
Map<String,Double> variables = new HashMap<>();
Expression exp = parse("x^2 - x + 2", variables);
for (double x = -20; x <= +20; x++) {
variables.put("x", x);
System.out.println(x + " => " + exp.eval());
}
}
Different datatypes:
Instead of double, you could change the evaluator to use something more powerful like BigDecimal, or a class that implements complex numbers, or rational numbers (fractions). You could even use Object, allowing some mix of datatypes in expressions, just like a real programming language. :)
All code in this answer released to the public domain. Have fun!
For my university project, I was looking for a parser / evaluator supporting both basic formulas and more complicated equations (especially iterated operators). I found very nice open source library for JAVA and .NET called mXparser. I will give a few examples to make some feeling on the syntax, for further instructions please visit project website (especially tutorial section).
https://mathparser.org/
https://mathparser.org/mxparser-tutorial/
https://mathparser.org/api/
And few examples
1 - Simple furmula
Expression e = new Expression("( 2 + 3/4 + sin(pi) )/2");
double v = e.calculate()
2 - User defined arguments and constants
Argument x = new Argument("x = 10");
Constant a = new Constant("a = pi^2");
Expression e = new Expression("cos(a*x)", x, a);
double v = e.calculate()
3 - User defined functions
Function f = new Function("f(x, y, z) = sin(x) + cos(y*z)");
Expression e = new Expression("f(3,2,5)", f);
double v = e.calculate()
4 - Iteration
Expression e = new Expression("sum( i, 1, 100, sin(i) )");
double v = e.calculate()
Found recently - in case you would like to try the syntax (and see the advanced use case) you can download the Scalar Calculator app that is powered by mXparser.
The correct way to solve this is with a lexer and a parser. You can write simple versions of these yourself, or those pages also have links to Java lexers and parsers.
Creating a recursive descent parser is a really good learning exercise.
HERE is another open source library on GitHub named EvalEx.
Unlike the JavaScript engine this library is focused in evaluating mathematical expressions only. Moreover, the library is extensible and supports use of boolean operators as well as parentheses.
You can also try the BeanShell interpreter:
Interpreter interpreter = new Interpreter();
interpreter.eval("result = (7+21*6)/(32-27)");
System.out.println(interpreter.get("result"));
You can evaluate expressions easily if your Java application already accesses a database, without using any other JARs.
Some databases require you to use a dummy table (eg, Oracle's "dual" table) and others will allow you to evaluate expressions without "selecting" from any table.
For example, in Sql Server or Sqlite
select (((12.10 +12.0))/ 233.0) amount
and in Oracle
select (((12.10 +12.0))/ 233.0) amount from dual;
The advantage of using a DB is that you can evaluate many expressions at the same time. Also most DB's will allow you to use highly complex expressions and will also have a number of extra functions that can be called as necessary.
However performance may suffer if many single expressions need to be evaluated individually, particularly when the DB is located on a network server.
The following addresses the performance problem to some extent, by using a Sqlite in-memory database.
Here's a full working example in Java
Class. forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite::memory:");
Statement stat = conn.createStatement();
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount");
rs.next();
System.out.println(rs.getBigDecimal(1));
stat.close();
conn.close();
Of course you could extend the above code to handle multiple calculations at the same time.
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount, (1+100)/20.0 amount2");
Another way is to use the Spring Expression Language or SpEL which does a whole lot more along with evaluating mathematical expressions, therefore maybe slightly overkill. You do not have to be using Spring framework to use this expression library as it is stand-alone. Copying examples from SpEL's documentation:
ExpressionParser parser = new SpelExpressionParser();
int two = parser.parseExpression("1 + 1").getValue(Integer.class); // 2
double twentyFour = parser.parseExpression("2.0 * 3e0 * 4").getValue(Double.class); //24.0
This article discusses various approaches. Here are the 2 key approaches mentioned in the article:
JEXL from Apache
Allows for scripts that include references to java objects.
// Create or retrieve a JexlEngine
JexlEngine jexl = new JexlEngine();
// Create an expression object
String jexlExp = "foo.innerFoo.bar()";
Expression e = jexl.createExpression( jexlExp );
// Create a context and add data
JexlContext jctx = new MapContext();
jctx.set("foo", new Foo() );
// Now evaluate the expression, getting the result
Object o = e.evaluate(jctx);
Use the javascript engine embedded in the JDK:
private static void jsEvalWithVariable()
{
List<String> namesList = new ArrayList<String>();
namesList.add("Jill");
namesList.add("Bob");
namesList.add("Laureen");
namesList.add("Ed");
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine jsEngine = mgr.getEngineByName("JavaScript");
jsEngine.put("namesListKey", namesList);
System.out.println("Executing in script environment...");
try
{
jsEngine.eval("var x;" +
"var names = namesListKey.toArray();" +
"for(x in names) {" +
" println(names[x]);" +
"}" +
"namesListKey.add(\"Dana\");");
}
catch (ScriptException ex)
{
ex.printStackTrace();
}
}
if we are going to implement it then we can can use the below algorithm :--
While there are still tokens to be read in,
1.1 Get the next token.
1.2 If the token is:
1.2.1 A number: push it onto the value stack.
1.2.2 A variable: get its value, and push onto the value stack.
1.2.3 A left parenthesis: push it onto the operator stack.
1.2.4 A right parenthesis:
1 While the thing on top of the operator stack is not a
left parenthesis,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Pop the left parenthesis from the operator stack, and discard it.
1.2.5 An operator (call it thisOp):
1 While the operator stack is not empty, and the top thing on the
operator stack has the same or greater precedence as thisOp,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Push thisOp onto the operator stack.
While the operator stack is not empty,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
At this point the operator stack should be empty, and the value
stack should have only one value in it, which is the final result.
This is another interesting alternative
https://github.com/Shy-Ta/expression-evaluator-demo
The usage is very simple and gets the job done, for example:
ExpressionsEvaluator evalExpr = ExpressionsFactory.create("2+3*4-6/2");
assertEquals(BigDecimal.valueOf(11), evalExpr.eval());
It seems like JEP should do the job
It's too late to answer but I came across same situation to evaluate expression in java, it might help someone
MVEL does runtime evaluation of expressions, we can write a java code in String to get it evaluated in this.
String expressionStr = "x+y";
Map<String, Object> vars = new HashMap<String, Object>();
vars.put("x", 10);
vars.put("y", 20);
ExecutableStatement statement = (ExecutableStatement) MVEL.compileExpression(expressionStr);
Object result = MVEL.executeExpression(statement, vars);
Try the following sample code using JDK1.6's Javascript engine with code injection handling.
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
public class EvalUtil {
private static ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");
public static void main(String[] args) {
try {
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || 5 >3 "));
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || true"));
} catch (Exception e) {
e.printStackTrace();
}
}
public Object eval(String input) throws Exception{
try {
if(input.matches(".*[a-zA-Z;~`#$_{}\\[\\]:\\\\;\"',\\.\\?]+.*")) {
throw new Exception("Invalid expression : " + input );
}
return engine.eval(input);
} catch (Exception e) {
e.printStackTrace();
throw e;
}
}
}
This is actually complementing the answer given by #Boann. It has a slight bug which causes "-2 ^ 2" to give an erroneous result of -4.0. The problem for that is the point at which the exponentiation is evaluated in his. Just move the exponentiation to the block of parseTerm(), and you'll be all fine. Have a look at the below, which is #Boann's answer slightly modified. Modification is in the comments.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)`
// | number | functionName factor | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else if (eat('^')) x = Math.pow(x, parseFactor()); //exponentiation -> Moved in to here. So the problem is fixed
else return x;
}
}
double parseFactor() {
if (eat('+')) return parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
eat(')');
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
x = parseFactor();
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
//if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation -> This is causing a bit of problem
return x;
}
}.parse();
}
import java.util.*;
public class check {
int ans;
String str="7 + 5";
StringTokenizer st=new StringTokenizer(str);
int v1=Integer.parseInt(st.nextToken());
String op=st.nextToken();
int v2=Integer.parseInt(st.nextToken());
if(op.equals("+")) { ans= v1 + v2; }
if(op.equals("-")) { ans= v1 - v2; }
//.........
}
I think what ever way you do this it's going to involve a lot of conditional statements. But for single operations like in your examples you could limit it to 4 if statements with something like
String math = "1+4";
if (math.split("+").length == 2) {
//do calculation
} else if (math.split("-").length == 2) {
//do calculation
} ...
It gets a whole lot more complicated when you want to deal with multiple operations like "4+5*6".
If you are trying to build a calculator then I'd surgest passing each section of the calculation separatly (each number or operator) rather than as a single string.
You might have a look at the Symja framework:
ExprEvaluator util = new ExprEvaluator();
IExpr result = util.evaluate("10-40");
System.out.println(result.toString()); // -> "-30"
Take note that definitively more complex expressions can be evaluated:
// D(...) gives the derivative of the function Sin(x)*Cos(x)
IAST function = D(Times(Sin(x), Cos(x)), x);
IExpr result = util.evaluate(function);
// print: Cos(x)^2-Sin(x)^2
package ExpressionCalculator.expressioncalculator;
import java.text.DecimalFormat;
import java.util.Scanner;
public class ExpressionCalculator {
private static String addSpaces(String exp){
//Add space padding to operands.
//https://regex101.com/r/sJ9gM7/73
exp = exp.replaceAll("(?<=[0-9()])[\\/]", " / ");
exp = exp.replaceAll("(?<=[0-9()])[\\^]", " ^ ");
exp = exp.replaceAll("(?<=[0-9()])[\\*]", " * ");
exp = exp.replaceAll("(?<=[0-9()])[+]", " + ");
exp = exp.replaceAll("(?<=[0-9()])[-]", " - ");
//Keep replacing double spaces with single spaces until your string is properly formatted
/*while(exp.indexOf(" ") != -1){
exp = exp.replace(" ", " ");
}*/
exp = exp.replaceAll(" {2,}", " ");
return exp;
}
public static Double evaluate(String expr){
DecimalFormat df = new DecimalFormat("#.####");
//Format the expression properly before performing operations
String expression = addSpaces(expr);
try {
//We will evaluate using rule BDMAS, i.e. brackets, division, power, multiplication, addition and
//subtraction will be processed in following order
int indexClose = expression.indexOf(")");
int indexOpen = -1;
if (indexClose != -1) {
String substring = expression.substring(0, indexClose);
indexOpen = substring.lastIndexOf("(");
substring = substring.substring(indexOpen + 1).trim();
if(indexOpen != -1 && indexClose != -1) {
Double result = evaluate(substring);
expression = expression.substring(0, indexOpen).trim() + " " + result + " " + expression.substring(indexClose + 1).trim();
return evaluate(expression.trim());
}
}
String operation = "";
if(expression.indexOf(" / ") != -1){
operation = "/";
}else if(expression.indexOf(" ^ ") != -1){
operation = "^";
} else if(expression.indexOf(" * ") != -1){
operation = "*";
} else if(expression.indexOf(" + ") != -1){
operation = "+";
} else if(expression.indexOf(" - ") != -1){ //Avoid negative numbers
operation = "-";
} else{
return Double.parseDouble(expression);
}
int index = expression.indexOf(operation);
if(index != -1){
indexOpen = expression.lastIndexOf(" ", index - 2);
indexOpen = (indexOpen == -1)?0:indexOpen;
indexClose = expression.indexOf(" ", index + 2);
indexClose = (indexClose == -1)?expression.length():indexClose;
if(indexOpen != -1 && indexClose != -1) {
Double lhs = Double.parseDouble(expression.substring(indexOpen, index));
Double rhs = Double.parseDouble(expression.substring(index + 2, indexClose));
Double result = null;
switch (operation){
case "/":
//Prevent divide by 0 exception.
if(rhs == 0){
return null;
}
result = lhs / rhs;
break;
case "^":
result = Math.pow(lhs, rhs);
break;
case "*":
result = lhs * rhs;
break;
case "-":
result = lhs - rhs;
break;
case "+":
result = lhs + rhs;
break;
default:
break;
}
if(indexClose == expression.length()){
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose);
}else{
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose + 1);
}
return Double.valueOf(df.format(evaluate(expression.trim())));
}
}
}catch(Exception exp){
exp.printStackTrace();
}
return 0.0;
}
public static void main(String args[]){
Scanner scanner = new Scanner(System.in);
System.out.print("Enter an Mathematical Expression to Evaluate: ");
String input = scanner.nextLine();
System.out.println(evaluate(input));
}
}
A Java class that can evaluate mathematical expressions:
package test;
public class Calculator {
public static Double calculate(String expression){
if (expression == null || expression.length() == 0) {
return null;
}
return calc(expression.replace(" ", ""));
}
public static Double calc(String expression) {
String[] containerArr = new String[]{expression};
double leftVal = getNextOperand(containerArr);
expression = containerArr[0];
if (expression.length() == 0) {
return leftVal;
}
char operator = expression.charAt(0);
expression = expression.substring(1);
while (operator == '*' || operator == '/') {
containerArr[0] = expression;
double rightVal = getNextOperand(containerArr);
expression = containerArr[0];
if (operator == '*') {
leftVal = leftVal * rightVal;
} else {
leftVal = leftVal / rightVal;
}
if (expression.length() > 0) {
operator = expression.charAt(0);
expression = expression.substring(1);
} else {
return leftVal;
}
}
if (operator == '+') {
return leftVal + calc(expression);
} else {
return leftVal - calc(expression);
}
}
private static double getNextOperand(String[] exp){
double res;
if (exp[0].startsWith("(")) {
int open = 1;
int i = 1;
while (open != 0) {
if (exp[0].charAt(i) == '(') {
open++;
} else if (exp[0].charAt(i) == ')') {
open--;
}
i++;
}
res = calc(exp[0].substring(1, i - 1));
exp[0] = exp[0].substring(i);
} else {
int i = 1;
if (exp[0].charAt(0) == '-') {
i++;
}
while (exp[0].length() > i && isNumber((int) exp[0].charAt(i))) {
i++;
}
res = Double.parseDouble(exp[0].substring(0, i));
exp[0] = exp[0].substring(i);
}
return res;
}
private static boolean isNumber(int c) {
int zero = (int) '0';
int nine = (int) '9';
return (c >= zero && c <= nine) || c =='.';
}
public static void main(String[] args) {
System.out.println(calculate("(((( -6 )))) * 9 * -1"));
System.out.println(calc("(-5.2+-5*-5*((5/4+2)))"));
}
}
How about something like this:
String st = "10+3";
int result;
for(int i=0;i<st.length();i++)
{
if(st.charAt(i)=='+')
{
result=Integer.parseInt(st.substring(0, i))+Integer.parseInt(st.substring(i+1, st.length()));
System.out.print(result);
}
}
and do the similar thing for every other mathematical operator accordingly ..
It is possible to convert any expression string in infix notation to a postfix notation using Djikstra's shunting-yard algorithm. The result of the algorithm can then serve as input to the postfix algorithm with returns the result of the expression.
I wrote an article about it here, with an implementation in java
Yet another option: https://github.com/stefanhaustein/expressionparser
I have implemented this to have a simple but flexible option to permit both:
Immediate processing (Calculator.java, SetDemo.java)
Building and processing a parse tree (TreeBuilder.java)
The TreeBuilder linked above is part of a CAS demo package that does symbolic derivation. There is also a BASIC interpreter example and I have started to build a TypeScript interpreter using it.
External library like RHINO or NASHORN can be used to run javascript. And javascript can evaluate simple formula without parcing the string. No performance impact as well if code is written well.
Below is an example with RHINO -
public class RhinoApp {
private String simpleAdd = "(12+13+2-2)*2+(12+13+2-2)*2";
public void runJavaScript() {
Context jsCx = Context.enter();
Context.getCurrentContext().setOptimizationLevel(-1);
ScriptableObject scope = jsCx.initStandardObjects();
Object result = jsCx.evaluateString(scope, simpleAdd , "formula", 0, null);
Context.exit();
System.out.println(result);
}
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class test2 {
public static void main(String[] args) throws ScriptException {
String s = "10+2";
ScriptEngineManager mn = new ScriptEngineManager();
ScriptEngine en = mn.getEngineByName("js");
Object result = en.eval(s);
System.out.println(result);
}
}
I have done using iterative parsing and shunting Yard algorithm and i have really enjoyed developing the expression evaluator ,you can find all the code here
https://github.com/nagaraj200788/JavaExpressionEvaluator
Has 73 test cases and even works for Bigintegers,Bigdecimals
supports all relational, arithmetic expression and also combination of both .
even supports ternary operator .
Added enhancement to support signed numbers like -100+89 it was intresting, for details check TokenReader.isUnaryOperator() method and i have updated code in above Link
Related
While working with an Android application where I have observed that the Unicode are the part of API response as:
{
"msg_txt":"Laurent Ruquier et l'\\u00e9quipe"
}
On the Android side, I have a simple TextView where I have to show the actual text by converting a Unicode into text.
Since you have the unicode characters in the response already you will need to parse it. I am not a friend of reinventing the wheel so you would usually use something like the apache commons unescapeJava. If you decide to add it to your gradle build as a dependency make sure to correctly configure your R8 shrinking (see here), otherwise you will add a gigantic amount of methods and classes to your release builds. This can also slow down debug builds considerably so beware.
While the method above does actually even more than just replace escaped unicode chars, if you just need this feature and don't want to add the apache dependency for just this code we can take a look at their implementation here and here. I have extracted the relevant part and converted it to Kotlin just for fun:
val inputString = "Laurent Ruquier et l'\\u00e9quipe"
val unescaped = translate(inputString)
//this is the main method that does the actual conversion on a character
// by character basis.
fun translate(input: CharSequence, index: Int, out: Writer): Int {
if (input[index] == '\\' && index + 1 < input.length && input[index + 1] == 'u') {
// consume optional additional 'u' chars
var i = 2
while (index + i < input.length && input[index + i] == 'u') {
i++
}
if (index + i < input.length && input[index + i] == '+') {
i++
}
if (index + i + 4 <= input.length) {
// Get 4 hex digits
val unicode = input.subSequence(index + i, index + i + 4)
try {
val value = unicode.toString().toInt(16)
out.write(value)
} catch (nfe: NumberFormatException) {
throw IllegalArgumentException("Unable to parse unicode value: $unicode", nfe)
}
return i + 4
}
throw IllegalArgumentException(
"Less than 4 hex digits in unicode value: '"
+ input.subSequence(index, input.length)
+ "' due to end of CharSequence"
)
}
return 0
}
//helper method for working directly with strings
fun translate(input: CharSequence): String {
val writer = StringWriter(input.length * 2)
translate(input, writer)
return writer.toString()
}
// this goes through the actual char sequence and passes every
// single char to the unicode transformer and swallows consumed chars
fun translate(input: CharSequence, out: Writer) {
var pos = 0
val len = input.length
while (pos < len) {
val consumed = translate(input, pos, out)
if (consumed == 0) {
// inlined implementation of Character.toChars(Character.codePointAt(input, pos))
// avoids allocating temp char arrays and duplicate checks
val c1 = input[pos]
out.write(c1.toInt())
pos++
if (Character.isHighSurrogate(c1) && pos < len) {
val c2 = input[pos]
if (Character.isLowSurrogate(c2)) {
out.write(c2.toInt())
pos++
}
}
continue
}
// contract with translators is that they have to understand codepoints
// and they just took care of a surrogate pair
for (pt in 0 until consumed) {
pos += Character.charCount(Character.codePointAt(input, pos))
}
}
}
Note this is really bare bones and might need to be adjusted a bit for actual production use. If e.g. your input will have both of the backslashes in there (like val inputString = "Laurent Ruquier et l'\\\\u00e9quipe") you will need to modify the method a bit.
I saw many questions about this, and tried to solve the problem, but after one hour of googling and a lots of trial & error, I still can't fix it. I hope some of you catch the problem.
This is what I get:
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:835)
at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:453)
at java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:392)
at java.util.ComparableTimSort.sort(ComparableTimSort.java:191)
at java.util.ComparableTimSort.sort(ComparableTimSort.java:146)
at java.util.Arrays.sort(Arrays.java:472)
at java.util.Collections.sort(Collections.java:155)
...
And this is my comparator:
#Override
public int compareTo(Object o) {
if(this == o){
return 0;
}
CollectionItem item = (CollectionItem) o;
Card card1 = CardCache.getInstance().getCard(cardId);
Card card2 = CardCache.getInstance().getCard(item.getCardId());
if (card1.getSet() < card2.getSet()) {
return -1;
} else {
if (card1.getSet() == card2.getSet()) {
if (card1.getRarity() < card2.getRarity()) {
return 1;
} else {
if (card1.getId() == card2.getId()) {
if (cardType > item.getCardType()) {
return 1;
} else {
if (cardType == item.getCardType()) {
return 0;
}
return -1;
}
}
return -1;
}
}
return 1;
}
}
Any idea?
The exception message is actually pretty descriptive. The contract it mentions is transitivity: if A > B and B > C then for any A, B and C: A > C. I checked it with paper and pencil and your code seems to have few holes:
if (card1.getRarity() < card2.getRarity()) {
return 1;
you do not return -1 if card1.getRarity() > card2.getRarity().
if (card1.getId() == card2.getId()) {
//...
}
return -1;
You return -1 if ids aren't equal. You should return -1 or 1 depending on which id was bigger.
Take a look at this. Apart from being much more readable, I think it should actually work:
if (card1.getSet() > card2.getSet()) {
return 1;
}
if (card1.getSet() < card2.getSet()) {
return -1;
};
if (card1.getRarity() < card2.getRarity()) {
return 1;
}
if (card1.getRarity() > card2.getRarity()) {
return -1;
}
if (card1.getId() > card2.getId()) {
return 1;
}
if (card1.getId() < card2.getId()) {
return -1;
}
return cardType - item.getCardType(); //watch out for overflow!
You can use the following class to pinpoint transitivity bugs in your Comparators:
/**
* #author Gili Tzabari
*/
public final class Comparators
{
/**
* Verify that a comparator is transitive.
*
* #param <T> the type being compared
* #param comparator the comparator to test
* #param elements the elements to test against
* #throws AssertionError if the comparator is not transitive
*/
public static <T> void verifyTransitivity(Comparator<T> comparator, Collection<T> elements)
{
for (T first: elements)
{
for (T second: elements)
{
int result1 = comparator.compare(first, second);
int result2 = comparator.compare(second, first);
if (result1 != -result2)
{
// Uncomment the following line to step through the failed case
//comparator.compare(first, second);
throw new AssertionError("compare(" + first + ", " + second + ") == " + result1 +
" but swapping the parameters returns " + result2);
}
}
}
for (T first: elements)
{
for (T second: elements)
{
int firstGreaterThanSecond = comparator.compare(first, second);
if (firstGreaterThanSecond <= 0)
continue;
for (T third: elements)
{
int secondGreaterThanThird = comparator.compare(second, third);
if (secondGreaterThanThird <= 0)
continue;
int firstGreaterThanThird = comparator.compare(first, third);
if (firstGreaterThanThird <= 0)
{
// Uncomment the following line to step through the failed case
//comparator.compare(first, third);
throw new AssertionError("compare(" + first + ", " + second + ") > 0, " +
"compare(" + second + ", " + third + ") > 0, but compare(" + first + ", " + third + ") == " +
firstGreaterThanThird);
}
}
}
}
}
/**
* Prevent construction.
*/
private Comparators()
{
}
}
Simply invoke Comparators.verifyTransitivity(myComparator, myCollection) in front of the code that fails.
It also has something to do with the version of JDK.
If it does well in JDK6, maybe it will have the problem in JDK 7 described by you, because the implementation method in jdk 7 has been changed.
Look at this:
Description: The sorting algorithm used by java.util.Arrays.sort and (indirectly) by java.util.Collections.sort has been replaced. The new sort implementation may throw an IllegalArgumentException if it detects a Comparable that violates the Comparable contract. The previous implementation silently ignored such a situation. If the previous behavior is desired, you can use the new system property, java.util.Arrays.useLegacyMergeSort, to restore previous mergesort behaviour.
I don't know the exact reason. However, if you add the code before you use sort. It will be OK.
System.setProperty("java.util.Arrays.useLegacyMergeSort", "true");
Consider the following case:
First, o1.compareTo(o2) is called. card1.getSet() == card2.getSet() happens to be true and so is card1.getRarity() < card2.getRarity(), so you return 1.
Then, o2.compareTo(o1) is called. Again, card1.getSet() == card2.getSet() is true. Then, you skip to the following else, then card1.getId() == card2.getId() happens to be true, and so is cardType > item.getCardType(). You return 1 again.
From that, o1 > o2, and o2 > o1. You broke the contract.
if (card1.getRarity() < card2.getRarity()) {
return 1;
However, if card2.getRarity() is less than card1.getRarity() you might not return -1.
You similarly miss other cases. I would do this, you can change around depending on your intent:
public int compareTo(Object o) {
if(this == o){
return 0;
}
CollectionItem item = (CollectionItem) o;
Card card1 = CardCache.getInstance().getCard(cardId);
Card card2 = CardCache.getInstance().getCard(item.getCardId());
int comp=card1.getSet() - card2.getSet();
if (comp!=0){
return comp;
}
comp=card1.getRarity() - card2.getRarity();
if (comp!=0){
return comp;
}
comp=card1.getSet() - card2.getSet();
if (comp!=0){
return comp;
}
comp=card1.getId() - card2.getId();
if (comp!=0){
return comp;
}
comp=card1.getCardType() - card2.getCardType();
return comp;
}
}
I had the same symptom. For me it turned out that another thread was modifying the compared objects while the sorting was happening in a Stream. To resolve the issue, I mapped the objects to immutable temporary objects, collected the Stream to a temporary Collection and did the sorting on that.
The origin of this exception is a wrong Comparator implementation. By checking the docs, we must implement the compare(o1, o2) method as an equivalence relation by following the rules:
if a.equals(b) is true then compare(a, b) is 0
if a.compare(b) > 0 then b.compare(a) < 0 is true
if a.compare(b) > 0 and b.compare(c) > 0 then a.compare(c) > 0 is true
You may check your code to realize where your implementation is offending one or more of Comparator contract rules. If it is hard to find it by a static analysis, you can use the data which cast the exception to check the rules.
If you try to run this code you will meet the kind this exception:
public static void main(String[] args) {
Random random = new Random();
List<Integer> list = new ArrayList<>();
for (int i = 0; i < 50000; i++) {
list.add(random.nextInt());
}
list.sort((x, y) -> {
int c = random.nextInt(3);
if (c == 0) {
return 0;
}
if (c == 1) {
return 1;
}
return -1;
});
}
Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeLo(TimSort.java:777)
at java.util.TimSort.mergeAt(TimSort.java:514)
at java.util.TimSort.mergeCollapse(TimSort.java:441)
at java.util.TimSort.sort(TimSort.java:245)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1462)
at Test.main(Test.java:14)
The reason is when implementing the Comparator, it may meet the case of A > B and B > C and C > A and the sort method will be run around to be broken. Java prevent this case by throw exception this case:
class TimSort<T> {
.
.
.
else if (len1 == 0) {
throw new IllegalArgumentException(
"Comparison method violates its general contract!");
.
.
.
In conclusion, to handle this issue. You have to make sure the comparator will not meet the case of A > B and B > C and C > A.
I got the same error with a class like the following StockPickBean. Called from this code:
List<StockPickBean> beansListcatMap.getValue();
beansList.sort(StockPickBean.Comparators.VALUE);
public class StockPickBean implements Comparable<StockPickBean> {
private double value;
public double getValue() { return value; }
public void setValue(double value) { this.value = value; }
#Override
public int compareTo(StockPickBean view) {
return Comparators.VALUE.compare(this,view); //return
Comparators.SYMBOL.compare(this,view);
}
public static class Comparators {
public static Comparator<StockPickBean> VALUE = (val1, val2) ->
(int)
(val1.value - val2.value);
}
}
After getting the same error:
java.lang.IllegalArgumentException: Comparison method violates its general contract!
I changed this line:
public static Comparator<StockPickBean> VALUE = (val1, val2) -> (int)
(val1.value - val2.value);
to:
public static Comparator<StockPickBean> VALUE = (StockPickBean spb1,
StockPickBean spb2) -> Double.compare(spb2.value,spb1.value);
That fixes the error.
I ran into a similar problem where I was trying to sort a n x 2 2D array named contests which is a 2D array of simple integers. This was working for most of the times but threw a runtime error for one input:-
Arrays.sort(contests, (row1, row2) -> {
if (row1[0] < row2[0]) {
return 1;
} else return -1;
});
Error:-
Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.base/java.util.TimSort.mergeHi(TimSort.java:903)
at java.base/java.util.TimSort.mergeAt(TimSort.java:520)
at java.base/java.util.TimSort.mergeForceCollapse(TimSort.java:461)
at java.base/java.util.TimSort.sort(TimSort.java:254)
at java.base/java.util.Arrays.sort(Arrays.java:1441)
at com.hackerrank.Solution.luckBalance(Solution.java:15)
at com.hackerrank.Solution.main(Solution.java:49)
Looking at the answers above I tried adding a condition for equals and I don't know why but it worked. Hopefully we must explicitly specify what should be returned for all cases (greater than, equals and less than):
Arrays.sort(contests, (row1, row2) -> {
if (row1[0] < row2[0]) {
return 1;
}
if(row1[0] == row2[0]) return 0;
return -1;
});
A variation of Gili's answer to check if the comparator satisfies the requirements described in the compare method's javadoc - with a focus on completeness and readability, e.g. by naming the variables the same as in the javadoc. Note that this is O(n^3), only use it when debugging, maybe just on a subset of your elements, in order to be fast enough to finish at all.
public static <T> void verifyComparator(Comparator<T> comparator, Collection<T> elements) {
for (T x : elements) {
for (T y : elements) {
for (T z : elements) {
int x_y = comparator.compare(x, y);
int y_x = comparator.compare(y, x);
int y_z = comparator.compare(y, z);
int x_z = comparator.compare(x, z);
// javadoc: The implementor must ensure that sgn(compare(x, y)) == -sgn(compare(y, x))
if (Math.signum(x_y) == -Math.signum(y_x)) { // ok
} else {
System.err.println("not holding: sgn(compare(x, y)) == -sgn(compare(y, x))" //
+ " | x_y: " + x_y + ", y_x: " + y_x + ", x: " + x + ", y: " + y);
}
// javadoc: The implementor must also ensure that the relation is transitive:
// ((compare(x, y)>0) && (compare(y, z)>0)) implies compare(x, z)>0.
if (x_y > 0 && y_z > 0) {
if (x_z > 0) { // ok
} else {
System.err.println("not holding: ((compare(x, y)>0) && (compare(y, z)>0)) implies compare(x, z)>0" //
+ " | x_y: " + x_y + ", y_z: " + y_z + ", x_z: " + x_z + ", x: " + x + ", y: " + y + ", z: " + z);
}
}
// javadoc: Finally, the implementor must ensure that:
// compare(x, y)==0 implies that sgn(compare(x, z))==sgn(compare(y, z)) for all z.
if (x_y == 0) {
if (Math.signum(x_z) == Math.signum(y_z)) { // ok
} else {
System.err.println("not holding: compare(x, y)==0 implies that sgn(compare(x, z))==sgn(compare(y, z)) for all z" //
+ " | x_y: " + x_y + ", x_z: " + x_z + ", y_z: " + y_z + ", x: " + x + ", y: " + y + ", z: " + z);
}
}
}
}
}
}
I had to sort on several criterion (date, and, if same date; other things...). What was working on Eclipse with an older version of Java, did not worked any more on Android : comparison method violates contract ...
After reading on StackOverflow, I wrote a separate function that I called from compare() if the dates are the same. This function calculates the priority, according to the criteria, and returns -1, 0, or 1 to compare(). It seems to work now.
What about doing something simpler like this:
int result = card1.getSet().compareTo(card2.getSet())
if (result == 0) {
result = card1.getRarity().compareTo(card2.getRarity())
}
if (result == 0) {
result = card1.getId().compareTo(card2.getId())
}
if (result == 0) {
result = card1.getCardType().compareTo(card2.getCardType())
}
return result;
You just need to order the comparisons in order of preference.
I am new to TensorFlow and cannot find out the solution of these questions.
How can I retrain the im2txt model for my new dataset such that the dataset on which the im2txt model was trained does not get lost and my new dataset is added to the MSCOCO dataset to caption the new images (i.e training dataset= MSCOCO dataset + My new dataset). Someone, please share the detailed procedure and the problems that I can face while retraining.
I have found out the TensorFlow tutorial for running inception V3 model in android on real-time datasets, Can this method be applied as well to the im2txt model i.e. can this be made to caption an image taken from a mobile in real time. Someone, please share the detailed steps how to do this.
After the weeks of struggle can be able to run and execute the im2txt model on Android.
Since I found the solutions from different blogs and different questions and answers, Felt that it might be useful if the all(maximum) solution is at one place.So, Sharing the steps followed.
You need to clone the tensorflow project https://github.com/tensorflow/tensorflow/releases/tag/v1.5.0 in order to freeze the graph and for some more utils.
Downloaded the im2txt model form https://github.com/KranthiGV/Pretrained-Show-and-Tell-model
Followed the steps described in the above link Can able to Run the inference to generate captions on the Linux desktop Successfully after renaming some variable in the graph(to overcome NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/bias not found in checkpoint types of errors)
Now we need to freeze the existing model to obtain the frozen graph in order to use in android/ios
from cloned tensorflow project using freeze_graph.py( tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py) one can freeze the graph from any model by giving the following command
An example of command-line usage is:
bazel build tensorflow/python/tools:freeze_graph && \
bazel-bin/tensorflow/python/tools/freeze_graph \
--input_graph=some_graph_def.pb \
--input_checkpoint=model.ckpt-8361242 \
--output_graph=/tmp/frozen_graph.pb --output_node_names=softmax
--input_binary=true
we need to supply the all the output_node_names which we required to run the model, from "Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_wrapper.py" we can list out the output node names as 'softmax', 'lstm/initial_state' and 'lstm/state'
when I run the freeze graph command by supplying the output node names as 'softmax', 'lstm/initial_state' and 'lstm/state' got the error "AssertionError: softmax is not in the graph"
From the answers for
How to freeze an im2txt model?(How to freeze an im2txt model?)
by Steph and Jeff Tang
The current model ckpt.data, ckpt.index and ckpt.meta files and a graph.pbtxt should be loaded in inference mode (see InferenceWrapper in im2txt). It builds a graph with the correct names 'softmax', 'lstm/initial_state' and 'lstm/state'. You save this graph (with the same ckpt format) and then you can apply the freeze_graph script to obtain the frozen model.
to do this in Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_utils\inference_wrapper.base.py, just add something like saver.save(sess, "model/ckpt4") after saver.restore(sess, checkpoint_path) in def _restore_fn(sess):. Then rebuild and run_inference and you'll get a model that can be frozen, transformed, and optionally memmapped, to be loaded by iOS and Android apps
Now I run the command as below
python tensorflow/python/tools/freeze_graph.py \
--input_meta_graph=/tmp/ckpt4.meta \
--input_checkpoint=/tmp/ckpt4 \
--output_graph=/tmp/ckpt4_frozen.pb \
--output_node_names="softmax,lstm/initial_state,lstm/state" \
--input_binary=true
and loaded the obtained ckpt4_frozen.pb file into the Android application and got the error
"java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'DecodeJpeg' with these attrs. Registered devices: [CPU], Registered kernels:
[[Node: decode/DecodeJpeg = DecodeJpegacceptable_fraction=1, channels=3, dct_method="", fancy_upscaling=true, ratio=1, try_recover_truncated=false]]"
From https://github.com/tensorflow/tensorflow/issues/2883
Since DecodeJpeg isn't supported as part of the Android tensorflow core, you'll need to strip it out of the graph first
bazel build tensorflow/python/tools:strip_unused && \
bazel-bin/tensorflow/python/tools/strip_unused \
--input_graph=ckpt4_frozen.pb \
--output_graph=ckpt4_frozen_stripped_graph.pb \
--input_node_names=convert_image/Cast,input_feed,lstm/state_feed\
--output_node_names=softmax,lstm/initial_state,lstm/state\
--input_binary=true
When I try to load ckpt4_frozen_stripped_graph.pb in android i faced errors so i followed Jeff Tang's answer (Error using Model after using optimize_for_inference.py on frozen graph)
Instead of tools:strip_unused I used graph transformation tool
bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=/tmp/ckpt4_frozen.pb \
--out_graph=/tmp/ckpt4_frozen_transformed.pb \
--inputs="convert_image/Cast,input_feed,lstm/state_feed" \
--outputs="softmax,lstm/initial_state,lstm/state" \
--transforms='
strip_unused_nodes(type=float, shape="1,299,299,3")
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms'
I can able load the obtained ckpt4_frozen_transformed.pb on android successfully.
when I supply the input as float array of RGB image pixels for input node "convert_image/Cast" and fetch the output from "lstm/initail_state" node successfully.
Now the challenge is to understand the beam search in "Pretrained-Show-and-Tell-model\im2txt\im2txt\inference_utils\caption_generator.py" and same should be implemented on the Android side.
If you observe the python script caption_generator.py at
softmax, new_states, metadata = self.model.inference_step(sess,input_feed,state_feed)
input_feed is an int32 bit array and state_feed is a multidimensional float array
on the android side, I tried feeding int32 bit array for "input_feed", since there is no Java API to feed multidimensional array so I fed float array to lstm/state_feed as it is which fetched previously from "lstm/initail_state" node.
Got two errors one is for the input_fedd is expecting int 64bit and
"java.lang.IllegalArgumentException: -input rank(-1) <= split_dim < input rank (1), but got 1" at lstm/state_feed.
For the first error, I changed the input_feed feed data type from int32 to int 64.
About second error it is expecting rank two tensor.
If you see tensorflow java sources the data type float array which we are feeding is converted to a rank one tensor, we should feed data type in such a way that rank two tensor should be created but at present, I didn't find any API to feed multidimensional float array there
When I browsing tensorflow java source I found the API which was not exposed as Android API where we can create a rank two tensor. so I rebuilt and both libtensorflow_inference.so and libandroid_tensorflow_inference_java.jar by enabling the rank two tensor creation call.(for building process refer https://blog.mindorks.com/android-tensorflow-machine-learning-example-ff0e9b2654cc)
Now I can able to run the inference on Android and get the one caption for the image.But the accuracy very low.
Reason limiting for the one caption is I didn't find a way to fetch the outputs as a multidimensional array which is required for generating more number of cations for a single image.
String actualFilename = labelFilename.split("file:///android_asset/")[1];
vocab = new Vocabulary(assetManager.open(actualFilename));
inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFilename);
final Graph g = c.inferenceInterface.graph();
final Operation inputOperation = g.operation(inputName);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + inputName + "'");
}
final Operation outPutOperation = g.operation(outputName);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + outputName + "'");
}
// The shape of the output is [N, NUM_CLASSES], where N is the batch size.
int numClasses = (int) inferenceInterface.graph().operation(outputName)
.output(0).shape().size(1);
Log.i(TAG, "Read " + vocab.totalWords() + " labels, output layer size is " + numClasses);
// Ideally, inputSize could have been retrieved from the shape of the input operation. Alas,
// the placeholder node for input in the graphdef typically used does not specify a shape, so it
// must be passed in as a parameter.
inputSize = inputSize;
// Pre-allocate buffers.
outputNames = new String[]{outputName + ":0"};
outputs = new float[numClasses];
inferenceInterface.feed(inputName + ":0", pixels, inputSize, inputSize, 3);
inferenceInterface.run(outputNames, runStats);
inferenceInterface.fetch(outputName + ":0", outputs);
startIm2txtBeamSearch(outputs);
//Implemented Beam search in JAVA
private void startIm2txtBeamSearch(float[] outputs) {
int beam_size = 1;
//TODO:Prepare vocab ids from file
ArrayList<Integer> vocab_ids = new ArrayList<>();
vocab_ids.add(1);
int vocab_end_id = 2;
float lenth_normalization_factor = 0;
int maxCaptionLength = 20;
Graph g = inferenceInterface.graph();
//node input feed
String input_feed_node_name = "input_feed";
Operation inputOperation = g.operation(input_feed_node_name);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + input_feed_node_name + "'");
}
String output_feed_node_name = "softmax";
Operation outPutOperation = g.operation(output_feed_node_name);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + output_feed_node_name + "'");
}
int output_feed_node_numClasses = (int) outPutOperation.output(0).shape().size(1);
Log.i(TAG, "Output layer " + output_feed_node_name + ", output layer size is " + output_feed_node_numClasses);
FloatBuffer output_feed_output = FloatBuffer.allocate(output_feed_node_numClasses);
//float [][] output_feed_output = new float[numClasses][];
//node state feed
String input_state_feed_node_name = "lstm/state_feed";
inputOperation = g.operation(input_state_feed_node_name);
if (inputOperation == null) {
throw new RuntimeException("Failed to find input Node '" + input_state_feed_node_name + "'");
}
String output_state_feed_node_name = "lstm/state";
outPutOperation = g.operation(output_state_feed_node_name);
if (outPutOperation == null) {
throw new RuntimeException("Failed to find output Node '" + output_state_feed_node_name + "'");
}
int output_state_feed_node_numClasses = (int) outPutOperation.output(0).shape().size(1);
Log.i(TAG, "Output layer " + output_state_feed_node_name + ", output layer size is " + output_state_feed_node_numClasses);
FloatBuffer output_state_output = FloatBuffer.allocate(output_state_feed_node_numClasses);
//float[][] output_state_output= new float[numClasses][];
String[] output_nodes = new String[]{output_feed_node_name, output_state_feed_node_name};
Caption initialBean = new Caption(vocab_ids, outputs, (float) 0.0, (float) 0.0);
TopN partialCaptions = new TopN(beam_size);
partialCaptions.push(initialBean);
TopN completeCaption = new TopN(beam_size);
captionLengthLoop:
for (int i = maxCaptionLength; i >= 0; i--) {
List<Caption> partialCaptionsList = new LinkedList<>(partialCaptions.extract(false));
partialCaptions.reset();
long[] input_feed = new long[partialCaptionsList.size()];
float[][] state_feed = new float[partialCaptionsList.size()][];
for (int j = 0; j < partialCaptionsList.size(); j++) {
Caption curCaption = partialCaptionsList.get(j);
ArrayList<Integer> senArray = curCaption.getSentence();
input_feed[j] = senArray.get(senArray.size() - 1);
state_feed[j] = curCaption.getState();
}
//feeding
inferenceInterface.feed(input_feed_node_name, input_feed, new long[]{input_feed.length});
inferenceInterface.feed(input_state_feed_node_name, state_feed, new long[]{state_feed.length});
//run
inferenceInterface.run(output_nodes, runStats);
//fetching
inferenceInterface.fetch(output_feed_node_name, output_feed_output);
inferenceInterface.fetch(output_state_feed_node_name, output_state_output);
float[] word_probabilities = new float[partialCaptionsList.size()];
float[] new_state = new float[partialCaptionsList.size()];
for (int k = 0; k < partialCaptionsList.size(); k++) {
word_probabilities = output_feed_output.array();
//output_feed_output.get(word_probabilities[k]);
new_state = output_state_output.array();
//output_feed_output.get(state[k]);
// For this partial caption, get the beam_size most probable next words.
Map<Integer, Float> word_and_probs = new LinkedHashMap<>();
//key is index of probability; value is index = word
for (int l = 0; l < word_probabilities.length; l++) {
word_and_probs.put(l, word_probabilities[l]);
}
//sorting
// word_and_probs = word_and_probs.entrySet().stream()
// .sorted(Map.Entry.comparingByValue())
// .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,(e1, e2) -> e1, LinkedHashMap::new));
word_and_probs = MapUtil.sortByValue(word_and_probs);
//considering first (beam size probabilities)
LinkedHashMap<Integer, Float> final_word_and_probs = new LinkedHashMap<>();
for (int key : word_and_probs.keySet()) {
final_word_and_probs.put(key, word_and_probs.get(key));
if (final_word_and_probs.size() == beam_size)
break;
}
for (int w : final_word_and_probs.keySet()) {
float p = final_word_and_probs.get(w);
if (p < 1e-12) {//# Avoid log(0).
Log.d(TAG, "p is < 1e-12");
continue;
}
Caption partialCaption = partialCaptionsList.get(k);
ArrayList<Integer> sentence = new ArrayList<>(partialCaption.getSentence());
sentence.add(w);
float logprob = (float) (partialCaption.getPorb() + Math.log(p));
float scroe = logprob;
Caption beam = new Caption(sentence, new_state, logprob, scroe);
if (w == vocab_end_id) {
completeCaption.push(beam);
} else {
partialCaptions.push(beam);
}
}
if (partialCaptions.getSize() == 0)//run out of partial candidates; happens when beam_size = 1.
break captionLengthLoop;
}
//clear buffer retrieve sub sequent output
output_feed_output.clear();
output_state_output.clear();
output_feed_output = null;
output_state_output = null;
output_feed_output = FloatBuffer.allocate(output_feed_node_numClasses);
output_state_output = FloatBuffer.allocate(output_state_feed_node_numClasses);
Log.d(TAG, "----" + i + " Iteration completed----");
}
Log.d(TAG, "----Total Iterations completed----");
LinkedList<Caption> completeCaptions = completeCaption.extract(true);
for (Caption cap : completeCaptions) {
ArrayList<Integer> wordids = cap.getSentence();
StringBuffer caption = new StringBuffer();
boolean isFirst = true;
for (int word : wordids) {
if (!isFirst)
caption.append(" ");
caption.append(vocab.getWord(word));
isFirst = false;
}
Log.d(TAG, "Cap score = " + Math.exp(cap.getScore()) + " and Caption is " + caption);
}
}
//Vocab
public class Vocabulary {
String TAG = Vocabulary.class.getSimpleName();
String start_word = "<S>", end_word = "</S>", unk_word = "<UNK>";
ArrayList<String> words;
public Vocabulary(File vocab_file) {
loadVocabsFromFile(vocab_file);
}
public Vocabulary(InputStream vocab_file_stream) {
words = readLinesFromFileAndLoadWords(new InputStreamReader(vocab_file_stream));
}
public Vocabulary(String vocab_file_path) {
File vocabFile = new File(vocab_file_path);
loadVocabsFromFile(vocabFile);
}
private void loadVocabsFromFile(File vocabFile) {
try {
this.words = readLinesFromFileAndLoadWords(new FileReader(vocabFile));
//Log.d(TAG, "Words read from file = " + words.size());
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
private ArrayList<String> readLinesFromFileAndLoadWords(InputStreamReader file_reader) {
ArrayList<String> words = new ArrayList<>();
try (BufferedReader br = new BufferedReader(file_reader)) {
String line;
while ((line = br.readLine()) != null) {
// process the line.
words.add(line.split(" ")[0].trim());
}
br.close();
if (!words.contains(unk_word))
words.add(unk_word);
} catch (IOException e) {
e.printStackTrace();
}
return words;
}
public String getWord(int word_id) {
if (words != null)
if (word_id >= 0 && word_id < words.size())
return words.get(word_id);
return "No word found, Maybe Vocab File not loaded";
}
public int totalWords() {
if (words != null)
return words.size();
return 0;
}
}
//MapUtil
public class MapUtil {
public static <K, V extends Comparable<? super V>> Map<K, V> sortByValue(Map<K, V> map) {
List<Map.Entry<K, V>> list = new ArrayList<>(map.entrySet());
list.sort(new Comparator<Map.Entry<K, V>>() {
#Override
public int compare(Map.Entry<K, V> o1, Map.Entry<K, V> o2) {
if (o1.getValue() instanceof Float && o2.getValue() instanceof Float) {
Float o1Float = (Float) o1.getValue();
Float o2Float = (Float) o2.getValue();
return o1Float >= o2Float ? -1 : 1;
}
return 0;
}
});
Map<K, V> result = new LinkedHashMap<>();
for (Map.Entry<K, V> entry : list) {
result.put(entry.getKey(), entry.getValue());
}
return result;
}
}
//Caption
public class Caption implements Comparable<Caption> {
private ArrayList<Integer> sentence;
private float[] state;
private float porb;
private float score;
public Caption(ArrayList<Integer> sentence, float[] state, float porb, float score) {
this.sentence = sentence;
this.state = state;
this.porb = porb;
this.score = score;
}
public ArrayList<Integer> getSentence() {
return sentence;
}
public void setSentence(ArrayList<Integer> sentence) {
this.sentence = sentence;
}
public float[] getState() {
return state;
}
public void setState(float[] state) {
this.state = state;
}
public float getPorb() {
return porb;
}
public void setPorb(float porb) {
this.porb = porb;
}
public float getScore() {
return score;
}
public void setScore(float score) {
this.score = score;
}
#Override
public int compareTo(#NonNull Caption oc) {
if (score == oc.score)
return 0;
if (score < oc.score)
return -1;
else
return 1;
}
}
//TopN
public class TopN {
//Maintains the top n elements of an incrementally provided set.
int n;
LinkedList<Caption> data;
public TopN(int n) {
this.n = n;
this.data = new LinkedList<>();
}
public int getSize() {
if (data != null)
return data.size();
return 0;
}
//Pushes a new element
public void push(Caption x) {
if (data != null) {
if (getSize() < n) {
data.add(x);
} else {
data.removeLast();
data.add(x);
}
}
}
//Extracts all elements from the TopN. This is a destructive operation.
//The only method that can be called immediately after extract() is reset().
//Args:
//sort: Whether to return the elements in descending sorted order.
//Returns: A list of data; the top n elements provided to the set.
public LinkedList<Caption> extract(boolean sort) {
if (sort) {
Collections.sort(data);
}
return data;
}
//Returns the TopN to an empty state.
public void reset() {
if (data != null) data.clear();
}
}
Even though accuracy is very low I am sharing this because it might be useful for some to load the show and tell models in android.
I want to test an application but it's too large to do by static approach.
I started googling. I found a interesting tool called Xposed Framework.
I read a lot of documents / examples. But I cant find the theads about getting method name of application.
My purpose is to log current method name when I press a button in app. Which parameters are sent when the methods are called?
For more information, the application I want to test is a chat application. I want to check that is it secured to use ? Is it true that developers claims the application uses blah blah blah encryption ? Is it real end-to-end encryption?.
According to the large of application, I need some tools to help me analyse this. When I send a message what methods are called ? what values are sent along with ?
If it is opensource you can easily insert a few logs in the source code and recompile. Some code coverage tools allow you to log the executed methods but i am unsure about the parameters (e.g. EMMA coverage).
If it is closed-source then you can do it with Xposed, but it has some challenges. Xposed allows you to hook Java methods, if it is opensource you can lookup the specific methods you want to intercept and print their parameters. If it is closed source, you can always check the method names through decompiling the app with apktool.
Check the Xposed tutorial on how to register hooks. Assuming you created your class that extends XC_MethodHook, the following methods should do the trick for primitive parameters:
#Override
protected void afterHookedMethod(MethodHookParam param) throws Throwable {
String threadname = Thread.currentThread().getName();
String d = _dateFormat.format(new Date());
Class<?> cls = param.this.getClass();
//Extend XC_MethodHook to receive the mthName in the constructor
//and store it as a static field.
Log.v("MethodInvocation", "[A][" + d + "]"
+ "[" + cls.getName() + "."
+ mthName
+ "(" + args(param) + ")" + " = " + result(param)
+ "]" + " [" + threadname + "]");
}
public boolean shouldPrintContent(Object o) {
if (o.getClass().isPrimitive()
|| o.getClass().getName().contains("java.lang"))
return true;
return false;
}
public String args(MethodHookParam param) {
String args = "";
int counter = 0;
if (param != null) {
for (Object o : param.args) {
if (counter > 0)
args += ",";
// argument can be null
if (o == null) {
args += "null";
} else { // if it is an object lets print its type and content.
args += printclean(o.getClass().getName());
if (shouldPrintContent(o)) {
args += ":" + printclean(o.toString());
} else
args += ":nonPrimitiveOrJavaLang";
}
counter++;
}
}
return args;
}
//avoid identation chars in strings
public String printclean(String str) {
char[] res = str.toCharArray();
for (int i = 0; i < str.length(); i++) {
if (res[i] == '\n' || res[i] == '\r' || res[i] == '\t'
|| res[i] == '[' || res[i] == ']') {
res[i] = '*';
}
}
return String.valueOf(res);
}
public String result(MethodHookParam param) {
String res = "";
Object retobj = param.getResult();
if (retobj == null) {
res += "null";
} else {
res += printclean(retobj.getClass().getName());
if (shouldPrintContent(retobj)) {
res += printclean(retobj.toString());
} else
res += "(nonPrimitiveOrJavaLang)";
}
return res;
}
Note that printing any object is not trivial. Here I printed only known types like primitives or other java.lang objects. More complex objects (including collections) you can try using Gson to represent them but this also comes with limitations (e.g. can't often handle reference loops).
Finally, be careful with what method you hook as hooking and logging methods that are called often will impact the performance of the app.
Good luck!
A new Xposed module called Inspeckage does (among many other useful inspections) what you want to do: you can use it to hook any method and view and even change its input and output.
As shown below, I have the following code as my constructor. But I want to duplicate it with different name. But I am getting error on this. Can I somehow modify to fit the requirement?
This is original code:
public HeyStatus(byte[] bytes)
{
this();
int offset = 7;
for (int i = 1; i < 9; ++i)
{
partition1Status[i-1].status = (bytes[offset + i] & 0x0F);
partition2Status[i-1].status = (bytes[offset + i + 9] & 0x0F);
}
if( (bytes[offset + 9] == 0) || (bytes[offset + 9] == 1)){
isPartitioned = (bytes[offset + 9] == 0) ? false : true;
}
partition1Status.status = (bytes[offset + 18] - 0x30);
partition2Status.status = (bytes[offset + 19] - 0x30);
String model = "" + (char)bytes[1] + (char)bytes[2];
if (model.equalsIgnoreCase("!A"))
modelNum = "T32";
}
I want to add this code after the previous code:
public HeyStatus(byte[] bytes2) <----the line I am getting error
{
this();
int offset = 7;
for (int i = 1; i < 9; ++i)
{
partition3Status[i-1].status = (bytes[offset + i] & 0x0F);
partition4Status[i-1].status = (bytes[offset + i + 9] & 0x0F);
}
if( (bytes[offset + 9] == 0) || (bytes[offset + 9] == 1)){
isPartitioned = (bytes[offset + 9] == 0) ? false : true;
}
partition3Status.status = (bytes[offset + 18] - 0x30);
partition4Status.status = (bytes[offset + 19] - 0x30);
String model = "" + (char)bytes[1] + (char)bytes[2];
if (model.equalsIgnoreCase("!A"))
modelNum = "T32";
}
This is a matter of method overloading (constructors are special kind of methods).
In short you can only differenciate two methods by their name, and type of arguments.
You cannot declare more than one method with the same name and the
same number and type of arguments, because the compiler cannot tell
them apart.
public HeyStatus(byte[] bytes)
public HeyStatus(byte[] bytes2)
Both are same declaration. You need to change the type of parameter(byte[]) to overload the constructor. Constructor or Method overloading is based on difference in signatures. Here you are using the same signature for both the constructor definitions , just changing the parameter name, and this is invalid.
Read Javadoc for more.
The Java programming language supports overloading methods, and Java can distinguish between methods with different method signatures. This means that methods within a class can have the same name if they have different parameter lists.
You did not change the constructor signature. Both of constructors has same types:
public HeyStatus(byte[] bytes)
public HeyStatus(byte[] bytes2)
If you provides different types of arguments, different numbers of arguments, or both, you can define multiple constructors with the same name. It is calling Constructor Overloaing.
If you want to have two constructors with the same paremeter count and types, you should use static factory method pattern. Make methods like
public static HeyStatus constructFromArray1(byte[] bytes) {...}
and
public static HeyStatus constructFromArray2(byte[] bytes) {...}
(you should name that methods properly, don't use the names that I proposed), that will construct objects properly, then you can construct objects from code like
HeyStatus heyStatus = HeyStatus.constructFromArray1(bytes)
(instead of HeyStatus heyStatus = new HeyStatus(bytes).
It's the best way to do what you want.
You can use only one constructor by adding an extra (for example) int parameter to specify which of the code blocks to be executed. Like this, you'll have one constructor, containing the two blocks of code you need:
public HeyStatus(byte[] bytes, int option){
switch (option){
case 1: {
//you can place here the code you used in the first constructor
break;
}
case 2:{
//you can place here the code you used in the second constructor
break;
}
default: break;
}
And when you'll call the constructor, you will specify the option in the parameters list.
Cheers!
Just use one constructor and when you call it put the appropriate bytes array within the parameters here is a simplified example:
byte[] bytesOne[]= new bytes[]{1,2,3};
byte[] bytesTwo[] =new bytes[]{4,5,6};
HeyStatus hStatusOne = new HeyStatus(bytesOne);
HeyStatus hStatusTwo = new HeyStatus(bytesTwo);
But only ONE contructor.
P.S. if you want to use more constructors parameters within it MUST be different from the first one, in type (not in name) and/or number. You can get more informations by googling overloading constructors.
Edit:
Just noticed the difference with partion1Status[] (and the others) put your partionStatus in the parameters.
example your constructors' head must look like this:
public HeyStatus(byte[] bytes, typeOfpartionStatus partitionOneStatus, typeOfpartionStatus partitionTwoStatus){
this();
int offset = 7;
for (int i = 1; i < 9; ++i)
{
partitionOneStatus[i-1].status = (bytes[offset + i] & 0x0F);
partitionTwoStatus[i-1].status = (bytes[offset + i + 9] & 0x0F);
}
if( (bytes[offset + 9] == 0) || (bytes[offset + 9] == 1)){
isPartitioned = (bytes[offset + 9] == 0) ? false : true;
}
partitionOneStatus.status = (bytes[offset + 18] - 0x30);
partitionTwoStatus.status = (bytes[offset + 19] - 0x30);
String model = "" + (char)bytes[1] + (char)bytes[2];
if (model.equalsIgnoreCase("!A"))
modelNum = "T32";
}
when you call it it must look like this.
HeyStatus hstatusOne = new (bytesOne,partion1Status,partition2Status);
HeyStatus hstatusTwo = new (bytesTwo,partion3Status,partition4Status);
You can't have two methods with the same signature in the same class. This also applies to constructors. There is no way for the compiler to know which one you're calling if you do something like this:
byte[] bytes = new bytes[]{1, 2, 3};
new HeyStatus(bytes);