Develop Reference

Kotlin Android allow only emojis and letters in a text

I've been trying to find a good way to be able to keep only emojis and letters in a given text, but every article I found, I didn't have success with .
I've tried to use regex, but seems that I can not make it work.
I've tried to use emoji4j but it seems that this library is working with emojis in this form ":)", which don't help me, because my emojis are groups of unicode characters.
The result I want is the following :
"This is. a text 👨‍👩‍👧‍👦,,1234" => "This is a text 👨‍👩‍👧‍👦"
"👨‍👩‍👧‍👦" => "👨‍👩‍👧‍👦"
"👨‍👩‍👧‍👦😃123abc👨‍👩‍👧‍👦" => "👨‍👩‍👧‍👦😃abc👨‍👩‍👧‍👦"
Here's the emoji regex : ?:[\u2700-\u27bf]|(?:[\ud83c\udde6-\ud83c\uddff]){2}|[\ud800\udc00-\uDBFF\uDFFF]|[\u2600-\u26FF])[\ufe0e\ufe0f]?(?:[\u0300-\u036f\ufe20-\ufe23\u20d0-\u20f0]|[\ud83c\udffb-\ud83c\udfff])?(?:\u200d(?:[^\ud800-\udfff]|(?:[\ud83c\udde6-\ud83c\uddff]){2}|[\ud800\udc00-\uDBFF\uDFFF]|[\u2600-\u26FF])[\ufe0e\ufe0f]?(?:[\u0300-\u036f\ufe20-\ufe23\u20d0-\u20f0]|[\ud83c\udffb-\ud83c\udfff])?)*|[\u0023-\u0039]\ufe0f?\u20e3|\u3299|\u3297|\u303d|\u3030|\u24c2|[\ud83c\udd70-\ud83c\udd71]|[\ud83c\udd7e-\ud83c\udd7f]|\ud83c\udd8e|[\ud83c\udd91-\ud83c\udd9a]|[\ud83c\udde6-\ud83c\uddff]|[\ud83c\ude01-\ud83c\ude02]|\ud83c\ude1a|\ud83c\ude2f|[\ud83c\ude32-\ud83c\ude3a]|[\ud83c\ude50-\ud83c\ude51]|\u203c|\u2049|[\u25aa-\u25ab]|\u25b6|\u25c0|[\u25fb-\u25fe]|\u00a9|\u00ae|\u2122|\u2139|\ud83c\udc04|[\u2600-\u26FF]|\u2b05|\u2b06|\u2b07|\u2b1b|\u2b1c|\u2b50|\u2b55|\u231a|\u231b|\u2328|\u23cf|[\u23e9-\u23f3]|[\u23f8-\u23fa]|\ud83c\udccf|\u2934|\u2935|[\u2190-\u21ff] .
If I try something like :
val regex = "the_whole_regex_above | [^a-zA-Z]".toRegex()
myText.replace(regex,""), it won't replace anything, basically every character will pass
Basically I want to achieve pretty much the same thing as in this question, but using Kotlin.

You want to remove all punctuation, symbols (other than those used to form emojis) and digits.
To do that, you may use
myText = myText.replace("""[\p{N}\p{P}\p{S}&&[^\p{So}]]+""".toRegex(), "")
See the online Kotlin demo.
Details
[ - start of a character class that matches:
\p{N} - any Unicode digit
\p{P} - any Unicode punctuation proper
\p{S} - any Unicode symbol
&&[^\p{So}] - BUT the Unicode symbols belonging to Symbol, other Unicode category that are mostly used to form emojis
]+ - 1 or more occurrences.

What's the difference between Android's Html.escapeHtml and TextUtils.htmlEncode ? When should I use one or the other?

Android has two different ways to escape / encode HTML characters / entities in Strings:
Html.escapeHtml(String), added in API 16 (Android 4.1). The docs say:
Returns an HTML escaped representation of the given plain text.
TextUtils.htmlEncode(String) For this one, the docs say:
Html-encode the string.
Reading the docs, they both seem to do pretty much the same thing, but, when testing them, I get some pretty mysterious (to me) output.
Eg. With the input: <p>This is a quote ". This is a euro symbol: €. <b>This is some bold text</b></p>
Html.escapeHtml gives:
<p>This is a quote ". This is a euro symbol: €. <b>This is some bold text</b></p>
Whereas TextUtils.htmlEncode gives:
<p>This is a quote ". This is a euro symbol: €. <b>This is some bold text</b></p>
So it seems that the second escapes / encodes the quote ("), but the first doesn't, although the first encodes the Euro symbol, but the second doesn't. I'm confused.
So what's the difference between these two methods ? Which characters does each escape / encode ? What's the difference between encoding and escaping here ? When should I use one or the other (or should I, gasp, use them both together ?) ?

You can compare their sources:
This is what Html.escapeHtml uses underneath:
https://github.com/android/platform_frameworks_base/blob/d59921149bb5948ffbcb9a9e832e9ac1538e05a0/core/java/android/text/Html.java#L387
This is TextUtils.htmlEncode:
https://github.com/android/platform_frameworks_base/blob/d59921149bb5948ffbcb9a9e832e9ac1538e05a0/core/java/android/text/TextUtils.java#L1361
As you can see, the latter only quotes certain characters that are reserved for markup in HTML, while the former also encodes non-ASCII characters, so they can be represented in ASCII.
Thus, if your input only contains Latin characters (which is usually unlikely nowadays), or you have set up Unicode in your HTML page properly, and can go along with TextUtils.htmlEncode. Whereas if you need to ensure that your text works even if transmitted via 7-bit channels, use Html.escapeHtml.
As for the different treating of the quote character (") -- it only needs to be escaped inside attribute values (see the spec), so if you are not putting your text there, you should be fine.
Thus, my personal choice would be Html.escapeHtml, as it seems to be more versatile.

Error using native smiles of Android in text field (Unity3d)

I make application with Unity3d and build it for Android, when I write in input field android native smiles - I got error in line
(invalid utf-16 sequence at 1411555520 (missing surrogate tail)):
r.font.RequestCharactersInTexture(chars, size, style);
chars contains string than contains native android smiles. How I may support native smiles? I use own class for Input Field.

Unfortunately, supporting emojis with Unity is hard. When I implemented this feature, it took about a month to finish it, with a custom text layout engine and string class. So, if this requirement is not particularly important, I would suggest axing this feature.
The reason behind this particular error is that Unity gets characters from the input string one by one, and updates the visual string every character. From the layman point of view, this makes complete sense. However, it doesn't take into account how UTF-16 encoding, which is used in C#, works.
UTF-16 encoding uses 16 bits per a single unicode characters. It is enough for almost all characters that you would normally use. (And, as every developer knows, "almost all" is a red flag that will lay dormant for a long time and then will explode and destroy everything you love.) But it so happens, that Emoji characters are do not fit into 16 bit UTF-16 character, and use a special case — surrogate pair:
Surrogate pair is a pair of UTF-16 characters that represent a single Unicode character. That means that they don't have any meaning on their own individually, and when you try to render a UTF-16 character that is a surrogate head or surrogate tail, you can expect to get an error like this, or something similar.
Essentially, what you need to implement is some kind of buffer, that will accept C# UTF-16 characters one by one, and then pass them to rendering code when it verifies that all surrogate pairs are closed.
Oh, and I almost forgot! Some Emoji characters, like country flags, are represented by two unicode characters. Which means that they can potentially take up to four UTF-16 characters. Aren't text encodings fun?

SQLite unicode slavic accented words Android

I'm trying to filter out accented words if user searches for them in local database. But I have problems, namely with slavic letters ČŠŽ. In my SQLite database I have a field "title" with value: "Želodček"
If I try to select LOWER(title) I always get back the same value "Želodček" whilst other words are correctly lower cased. Only if the word begins with ČŽŠ then it doesn't get lower cased. This only persists with words which have leading accented letters.
Database records
Stomach
Želodček
Uppercase with UPPER()
STOMACH
ŽELODčEK
Lowercase with LOWER()
stomach
Želodček
I've already tried setting localization with setLocale() with no luck. I also tried different collation like NOCASE, UNICODE, LOCALIZED but nothing worked. I'm wondering why when lower cased the first letter is not lower cased and when upper cased other accented words are lowercase.
I've solved the problem with LIKE searches where I replace accented words with their lower cased counterpart. But I have problem with full text(FTS3) searching because I can't use the same trick with MATCH.
-- works but it's a hack
SELECT title FROM articles WHERE REPLACE(LOWER(title),'Ž','ž') LIKE '%želodček%'
-- can't seem to get it work
SELECT title FROM articles WHERE title MATCH 'želodček' COLLATE NOCASE
Is there any solution to this or is there a bigger problem?
Update:
No optimal solution yet.
Un-optimal solution 1:
I decided to deal with the problem directly by changing data in the select query. While this doesn't work for all cases (and I would have to cover all accents) it suits my case for now. So I'm posting it:
-- LIKE query
SELECT title FROM articles WHERE (REPLACE(REPLACE(REPLACE(LOWER(title),'Č','č'),'Š','š'),'Ž','ž') LIKE ? COLLATE NOCASE))
-- MATCH query (FTS)
-- In this case I programmatically replace searched word with 2 word variation (one that starts with lowercase and one that starts with uppercase) ie: title='želodček OR Želodček'
SELECT title FROM articles WHERE title MATCH ? COLLATE UNICODE
Un-optimal solution 2:
As suggested by user CL. to insert in normalized form (didn't work for me because normalized form was basically the original unicode form). I took it futher and insert title stripped of of accents (basically ASCII form). This is maybe better than solution one in ways of general solution. Since I only cover some accents in the first.
But there are downsides:
data doubles (one unicode title and one ASCII title). Which can be a problem if you have a lot of data.
some characters are not supported (like chinese characters will be gone after normalization and stripping)
ambiguity which you get by stripping accents (ie. two words "zelo" and "želo" have different meanings but will both turn up when searching).
Here's the Java code for it:
// Gets you the ASCII version of unicode title which you insert into different column
String titleAsciiName = Normalizer.normalize(title, Normalizer.Form.NFD)
.replaceAll("[^\\p{ASCII}]", "");

LIKE never uses a custom collation.
FTS can use a custom tokenizer, but you have to check whether unicode61 is available in all Android versions you want to support.
The Android database API does not allow to create custom implementations of LIKE or of a FTS tokenizer.
You might want to store a normalized version of your strings in the database.

Problem with special characters

How can I change the font on android to allow to show special characters like "'" or "à"?
Actually the strings that contains these characters are stored in the sqlite database.

When you load the text into your TextView, will this work for you?
textView.setText(new String(textFromDatabase, "UTF-8"));
This uses the String constructor to set the charset name. You can change "UTF-8" to a different Character encoding -- Also, look at the javadoc for String.
String(byte[] bytes, String charsetName) -
Constructs a new String by decoding the specified array of bytes using the specified charset.

The Droid font supports the "'", "à" and many others characters. I use them all the time (pt language).
Actually, I'm quite sure they support all the Basic Latin, Latin 1 supplement and the first extended latin range. They also support many others like hebrew etc., although I'm not sure if that changed between SDK versions.
You can also download the Unicode Map app in the Market to check which characters are available in your particular device. I also store unicode text in sqlite all the time, and still I don't have any problems.
One thing to consider: check that the encoding you are setting match the encoding of your source. It may be a text or a URL... an example:
BufferedReader b = new BufferedReader(new InputStreamReader(url.openStream(), MY_ENCODING));
Are you sure it's not a problem somewhere?

You should use '' instead of ' to store it into Sqlite database.
For example if you want to store 5 o'clock into database then you have to write this as 5 O''clock. Take a look here, for more information about it.

By default Android SQLite uses UTF-8.
I had this problem because when I populated the database on the first launch I used a txt file with another charset.