I am facing Performance issues with iOS Swift function to the word-wrap streaming text received in real-time**
We have written an iOS function to perform word-wrapping based on character count on streaming text received in real-time, to display on an OLED. We are trying to replicate what the Android StringTokenizer class and methods do, since iOS Swift does not have an equivalent class/method. But we have performance issues with this function (delay in the output of wrapped words compared to the streaming text being received). We are looking for the best possible method/solution/class in Swift which can handle the entire process of word wrapping of streaming text with minimum delay.
Background:
We are developing an iOS mobile app to convert speech to text in real-time using cloud-based automatic speech-to-text (STT) model APIs including STT APIs from Google, Azure & IBM, and display the transcribed text on an OLED display via a BLE device.
The app uses the mobile device’s microphone to capture “endless” or “infinite” streaming audio feeds, streams the audio to the cloud STT API, and receives stream speech recognition (transcription) results in real-time as the audio is processed.
Unlike standard speech-to-text implementations in voice assistants which listen to voice commands, wait for silence and then give the best output, in infinite streaming, there is a continuous stream of input audio being sent to the STT API which returns interim and final transcription results continuously.
As the transcription data is received from the API (interim/final), the app performs word-wrapping logic to prepare the text for display on the OLED. This logic depends on the number of characters that can be displayed on one line of the OLED and the number of lines in the display.
For example, on our current OLED, we can display 18 characters in one line and a maximum of 5 lines at a time. So if a string of than 18 characters is received from the STT API, the app logic has to find the appropriate space character in the string after which to break it and display the remaining string on the next line(s). . Here’s an example of a string received from the STT API:
TranscribeGlass shows closed captions from any source in your field of view
The expected result on the 5 lines of the OLED after performing word wrapping:
TranscribeGlass
shows closed
captions from any
source in your
field of view
Explanation:
Line 1 - “TranscribeGlass” - because “TranscribeGlass shows” exceeds 18 characters so break the string after the space before “shows”, and wrap it to the next line
Line 2 - “shows closed” - because“shows closed captions” exceeds 18 characters, break the string after the space before “captions”
Line 3 - “captions from any” - because “captions from any source” exceeds 18 characters, break the string after the space before “source”
Line 4 - “source on in your” - because “source on in your field” exceeds 18 characters, break the string after the space before “field”
Line 5 - “field of view” - no wrapping needed since the string is <= 18 characters
In Android, we have a ready-made StringTokenizer class & methods available that allows an application to break a string into tokens.
Our iOS Swift function:
We have written a function WriteOnlyCaption(), which takes a string as input and performs the word wrapping. See the code below:
For strings which has less than 18 characters, we are using the below function
public func writeOnlyCaption(text:String, isFinalBLE: Bool) {
print("TEXT IS NOW:\(text)")
var sendString:String = " "
var previousWrapLocation = 0
var currentWrapLocationIndex = characterPerLine
while(text.count - previousWrapLocation > characterPerLine) {
while(text[currentWrapLocationIndex] != " ") {
//print("While CurrentWrapLocationIndex Index \(currentWrapLocationIndex)")
currentWrapLocationIndex -= 1
}
sendString = sendString + String(text[previousWrapLocation...currentWrapLocationIndex]) + "\n"
line.addLineIndex(textIndex: currentWrapLocationIndex)
currentWrapLocationIndex += 1
previousWrapLocation = currentWrapLocationIndex
currentWrapLocationIndex = currentWrapLocationIndex + characterPerLine
}
sendString = sendString + String(text[previousWrapLocation..<text.count]) + "\n"
print("sendSting:->\(sendString)")
line.addLineIndex(textIndex: sendString.count)
line.setSendString(sendText: sendString)
line.tooString()
if previousLine == 0 {
previousLine = line.getLines().count
}
print("previousLine:->\(previousLine)")
var seed = ""
var counter = 0
var stringData = line.getSendString().components(separatedBy: "\n")
stringData.removeLast()
if stringData.count < 5 {
repeat {
if counter >= 0 {
seed = seed + stringData[counter] + "\n"
}
counter += 1
} while(stringData.count != counter)
} else {
counter = stringData.count - 5
repeat {
if counter > 0 {
seed = seed + stringData[counter] + "\n"
}
counter += 1
} while(stringData.count != counter)
}
print("Seeeddddddd:->\(seed)")
print("counter: \(counter)")
if seed != "" {
if isFinalBLE {
var seedBytes = seed.bytes
if previousLine == line.getLines().count {
seedBytes.append(0x03)
} else if line.getLines().count > previousLine {
seedBytes.append(0x02)
seedBytes.append(0x03)
}
seedBytes.append(0x00)
print("-- seedBytes: \(seedBytes) --")
previousLine = line.getLines().count
self.writeDataWithResponse(writeData: Data(seedBytes), characteristic: self.getCharsticFrom(UUID: CHAR.GLASS_WRITE)!)
} else {
var seedBytes = seed.bytes
if previousLine == line.getLines().count {
//seedBytes.append(0x00)
} else if line.getLines().count > previousLine {
if line.getLines().count - previousLine == 1 {
seedBytes.append(0x02)
} else if line.getLines().count - previousLine == 2{
seedBytes.append(0x02)
seedBytes.append(0x02)
} else {
seedBytes.append(0x02)
}
}
seedBytes.append(0x00)
print("-- seedBytes: \(seedBytes) --")
previousLine = line.getLines().count
self.writeDataWithResponse(writeData: Data(seedBytes), characteristic: self.getCharsticFrom(UUID: CHAR.GLASS_WRITE)!)
}
}
}
As you can see, we are using four while loops to perform the word wrapping correctly and send it to the OLED.
The main issue with this function is that four while loops and different if-else conditions create lots of delays (varies from 450ms to 900ms) while processing the text string, and this results in poor performance - either a lag in the display of text on the OLED compared to the streaming audio or entire chunks of transcribed text get skipped.
Related
I know Android, and many users, say not to use $event.keyCode || $event.charCode because the returned characters can't be trusted. And its true, it can't. But what about using a regexp to test for the presence of certain characters?
As a case example, US zip codes can be in several formats: 12345, 12345-6789 or 12345 6789. These three formats allow numbers, space and minus sign. But the minus sign could also be the dash sign. So with a physical keyboard you can test for $event.keyCode to match keyCodes [32,109,189] (ie: space, minus, keypad-minus).
But on a softkey board, particularly android soft keyboard, the returned key for almost ALL keys is never correct. On top of which, on android keypress isn't supported, only keydown and keyup - and those present their own challenges. So instead of capturing the keycodes, why not just evaluate the last pressed key and test that character against a regexp?
In my code I have changed things to:
<input id="zip" ng-model="userInfo.zip" ng-keyup="checkKeyCode($event)" ng-blur="setInfo($event,'zip')" >
$scope.checkKeyCode = function() {
var lastChar = $scope.userInfo.zip.slice(-1) ;
var regex1 = /^[\n]*$/ ; // enter key
var regex2 = /^[\d\s-]*$/ ; // numbers, space and minus
if (regex1.test(lastChar)) {
$event.target.blur($event,'zip') ; //force to blur, process data
return ;
}
if (regex2.test(lastChar) == false) {
$scope.errMsg = "Only numbers, space and dash/minus characters are permitted" ;
$scope.errColor = "red" ;
var sLen = $scope.userInfo.zip.length - 1 ;
$scope.userInfo.zip = $scope.userInfo.zip.substring(0,sLen) ; //remove invalid char
return ;
}
if ($scope.userInfo.zip.length > 10) {
$scope.errMsg = "Max length is 10 characters" ;
$scope.errColor = "red" ;
$scope.userInfo.zip = $scope.userInfo.zip.substring(0,10) ;
return ;
}
}
Even though my code works, I have a feeling there are some pitfalls to this method. For example keyCode 109 and 189 are both the minus sign, but does the above regex2 capture both of those or do I need to account a minus sign and a dash - does the regexp treat them differently?
What are the pitfalls and reasons to not do it this way? And last, if this isn't a solid method, then how does one account for keyCodes for Android soft keyboards?
I am working on a inventory management system where it's required to write asset ids (with length of maximum 17) in to an RFID tag.
First the problem was that when I write on a tag an ID with shorter length than the already written one, it keeps the non overridden characters in the tag.
for ex: if the tag has the ID "123456789" written on it and I write id "abc" on the tag. The tag's asset id becomes abc456789. I tried killing and erasing the tag before writing but it didn't work.
After that, I though of appending zeros before the target ID until it reaches the maximum length (17) so that this way no asset id with shorter length will be written on the tag and after reading I remove all preceding zero's. This worked well with a certain tag but not with another one, I figured out that the other tag can't be written on with more than 12 characters but I don't get why and the problem isn't in the RFID tag since it works well in another application and can be written on with more than 12 characters.
I would be really thankful if anyone could help me identify why this tag has only 12 characters written on it and the rest of the asset ID is neglected even though the same code works with another RFID tag.
Here's the write tag method:
fun writeTag(sourceEPC: String?, targetData: String): TagData? {
errorMessage = ""
try {
val tagData = TagData()
val tagAccess = TagAccess()
val writeAccessParams = tagAccess.WriteAccessParams()
writeAccessParams.accessPassword = 0
writeAccessParams.memoryBank = MEMORY_BANK.MEMORY_BANK_EPC
writeAccessParams.offset = 2
var paddedTargetData = padLeftZeros(targetData,17)
val targetDataInHex = HexStringConverter.getHexStringConverterInstance().stringToHex(if (paddedTargetData.length % 2 != 0) "0$paddedTargetData" else paddedTargetData)//if ODD
val padded = targetDataInHex + RFID_ADDED_VALUE
writeAccessParams.setWriteData(padded)
writeAccessParams.writeRetries = 1
writeAccessParams.writeDataLength = padded.length / 4 // WORD EQUALS 4 HEX
reader!!.Actions.TagAccess.writeWait(sourceEPC, writeAccessParams, null, tagData)
return tagData
} catch (e: InvalidUsageException) {
errorMessage = "InvalidUsageException=" + e.vendorMessage + " " + e.info
println(errorMessage)
return null
} catch (e: OperationFailureException) {
errorMessage = "InvalidUsageException=" + e.vendorMessage + " " + e.results
println(errorMessage)
return null
} catch (e: UnsupportedEncodingException) {
errorMessage = if (e.message == null) "" else e.message!!
println(errorMessage)
return null
}
}
Read Full Tag method:
fun readFullTag(sourceEPC: String): TagData? {
errorMessage = ""
try {
val tagAccess = TagAccess()
val readAccessParams = tagAccess.ReadAccessParams()
readAccessParams.accessPassword = 0
readAccessParams.memoryBank = MEMORY_BANK.MEMORY_BANK_TID
readAccessParams.offset = 0
return reader?.Actions?.TagAccess?.readWait(sourceEPC, readAccessParams, null, false)
} catch (e: InvalidUsageException) {
errorMessage = "InvalidUsageException=" + e.vendorMessage + " " + e.info
println(errorMessage)
return null
} catch (e: OperationFailureException) {
errorMessage = "InvalidUsageException=" + e.vendorMessage + " " + e.results
println(errorMessage)
return null
}
}
Handle Tag Data method:
override fun handleTagData(tagData: Array<TagData?>?) {
var readValue = ""
if (!tagData.isNullOrEmpty()) readValue = tagData[0]!!.tagID.trimIndent().replace("\n", "")
if (isWritingRFID) {
isWritingRFID = false
if (currentRFIDAssetCode.isNotEmpty())
writeRFID(readValue, currentRFIDAssetCode)
} else {
CoroutineScope(Dispatchers.Main).launch {
if (!pDialog.isShowing) readFullRFID(readValue)
}
}
}
Feel free to ask for any additional code or info.
Another way to solve this problem instead of padding with zeros is use a technique used a lot in NFC and the first byte of your data is the value of the length of the data (in Hex).
Therefore it does not matter if the old data is not zero'd out and you won't have a problem detecting if it is a zero for blanking or a real zero
e.g.
0A 31 32 33 34 35 36 37 38 39 30
or in text
10 1 2 3 4 5 6 7 8 9 0
would be overwritten with
03 41 42 43
or in text
3 A B C
resulting in text in memory of
3 A B C 4 5 6 7 8 9 0
But you would read the first byte to get the length of 3 and then read 3 more bytes to get in text
A B C
To substitute my usage of native apps that allow to keep track of my position, I wanted to generate a PWA using the HTML5 Goelocation API.
The result I have been able to achieve so far seems to point a inferior functionality of the HTML5 Goelocation API compared to native Android API.
Below is the code I have used and the issue is, that the PWA/ website application only receives infrequent updates. Additionally the app only receives position while the screen is not off.
This puts a huge obstacle into having a PWA being to track for instance my bike tour, since I cannot keep the screen and browser in the foreground, while ideed I wished the PWA would simply continues running even when the screen is off.
Now I am aware that in most cases a device user and privacy aware person would benefit from the useragent/browser to cut the waste of resources and limit privacy loss by disabling the very feature I search.
In essence however I have looked over the MDN documentation and besides the PositionOptions I was yet unable to find any clue about the guarantees of the API.
Find below the way I have sought to make it work as a code.
Does + Should HTML5 Goelocation API work when screen of on a mobile?
Is there a concrete information about if and how much geolocation information is returedn? like frequency/delay of update and like geofencing imprecision ?
Could for instance google maps navigation work in the browser itself?
My platform is Gecko on Android. If better results could be achieved in Chromium on Android I would be happy to hear about that too.
On Android I use firefox. In it I wanted to have my website provide a page that keeps track of my movements via the geolocation API, in order to replace my strave.
window.addEventListener("load",function(){
var a= document.createElement("div");
var textarea = document.createElement("textarea");
//textarea.value="aaaa"
textarea.style.display="block"
textarea.style.minHeight="5cm"
textarea.style.width="100%"
document.body.appendChild(a);
document.body.appendChild(textarea);
if(confirm("reset data?")){
localStorage.clear() ;
}
localStorage.setItem("start"+Date.now(),startInfo);
var startInfo = Object.keys(localStorage).length+ " " +Date.now()+ " " + (Date.now() % 864000);
var lastTime=0,lastLat=0,lastLon=0,count=0,lastDistance=0;
var startTime = Date.now();
var distance = 0;
if('geolocation' in navigator) {
a.innerHTML="<h2>super wir haben geolocation!</h2>";
setInterval(()=>{
navigator.geolocation.getCurrentPosition((position) => {
// var a = document.createElement("div");
count++;
a.innerHTML="<h1>"+(((Date.now()-startTime)/1000)|0)+" "+((distance*100|0)/100)+"</h1><h3>"+startInfo+"</h3><h2>date="+(new Date()).toString()+"<br>lang="+position.coords.latitude+" long="+ position.coords.longitude+"</h2>";
var lat = ((position.coords.latitude * 10000000) | 0)
var lon = ((position.coords.longitude * 10000000) | 0)
var time = Date.now();
var deltaTime = time - lastTime;
var deltaLat = lat - lastLat;
var deltaLon = lon - lastLon;
if(Math.abs(deltaLat)>100000 || Math.abs(deltaLon) > 100000)
{
} else{
distance += Math.sqrt(deltaLat*deltaLat+deltaLon*deltaLon);
}
var deltaDistance = distance - lastDistance;
lastLat=lat;
lastLon=lon;
lastTime=time;
lastDistance=distance;
newline = (((Date.now()-startTime)/1000)|0) + " dist=" + distance + "("+deltaDistance+") lat=" + lat + "("+deltaLat+") lon=" + lon + "("+deltaLon+") ";
textarea.value = newline + "\n" + textarea.value;
localStorage.setItem("P"+(Date.now()%864000),deltaLat+" "+deltaLon+" "+deltaTime);
},function(){},{timeout:900});
},1000);
} else {
a.innerHTML="<h2> shit</h2>";
}
});
I'm working on an Android app, and I do not want people to use emoji in the input.
How can I remove emoji characters from a string?
Emojis can be found in the following ranges (source) :
U+2190 to U+21FF
U+2600 to U+26FF
U+2700 to U+27BF
U+3000 to U+303F
U+1F300 to U+1F64F
U+1F680 to U+1F6FF
You can use this line in your script to filter them all at once:
text.replace("/[\u2190-\u21FF]|[\u2600-\u26FF]|[\u2700-\u27BF]|[\u3000-\u303F]|[\u1F300-\u1F64F]|[\u1F680-\u1F6FF]/g", "");
Latest emoji data can be found here:
http://unicode.org/Public/emoji/
There is a folder named with emoji version.
As app developers a good idea is to use latest version available.
When You look inside a folder, You'll see text files in it.
You should check emoji-data.txt. It contains all standard emoji codes.
There are a lot of small symbol code ranges for emoji.
Best support will be to check all these in Your app.
Some people ask why there are 5 digit codes when we can only specify 4 after \u.
Well these are codes made from surrogate pairs. Usually 2 symbols are used to encode one emoji.
For example, we have a string.
String s = ...;
UTF-16 representation
byte[] utf16 = s.getBytes("UTF-16BE");
Iterate over UTF-16
for(int i = 0; i < utf16.length; i += 2) {
Get one char
char c = (char)((char)(utf16[i] & 0xff) << 8 | (char)(utf16[i + 1] & 0xff));
Now check for surrogate pairs. Emoji are located on the first plane, so check first part of pair in range 0xd800..0xd83f.
if(c >= 0xd800 && c <= 0xd83f) {
high = c;
continue;
}
For second part of surrogate pair range is 0xdc00..0xdfff. And we can now convert a pair to one 5 digit code.
else if(c >= 0xdc00 && c <= 0xdfff) {
low = c;
long unicode = (((long)high - 0xd800) * 0x400) + ((long)low - 0xdc00) + 0x10000;
}
All other symbols are not pairs so process them as is.
else {
long unicode = c;
}
Now use data from emoji-data.txt to check if it's emoji.
If it is, then skip it. If not then copy bytes to output byte array.
Finally byte array is converted to String by
String out = new String(outarray, Charset.forName("UTF-16BE"));
For those using Kotlin, Char.isSurrogate can help as well. Find and remove the indexes that are true from that.
Here is what I use to remove emojis. Note: This only works on API 24 and forwards
public String remove_Emojis_For_Devices_API_24_Onwards(String name)
{
// we will store all the non emoji characters in this array list
ArrayList<Character> nonEmoji = new ArrayList<>();
// this is where we will store the reasembled name
String newName = "";
//Character.UnicodeScript.of () was not added till API 24 so this is a 24 up solution
if (Build.VERSION.SDK_INT > 23) {
/* we are going to cycle through the word checking each character
to find its unicode script to compare it against known alphabets*/
for (int i = 0; i < name.length(); i++) {
// currently emojis don't have a devoted unicode script so they return UNKNOWN
if (!(Character.UnicodeScript.of(name.charAt(i)) + "").equals("UNKNOWN")) {
nonEmoji.add(name.charAt(i));//its not an emoji so we add it
}
}
// we then cycle through rebuilding the string
for (int i = 0; i < nonEmoji.size(); i++) {
newName += nonEmoji.get(i);
}
}
return newName;
}
so if we pass in a string:
remove_Emojis_For_Devices_API_24_Onwards("😊 test 😊 Indic:ढ Japanese:な 😊 Korean:ㅂ");
it returns: test Indic:ढ Japanese:な Korean:ㅂ
Emoji placement or count doesn't matter
My app reads in large amounts of data from text files assets and displays them on-screen in a TextView. (The largest is ~450k.) I read the file in, line-by-line into a SpannableStringBuffer (since there is some metadata I remove, such as section names). This approach has worked without complaints in the two years that I've had the app on the market (over 7k active device installs), so I know that the code is reasonably correct.
However, I got a recent report from a user on a LG Lucid (LGE VS840 4G, Android 2.3.6) that the text is truncated. From log entries, my app only got 9,999 characters in the buffer. Is this a known issue with a SpannableStringBuffer? Are there other recommended ways to build a large Spannable buffer? Any suggested workarounds?
Other than keeping a separate expected length that I update each time I append to the SpannableStringBuilder, I don't even have a good way to detect the error, since the append interface returns the object, not an error!
My code that reads in the data is:
currentOffset = 0;
try {
InputStream is = getAssets().open(filename);
BufferedReader br = new BufferedReader(new InputStreamReader(is));
ssb.clear();
jumpOffsets.clear();
ArrayList<String> sectionNamesList = new ArrayList<String>();
sectionOffsets.clear();
int offset = 0;
while (br.ready()) {
String s = br.readLine();
if (s.length() == 0) {
ssb.append("\n");
++offset;
} else if (s.charAt(0) == '\013') {
jumpOffsets.add(offset);
String name = s.substring(1);
if (name.length() > 0) {
sectionNamesList.add(name);
sectionOffsets.add(offset);
if (showSectionNames) {
ssb.append(name);
ssb.append("\n");
offset += name.length() + 1;
}
}
} else {
if (!showNikud) {
// Remove nikud based on Unicode character ranges
// Does not replace combined characters (\ufb20-\ufb4f)
// See
// http://en.wikipedia.org/wiki/Unicode_and_HTML_for_the_Hebrew_alphabet
s = s. replaceAll("[\u05b0-\u05c7]", "");
}
if (!showMeteg) {
// Remove meteg based on Unicode character ranges
// Does not replace combined characters (\ufb20-\ufb4f)
// See
// http://en.wikipedia.org/wiki/Unicode_and_HTML_for_the_Hebrew_alphabet
s = s.replaceAll("\u05bd", "");
}
ssb.append(s);
ssb.append("\n");
offset += s.length() + 1;
}
}
sectionNames = sectionNamesList.toArray(new String[0]);
currentFilename = filename;
Log.v(TAG, "ssb.length()=" + ssb.length() +
", daavenText.getText().length()=" +
daavenText.getText().length() +
", showNikud=" + showNikud +
", showMeteg=" + showMeteg +
", showSectionNames=" + showSectionNames +
", currentFilename=" + currentFilename
);
After looking over the interface, I plan to replace the showNikud and showMeteg cases with InputFilters.
Is this a known issue with a SpannableStringBuffer?
I see nothing in the source code to suggest a hard limit on the size of a SpannableStringBuffer. Given your experiences, my guess is that this is a problem particular to that device, due to a stupid decision by an engineer at the device manufacturer.
Any suggested workarounds?
If you are distributing through the Google Play Store, block this device in your console.
Or, don't use one massive TextView, but instead use several smaller TextView widgets in a ListView (so they can be recycled), perhaps one per paragraph. This should have the added benefit of reducing your memory footprint.
Or, generate HTML and display the content in a WebView.
After writing (and having the user run) a test app, it appears that his device has this arbitrary limit for SpannableStringBuilder, but not StringBuilder or StringBuffer. I tested a quick change to read into a StringBuilder and then create a SpannableString from the result. Unfortunately, that means that I can't create the spans until it is fully read in.
I have to consider using multiple TextView objects in a ListView, as well as using Html.FromHtml to see if that works better for my app's long term plans.