I've observed this issue for years now, not knowing where it came from. I am concerned that this bug is still observable in the new versions of Android, in 2011, and I hope you can finally help me to fully understand it, if not solve it.
Let's consider the given (real) situation. Mister "A" is using a custom SMS/MMS app from Sony on his Xperia Arc (official 2.3.3). Mister B is using the android SMS/MMS stack app on his Milestone (Cyanogen 6.12, unofficial 2.2). Both of them use Android in French (if that matters).
When A sends a sms to B containing special characters like "ç", "ê", B receives a message with these characters replaced by a space. Characters like "é" are working fine though.
When B sends the sms to A, everything works fine.
When A sends this sms to himself, everything works fine.
Conclusion : this is not the mobile provider's fault since it works in one way and not the other.
So, I guessed at first that something was wrong with A's custom app. Replaced it with the apk from B's phone. Everything remained the same. I decompiled the app and I didn't find where the encoding of the sms string was done. I concluded the bug is not coming from the app, but from the way Android encodes the strings...
I ran another test :
I wrote an sms with only standard characters, something like 250 characters in 1.5 sms. Then, I append a "ç" to the sms.
On A's phone : the counter says it consumed 10 characters.
On B's phone : the counter says the sms now takes 3 sms : the string size doubled !
Conclusion :
On A's phone, the default charset includes "ç".
On B's phone, when "ç" appears, the charset changes and each character needs then twice the original space.
(Or am I missing something ?)
Questions :
Why different version of Android aren't using the same default charset ?
On Android, are these default charset depending on the rom, for example ?
Can we configure/change these charset somewhere (in the menu or directly on a rooted phone) ?
Is there another easy way to fix this ?
Any help, explanation or experience is welcome :)
You are suffering from encoding problems. From the description it looks like 'A' is sending data in one charset and not including information about what charset that is. The root cause is that to pass extended (non-ascii) characters between two systems they have to agree on an encoding to use. If you are restricted to 8 bit values then the systems agree to use the same codepages. In SMS there is a special GSM codepage for 7 or 8 bit encodings or UTF-16 can be used which uses 2 bytes to represent each character. What you see when you enter 250 characters followed by a single extended character shows you what is happening in the application. An SMS message is restricted to 140 octets. When you are using an 8 bit encoding your 250 chars fit into 2 messages (250 < 280) however once you added the "ç" the app changed to using UTF-16 encoding so suddenly all your characters are taking 2 octets and you can only fit 70 characters into a message. Now it takes 3.5 SMS messages to transfer the entire message.
On Android the decoding of the SMS message is part of the framework telephony code in SmsCbMessage.java. It works out the language code and encoding of the message body. If this is incorrect (the message was encoded with an english codepage but uses french extended chars) then you can get odd characters appearing.
You are right that this is not the mobile network at fault. I suspect it is phone A's messaging application although it is possible that Android is failing to correctly identify the encoding of a valid SMS. I wonder how it works between A and an iPhone or some other manufacturers device.
I have encountered the same problem when I had to show a few special characters in an sms unicode app. The method I used was take the string that I need to send as sms, run it in a for loop to take each character , find its ascii code , use that integer value to encode that string using a delimiter. This string can be sent as sms, which needs to be decoded using the same delimiter that is used for sending, then convert each ascii code char in it to characters (language specific), form a string by appending the converted chars. This text will be same as the one that was sent as sms.
Regards
Related
I need to send a SMS that contains a JSON in my Android app. Here is the code:
SmsManager sms = SmsManager.getDefault();
String message = "{\"phone\":\"9999-9999\"}";
sms.sendTextMessage(phoneNumber, null, message, null, null);
I tested with some phones and most part of them receive the right message:
{"phone":"9999-9999"}
But one model (LG G2 Mini) changes the '{' character when the SMS is received:
ä"phone":"9999-9999"ñ
Does anybody know why it's happening?
Any tip will be very helpful,
Thanks
There are special charsets for SMS, see this link from Wikipedia: https://en.wikipedia.org/wiki/GSM_03.38
The characters { and }, that are wrong on your phone, are in the Basic Character Set Extension. Try other are characters like €, |, ] and so on. When these characters also do not work on your phone, I suppose your problem is related to this charset extension.
Characters from the extension need to be escaped with 0x1B (escape character in SMS). This is only a conjecture, maybe there could be a problem with the escaping.
Looks like the LG G2 Min has not correctly implemented the 3GPP TS 23.038.
SMS uses different character sets. The mostly used is the GSM 7 bit default alphabet. It codes in 7 bits most of the usual characters and special characters used by US and some European languages. Some special characters (^{}\<[~]€) did not fit into this 7 bit table. The escape character switch to GSM 7 Bit default alphabet extension table that contains the above mentioned special character. These characters are codes by two 7-Bit character, ESC and the code for e.g. ‘{‘.
What you can try, if possible, is to use Unicode. You may force Unicode by using a Unicode character in your message that is not part of the 7-Bit GSM character code and its extension table.
I use Cloudhopper library to connect to SMSC and submit messages.
I split long message in a standard way:
split text into small chunks of 134 bytes
generate next reference number
create UDH 6 bytes with ref number
send parts one by one
I have several connections open in transceiver mode to SMSC. If I use only one connection to send all the parts, then most of the time the messages are merged correctly both on Android 5.0.1 and iOS 8.4.1, though sometimes quite rarely the concatenated messages have the last part from the wrong message.
If I use different connections to send the parts, then most of the time messages are concatenated in a wrong way, usually the last part belongs to some other message. So it looks like mobiles don't honor the included reference number at all.
I know that if the reference number is the same and there are hanging old parts in the database on mobile then it might concatenate wrongly, but this doesn't seem to be the case in my tests since the reference number is sequential and controlled to be different for wrongly concatenated parts.
What could be the problem or how could I approach it?
I have a technical issue that I can't resolve because the problem only happens on the other side of the planet that I'm on.
So I am hoping someone with a lot more Experience with these technologies can suggest an alternate approach or path to take in order to rectify it, or possibly right out "RECOGNIZE" the source of the problem.
THE SCENARIO :
I have developed an application for the Samsung Gear2 that sends SMS messages using its host companion android application.
In all tests I perform with the app pair, everything works as expected.
Messages are sent by the android device's SMS manager and received by the recipient no problem, even when adding and removing the leading 1.
10 and 11 digit numbers to and from the USA, with and without the leading + sign all pass.
However, when sending the apps to Samsung for review and testing, they keep having problems sending certain numbers. As far as I can tell the only thing different between the numbers that send and the numbers that fail is the fact that the first has a leading 0 and the second does not.
This would have to be coupled with the fact that they are testing this functionality
in a different Continent/Country than the 1 where my tests are being done(USA).
As my tests succeed to send and receive the messages regardless of whether
the number is a 10 digit version or 11 digit version of the same number.
Also discrepancies surfacing from the use of different Carriers (possibly Telecom in Asia) other than the one that I am testing with(Bell - USA)...
To be clear, I am simply composing messages via the Gear2 app and sending it to the android device via Samsung's Accessory Protocol which works flawlessly.
From there I use the basic implementation of the standard
Android's SMSManager
getDefault()
and
sendTextMessage(String destinationAddress, String scAddress, String
text, PendingIntent sentIntent, PendingIntent deliveryIntent)
with all null values exept the "destinationAddress","text",
and the "sentIntent".
THE QUESTION(s) : in order of most to least important
1. IS THERE a specific "routine" SMS app developers use to maximize the
success rate
of sent sms messages no matter what Country or Carrier the user sends the message from or to, and no matter if the entered number contains the leading exit and Country codes?
(ie: Adding or removing a plus(+) sign, per-examining the Locale
being used and adjusting the number accordingly using some sort of
"SMS Numbering Standard", Altering the Service Center Address / SMSC
Address (AT+CSCA), ect...)?
2. IS THERE some sort of information source detailing general "GOTCHAS" to look for
dependent upon the various Cell/SMS service providers/Carriers? (at least 4 that I know of).
3. Why would Android's SMSManger report "SMS Sent" if the message wasn't actually delivered.
NOTE: I realize that one can also listen for the "second" broadcast android sends
confirming that the message was "actually" received by the recipient, but that raises questions concerning how to rectify that situation, if it's at all rectifiable (at least from a programmatic perspective.)
4. Should my app, which is actually a Samsung Gear2 app integrated with an android app,
and merely delegates the responsibility of sending the messages to the android devices SMSManager, be held responsible for the message not being sent?
NOTE: I can't help but feel that Samsung is requiring me to handle
issues outside the scope and unrelated to the apps intended
functionality, Albeit , I do understand that at least part of the
issue has to fall back on me by Default, being the nature of the Gear2
apps dependance on the android device to complete the intended action
started by the Gear2 app, but how far must I go to ensure the standard
SMS abilities of a users android device beyond giving helpful feedback
as to what the underlying problem may be. I have to assume this
problem would exist using the device itself to compose the message
instead of the Gear2 app, given the exact same address/number .
5. Assuming that the app is being tested in Asia (South Korea, Samsung's Headquarters)... And assuming that the carrier being used to test SMS is SK Telecom
(Which I think is South Korea's top Carrier)...
Is it Mandatory to start all numbers with a zero in order to successfully send an SMS within the same Country? And if so is that the same in all Countries besides the USA and Canada(Which seem to work either way)?
6. Is this a common issue to deal with generally when developing sms applications, or can I single this particular situation out from the rest based on the unique variables introduced by Samsung's over diligent testing practices along with there unfamiliar
Telephony Service Providers and even more unfamiliar Geo-location/Country-code and numbering schemes.
THE PROBLEM *(only happens during Samsung testing) :
TEST 1:
Send SMS from stored contacts with number 010-6627-xxxx (11 digits)
Result:
SMS is reported by android's SMSManager as sent. The recipient
immediately receives SMS message.
Conclusion:
SMS delivery success.
TEST 2:
Send SMS from manually entering number 10-6627-xxxx (10 digits) (same number as previous without first 0)
Result:
SMS is reported by android's SMSManager as sent. However, the
recipient never receives SMS message.
Conclusion:
?
Unable to duplicate this problem (at least not in the USA) as both :
11 digit numbers (leading 1) AND 10 digit numbers
(no leading 1)
Successfully send the message as expected.
The problem only happens during device testing/certification by Samsung's app testing
department, and seemingly only when manually entering the destination address/number.
Stored Contacts (for some reason) always work.
After being denied certification of an otherwise "GOOD TO GO APPLICATION" 3 times by Samsung for issues UN-recreatable in my own testing environment I find myself turning the good folks at SOF with this "HAIL MARY PASS".
I do not change anything at all about the numbers that are sent to the SMSManager, But I'm wondering if maybe THAT'S the problem.
EXTRA INFO :
All devices used are using android 4.2 and higher.
My tests were all done using T-Mobile Galaxy phones and sometimes google/Gmail/GoogleVoice
sms/mms service From North Fla. USA.
ON A SIDE NOTE ABOUT SAMSUNG...
Samsung's testers are very sparse with the details of their testing enviroment including what may possibly be the issue from their standpoint which is unfortunate because I'm becoming certain that this issue is Trivial at best and probably easily fixable with a basic understanding of your own Country/Carriers methods for sending successful SMS messages.
It seems they are not even making an effort to enlighten me or TO make any neccesary
adjustments to their test numbers as any user would do in order to have a successful result.
ANY INSIGHT INTO THIS ISSUE WHATSOEVER WOULD BE IMMENSELY APPRECIATED!!!
Interesting question. I understand that the Samsung testing process can be quite rigorous.
From reading your info .. have you tried including that leading "0" yourself? E,g when a person manually enters 10 digits without a leading 0, you detect it and add this "0" . Also, how are this digits entered on a gear 2 device ? Do you implement your own keypad and ensure it captures the right digits ?
I guess that would be a starting point ..
V.
I'm developing an Android application which sends commands to a remote equipment through SMS. The commands are all regular text messages, and some of them start with the prefix A##. To test the app I sent some "commands" to other phones using an Android 4.3 phone and also an Android 2.3 phone.
When I run the app on the Android 4.3 phone, the SMS on the receiving end shows just fine on any device, but if I use the Android 2.3 to send the commands they get received as A¿¿ on an Android 4.3 phone but arrive as normally as A## either on an Android 2.3 or an iPhone. On the target equipment (it uses a GSM modem) the message gets like A (character "A" plus two spaces - ASCII 0x20), so I suspect the sender is using a different encoding. What I find strange is the # symbol is not even an extended ASCII char, so I'm wondering why it would be encoded in some other charset than ASCII anyway.
Can anyone explain what is happening here? If the Android 2.3 device is really using another encoding, is there a way to force it to ASCII before sending the SMS?
The sending code is as follows:
#Override
public void sendCommand(String command) {
//TODO: Send SMS with 'command' as its text message
SmsManager sms=SmsManager.getDefault();
PendingIntent piSent=PendingIntent.getBroadcast(this, 0,
new Intent("SMS_SENT"), 0);
PendingIntent piDelivered=PendingIntent.getBroadcast(this, 0,
new Intent("SMS_DELIVERED"), 0);
String phone = txtPhone.getText().toString();
sms.sendTextMessage(phone, null, command, piSent, piDelivered);
}
Where the parameter command is always the concatenation of the prefix with some other text, like this:
String SmsPrefix = new String("A##");
sendCommand(SmsPrefix + "AT+DEACT");
UPDATE:
I had a hint from someone that problem might be related to the carrier, not to the Android system itself. I live in Brazil and my Android 2.3 device was using the carrier TIM, just the same as the iPhone we used. The Android 4.3 device was using the carrier Claro. What I found is that, if I get the TIM SIM card and put it on the Android 4.3 device the receiving end also shows the garbled #, so it seems the carrier TIM is messing up the SMS sent through their network. I will try the new suggestions from #PMunch below so we can possibly find a workaround, but we can be sure already it wasn't really some kind of bug corrected from Android 2.3 to 4.3.
Really seems like an encoding issue. It might be that you try to send as ASCII but the receiver tries to parse it in a different encoding. If you explicitly specify encoding on both sender and receiver end it should work.
EDIT:
This would get the character array and create a string from it all using US-ASCII encoding.
String newString = new String(oldString.getBytes("US-ASCII"), "US-ASCII"));
EDIT2:
Turns out GSM doesn't use regular US-ASCII, rather it uses it's own GSM alphabet. What seems to be happening is that the # (ASCII 0x40) is translated into the GSM alphabet directly as ¡ (Upside down exlamation mark, GSM 0x40). This won't affect regular text characters as they share the same addresses (same for the plus sign 0x2B). Then when converted back it tries to convert it frow what it assumes is GSM-alphabet to ASCII, meaning the 0x40 of the earlier # sign is now an upside down exclamation mark. This is a sign that does not exist in regular ASCII and is therefore replaced by an unknown character symbol, apparently an upside down question mark in Android 2.3 and a space in the GSM receiver. This lack of conversion from ASCII to GSM seems to have been fixed between Android 2.3 and Android 4.3.
If you try to specify to Android that this is an ASCII string by using new String ("A##","ISO-8859-1") it might do the conversion in it's own. If not you might have to do it yourself (Something like this might help). If # is the only special character you need to support then you could of course encode that single character yourself(\0\0 for ##).
EDIT3:
Edit2 included multiple actions, what did you try? To explain the whole GSM/ASCII thing:
ASCII uses it's first 32 characters as control characters. These characters were deemed unnecessary for GSM and so they were replaced with other characters. The null character, on computers used to terminate a string, is not used for text messages. They are set at 140 octets and any empty space is simply filled with filler characters. So the null character 0x00 in ASCII is used for something else, the # character. If you look at the GSM alphabet and the ASCII alphabet you will see that the 32 first characters are replaced by Greek characters and some others. If you look at the remaining characters they are mostly in the right place, the # character is one of the ones that aren't. If you for example try to type in _ you should get similar results. When you say that # comes out as A do you then mean that A## becomes A or that it becomes AAA? I also found something interesting while looking through Unicode conversion supplied by Unicode Inc.:
0x00 is NULL (when followed only by 0x00 up to the
end of (fixed byte length) message, possibly also up to
FORM FEED. But 0x00 is also the code for COMMERCIAL AT
when some other character (CARRIAGE RETURN if nothing else)
comes after the 0x00.
So if you tried to send only A## then the two last #s might be interpreted as padding characters instead of # characters. Regardless it seems like the carriers in your area does some conversion between them, have you tried to send the characters with sendDataMessage as raw data bytes? The function byte[] stringToGsm7BitPacked(String data) throws EncodeException from telephony.GsmAlphabet should help convert your string to the GSM alphabet.
I've got an app that lets users send sms messages. Works great when the message < 160 characters. After that, things work less-perfectly. Seems like there are a few options here:
Manually break the message up into multiple SMSs, send each part as a separate SMS.
Use the multi-part send SMS function (sendMultipartTextMessage()).
Send the message as an MMS message (sendDataMessage()?).
Here's my novice take on it:
1)
most well supported across carriers. Users may get mad that you just cost them N separate messages though, instead of converting to MMS or something.
2)
not sure if this is supported by different carriers, and read that once the message is greater than 3 * 160 chars in length, gets converted to MMS anyway by different SMS apps - maybe stay away from this altogether.
3)
not sure how to do this, and older phones might not support MMS. To send an MMS using the android SDK, do we just use the SmsManager.sendDataMessage() method?
Thanks
This is quite an old post but it's high up on Google when searching for Android multipart sms, so maybe it helps someone.
Regarding part 1 and 2, it's pretty much the same thing. To use sendMultipartTextMessage, you need to break up the long message into an ArrayList of Strings. It then sends as many SMS as needed. In short:
SmsManager sms = SmsManager.getDefault();
ArrayList<String> parts = sms.divideMessage(longMessage);
sms.sendMultipartTextMessage(phoneNumber, null, parts, null, null);
Part 3: MMS is not an option, as it has been pointed out. The charges and all.
seems to me that the first option is what most mobile phones do by default. sms messages by design can only send a certain amount of characters (160 probbaly), just inform the user that the message is too big and if he still wants to send it anyway (informing also how many sms would the total be).
as for MMS and multipart as you said not every carrier supports it, so they dont seem to be the best option.
EDIT: as for how does MMS work on android-sdk check this thread out: Android SDK MMS
I would suggest the usage of option 2 when you are working on Androids GSM based handsets.
GSM based Mobile device takes care of segmentation in which breaking the messages to multiparts for sending in performed and also assembling of the multipart messages in to one message on receipt.
If you have a method which takes care of sending text messages, than by default use the options of manager.divideMessage as it will work even if the message segments required are only 1 in number.
I dont think you should have any problem sending messages using option 2 and it will also ensure that the receiver receives the message as a single message.
Else we need to write your our own protocol stack where in you write reference number and number of messages for the receipient to understand and recreate the complete message; which is not very tough. We can use byte arrays with headers and the messages can be sent as base64 encoded.
Also I dont know much about the limits the carriers enforce on the number of segments in multipart message; based on my test I was able to send and receive 160*8 segments properly. Based on the GSM standards the segments can be upto 255 but that count might depend on the service provider implementation.