My application subscribes to an Observable<Timestamped<byte[]>> of data packets arriving in sequence, and assembles them into larger frames. It must examine each packet to find the "Start of Frame" header and do some minor processing to assemble the packets into a valid frame.
How can I create a new Observable<Frame> that will emit these completed frames to a Subscriber?
Update: the suggested answer doesn't want to work for me. Some details:
My source Observable emits Timestamped<byte[]> packets.
Desired Output is an Observable of DataFrame objects, each including the data from several packets along with some other fields.
I have a class FrameAssembler with a method DataFrame receivePacket( Timestamped<byte[]> packet ). It returns null until it has assembled a frame, which it then returns and gets ready for the next one.
I can't create the output Observable. I'm trying this
Observable<DataFrame> source = Observable
.just( new Timestamped<byte[]>(100, new byte[10]) ) // sample packet
.scan( new FrameAssembler(), (acc, packet) -> acc.receivePacket( packet ))
.filter( frame -> frame != null )
but the lambda is underlined, with the message "Bad return type in lambda expression: DataFrame cannot be converted to TestScan.FrameAssembler".
I'm thoroughly stumped by this. What is acc and what's it doing there? Why does it want to convert the DataFrame returned by receivePacket into FrameAssembler? And why is new FrameAssembler() used as the first argument to scan()?
You probably want to use the 2-parameter scan operator:
class ByteAccumulator {
private byte[] buffer = ...
public byte[] receivePacket(byte[] receivedPacket) {
// add the received packet to the buffer
if(containsFullFrame(buffer)) {
return extractFrameAndTrimBuffer();
} else {
return null;
}
}
}
Observable<byte[]> source = ...
source.scan(new ByteAccumulator(), ByteAccumulator::receivePacket)
.filter(frame -> frame != null)
...
Edit: You need an intermediate class to adapt your FrameAssembler to what scan expects:
public FrameScanner {
private final FrameAssembler assembler;
private final DataFrame frame;
public FrameScanner() {this(new FrameAssembler(), null);}
public FrameScanner(FrameAssembler assembler,DataFrame frame) {
this.frame=frame; this.assembler=assembler;
}
public getFrame() {return frame;}
public FrameScanner scan(Timestamped<byte[]> nextBytes) {
return new FrameScanner(assembler, assembler.receivePacker(nextBytes));
}
}
Now you should be able to use it like this:
.scan(new FrameScanner(), FrameScanner::scan)
.map(FrameScanner::getFrame)
.filter(Objects::nonNull)
Hmm... now that I think about it, instead of the abofethis might also work:
FrameAssembler assembler=new FrameAssembler();
...
.scan((DataFrame)null, (ignore, packet) -> assembler.receivePacket( packet))
.filter(Objects::nonNull)
I couldn't get the proposed solution using the scan() operator to work. I believe the problem was the null being returned until a complete set of packets was received. Observable operator chains don't seem to like nulls.
How I solved it:
In the onNext() handler of the data packet Observable subscription:
Thread.currentThread().setPriority( DATA_RX_PRIORITY );
packetArrayList = DataOps.addPacket( packetArrayList, dataPacket );
if( packetArrayList != null ) { // we have a new complete packet buffer
DataOps.DataFrame frameReturned = DataOps.pBuf2dFrame( packetArrayList );
frameRelayer.onNext( frameReturned ); // send the new frame to the BehaviorSubject
}
The addPacket() routine adds each received packet to an ArrayList, but returns null except when a complete Frame's packets have been accumulated, when it returns the filled ArrayList.
When a non-null ArrayList is received, the pBuf2dFrame() method parses the packets and assembles them into a new DataFrame object.
Then comes the trick that converts the Observable of packets into an Observable of DataFrames: frameRelayer is a BehaviorSubject (an RxJava object that can function as both an Observable and a Subscriber). All you have to do is call its onNext() method with the new DataFrame to have it passed on to any Subscribers to frameRelayer.
Related
I have a dummy Stream<int> called intialStream and I am mapping it into another stream, called mappedStream. which is supposed to be a new stream as I quote from the documentation of the map method:
Creates a new stream that converts each element of this stream to a
new value using the convert function, and emits the result.
void main() {
Stream<int> initialStream = Stream<int>.value(0);//using broadcast stream instead, makes the code work, as if dart considers the mapped stream to be the same stream as this one when I am using non-broadcast streams !
initialStream.listen(print);
var mappedStream = initialStream.map((nb) => nb + 1);
print(initialStream == mappedStream); // prints false !
mappedStream.listen(print);//error here
}
The problem is that when I try to listen to the mapped stream, I get an error:
Bad state: Stream has already been listened to.
Isn't this supposed to be a new stream and thus I can listen to it separately?
Yes, you do create a new stream when you use map() method, but the new stream also listens on the original stream. So, you cannot listen on the new stream that is created using the methods (map(), where(), expand()...) after adding a subscription to the original stream which is single-subscription.
It's not explicitly stated in docs, but check out the following lines that you might get a general idea:
Methods that modify a stream
The following methods on Stream return a new stream based on the
original stream. Each one waits until someone listens on the new
stream before listening on the original:
Stream<R> cast<R>();
Stream<S> expand<S>(Iterable<S> Function(T element) convert);
Stream<S> map<S>(S Function(T event) convert);
Stream<T> skip(int count);
Stream<T> skipWhile(bool Function(T element) test);
Stream<T> take(int count);
Stream<T> takeWhile(bool Function(T element) test);
Stream<T> where(bool Function(T event) test);
Here, you can say that the new stream will listen on the original one once a subscription is added. Therefore, it will throw error if the original stream is single-subscription and it already has a listener.
In my Android App I have a presenter which handles user interactions, contains kind of request manager and if needed sends user input over request manager to request manager.
Request manager itself contains server API and handles server request using this RxJava.
I have a code, which sends a request to server everytime a user enters a message and show the response from server:
private Observable<List<Answer>> sendRequest(String request) {
MyRequest request = new MyRequest();
request.setInput(request);
return Observable.fromCallable(() -> serverApi.process(request))
.doOnNext(myResponse -> {
// store some data
})
.map(MyResponse::getAnswers)
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
}
However now I need to have kind of queue. The user may send a new message before the server has responded. Each message from the queue should be processed sequentially. I.e. the second message will be sent after we've got a response to the first message and so on.
In case an error occurs no further requests should be handled.
I also need to display the answers within a RecyclerView.
I have no idea how to change the code above to achieve the handling described above
I see kind of problem. On one hand, this queue can be anytime updated by the user, on the other hand anytime server sent a response the message should be removed from the queue.
Maybe there is a rxjava operator or special way I just missed.
I saw a similar answer here, however, the "queue" there is constant.
Making N sequential api calls using RxJava and Retrofit
I'll be very thankful for any solution or link
I don't fnd any elegant native-RxJava solution. So I will custom a Subscriber to do your work.
For your 3 points:
For sequential execution, we create a single thread scheduler
Scheduler sequential = Schedulers.from(Executors.newFixedThreadPool(1));
For stop all requests when error occur, we should subscribe all request together instead of create a Flowable every time. So we define following functions (here I request is Integer and response String):
void sendRequest(Integer request)
Flowable<String> reciveResponse()
and define a field to make association of request and response flow:
FlowableProcessor<Integer> requestQueue = UnicastProcessor.create();
For re-run the not-sent request, we define the rerun function:
void rerun()
Then we can use it:
reciveResponse().subscribe(/**your subscriber**/)
Now let us implement them.
When send request, we simply push it into requestQueue
public void sendRequest(Integer request) {
requestQueue.onNext(request);
}
First, to do the request sequentialy, we should schedule work to sequential:
requestQueue
.observeOn(sequential)
.map(i -> mockLongTimeRequest(i)) // mock for your serverApi.process
.observeOn(AndroidSchedulers.mainThread());
Second, to stop request when error occur. It's a default behavior. If we do nothing, an error will broken the subscription and any futher items will not be emitted.
Third, to re-run the not-sent requests. First because that the native operator will cancel the stream, like MapSubscriber do (RxJava-2.1.0-FlowableMap#63):
try {
v = ObjectHelper.requireNonNull(mapper.apply(t), "The mapper function returned a null value.");
} catch (Throwable ex) {
fail(ex);// fail will call cancel
return;
}
We should wrap the error. Here I use my Try class to wrap the possible exception, you can use any other implementation that can wrap the exception instead of throw it:
.map(i -> Try.to(() -> mockLongTimeRequest(i)))
And then it's the custom OnErrorStopSubscriber implements Subscriber<Try<T>>, Subscription.
It request and emits items normally. When error occur(in fact is a failed Try emitted) it stopped there and won't request or emit even downstream request it. After call rerun method, it will back to the running statu and emit normally. The class is about 80 lines. You can see the code on my github.
Now we can test our code:
public static void main(String[] args) throws InterruptedException {
Q47264933 q = new Q47264933();
IntStream.range(1, 10).forEach(i -> q.sendRequest(i));// emit 1 to 10
q.reciveResponse().subscribe(e -> System.out.println("\tdo for: " + e));
Thread.sleep(10000);
q.rerun(); // re-run after 10s
Thread.sleep(10000);// wait for it complete because the worker thread is deamon
}
private String mockLongTimeRequest(int i) {
Thread.sleep((long) (1000 * Math.random()));
if (i == 5) {
throw new RuntimeException(); // error occur when request 5
}
return Integer.toString(i);
}
and output:
1 start at:129
1 done at:948
2 start at:950
do for: 1
2 done at:1383
3 start at:1383
do for: 2
3 done at:1778
4 start at:1778
do for: 3
4 done at:2397
5 start at:2397
do for: 4
error happen: java.lang.RuntimeException
6 start at:10129
6 done at:10253
7 start at:10253
do for: 6
7 done at:10415
8 start at:10415
do for: 7
8 done at:10874
9 start at:10874
do for: 8
9 done at:11544
do for: 9
You can see it runs sequentialy. And stopped when error occur. After call rerun method, it continue handle the left not-sent request.
For complete code, see my github.
For this kind of behaviour I'm using Flowable backpressure implementation.
Create outer stream that is parent for your api request stream, flatMap the api request with maxConcurrency = 1 and implement some sort of buffer strategy, so your Flowable doesn't throw exception.
Flowable.create(emitter -> {/* user input stream*/}, BackpressureStrategy.BUFFER)
.onBackpressureBuffer(127, // buffer size
() -> {/* overflow action*/},
BackpressureOverflowStrategy.DROP_LATEST) // action when buffer exceeds 127
.flatMap(request -> sendRequest(request), 1) // very important parameter
.subscribe(results -> {
// work with results
}, error -> {
// work with errors
});
It will buffer user input up to given threshold, and then drop it(if you don't do this it will throw exception, but it is highly unlikely that user will exceed such buffer), it will execute sequentially 1 by 1 like a queue. Don't try to implement this behaviour yourself if there are operators for thing kind of behaviour in libary itself.
Oh I forgot to mention, your sendRequest() method must return Flowable or you can convert it to Flowable.
Hope this helps!
My solutions would be as follows (I did something similar in Swift before):
You will need a wrapper interface (let's call it "Event") for both requests and responses.
You will need a state object (let's make it class "State") that will contain request queue and the latest server response, and a method that will accept "Event" as parameter and return 'this'.
Your main processing chain will look like Observable state = Observable.merge(serverResponsesMappedToEventObservable, requestsMappedToEventObservable).scan(new State(), (state, event) -> { state.apply(event) })
Both parameters of the .merge() method will probably be Subjects.
Queue processing will happen in the only method of "State" object (pick and send request from the queue on any event, add to queue on request event, update latest response on response event).
i suggest to create asynchronous observable methods , here a sample :
public Observable<Integer> sendRequest(int x){
return Observable.defer(() -> {
System.out.println("Sending Request : you get Here X ");
return storeYourData(x);
});
}
public Observable<Integer> storeYourData(int x){
return Observable.defer(() -> {
System.out.println("X Stored : "+x);
return readAnswers(x);
}).doOnError(this::handlingStoreErrors);
}
public Observable<Integer> readAnswers(int h){
return Observable.just(h);
}
public void handlingStoreErrors(Throwable throwable){
//Handle Your Exception.
}
the first observable will send request when he get response will proceed the second one and you can chain , you can customize each method to handle errors or success, this sample like queue.
here the result for execution :
for (int i = 0; i < 1000; i++) {
rx.sendRequest(i).subscribe(integer -> System.out.println(integer));
}
Sending Request : you get Here X
X Stored : 0
0
Sending Request : you get Here X
X Stored : 1
1
Sending Request : you get Here X
X Stored : 2
2
Sending Request : you get Here X
X Stored : 3
3
.
.
.
Sending Request : you get Here X
X Stored : 996
996
Sending Request : you get Here X
X Stored : 997
997
Sending Request : you get Here X
X Stored : 998
998
Sending Request : you get Here X
X Stored : 999
999
I'm communication with a server through a tcp socket connection, i'm able to read lines that ends with \n fine, however when the line is not terminated (ends in \n) i'm not able to read it. I tried the following but it didn't work and caused my app to freeze at startup:
private Socket socket;
private BufferedReader input;
public boolean isConnected;
#Override
public void onCreate()
{
try
{
socket = new Socket ("server.ip.add.ress", 23456);
input = new BufferedReader (new InputStreamReader (socket.getInputStream());
handshake();
isConnected = true;
}
catch // Handle IOException and UnknownHostException
}
// custom runnable to read availabe input from the server
private class MyRunnable implements Runnable
{
private volativle String value;
public String getValue()
{
return value;
}
#Override
public void run()
{
int count;
char[] buffer = new char[10]; // expected message 'username: '
try
{
count = input.read (buffer, 0, 10);
if (count > 0) value = new String (buffer);
}
catch // IOException
}
}
// when connection is established with server expect 'username: ' from
// the server and send the user name back to it
public void handshake()
{
MyRunnable runnable = new MyRunnable();
try
{
Thread thread = new Thread (runnable);
thread.start();
thread.join();
String greeting = runnable.getValue();
if (greeting.equals ("username: ")) // Send username back
}
catch // InterruptedException
}
why is it hanging? and how can i read a non terminated line?
Edit:
To clarify: The server sends the greeting message username: immediately after the connection is established with a client, the client wait for the greeting and send back it's username when received (that's what handshake() does), if no handshake the client disconnects otherwise it start listening for incoming messages. Because i need to know if handshake is complete before starting the listener i had to use Thread.join().
The problem: Thanks for the comments and answers below, it turned out that BufferedReader.read() blocks the thread and waits until something is sent from the server and if nothing is being sent it causes the app to hang, Therefor there's no way to find out if the line has ended.
The solution: In my specific situation i just wanted to know if a specific message is sent "username: " so i used read (buffer, 0, 10) to read exactly 10 characters (the length of "username: "), and because it blocks if nothing is sent i used Thread.join (1000) which waits only one second and then if nothing received i disconnect the client.
Why is it hanging?
This is what it is suppose to be. It will block the thread if no data is available to read. This is also why you want to put it in a background thread.
Can it not just return if nothing is available?
What you are looking for is ready(), which will tell you whether there is available data or not.
Indicates whether this reader is ready to be read without blocking.
Returns
true if this reader will not block when read is called, false if unknown or blocking will occur.
But you should be very careful when using this function. Because networking is a lot about timing. The fact that you don't have any data to read at this second doesn't necessary mean that it won't be any data in the next second.
So a better design of the server should be more or less as the following:
If the username is found, return the username
If the username is not found, return an error message to let the client side know that the username is not found
There's no need for the thread. Your goal is to wait until you've read what you've been waiting for. Why not just let read() perform the wait for you?
What you're struggling with is the classic problem of TCP communication: "when do I know that I've got everything the server sent?"
In your case, you're expecting to read bytes until the collection of bytes ends with "username: ". So, change your algorithm to perform 1 byte reads (filling a buffer as you go) until that buffer ends with "username: ".
You can make a more complicated algorithm -- which would be more efficient -- that would attempt to read multiple bytes at a time and append them to a buffer -- performing your check each time. But either strategy is logically equivalent.
I also recommend just using the InputStreamReader. It has various read() methods. I am a bit suspicious about the BufferedInputReader, especially when dealing with data that isn't newline terminated. I'm probably just paranoid. I've just never used it when writing TCP client/server programs, so I'm not sure.
My problem is i can't get infinite stream with Retrofit. After i get credentials for initial poll() request - i do initial poll() request. Each poll() request responds in 25 sec if there is no change, or earlier if there are any changes - returning changed_data[]. Each response contains timestamp data needed for next poll request - i should do new poll() request after each poll() response. Here is my code:
getServerApi().getLongPollServer()
.flatMap(longPollServer -> getLongPollServerApi(longPollServer.getServer()).poll("a_check", Config.LONG_POLLING_SERVER_TIMEOUT, 2, longPollServer.getKey(), longPollServer.getTs(), "")
.take(1)
.flatMap(longPollEnvelope -> getLongPollServerApi(longPollServer.getServer()).poll("a_check", Config.LONG_POLLING_SERVER_TIMEOUT, 2, longPollServer.getKey(), longPollEnvelope.getTs(), "")))
.retry()
.subscribe(longPollEnvelope1 -> {
processUpdates(longPollEnvelope1.getUpdates());
});
I'm new to RxJava, maybe i don't understand something, but i can't get infinite stream. I get 3 calls, then onNext and onComplete.
P.S. Maybe there is a better solution to implement long-polling on Android?
Whilst not ideal, I believe that you could use RX's side effects to achieve a desired result ('doOn' operations).
Observable<CredentialsWithTimestamp> credentialsProvider = Observable.just(new CredentialsWithTimestamp("credentials", 1434873025320L)); // replace with your implementation
Observable<ServerResponse> o = credentialsProvider.flatMap(credentialsWithTimestamp -> {
// side effect variable
AtomicLong timestamp = new AtomicLong(credentialsWithTimestamp.timestamp); // computational steering (inc. initial value)
return Observable.just(credentialsWithTimestamp.credentials) // same credentials are reused for each request - if invalid / onError, the later retry() will be called for new credentials
.flatMap(credentials -> api.query("request", credentials, timestamp.get())) // this will use the value from previous doOnNext
.doOnNext(serverResponse -> timestamp.set(serverResponse.getTimestamp()))
.repeat();
})
.retry()
.share();
private static class CredentialsWithTimestamp {
public final String credentials;
public final long timestamp; // I assume this is necessary for you from the first request
public CredentialsWithTimestamp(String credentials, long timestamp) {
this.credentials = credentials;
this.timestamp = timestamp;
}
}
When subscribing to 'o' the internal observable will repeat. Should there be an error then 'o' will retry and re-request from the credentials stream.
In your example, computational steering is achieved by updating the timestamp variable, which is necessary for the next request.
I am writing an android app that recieves data over bluetooth. The bytes comming in can be of any size example: 00023>024935928598235>9284>
As you can see each set is seperated by ">". The data comes in extremely fast. I would like some ideas for an implementation. See my problem is that I need to read the data into a byte array that can and then convert it to a string and split them according to the delimeter of ">".
so in the above example:
00023
024935928598235
9284
If i set byte[] data = new byte[8] then when reading the incomming data it might get 00023>02 which is not what i want. I'm not sure how to implement something like this. Any ideas?
Here's one approach. You'll have to implement the readDataFromBluetooth() and somehow set dataAvailable, but this should get you on the right track.
byte[] data = new byte[1024];
List<String> chunks = new LinkedList<String>();
StringBuilder chunk = new StringBuilder();
while (dataAvailable) {
data = readDataFromBluetooth();
for (byte b : data) {
if (b == '<') {
chunks.add(chunk.toString());
chunk.setLength(0);
} else {
chunk.append(b);
}
}
}
if (chunk.length() > 0)
chunks.add(chunk.toString());
I would recommend using a buffered stream, but maybe a bit bigger that 8 bytes, as you suggest, and the read one and one character from the beginning of the stream, accumulating the string. When you encounter a ">", send the value you have accumulated off to a queue for a background thread processing. Use standard producer/consumer implementation techniques (e.g. the Monitor pattern) to communicate via the queue.