Xtreams framework for Squeak

By: on November 29, 2010

At the last UKSTUG meeting, Michael Lucas-Smith presented a new streaming framework, Xtreams, that he and Martin Kobetic wrote.

He had some examples of some crazy things you can do with the framework, like writing a base64 encoder in a single method (XTReadStream >> #encodingBase64).

I’ll try give a flavour of Nicolas Cellier’s Squeak port of Xtreams. Some parts are as yet unported, and I’m not sure which parts those are because I haven’t seen the original VisualWorks code.

First, how to load it. Noone’s written a Metacello loading script yet, but I managed to easily figure out a working load order:

  1. Xtreams-Core
  2. Xtreams-CoreTests
  3. Xtreams-Substreams
  4. Xtreams-SubstreamsTests
  5. Xtreams-Terminals
  6. Xtreams-TerminalsTests
  7. Xtreams-Transforms
  8. Xtreams-TransformsTests
  9. Xtreams-SqueakTextConverter
  10. Xtreams-SqueakExternals
  11. Xtreams-VWCompatible

Xtreams has a very lazy feel about it. You stack streams one on the other, each one providing some additional functionality, and each one not actually doing anything until you try read or write to the stream. Apart from being a neat way to compose functionality, this also means you don’t copy buffers from stream to stream, making Xtreams rather efficient.

At the very bottom of your stack of streams lie Terminals – files, sockets, collections and the like – from which you read or write. Streams are either read streams OR write streams, never both.

So let’s have some fun.

We can simulate a generator with a block (Smalltalk’s term for a closure):

n := 0.
[n := n + 1] value

Every time you evaluate the second line, you get the next integer.

It’s easy enough to lock your image up:

n := 0.
[n := n + 1] reading rest

Or, “please return a collection containing every single integer greater than zero”. Given that Squeak seamlessly handles arbitrarily large integers, the resulting collection could contain (depending on your amount of memory) rather more than 2^32 integers.

But if we don’t want to do that, we can do other things:

n := 0.
s := ([n := n + 1] reading collecting: #even)
injecting: 0
into: [:total :each | each + total].

(#inject:into: is known elsewhere as reduce, or left fold.)

Note that we haven’t done anything yet: we’ve set up a stack of streams that will take a stream of integers, filter out the even integers, and then sum those integers. When we read 10 elements from the stream we get (unlike a standard reduce on a collection) a collection of partial results:

s read: 10
    => #(2 6 12 20 30 42 56 72 90 110)

Something interesting about #injecting:into: is that it’s still lazy: you only calculate the reduction of the items you’ve read so far. We can happily work with an infinite list of integers, and ask for the sums of the first N integers:

n := 0.
([n := n + 1] reading injecting: 0 into: [:total :each | total + each]) read: 10
    =>  #(1 3 6 10 15 21 28 36 45 55)

Like sockets, using blocks results in a stream where you can’t go backwards. These kinds of streams are not positionable, unlike a stream on a file. So what happens if you need to backtrack? Stack a positionable stream on your stream!

n := 0.
s := [n := n + 1] reading positioning.
s read: 5.
    => #(1 2 3 4 5) "Read 5 things"
s -- 3.                    "Go back 3 places"
s read: 5.
    => #(3 4 5 6 7) "And read the next 5 things"

By default, the buffer will grow to the size of the underlying stream. You probably don’t want this, in which case you can use a ring buffer:

n := 0.
s := [n := n + 1] reading positioning buffer: (XTRingBuffer on: (Array new: 3)).
s read: 20.
[s -- 5.] on: XTIncomplete do: []. "Try go back 5, and prepare to eat an Exception."
s read: 5.
    => #(18 19 20 21 22)

The first read results in the first twenty integers – 1, 2, …, 19, 20. We want to go back 5 places, but our ring buffer’s only three elements in length. We get an XTIncomplete exception, signalling that we’ve tried to read past the beginning of the XTRingBuffer. #on:do: swallows that exception, and we read the next 5 integers from as far back as we can go, namely the 5 integers after and including 18. Notice that you still get 5 items; the ring buffer simply controls how positionable your stream is.

That’s a very brief introduction to Xtreams. I hope to write another post soon, delving into another part of the framework.

As I mentioned earlier, there are some parts missing in the Squeak port. For my interests, the biggest mising bit is Xtream’s PEG parser. (This would also bring the number of Squeak PEG parsers to three, joining Ometa and PetitParser.)



Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>