jeudi 19 mars 2015

Scala - best way to buffer data


Data flows into my Scala program in the form of a stream of events (case classes), and I'd like to commit this data into my database (Neo4J) in batches, to avoid starting a transaction for each element of data.


I'd like a batch to be written whenever it exceeds a size, or an age in milliseconds



val batcher = SomeKindOfBatcher[MyEvent](maxSize = 100, maxAge = 1 second)

batcher.onBatch(db.writeBatch) // writeBatch accepts a Seq[MyEvent]

while(input.hasNext()) {
batcher.enqueue(input.next())
}


I'm looking for the most appropriate library that provides an interface like SomeKindOfBatcher above





Aucun commentaire:

Enregistrer un commentaire