Module Mapred_io_bgreader
module Mapred_io_bgreader:sig..end
Background reader for
Mapred_ioThis module reads the blocks of a file in a background kernel thread, for increasing the resource utilization of mapred tasks. While the task is doing computations, the kernel thread can already read the next blocks from the filesystem.
type
val create : Mapred_fs.filesystem ->
string -> int64 -> int64 -> int -> bgreadercreate fs name startblock endblock bufnum
The new reader uses a copy of the client fs for the fs access.
The file is name, which is read from startblock to
endblock-1. bufnum is the length of the buffer in blocks
(for each of the two buffers, foreground and background).
After create the first returned buffer is empty. Call read
to read the first blocks.
val extend : bgreader -> int64 -> unitextend bgreader endblock: Sets a new endblock (higher value than the
old endblock)val read : bgreader -> intread bgreader: reads the next blocks into the buffer. Returns the
number of blocks - 0 if at eofval buffer : bgreader -> Netsys_mem.memory
Returns the buffer. Note that the module uses double buffering: There
are actually two buffers, one foreground buffer, and one background
buffer. This function returns the current foreground buffer which
is not touched anymore by the reader. The two buffers are normally
swapped when
read is called.val buffer_block : bgreader -> int64
Returns the block position of
bufferval buffer_len : bgreader -> int
Returns the length of
buffer in blocksval close : bgreader -> unit
Closes the reader
