Module Mapred_toolkit.Place

module Place: sig .. end

type 'a t 

A place for storing records of type 'a

type 'a codec = ('a -> string) Mapred_rfun.rfun *
       (string -> int -> int -> 'a) Mapred_rfun.rfun 

A codec (coder/decoder) says how to convert a record to string, and how to read it back. It is given as two functions (coder,decoder) which must both be registered (rfun). The decoder takes a string and a range with position and length.

type ext_location = [ `Deep_dir of string | `File of string | `Flat_dir of string ] 

type location = [ `Deep_dir of string | `File of string | `Flat_dir of string | `Notebook ] 

val create : ?prefix:string ->
       ?repl:int ->
       Mapred_fs.filesystem ->
       'a codec ->
       Mapred_toolkit.format ->
       Mapred_io.record_config ->
       ext_location -> 'a t

Creates a new empty place. The file or directory is created. The place can be used to store new files

• prefix: New files will get this prefix (ignored for `File location)

val from : ?prefix:string ->
       ?repl:int ->
       Mapred_fs.filesystem ->
       'a codec ->
       Mapred_toolkit.format ->
       Mapred_io.record_config ->
       ext_location -> 'a t

Picks up an existing location, and makes the files there accessible as place

• prefix: New files will get this prefix (ignored for `File location)

val notebook_place : unit -> 'a t

A new in-memory place for notebooks

val location : 'a t -> location

val get_codec : 'a t -> 'a codec

val get_format : 'a t -> Mapred_toolkit.format

val get_fs : 'a t -> Mapred_fs.filesystem

val get_rc : 'a t -> Mapred_io.record_config

val is_file_place : 'a t -> bool

Whether the place can store files

val files : 'a t -> string list

Returns the files of the place

val create_file : ?filename:string -> 'a t -> string

create_file pl:

Create a new empty file at the place. The file name is automatically determined, but starts with the previously set prefix. Also, the file gets a suffix that is derived from the format (".var" or ".fixed<n>").

If the place refers to a `File location, it is only possible to call create_file once.

This function fails for notebooks.

The optional filename argument influences the filename.

• If passed, and if prefix ^ filename ^ suffix does not exist yes, this file name is chosen. Otherwise:
• If passed, a fragment uuid is generated so that prefix ^ filename ^ uuid ^ suffix is the new file
• If not passed, the name is prefix ^ uuid ^ suffix.

val clear : 'a t -> unit

Deletes all files at this place the files function returns. Note that this excludes ignored files starting with "_" and ".", and files in directories starting with these characters. Only files are deleted, and the directories are left in place.

This function works for notebook places, and just deletes the notebooks.

For deleting directories entirely, please use Mapred_io.delete_rec.